mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it
@ 2026-11-19 21:10 Usama Arif
  2026-11-19 21:10 ` [PATCH 1/3] perf/core: Fix perf_sample_data reuse across software events Usama Arif
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Usama Arif @ 2026-11-19 21:10 UTC (permalink / raw)
  To: peterz, mingo, acme, namhyung, mark.rutland, alexander.shishkin,
	jolsa, irogers, adrian.hunter, james.clark, linux-perf-users,
	linux-kernel, bpf, rkeilty, atish.patra, dgreid, riel,
	shakeel.butt, kernel-team
  Cc: Usama Arif

do_perf_sw_event() builds samples for software events (context switches,
page faults, ...) in an on-stack struct perf_sample_data. This series
fixes two bugs around that struct and then stops zero-initializing it,
a 256-byte memset that every context switch pays once a context-switch
counter is open anywhere.

Patch 1 initializes the struct for each matching event. Today a sampling
event inherits the sample ID, timestamp, pid/tid and callchain prepared
for the event before it, and sometimes a record size smaller than what
is then written, overrunning the records that follow in its ring buffer.
Two concurrent "perf record -e page-faults,context-switches -c 1 -a"
sessions make the older one fail. An unprivileged user's page-fault
event can do this to root's CPU-wide session. Broken since v6.3.

Patch 2 stops BPF programs from reading an unset data->addr through
bpf_perf_event_data::addr. On INIT_STACK_NONE kernels this returned
leftover stack, mostly kernel pointers, on almost all context-switch
and cpu-clock samples in my tests. INIT_STACK_ALL_ZERO mostly hides it.
It affects all event types, needs CAP_BPF and CAP_PERFMON, and has been
there since v6.1.

Patch 3 marks the struct __uninitialized. On every path from do_perf_sw_event(),
each field is written before it is read, counting events never read the
struct, and it is never copied as a whole, so its padding never leaves
the kernel. A privileged BPF program using bpf_cast_to_kern_ctx() can
still read unprepared fields, and would see stack contents instead of 0,
as it already does on INIT_STACK_NONE kernels.

Patches 1-2 are for perf/urgent and stable, patch 3 for perf/core.

Performance: with sched_yield() ping-pong on one CPU, software counters
add on average 54 ns to a ~627 ns context switch, and patch 3 brings that
to 44 ns (10 rounds, spec_rstack_overflow=off).
 
Usama Arif (3):
  perf/core: Fix perf_sample_data reuse across software events
  perf/core: Don't let BPF programs read an uninitialized sample address
  perf/core: Don't zero perf_sample_data in do_perf_sw_event()

 kernel/events/core.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 1/3] perf/core: Fix perf_sample_data reuse across software events
  2026-11-19 21:10 [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it Usama Arif
@ 2026-11-19 21:10 ` Usama Arif
  2026-11-19 21:10 ` [PATCH 2/3] perf/core: Don't let BPF programs read an uninitialized sample address Usama Arif
  2026-11-19 21:10 ` [PATCH 3/3] perf/core: Don't zero perf_sample_data in do_perf_sw_event() Usama Arif
  2 siblings, 0 replies; 4+ messages in thread
From: Usama Arif @ 2026-11-19 21:10 UTC (permalink / raw)
  To: peterz, mingo, acme, namhyung, mark.rutland, alexander.shishkin,
	jolsa, irogers, adrian.hunter, james.clark, linux-perf-users,
	linux-kernel, bpf, rkeilty, atish.patra, dgreid, riel,
	shakeel.butt, kernel-team
  Cc: Usama Arif, stable

do_perf_sw_event() passes one on-stack struct perf_sample_data to every
matching event. perf_prepare_sample() fills in the fields an event
samples, marks them in data->sample_flags and skips fields that are
already marked, and those marks survive from one event to the next.

So a sampling event later in the list emits the sample ID, timestamp,
pid/tid and callchain prepared for an earlier one. Its record size is
computed from the earlier event's dynamic fields as well and can come
out smaller than what is written, in which case the kernel writes past
its ring-buffer reservation over the records that follow.

Two "perf record -e page-faults,context-switches -c 1 -a" sessions
running at the same time hit this: the one started first gets the
other's sample IDs and fails with "Bad address".

Commit 1d1bfe30dad5 ("perf/core: Fix perf_sample_data not properly
initialized for different swevents in perf_tp_event()") fixed the same
problem for tracepoints. Do the same here: initialize the struct for
each matching event. While at it, drop the unused data argument of
perf_swevent_match().

Fixes: bb447c27a467 ("perf/core: Set data->sample_flags in perf_prepare_sample()")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/events/core.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 687bd398275ba3..1c1ffd103c74a0 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -11142,7 +11142,6 @@ int perf_exclude_event(struct perf_event *event, struct pt_regs *regs)
 static int perf_swevent_match(struct perf_event *event,
 				enum perf_type_id type,
 				u32 event_id,
-				struct perf_sample_data *data,
 				struct pt_regs *regs)
 {
 	if (event->attr.type != type)
@@ -11207,11 +11206,10 @@ find_swevent_head(struct swevent_htable *swhash, struct perf_event *event)
 }
 
 static void do_perf_sw_event(enum perf_type_id type, u32 event_id,
-				    u64 nr,
-				    struct perf_sample_data *data,
-				    struct pt_regs *regs)
+				    u64 nr, struct pt_regs *regs, u64 addr)
 {
 	struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
+	struct perf_sample_data data;
 	struct perf_event *event;
 	struct hlist_head *head;
 
@@ -11221,8 +11219,10 @@ static void do_perf_sw_event(enum perf_type_id type, u32 event_id,
 		goto end;
 
 	hlist_for_each_entry_rcu(event, head, hlist_entry) {
-		if (perf_swevent_match(event, type, event_id, data, regs))
-			perf_swevent_event(event, nr, data, regs);
+		if (perf_swevent_match(event, type, event_id, regs)) {
+			perf_sample_data_init(&data, addr, 0);
+			perf_swevent_event(event, nr, &data, regs);
+		}
 	}
 end:
 	rcu_read_unlock();
@@ -11243,13 +11243,10 @@ void perf_swevent_put_recursion_context(int rctx)
 
 void ___perf_sw_event(u32 event_id, u64 nr, struct pt_regs *regs, u64 addr)
 {
-	struct perf_sample_data data;
-
 	if (WARN_ON_ONCE(!regs))
 		return;
 
-	perf_sample_data_init(&data, addr, 0);
-	do_perf_sw_event(PERF_TYPE_SOFTWARE, event_id, nr, &data, regs);
+	do_perf_sw_event(PERF_TYPE_SOFTWARE, event_id, nr, regs, addr);
 }
 
 void __perf_sw_event(u32 event_id, u64 nr, struct pt_regs *regs, u64 addr)
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 2/3] perf/core: Don't let BPF programs read an uninitialized sample address
  2026-11-19 21:10 [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it Usama Arif
  2026-11-19 21:10 ` [PATCH 1/3] perf/core: Fix perf_sample_data reuse across software events Usama Arif
@ 2026-11-19 21:10 ` Usama Arif
  2026-11-19 21:10 ` [PATCH 3/3] perf/core: Don't zero perf_sample_data in do_perf_sw_event() Usama Arif
  2 siblings, 0 replies; 4+ messages in thread
From: Usama Arif @ 2026-11-19 21:10 UTC (permalink / raw)
  To: peterz, mingo, acme, namhyung, mark.rutland, alexander.shishkin,
	jolsa, irogers, adrian.hunter, james.clark, linux-perf-users,
	linux-kernel, bpf, rkeilty, atish.patra, dgreid, riel,
	shakeel.butt, kernel-team
  Cc: Usama Arif, stable

A BPF program attached to a perf event gets a struct
bpf_perf_event_data, and reading its addr field turns into a load from
the perf_sample_data the sample is being assembled in. The verifier
allows that load no matter which fields the event samples.

perf_sample_data_init() used to store the address unconditionally, so
the program read 0 when there was none. To touch fewer cachelines, it
now stores the address only when it is non-zero, and
perf_prepare_sample() fills it in only when the event samples it
(PERF_SAMPLE_ADDR, PHYS_ADDR or DATA_PAGE_SIZE). For all other events,
including context switches, cpu-clock and hardware counters like
cycles, the program reads whatever was left in that stack slot, often a
kernel pointer.

Kernels built with CONFIG_INIT_STACK_ALL_ZERO, the default when the
compiler supports it, mostly hide this by clearing the struct on
function entry. With INIT_STACK_ALL_PATTERN the program reads the fill
pattern instead.

Clear the address before running the program if nobody filled it in,
the same way the SIGTRAP path already checks PERF_SAMPLE_ADDR before
using it. Events without a BPF program are not affected.

Fixes: 7b0846301531 ("perf: Use sample_flags for addr")
Cc: stable@vger.kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/events/core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 1c1ffd103c74a0..57ea7e3c01a025 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -10819,6 +10819,9 @@ static int bpf_overflow_handler(struct perf_event *event,
 	prog = READ_ONCE(event->prog);
 	if (prog) {
 		perf_prepare_sample(data, event, regs);
+		/* bpf_perf_event_data::addr is readable without PERF_SAMPLE_ADDR */
+		if (!(data->sample_flags & PERF_SAMPLE_ADDR))
+			data->addr = 0;
 		ret = bpf_prog_run(prog, &ctx);
 	}
 	rcu_read_unlock();
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 3/3] perf/core: Don't zero perf_sample_data in do_perf_sw_event()
  2026-11-19 21:10 [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it Usama Arif
  2026-11-19 21:10 ` [PATCH 1/3] perf/core: Fix perf_sample_data reuse across software events Usama Arif
  2026-11-19 21:10 ` [PATCH 2/3] perf/core: Don't let BPF programs read an uninitialized sample address Usama Arif
@ 2026-11-19 21:10 ` Usama Arif
  2 siblings, 0 replies; 4+ messages in thread
From: Usama Arif @ 2026-11-19 21:10 UTC (permalink / raw)
  To: peterz, mingo, acme, namhyung, mark.rutland, alexander.shishkin,
	jolsa, irogers, adrian.hunter, james.clark, linux-perf-users,
	linux-kernel, bpf, rkeilty, atish.patra, dgreid, riel,
	shakeel.butt, kernel-team
  Cc: Usama Arif

Kernels built with CONFIG_INIT_STACK_ALL_ZERO, the default when the
compiler supports it, clear every local variable on function entry. For
the 256-byte struct perf_sample_data in do_perf_sw_event() that is a
memset, and it runs on every entry, before the list of events is even
looked up.

do_perf_sw_event() is entered for a software event, e.g. a context
switch, as soon as one event of that kind is open anywhere in the
system. Counting events add to event->count and never read the struct.
For every matching event, perf_sample_data_init() resets sample_flags,
period and dyn_size and stores the address if there is one; for a
sampling event, perf_prepare_sample() then fills in what its
sample_type asks for.

A single context-switch counter, e.g. one kept open by a monitoring
daemon, is enough for every context switch on the machine to pay for
the clearing. On hosts running many small VMs, ___perf_sw_event() is
0.3% of all kernel cycles, and the memset about a seventh of that.

Mark the struct __uninitialized, which is what kernels without stack
auto-initialization get anyway. Because its contents are copied into a
ring buffer that userspace maps, every field that is read must have
been written first. That holds for perf_output_sample(), and with the
previous patch also for what a BPF program can read through
bpf_perf_event_data.

In a KVM guest on AMD Zen 5 with software counters open on every CPU,
this cuts the time they add to a context switch from 54 ns to 44 ns
(sched_yield() between two threads on one CPU, with
spec_rstack_overflow=off).

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 kernel/events/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 57ea7e3c01a025..fa10ad7cfd99af 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -11212,7 +11212,7 @@ static void do_perf_sw_event(enum perf_type_id type, u32 event_id,
 				    u64 nr, struct pt_regs *regs, u64 addr)
 {
 	struct swevent_htable *swhash = this_cpu_ptr(&swevent_htable);
-	struct perf_sample_data data;
+	struct perf_sample_data data __uninitialized;
 	struct perf_event *event;
 	struct hlist_head *head;
 
-- 
2.53.1-Meta


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-19 21:12 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:10 [PATCH 0/3] perf/core: Fix software event sample data, then stop zeroing it Usama Arif
2026-11-19 21:10 ` [PATCH 1/3] perf/core: Fix perf_sample_data reuse across software events Usama Arif
2026-11-19 21:10 ` [PATCH 2/3] perf/core: Don't let BPF programs read an uninitialized sample address Usama Arif
2026-11-19 21:10 ` [PATCH 3/3] perf/core: Don't zero perf_sample_data in do_perf_sw_event() Usama Arif

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®