Leon Hwang says:
====================
bpf: Introduce global percpu data
This patch set introduces global percpu data, similar to commit
6316f78306 ("Merge branch 'support-global-data'"), to reduce restrictions
in C for BPF programs.
With this enhancement, it becomes possible to define and use global percpu
variables, like the DEFINE_PER_CPU() macro in the kernel
include/linux/percpu-defs.h.
The section name for global peurcpu data is ".percpu". Even though, a one-byte
percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash
with Clang 17 [1], users are expected to use such small variables as global
percpu data with newer Clang versions, which don't have the issue.
The idea stems from the bpfsnoop [2], which itself was inspired by
retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR
(Last Branch Record) entries were observed related to the
bpf_get_smp_processor_id() helper.
Since commit 1ae6921009 ("bpf: inline bpf_get_smp_processor_id() helper"),
the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing
the overhead and consequently minimizing these two LBR records.
However, the introduction of global percpu data offers a more robust
solution. By leveraging the percpu_array map and percpu instruction,
global percpu data can be implemented intrinsically.
This feature also facilitates sharing percpu information between tail
callers and callees or between freplace callers and callees through a
shared global percpu variable. Previously, this was achieved using a
1-entry percpu_array map, which this patch set aims to improve upon.
Links:
[1] https://lore.kernel.org/bpf/fd1b3f58-c27f-403d-ad99-644b7d06ecb3@linux.dev/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop
Changes:
v11 -> v12:
* Improve feature check in bpf_object__create_maps() in libbpf.
* Add percpu_array map support in bpf_map__set_value_size() in libbpf.
* Exercise bpf_map__set_value_size() in selftest.
* Drop dead warning in bpf_object__populate_internal_map() in libbpf.
(Sashiko)
* v11: https://lore.kernel.org/bpf/20260806163125.11172-1-leon.hwang@linux.dev/
v10 -> v11:
* Drop env->prog->jit_requested check when inlining insns for global
percpu data.
* Do not autocreate percpu_array map when kernel does not have global
percpu data support in libbpf.
* Check map->btf_value_type_id in bpftool's is_skel_data().
* Exercise bpf_map__lookup_elem() in selftest.
* Collect Reviewed-by tags from Emil, thanks.
* Drop all duplicate blank lines in kernel/bpf/*.c. (Emil)
* Factor out check_map_mem_read() helper. (Emil)
* Check bpf_jit_supports_percpu_insn() first in
percpu_array_map_direct_value_addr/meta(). (Emil)
* Add comment for 'map->libbpf_type == LIBBPF_MAP_PERCPU' in libbpf's
map_is_mmapable(). (Emil)
* Init update_flags as a const var in libbpf's
bpf_object__populate_internal_map(). (Emil)
* Keep is_mmapable_map() beyond is_skel_data() in bpftool. (Emil)
* Add 'run' and 'cpu_id' in selftest. (Emil)
* Drop subskel test. Verify the generated subskel manually. (Emil)
* Add comment to the raw insns in selftest. (Emil)
* v10: https://lore.kernel.org/bpf/20260715153254.92010-1-leon.hwang@linux.dev/
v9 -> v10:
* Rebase latest bpf-next tree to resolve code conflict in verifier in
patch #1.
* v9: https://lore.kernel.org/bpf/20260713154024.30851-1-leon.hwang@linux.dev/
v8 -> v9:
* Use real name for percpu data maps in libbpf in patch #4.
* Add long map name test in patch #6.
* Move parse_cpu_mask_file() to test_percpu_data_on_cpus() in test in
patch #6.
* Validate map type in get_map_ident() for percpu data maps in patch #5.
* Update code comment in verifier in patch #2. (per Andrii)
* Pass 'type' to internal_map_name in libbpf in patch #4. (per Andrii)
* Factor out the helper is_skel_data() in bpftool in patch #5.
(per Quentin and Andrii)
* v8: https://lore.kernel.org/bpf/20260629152406.52582-1-leon.hwang@linux.dev/
v7 -> v8:
* Send patch #1 and #2 separately that fix interpreter fallback issues.
(Andrii)
* Use 'array->elem_size' to avoid 'range' local variable in
percpu_array_map_direct_value_meta(). (Andrii)
* Keep original map name for percpu data's map in libbpf. (Andrii)
* Factor out helper bpf_map_is_skel_data() in bpftool. (Andrii)
* Update commit message of direct access read-only percpu_array map.
(Andrii)
* Add test to verify that it is disallowed to directly write data of
read-only percpu_array map. (Andrii)
* Drop unused 'num_cpus' in test. (bot+bpf-ci)
* Factor out helper test_percpu_data_on_cpus() in test. (bot+bpf-ci)
* v7: https://lore.kernel.org/bpf/20260622143557.22955-1-leon.hwang@linux.dev/
v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/20260615152646.27639-1-leon.hwang@linux.dev/
v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in
interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling
bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton.
(Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without
test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/20260608145113.65857-1-leon.hwang@linux.dev/
v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in
interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read()
in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to
bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the
off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped
has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/20260414132421.63409-1-leon.hwang@linux.dev/
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag
is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf
(per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/20250526162146.24429-1-leon.hwang@linux.dev/
v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/20250213161931.46399-1-leon.hwang@linux.dev/
v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/20250127162158.84906-1-leon.hwang@linux.dev/
rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(),
and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and
PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/20250113152437.67196-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260813152324.97937-1-leon.hwang@linux.dev
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
If the arch, like s390x, does not support percpu insn, these cases won't
test global percpu data by checking FEAT_PERCPU_DATA support.
The following APIs have been tested for global percpu data:
1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. bpf_map__set_value_size()
4. generated percpu struct pointer pointing to internal map's mmaped data
5. bpf_map__lookup_elem() for global percpu data map
6. bpf_map_lookup_elem_flags() for global percpu data map
At the same time, the case is also tested with 'bpftool gen skeleton -L'.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-8-leon.hwang@linux.dev
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev
Add support for global percpu data in libbpf by adding a new ".percpu"
section, similar to ".data". It enables efficient handling of percpu
global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array
map used for global percpu data using BPF_F_ALL_CPUS, in order to update
values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked
as read-only after populating the percpu_array map. Thereafter, users can
read those initialized percpu data after loading prog. If they want to
update the percpu data after loading prog, they have to update the
percpu_array map using key=0 instead.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev
Introduce global percpu data, inspired by the commit
6316f78306 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
The chained_global_func_calls_success() test hardcodes the instruction
counts reported by the verifier's per-subprog stats:
subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack
processed 14 insns
global_good() does 'return arr[0]', where arr[] is an int array and the
return type is long. Without cpu v4 this is a zero-extending load
followed by a <<32/s>>32 sign-extension pair. With -mcpu=v4 llvm emits a
single sign-extending load instead:
18: (18) r1 = 0xffa00000008eb000
20: (81) r0 = *(s32 *)(r1 +0)
21: (95) exit
so the subprog is 3 insns rather than 5, and the whole program is
12 processed insns rather than 14. test_progs-cpuv4 fails with:
EXPECTED REGEX: 'subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack'
#606/1 verifier_global_subprogs/chained_global_func_calls_success:FAIL
Select the expected counts based on __BPF_CPU_VERSION__.
Fixes: c2e6c7de88 ("bpf: Show more useful info in stack depth stats")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260813150641.3347662-1-yonghong.song@linux.dev
The commit 7619a0ee93 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Add resolve_btfids support for __arena kfunc suffix
Use __arena/__arena__nullable suffixes to emit address_space(1)
annotations on kfunc definitions in vmlinux.h. See commits for details.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/20260809085155.3305519-1-memxor@gmail.com
* Avoid enumerating all the ways resolve_btfids can emit the
"address_space(1)" attribute in its header comment and in kfuncs.rst.
(Ihor)
* Drop the kfunc_has_arena_arg() helper: add_arena_tagged_proto()
returns the original prototype when nothing needs tagging, so it can
be invoked unconditionally for every kfunc. (Ihor)
* Add resolve_btfids selftest cases with mixed tagged and untagged
arguments, and a kfunc that combines the KF_ARENA_RET flag with
suffixed arena arguments. (Ihor)
====================
Link: https://patch.msgid.link/20260812193842.2879226-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a suffix-only kfunc declaration with arena annotations on all five
arguments. Verify that resolve_btfids emits address_space(1) type tags
for every position without KF_ARENA_ARG flags in the BTF ID sets.
Represent expected arena arguments as a per-parameter bitmap so the
test covers suffixes beyond the two positions expressible by flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Kfunc declarations can identify arena arguments through parameter name
suffixes without repeating KF_ARENA_ARG flags in their BTF ID sets.
resolve_btfids currently misses those arguments when synthesizing the
address_space(1) attributes used by generated vmlinux.h files.
Teach the arena prototype rewrite to recognize __arena and
__arena__nullable directly on each parameter. Keep KF_ARENA_ARG1 and
KF_ARENA_ARG2 handling for explicitly flagged kfuncs, while allowing
suffixes on any argument without synthesizing kfunc flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Hui Zhu says:
====================
bpf: Fix trampoline image UAF on multi detach failure
From: Hui Zhu <zhuhui@kylinos.cn>
This series fixes a UAF in bpf_trampoline_multi_attach_free() where
old_image is freed while ftrace still calls into it, and makes
bpf_trampoline_multi_detach() return void as suggested by Jiri Olsa.
Patch 1 fixes the UAF. Patch 2 is an independent cleanup that
changes the return type to void and drops the WARN_ON_ONCE at the
call site.
Changelog:
v5:
According to the comments of bot+bpf-ci, split the single patch into
two: the bug fix and the return-type cleanup.
v4:
According to the comments of bot+bpf-ci, add Fixes: and update comments
of bpf_trampoline_multi_attach_free.
v3:
According to the comments of Jiri Olsa, drop patches 2/3 and the
prog-side machinery.
keep only the simplified image-side fix in
bpf_trampoline_multi_attach_free() and make
bpf_trampoline_multi_detach() return void.
v2:
Folded v1's two detach patches into patch 1.
According to the comments of Jiri Olsa, Pin the prog (pinned_prog) on
cur_image so it stays alive while ftrace may still call into it.
Make bpf_trampoline_multi_detach() return void.
Fix the same UAF in standard (non-multi) trampolines.
According to the comments of sashiko, Fix the prog UAF in
bpf_trampoline_multi_attach() rollback.
Leak the trampoline in bpf_trampoline_put() when cur_image is left
by a rollback.
====================
Link: https://patch.msgid.link/cover.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Improve stack depth verification stats output
Some improvements for more clarity in the stack depth verification
statistics output. See commit logs for details.
For example, ./test_progs -t subprogs/subprogs_alone loads prog4,
which has a main program, two static subprograms, and two independently
verified global subprograms. A sample run produces:
verification time 1765 usec
stack depth max 48
subprog 0 (prog4) main insns_self 29 insns_total 51 stack 8
subprog 1 (get_task_tgid) global insns_self 9 insns_total 9 stack 8
subprog 2 (sub4) static insns_self 15 insns_total 22 stack 8
subprog 3 (sub3) static insns_self 7 insns_total 7 stack 0
subprog 4 (sub1) global insns_self 10 insns_total 10 stack 8
processed 70 insns (limit 1000000) max_states_per_insn 0 total_states 7 peak_states 7 mark_read 0
The insns_self counts account for every processed instruction exactly once:
29 + 9 + 15 + 7 + 10 = 70
The main program and global subprograms are independent exploration roots,
so their insns_total counts also account for the full processed budget:
51 + 9 + 10 = 70
Static subprogram totals provide a nested, top-down breakdown inside their
root. In this example:
sub4: 22 = 15 self + 7 in sub3
prog4: 51 = 29 self + 22 in sub4
The global subprogram bodies are accounted in their own root totals rather
than being included in prog4 or the static callees which call them.
Asynchronous callbacks start from fresh frame-zero verifier states, but the
work remains part of the do_check_common() invocation for the main or global
verification root under which it was scheduled. Running:
./test_progs -t verifier_subprog_insn_stats/stats_async_nested -v
produces the following stats:
stack depth max 0
subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0
subprog 1 (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0
subprog 2 (stats_async_outer) static insns_self 7 insns_total 7 stack 0
subprog 3 (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0
processed 25 insns
Here, 9 + 2 + 7 + 7 = 25. The main root total is the complete verifier
budget for its do_check_common() invocation, including both directly and
transitively scheduled asynchronous callbacks. Static subprogram and
callback totals remain local to their synchronous paths.
Changelog:
----------
v7 -> v8
v7: https://lore.kernel.org/bpf/20260808062601.1070988-1-memxor@gmail.com
* Move the insns_total snapshot and delta for main and global roots into
do_check_common() and explain why the override is needed for async
subprograms. (Eduard)
* Avoid splitting __msg string literals in the stack-depth stats tests.
(Eduard)
* Add a comment explaining why both the new per-subprogram records and the
legacy one-line format are matched in veristat's parse_verif_log().
(Eduard)
v6 -> v7
v6: https://lore.kernel.org/bpf/20260805011517.1717238-1-memxor@gmail.com
* Rename insns_own to insns_self throughout. (Andrii)
* Drop the async accounting call stack and attribute callback work to the
scheduling main or global verification root using its processed-insn
delta. (Eduard, Andrii)
* Skip missing frames when folding instruction totals after a partial
verifier state copy. (BPF CI Bot)
* Use explicit callback argument operands in deterministic instruction-count
tests and update tests and examples for root attribution. (BPF CI Bot)
v5 -> v6
v5: https://lore.kernel.org/bpf/20260804081114.3871564-1-memxor@gmail.com
* Track self and inclusive instruction counts for main, global, and static
subprograms. (Andrii, Eduard)
* Keep instruction subtotals path-local across verifier state copies.
* Propagate async callback budget through nested scheduling chains. (Andrii)
* Split per-subprogram instruction accounting into a preparatory patch.
* Add deterministic selftests with exact self, total, and processed counts.
v4 -> v5
v4: https://lore.kernel.org/bpf/20260803072733.191502-1-memxor@gmail.com
* Change the format to combine instruction counts and stack depths into
per-program records. (Andrii)
* Adjust veristat for the new format while retaining support for the legacy
format.
* Explain why the legacy stack parsing buffer is zero-initialized. (BPF CI
Bot)
v3 -> v4
v3: https://lore.kernel.org/bpf/20260803031457.3115812-1-memxor@gmail.com
* Read names from subprog_info directly to avoid an out-of-bounds access
when func_info validation fails. (BPF CI Bot)
v2 -> v3
v2: https://lore.kernel.org/bpf/20260802225209.2511758-1-memxor@gmail.com
* Reuse subprog_name() to fetch subprogram names. (BPF CI Bot)
v1 -> v2
v1: https://lore.kernel.org/bpf/20260801230400.850271-1-memxor@gmail.com
* Use multi-line format. (Eduard)
* Adjust veristat to work with old and new format.
* Adjust selftest log_level without new option. (Eduard)
====================
Link: https://patch.msgid.link/20260812221925.3358041-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add small verifier programs with deterministic instruction streams to
exercise per-subprogram self and inclusive instruction accounting. Use
assembly for normal call chains and straight-line callback bodies containing
only moves, calls, and returns or exits, so control-flow pruning does not make
the expected counts unstable. Pass callback arguments as explicit assembly
operands so the compiler keeps their registers live across the asm block.
Cover asynchronous callback attribution separately: main verification-root
totals include all callback exploration, while static and callback totals
remain local to their synchronous paths.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Test the per-program insns_self, insns_total, and stack depth statistics
emitted when BTF function info does not provide subprogram names. Check that
the subprog 0 main record and static-subprogram records use <unknown>.
Make VERBOSE_ACCEPT request verifier statistics so the raw-insn test can
validate the output without a test-specific log level.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier now reports instruction and stack depth statistics using
uniform "subprog <id> (<name>) <kind>" records. Subprogram 0 is classified
as main, while other records are global or static. Each record carries
insns_self, insns_total, and stack depth.
Teach veristat to parse the new records while retaining support for the
legacy one-line stack depth format used by older kernels. Skip both
instruction counts and match only through the stack value so fields can
still be appended without breaking parsing.
Increase the bounded backward scan so it can include all 256 per-subprogram
records.
Zero-initialize the legacy stack buffer because logs using the new format do
not populate it before the trailing tokenizer loop. This makes the loop see
an empty string instead of reading uninitialized data.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
vmtest.sh captures the trailing command with command="$@", which flattens
the arguments into a single space-separated string, and then pastes it
into the generated guest init script:
cd /root/bpf
echo ${command}
stdbuf -oL -eL ${command}
That here-doc is unquoted, so the host expands ${command} and the
flattened text lands in the script verbatim. The guest bash then parses
those lines as shell source, re-splitting the text on whitespace and
glob-expanding it against /root/bpf. As a result any command with a glob
or an argument containing spaces is corrupted before it reaches the test
binary. For example:
vmtest.sh -- ./test_progs -a 'verifier_*'
has 'verifier_*' expanded in the guest into the matching object/skeleton
files (verifier_align.bpf.o verifier_align.skel.h ...), so test_progs is
handed a list of filenames instead of the intended name filter and runs no
matching tests.
Quote each argument with printf '%q ' so the command is reproduced
verbatim inside the VM: the escaped text goes through exactly one round
of quote removal when the guest parses the init script, yielding the
original argv with globs and special characters intact. The common case
(e.g. -t <name>) is unaffected.
Only do this when there is a command to quote. printf '%q ' with no
arguments still applies the format once and emits '', which the -s
(debug shell) path would take for a real command and try to run.
Note this makes the trailing command strictly an argv rather than a shell
snippet: passing it pre-quoted as one word, e.g.
vmtest.sh -- "./test_progs -t foo"
no longer works, and neither does embedding guest-side shell syntax such
as ';' or a redirection. 'sh -c ...' still works.
The RV64 recipe in README.rst does depend on the old double parse: it
wraps the denylist in \" so the literal quotes reach the guest, whose
second parse of the init script removes them. Under %q those quotes now
survive into argv, and parse_test_list() strtok_r()s on ',' turns them
into junk filters:
-d ",exceptions," -> ["] [exceptions] ["]
That is harmless for DENYLIST.riscv64 only because its first line is a
comment, so the leading field is empty. A denylist starting with a real
entry would silently lose it - ["*arena*] never matches - so drop the
backslashes and let the host consume the quotes instead.
Fixes: c9709f5238 ("bpf: Helper script for running BPF presubmit tests")
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807204434.1036279-5-vineet.gupta@linux.dev
The final summary line is asymmetric: the PASSED field reports both the
number of top-level tests and the number of subtests within them, while
the FAILED field reports only top-level tests:
Summary: 640/5750 PASSED, 7760 SKIPPED, 100 FAILED
There is no way to tell whether those 100 failing tests amount to 100
broken subtests or 1000. So count subtests with a non-zero error_cnt
into a new sub_fail_cnt and print it alongside fail_cnt:
Summary: 640/5750 PASSED, 7760 SKIPPED, 100/342 FAILED
^^^^^
This is correct for -j runs, as subtest_states[] is populated both in
sequential and parallel modes.
A test that fails without declaring any subtests contributes 0 to
sub_fail_cnt. That mirrors the existing behaviour of sub_succ_cnt for
tests that pass without subtests, keeping the two numerators
comparable.
Also emit the new count as a "failed_subtest" field in the JSON output,
for parity with the existing "success_subtest".
Note that this changes the trailing field of the summary line from a bare
integer to "A/B", so anything scraping "N FAILED" out of it needs updating.
While here, fix the fail_cnt comment in struct test_env, which claims it
counts "total failed tests + sub-tests".
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807204434.1036279-4-vineet.gupta@linux.dev
By default test_progs re-prints the aggregated error logs of all failed
tests at the end of the run (when not in verbose mode), starting with
"All error logs:".
With bpf-gcc the current failures and a couple runaway 1M fails cause a
huge print overhead/delay at the end.
Add a subtractive --no-error-summary flag, gated on a new
env.error_summary field which defaults to true, so the default behavior
is unchanged. Passing --no-error-summary suppresses the final
"All error logs:" dump.
Only the human readable output is elided. dump_test_log() also emits the
per-test and per-subtest entries of the --json-summary "results" array,
so it keeps being called (via a new @quiet argument) and the JSON report
is bit for bit what it was before.
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260807204434.1036279-3-vineet.gupta@linux.dev
reject_scalar_store_to_kptr stores a scalar constant to a kptr field:
*(volatile u64 *)&v->unref_ptr = 0xBADC0DE;
Compilers generate one of two encodings for that:
1. Materialize the constant into a register and emit BPF_STX:
r1 = 0xbadc0de
*(u64 *)(r0 + 0x8) = r1
2. Or fold it into a single BPF_ST (store immediate):
*(u64 *)(r0 + 0x8) = 0xbadc0de
These go through different rejection paths and output different
messages.
- BPF_STX goes through map_kptr_match_type(), which prints
"invalid kptr access, R...".
- BPF_ST only gets the immediate check printing
"BPF_ST imm must be 0 when storing to kptr"
The test only expects the BPF_STX message, so it fails on a toolchain
that folds the constant - bpf-gcc, and clang -mcpu=v4:
7: (7a) *(u64 *)(r0 +8) = 195936478
BPF_ST imm must be 0 when storing to kptr at off=8
...
EXPECTED SUBSTR: 'invalid kptr access, R'
Pick the expected message with __BPF_FEATURE_ST, which clang and bpf-gcc
both define exactly when BPF_ST codegen is available - cpuv4 for clang,
and by default for bpf-gcc, whose default cpu is v4.
bpf-gcc, before: #229/20 map_kptr/reject_scalar_store_to_kptr:FAIL
bpf-gcc, after : #229/20 map_kptr/reject_scalar_store_to_kptr:OK
Two caveats worth noting:
- On a BPF_ST toolchain the test now only exercises the imm != 0 check
and never reaches map_kptr_match_type(), so the scalar-vs-PTR_TO_BTF_ID
rejection the test is named for is only covered by the non-ST builds.
The imm path itself is already covered compiler-independently by
verifier/map_kptr.c ("map_kptr: BPF_ST imm != 0").
- __BPF_FEATURE_ST says the compiler *can* emit BPF_ST, not that it will.
The encoding also depends on the optimization level: clang -mcpu=v4 -O0
still emits BPF_STX, which would send the #ifdef down the wrong branch
and fail the test. Selftests always build BPF objects at -O2 so this
does not bite today, but it is a latent failure mode if that changes.
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Link: https://lore.kernel.org/bpf/20260807204434.1036279-2-vineet.gupta@linux.dev
Add stream_arena_xchg_fault and stream_arena_cmpxchg_fault next to the
existing read, write and load-acquire fault tests, covering the two
places a read-modify-write can deposit the old value: src_reg for a
BPF_XCHG and r0 for a BPF_CMPXCHG. Both cover both halves of the JIT
bug that left the fetch destination alone when a RMW on an arena pointer
faulted:
- the fault has to be reported as a WRITE, and at the address held by
the destination register, which __stderr() and test_address() check
- the register receiving the fetched value has to be cleared by the
fault handler, which the programs check by poisoning it before the
atomic and returning it, so __retval(0) fails if it is left untouched
The __stderr() annotation can only wildcard the faulting address since
the arena base is not known until runtime, hence the two test_address()
subtests on top, which pin it to the address held by dst_reg rather than
src_reg.
Note, the atomics are open coded since linux/filter.h cannot be included
alongside vmlinux.h.
# LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream
[...]
#464/1 stream_arena_fault_address/read_fault:OK
#464/2 stream_arena_fault_address/write_fault:OK
#464/3 stream_arena_fault_address/load_acquire_fault:OK
#464/4 stream_arena_fault_address/xchg_fault:OK
#464/5 stream_arena_fault_address/cmpxchg_fault:OK
#464 stream_arena_fault_address:OK
[...]
#466/5 stream_success/stream_arena_write_fault:OK
#466/6 stream_success/stream_arena_read_fault:OK
#466/7 stream_success/stream_arena_load_acquire_fault:OK
#466/8 stream_success/stream_arena_xchg_fault:OK
#466/9 stream_success/stream_arena_cmpxchg_fault:OK
[...]
Summary: 4/22 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Puranjay Mohan <puranjay@kernel.org>
Link: https://patch.msgid.link/20260811131600.506721-6-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Same missing register clear as on riscv64. A RMW atomic on an arena pointer
is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but
bpf_jit_probe_atomic_pre() only fills in the arena base and the probe
offset, leaving probe->reg at the -1 that bpf_jit_probe_init() set, which
bpf_jit_probe_post() writes into the entry and ex_handler_bpf() then reads
back as "there is nothing to clear".
That is right for a plain BPF_{ADD,AND,OR,XOR}, which only writes memory,
but an RMW carrying BPF_FETCH also reads the old value into a register:
src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and r0 for
BPF_CMPXCHG. So on a fault over an unmapped arena page the program resumes
at the landing pad with whatever that register held before the atomic
instead of the 0 that every other BPF_PROBE_* access delivers.
Fill probe->reg in from bpf_atomic_load_reg(). Unlike x86-64 and arm64,
s390x does not report arena violations from its exception handler, so there
is no access direction to correct here, only the missing register clear.
Fixes: 2f9469484a ("s390/bpf: Support arena atomics")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Ilya Leoshkevich <iii@linux.ibm.com>
Link: https://patch.msgid.link/20260811131600.506721-5-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Same problem as on x86-64: add_exception_handler() folds "there is no
destination register to clear" and "this is a store" into one DONT_CLEAR
value ...
if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn))
dst_reg = DONT_CLEAR;
... which ex_handler_bpf() then reads back as the access direction:
bool is_write = (dst_reg == DONT_CLEAR);
A RMW carrying BPF_FETCH is both. emit_lse_atomic() reads the old value
into src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and into
r0 for BPF_CMPXCHG, so a fault over an unmapped arena page is correctly
reported as a WRITE but leaves that register holding a stale value instead
of the 0 that every other BPF_PROBE_* access delivers. Same as on x86-64,
add a separate ARENA_WRITE bit for the direction.
FIXUP_REG is now filled in by the callers of add_exception_handler(), the
BPF_PROBE_ATOMIC one deriving it from bpf_atomic_load_reg(), so that the
helper only has to determine the direction. This is how the riscv64 JIT
already does it, and it stops the two store callers from handing in a
dst_reg that was only going to be overwritten with DONT_CLEAR anyway.
Fixes: e612b5c1d3 ("bpf, arm64: Add support for lse atomics in bpf_arena")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Link: https://patch.msgid.link/20260811131600.506721-4-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
populate_extable() encodes "there is no destination register to clear" as
DONT_CLEAR in the DST_REG field of the exception table metadata, and later
ex_handler_bpf() then reuses that very value to derive the direction it
reports the fault with is_write = (reg == DONT_CLEAR). The two coincide
for a plain load or store, but not for a RMW carrying BPF_FETCH. Such an
atomic writes memory, so it has to be reported as a WRITE, and it also reads
the old value into a register, src_reg for BPF_ADD | BPF_FETCH and BPF_XCHG,
r0 for BPF_CMPXCHG, so that register has to be cleared on fault. A single
DONT_CLEAR cannot say both, and the store branch picks it unconditionally:
[...]
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
[...]
The reported direction is therefore right, but on a fault over an unmapped
arena page the fetch destination keeps whatever it held before the atomic,
where every other BPF_PROBE_* access delivers 0. Give the metadata its own
ARENA_WRITE bit so that the reported direction no longer depends on whether
there is a register to clear, and fill DST_REG in from bpf_atomic_load_reg().
BPF_{AND,OR,XOR} | BPF_FETCH need no handling here, bpf_jit_supports_insn()
already rejects those in the arena.
Fixes: d503a04f8b ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-3-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
A RMW atomic on an arena pointer is converted to BPF_PROBE_ATOMIC and
gets an exception table entry, but that entry records no destination
register to clear unless the instruction is a load-acquire today. That
is right for a plain BPF_{ADD,AND,OR,XOR}, which only writes memory,
but an RMW carrying BPF_FETCH also reads the old value into a register:
src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and r0 for
BPF_CMPXCHG. emit_atomic_rmw() emits it that way, e.g.:
[...]
case BPF_XCHG:
ctx->ex_insn_off = ctx->ninsns;
emit(is64 ? rv_amoswap_d(rs, rs, rd, 1, 1) :
rv_amoswap_w(rs, rs, rd, 1, 1), ctx);
[...]
Thus, a fault over an unmapped arena page ex_handler_bpf() jumps over
the access but leaves rs untouched, and the program resumes with
whatever it held before the atomic instead of the 0 that every other
BPF_PROBE_* access delivers. Fill the exception table entry in from
bpf_atomic_load_reg(), which returns the BPF register an atomic reads
the memory operand into or -1 when it has none. A load-acquire ends up
with the same register it gets today, it just goes through the helper.
Unlike x86-64 and arm64, riscv64 does not report arena violations from
its exception handler, so there is no access direction to correct here,
only the missing register clear.
Fixes: fb7cefabae ("riscv, bpf: Add support arena atomics for RV64")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://patch.msgid.link/20260811131600.506721-2-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register
it reads the old value into, the BPF_STX case of insn_def_regno() open codes
the very same mapping a second time, the const folding and the liveness
transfer functions a third and a fourth time, and BPF JITs need it as well
to know which register a faulting BPF_PROBE_ATOMIC has to clear.
Add a small helper so that all of them can share it. No functional change.
The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction
classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the
helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally
accepts cannot be seen there as it is only set from bpf_do_misc_fixups(),
that is, after const folding has run. arg_track_xfer() keeps its mode test
since that also guards the stack clearing next to it.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Ning Ding says:
====================
bpf: Compare iterator types during state pruning
Iterator stack slots can be marked MEM_RCU or PTR_UNTRUSTED. The
STACK_ITER check in stacksafe() does not compare this type, so state
pruning can treat these states as equal and prune an unsafe path.
Compare the type and add a test where RCU protection has a gap.
---
Changes in v2:
- Convert the regression test to inline assembly so its verifier-sensitive
control-flow layout is stable.
- Add Eduard Zingerman's Acked-by tag to patch 1.
v1: https://lore.kernel.org/bpf/20260807004320.134069-1-dingning04@gmail.com/
====================
Link: https://patch.msgid.link/20260811035955.132989-1-dingning04@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Add arena argument support to kfuncs and struct_ops
This is a continuation of patches in [0], with mostly minor changes and
reordering. The motivation is covered in that link. A major change is
moving to two tags (__arena and __arena__nullable) and moving the changes
to JIT to emit more optimized sequences.
Please see commit logs for details.
[0]: https://lore.kernel.org/bpf/20260713024414.3759854-1-tj@kernel.org
Changelog:
----------
v4 -> v5
v4: https://lore.kernel.org/bpf/20260805210427.3218326-1-memxor@gmail.com
* Remove the redundant patch-8 capability comment and duplicate
nullable kfunc test coverage. (Eduard)
* Introduce the final bpf_tramp_arena_base() interface directly with
function-model argument flags, avoiding temporary slot bitmaps and
arena_nullable state; simplify struct_ops pointer validation. (Eduard)
* Simplify kfunc arena nullability classification by using the common
nullable path for both arena suffixes while leaving the function model
to distinguish JIT NULL preservation. (Amery)
* Keep bpf_prog_has_arena_ctx_arg() in bpf_verifier.h from its
introduction so trampoline and verifier users share one inline
definition, avoiding BPF_JIT/BPF_SYSCALL link dependencies.
(Eduard, BPF CI Bot)
* Reject both tracing and extension attachments to struct_ops programs
with arena context arguments, and add fentry, fexit, and freplace
rejection tests. (Eduard, Sashiko)
v3 -> v4
v3: https://lore.kernel.org/bpf/20260803125115.2264733-1-memxor@gmail.com
* Rename __arena_nullable to __arena__nullable and prioritize the
composite suffix over __nullable during argument classification.
(Sashiko, Eduard)
* Resolve instructions before collecting subprograms and kfuncs so kfunc
prototype validation can use associated arena state.
* Move the arena kfunc and JIT-sequence test entry points into
prog_tests/verifier.c. (Eduard)
* Match the generated L0 target and call in nullable JIT assertions.
(Eduard)
* Route arena kfunc validation through the common argument-checking path.
(Amery)
* Reuse btf_func_model argument flags for struct_ops arena arguments
instead of maintaining separate trampoline slot metadata. (Eduard)
* Check the generic-trampoline arena argument invariant at link time and
warn once on violations. (Eduard)
* Reject tracing attachments to struct_ops programs with arena context
arguments whose indirect trampolines convert the pointers. (Sashiko)
v2 -> v3
v2: https://lore.kernel.org/bpf/20260726013105.3689867-1-memxor@gmail.com
* Rebase onto current bpf-next to resolve conflicts.
v1 -> v2
v1: https://lore.kernel.org/bpf/20260715220052.1590783-1-memxor@gmail.com
* Fix documentation to only mention x86 for now. (Sashiko)
* Move arg bitmap from insn_aux_data to kfunc descriptor. (Eduard)
====================
Link: https://patch.msgid.link/20260808003938.3486067-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Reject tracing and freplace attachments to a target program with arena
context arguments. The struct_ops indirect trampoline converts those
arguments before entering the target, so a generic tracing trampoline
would otherwise expose arena offsets using the target BTF pointer type.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-14-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a test_arena_stack member with eight leading scalar arguments so the
arena pointer is passed on the stack.
The callback validates the first and last scalar ctx slots before
dereferencing the pointer in ctx[8]. This exercises the indirect
trampoline stack layout and arena conversion together, and prevents a
regression where stack arguments are read one slot late.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tested-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-13-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
save_args() reads stack-passed arguments relative to rbp assuming two
return addresses sit between the saved rbp and the arguments, which
holds when the trampoline is entered through the fentry call from a
traced function. An indirect trampoline is called through a function
pointer, so only the caller's return address is on the stack and the
arguments start at rbp + 16, not rbp + 24. Every stack-passed argument
of a struct_ops callback with more than six argument slots is read one
slot off.
This has gone unnoticed because no in-tree struct_ops member passes
arguments on the stack. The jmp-entry form already accounts for having
a single return address; treat BPF_TRAMP_F_INDIRECT the same way.
Fixes: 473e3150e3 ("bpf, x86: allow function arguments up to 12 for TRACING")
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Tested-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-12-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add test_arena and test_arena_nullable members to bpf_testmod_ops3 with
arena-tagged stub arguments and kfuncs that forward a caller-provided
pointer to them. The kfuncs take arena-tagged arguments, so each round
trip exercises both conversion directions end to end: the kfunc receives
a kernel arena address and the trampoline converts it back to an arena
pointer for the callback.
The non-nullable callback dereferences its argument with no NULL branch
and captures the raw ctx value, which the trigger program compares
against the arena offset of the passed object, pinning the exact
(u32)(kaddr - kern_vm_start) conversion. The nullable callback verifies
that only a true kernel NULL arrives as NULL. Failure coverage: a
program with no arena is rejected when it loads. The tests run on x86-64
and skip elsewhere, as the programs fail verification where the JIT
lacks arena argument support.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-11-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Pin the exact rebase sequences the JITs emit for __arena and
__arena__nullable kfunc arguments with __jited assertions on x86-64: the
unconditional truncate-and-add, the nullable test-and-skip variant, and
all five argument registers in one call, which also covers the
REX-prefixed encoding of r8 on x86. The capture kfuncs take the argument
without dereferencing, so only the emitted code is under test. The
tests skip without LLVM disassembler support.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-10-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add arena-argument kfuncs to bpf_testmod, which also exercises the
argument rebasing on module kfuncs, and tests covering the accepted
argument forms (arena pointer, low 32 bits as a scalar, full user
address as a scalar), the exact rebase semantics via capture kfuncs
returning the raw argument (zero low 32 bits arrive as the arena kernel
base under __arena and as NULL under __arena__nullable), five arena
arguments in one call, a mixed __arena plus __arena__nullable call
exercising both bitmasks on one call site, a kernel-side dereference of
an unpopulated page recovering through the scratch page, and the
rejections (no arena in the program, incompatible register type).
The tests run on x86-64 and skip elsewhere, as programs with
arena-tagged kfunc args fail verification where the JIT lacks support.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-9-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Implement the struct_ops arena argument conversion on x86. save_args()
receives the arena base from bpf_tramp_arena_base() and consults the
btf_func_model argument flags as it copies each native argument into the
BPF ctx, routing a marked argument through RAX:
movl %esrc, %eax /* truncate and clear the upper 32 bits */
subl $base_lo, %eax
movq %rax, ctx_slot
A nullable argument tests the full 64-bit kernel pointer first:
movq %rsrc, %rax
testq %rax, %rax
jz 1f
subl $base_lo, %eax
1:
movq %rax, ctx_slot
The 32-bit subtraction is sufficient since (u32)(kaddr - base) ==
(u32)kaddr - (u32)base, and it clears the upper half as the JITs require
of arena pointer registers. Stack-passed arguments already reload
through RAX, so only the subtraction (and the NULL test) is inserted
there.
Keep arena and nullable classification in btf_func_model.
bpf_tramp_arena_base() returns a base only for a single-program
struct_ops indirect trampoline; other trampolines pass zero and perform
no conversion. The size probe reruns the same emission with the same
model and nodes, so the image size matches by construction.
With both the kfunc and struct_ops directions implemented, flip
bpf_jit_supports_arena_args() on for x86.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260808003938.3486067-8-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
