Record material register and outgoing stack argument changes so diagnostics can
explain how a value reached its current type, bounds, or unreadable state.
Store old and new register types, scalar ranges, tnum value and mask, map and
BTF type identity, and basic operand metadata in the environment-owned
diagnostic event stream.
Record invalidations when packet data moves, references are released, or
borrowed references leave their protected region. Register-scoped history
starts at the latest matching modification and then shows later branch
outcomes.
Also record fixed stack spills and overwrites, and tag register fills from
stack so register-scoped history can follow value flow through spilled stack
slots.
The type_is_map_ptr() helper previously lived as a static function in
kernel/bpf/log.c since commit 0c95c9fdb6 ("bpf: emit map name in register
state if applicable and available"). Move it verbatim to
include/linux/bpf_verifier.h as a static inline, next to the other type
classifiers, so diagnostics.c can reuse it without duplicating the case list.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Save the diagnostic event-log position with each verifier stack entry and
reset the environment-owned stream together with the normal verifier log
when a queued state is popped. Also reset the diagnostic stream after
successful subprogram verification even when level-2 logging preserves the
normal verifier log.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add an environment-owned diagnostic history for verifier reports. Event
payloads keep the user-facing branch history shape, while storage lives
in bpf_verifier_env and follows the active verifier path.
Grow the event array geometrically up to a 64 MiB limit. Once storage
reaches the limit, or an allocation fails, overwrite the oldest event so
diagnostics retain the newest useful suffix without adding per-event
metadata.
Represent saved positions as absolute logical sequence numbers. A restore
truncates to a retained position. If its prefix has already been evicted,
clear the abandoned suffix and preserve the missing-history position. This
keeps marks stable across rotation without increasing their size.
Add the branch event renderer and branch recording.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Teach verifier diagnostics to annotate an instruction with BTF source
line information and nearby BPF instructions. The renderer keeps source
text in a fixed-width lane and prints instructions in a stable right-hand
gutter.
Wrap annotation text under the source line so long error labels remain
readable while the source and instruction lanes keep their fixed layout.
Keeping source and instruction context in one commit preserves the visual
layout contract that later diagnostic reports rely on.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add the initial diagnostics renderer for verifier reports and wire it into
the BPF build. The helper emits the common failure header through the
verifier log.
Later patches add prose wrapping, reusable report sections, and source and
instruction context for category-specific diagnostics.
Gate the helpers on normal verifier log output from the start, so
BPF_LOG_STATS-only loads do not collect or render diagnostics.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260815064612.378577-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Vineet Gupta says:
====================
bpf, x86: fix per-CPU address resolution into an extended register
The JIT resolves a per-CPU address with
add <dst>, gs:[this_cpu_off]
but builds the REX prefix with add_1mod(), which sets REX.B. The
destination is encoded in ModRM.reg, which REX.R extends, and the memory
operand is disp32 with no base, so REX.B does nothing and the high
register bit is dropped. Every extended destination therefore resolves
into whichever register shares the low three bits:
R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI
The address is left unadjusted and an unrelated register is clobbered.
Patch 1 switches to add_2mod() so the bit goes through REX.R.
Clang reloads the address into R1 before each per-CPU access, so the
destination is never an extended register and the bug has been dormant
since v6.10. GCC keeps several per-CPU addresses live at once, which is
how it turned up: test_progs-bpf_gcc panics the kernel in
global_percpu_data/init, with the address of a .percpu variable in R5.
Patch 2 covers every register. A functional test only catches this if
the address happens to land in an extended register, so the test matches
the JITed add instead.
Changes in v3:
- Fold the five per-register programs into one that loads every
register, and drop the comment explaining the register choice
(Eduard Zingerman).
- Move the percpu_data declaration inside the arch guard, so other
targets no longer carry a .percpu section and an unused map (bpf-ci).
- Match the movabsq of each address as well as the add, so the matchers
stay on consecutive lines and the pair is checked to use the same
register.
- Restore the Reviewed-by on patch 1, dropped by mistake in v2.
Changes in v2:
- Add the selftest, patch 2/2 (Eduard Zingerman). It uses __jited()
rather than __xlated(): the xlated stream is identical for every
register, and the wrong prefix is only visible in the native encoding.
- No functional change to patch 1.
====================
Link: https://patch.msgid.link/20260814220254.3797467-1-vineet.gupta@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
An ld_imm64 of a per-CPU map value is followed by a mov_percpu_addr that
reuses the same register, so which register the address lands in decides
how the JIT encodes the add. Getting the REX prefix wrong there is
invisible to a functional test unless the address happens to land in an
extended register, which is why this went unnoticed.
Load a .percpu variable into every register in one program and match the
JITed add against the register each one must resolve into.
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Link: https://patch.msgid.link/20260814220254.3797467-3-vineet.gupta@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The destination of the per-CPU address MOV is encoded in ModRM.reg,
which is extended by REX.R, but the REX prefix is built with
add_1mod(), which sets REX.B. REX.B extends ModRM.rm and SIB.base, and
this instruction addresses memory as disp32 with no base, so the bit
has no effect at all and the high register bit is simply lost.
Every is_ereg() destination therefore resolves to the wrong register,
picking whichever one shares the low three bits:
R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI
With BPF_REG_5, whose reg2hex is 0, the emitted
65 49 03 04 25 <off> add %gs:<off>,%rax
adds the per-CPU offset to RAX rather than R8. The destination keeps
the unadjusted address and RAX is clobbered, so the program goes on to
dereference a pointer that was never made per-CPU:
BUG: unable to handle page fault for address: 0000607e386a8894
RIP: bpf_prog_707837aafd2aa9ae_update_percpu_data+0x93/0xc9
Call Trace:
__bpf_prog_test_run_raw_tp+0x2dc/0x7d0
__flush_smp_call_function_queue+0x1e9/0xc80
Kernel panic - not syncing: Fatal exception in interrupt
R5 is the mildest of the four, aliasing a scratch register and faulting
at the store. R7 aliases RBP and would corrupt the frame pointer, R8
and R9 alias the argument registers.
Use add_2mod() so the register goes through REX.R, matching how
add_2reg() places it in ModRM.reg and how emit_priv_frame_ptr()
hardcodes 0x4c for the same instruction with R9. Encodings for the
non-extended registers are unchanged.
Problem showed up when trying to resurrect BPF_GCC CI (selftests built
with BPF_GCC).
This has gone unnoticed because clang reloads the address into R1
before each per-CPU access, so the destination is never an extended
register. GCC keeps several per-CPU addresses live at once, and
test_progs-bpf_gcc panics the kernel in global_percpu_data/init, where
the address of a .percpu variable ends up in R5.
Fixes: 7bdbf74463 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260814220254.3797467-2-vineet.gupta@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.
The handler is reached through a new optional ->map_mmap_fault callback.
Maps that provide it get a vm_operations_struct with a .fault handler;
maps that populate their mapping eagerly keep the one they had. Both
share the same open/close callbacks, so the existing VMA accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized
rather than each map installing its own vm_operations_struct.
Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.
Time for one mmap()+munmap() of an 8MiB mmap-able array map:
before after
no MAP_POPULATE, no access 226us 1.1us
no MAP_POPULATE, access all pages 236us 1341us
MAP_POPULATE, no access 312us 493us
MAP_POPULATE, access all pages 318us 519us
Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org
ringbuf_process_ring() walks the records between the consumer and the
producer with an ordering comparison:
while (cons_pos < prod_pos) {
cons_pos and prod_pos mirror the kernel's ring positions and are
unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic.
When producer_pos has wrapped and consumer_pos has not, prod_pos is the
smaller of the two, the loop body never runs and no record is consumed.
Since consumer_pos only advances inside that loop, it never wraps either
and the consumer stops delivering samples for good, with no error
returned to the caller: ring_buffer__poll() keeps reporting zero
records while the kernel side fills up and starts dropping.
Compare the distance instead. The consumer never runs ahead of the
producer, so prod_pos - cons_pos is the amount of unconsumed data and
stays correct across the wrap.
64-bit hosts are unaffected in practice: the counters would need 16 EiB
to wrap. This is the userspace counterpart of the kernel-side walk fixed
in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit
consumer hits whichever of the two comes first.
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com
In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer
and overwrite positions before measuring how much data is available:
return prod_pos - max(cons_pos, over_pos);
max() is an ordering comparison, and consumer_pos, producer_pos and
overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures,
where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the
two positions has wrapped and the other has not, max() returns the older
one: the result is then a modular difference close to 2^32, so the
function reports far more available data than the ring can hold. Pollers
using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be
woken with nothing to read.
Compare distances rather than positions. prod_pos - X is the amount of
data produced since X for either position, wrap or no wrap, so the newer
position is simply the one with the smaller distance, which is also the
value the function wants to return.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap. Found by review of the same class of bug fixed in
"bpf: Fix pending_pos walk on 32-bit ring position wrap".
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com
The reservation path caches the position of the oldest not-yet-committed
record in rb->pending_pos and advances it past already committed records
on every reservation:
while (pend_pos < prod_pos) {
consumer_pos, producer_pos and pending_pos are unsigned long, i.e.
32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states
that these counters may wrap around there. Every other comparison in the
file is written as a difference, so modular arithmetic keeps them
correct across the wrap. This one is an ordering comparison, and it is
not wrap-safe.
Once producer_pos wraps past 2^32, prod_pos is small while pend_pos
still holds its pre-wrap value, so the loop condition is false and
pending_pos is never advanced again. Reservations keep succeeding for a
while, because bpf_ringbuf_has_space() uses differences, but
new_prod_pos - pend_pos grows as the producer advances, and once it
exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails:
the kernel believes a pending record spans the whole buffer. The ring
never recovers, bpf_ringbuf_output() drops every event from then on, and
nothing is logged.
Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a
tracepoint-based collector with a 512 KiB ring and 160-byte records.
Every one of them stopped delivering after exactly 26846821 records and
4295491360 bytes had passed through the ring, at event rates between 441
and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is
the byte count, not time or load. That figure is 2^32 plus 524064 bytes,
and the excess is one ring's worth of grace period, as expected while
new_prod_pos - pend_pos is still below rb->mask. The last reservation
that fits is the largest record boundary X with X + 160 <= 524287, and
since 2^32 mod 160 = 96 the boundaries after the wrap sit at
X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally
until the producer stopped, then read zero records for good. With this
patch applied, one of the four devices took 10 GiB through the same ring
with no stall, while the three unpatched ones kept wedging at the same
byte count.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap.
Compare the two positions as a difference instead. pending_pos never
runs ahead of producer_pos, so the unsigned difference is the real
distance between them and stays correct across the wrap.
Fixes: cfa1a2329a ("bpf: Fix overrunning reservations in ringbuf")
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-2-i.tellez@btesa.com
Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz().
This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap()
when the old and new value sizes occupy the same number of pages.
Fix some typos btw:
* mmapble -> mmapable
* satisified -> satisfied
* relocatin -> relocation
* atach_btf_obj_fd -> attach_btf_obj_fd
* len_secnd -> len_second
* precendence -> precedence
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev
On RV64, the ABI requires sign-extension for signed 1-byte and 2-byte kfunc
args. However, the RV64 JIT currently does not perform sign-extension for
such kfunc args.
Before commit 7ce090afbf ("bpf: Infer zext_dst based on static register
liveness analysis"), state pruning could potentially omit zero-extension
of 32-bit subregisters, which inadvertently masked the above issue by making
the args appear as if they had been properly sign-extended. After that
commit, the problem is exposed, causing the kfunc_call/kfunc_call_test4
selftest to fail.
Fix this by extending the existing sign-extension logic to handle signed
1-byte and 2-byte kfunc args as well.
Fixes: 443574b033 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260814064726.3607615-1-pulehui@huaweicloud.com
pyperf600 has been running into 8K BPF_COMPLEXITY_LIMIT_JMP_SEQ limitations
for a long while now, after some internal compiler changes.
Until BPF verifier is bestowed with scalar evolution logic, make that test
actually work by doing what would anyone should do in such situations: by
moving repeatable per-iteration work into independently verified global
functions.
`void *` argument is a problem for global funcs, but a static function wrapper
doing necessary casts and a bit of __arg_nonnull magic dust is all it takes.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Tested-by: Pu Lehui <pulehui@huawei.com> # riscv
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20260813232943.581283-1-andrii@kernel.org
lwt_ip_encap hardcodes the ping6 binary for its IPv6 pings. iputils
merged ping6 into ping long ago and distros have started dropping the
compat symlink -- Arch's iputils 20250605 ships only arping, clockdiff,
ping and tracepath. There, every lwt_ip_encap subtest fails:
check_ping_ok:FAIL:ip netns exec ns-lwt-ip-encap-1-0101330 ping6 -c 1 \
-W1 -I veth1 fb04::1 > /dev/null unexpected error: 256 (errno 2)
#217/1 lwt_ip_encap_ipv4/egress:FAIL
The IPv4 subtests fail too, because check_ping_ok() pings both families.
SYS() runs the command through system(), so a missing binary is
indistinguishable from an unreachable peer.
network_helpers.c has had ping_command() for exactly this since commit
372642ea83 ("selftests/bpf: Move netcnt test under test_progs"): it
falls back to "ping -6" when ping6 is not present. lwt_ip_encap.c is the
last hardcoded ping6 user. Fix that.
Fixes: f5e288943e ("selftests/bpf: Move test_lwt_ip_encap to test_progs")
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260813213558.3103179-1-andrii@kernel.org
Puranjay Mohan says:
====================
bpf, arm64: __arena kfunc and struct_ops arguments
The x86-64 JIT recently gained support for the __arena and
__arena__nullable argument suffixes on kfuncs and struct_ops stubs. This
adds the arm64 side and flips bpf_jit_supports_arena_args() on, so the
verifier stops rejecting these programs on arm64.
Patch 1 is an independent fix. save_args() reads stack-passed arguments
at FP + 32, which only holds when the trampoline is entered through the
fentry call and two frame records are pushed. A struct_ops trampoline is
entered via blr and pushes one frame fewer, so its stack arguments start
at FP + 16 and every one of them was read two slots off. No struct_ops
member passed arguments on the stack until the test added by commit
2d4de9a493, which is why this went unnoticed. It carries a Fixes tag
and can be taken separately; note that the test covering it only runs on
arm64 once the rest of this series lands.
Patch 2 adds an ADD/SUB (extended register) encoder to the insn library,
so the JIT can zero-extend and add in one instruction.
Patches 3 and 4 are the JIT work. A kfunc argument is rebased onto the
arena base at the call site:
add xN, x28, wN, uxtw
and a nullable one skips the add so NULL stays NULL:
mov wN, wN
cbz wN, 1f
add xN, x28, wN, uxtw
1:
A struct_ops callback converts in the other direction, in the trampoline
while saving arguments into the BPF ctx, with the low half of the arena
base kept in x11:
sub w10, wsrc, w11
str x10, [sp, #slot]
Patches 5 and 6 add arm64 JIT-sequence assertions and drop the x86-64
gating from the existing arena argument tests. Patch 7 is arch-neutral:
it adds a struct_ops member whose first argument is a 16-byte struct
passed by value, so the arena pointer does not land at the ctx slot its
argument index suggests. Nothing covered that before, and it is the case
patch 4 has to get right.
Changelog:
V1: https://lore.kernel.org/bpf/20260810190922.3408757-1-puranjay@kernel.org/
Changes in v2:
- patch 2: fix the decode masks for the new extended-register predicates,
0x7F200000 -> 0x7FE00000. opt in bits 23:22 is part of the opcode here
rather than a shift type, and any value other than 00 is unallocated
(Xu Kuohai). Also noted in the commit message. No functional change: the
masks only feed aarch64_insn_is_*_ext(), which has no in-tree callers,
while the encoder uses aarch64_insn_get_*_ext_value().
- patch 4: comment why the conversion in the stack-argument loop is not
guarded by for_call_origin (Xu Kuohai).
- collect Reviewed-by/Acked-by from Xu Kuohai.
- rebase onto current bpf-next.
====================
Link: https://patch.msgid.link/20260813190356.335181-1-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The trampoline reads the __arena flag from the btf_func_model per
argument but stores the ctx one register slot at a time, so the two only
line up if every preceding argument occupies exactly one slot. Every
arena-bearing member of bpf_testmod_ops3 takes single-slot arguments, so
nothing exercises the mapping and a mis-indexed arg_flags lookup would
go unnoticed on any architecture.
Add test_arena_multislot(), whose first argument is a 16-byte struct
passed by value. It fills ctx[0] and ctx[1], putting the arena pointer
at argument index one but slot two. The callback checks both halves of
the struct before dereferencing ctx[2], so a JIT that walks registers
instead of arguments converts the wrong slot and fails the test.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-8-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The arena kfunc and struct_ops argument tests were restricted to x86-64
because it was the only JIT that implemented the conversions. arm64 does
now, so let them run there too: tag every program in arena_kfunc.c with
__arch_arm64 in addition to __arch_x86_64, and widen the __x86_64__
guards in the struct_ops arena test.
Without this the tests report SKIP on arm64 rather than exercising the
newly added JIT support.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-7-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Pin the arm64 counterparts of the x86-64 rebase sequences: the single
extended-register add for an unconditional argument, the nullable
truncate-test-and-skip variant, and all five argument registers in one
call. The nullable cases use a local label so the branch is pinned to
the instruction right after the add, and the label line does not spell
out the call because arm64 emits either a direct bl or a materialize-
and-blr pair depending on the distance to the kfunc.
Note that on arm64 an unconditional argument is one instruction with
nothing to anchor it against, so arena_arg_jit_rebase alone cannot tell
the two forms apart; it only requires that nothing is emitted between
the rebase and the call. The args5 test is what pins the distinction,
since its four consecutive adds leave no room for a nullable
truncate-and-branch pair between them.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-6-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Implement the struct_ops arena argument conversion on arm64. save_args()
receives the arena base from bpf_tramp_arena_base() and consults the
btf_func_model argument flags as it copies each native argument into the
BPF ctx, routing a marked argument through x10 with the low half of the
base materialized once into x11:
sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */
str x10, [sp, #slot]
A nullable argument tests the full 64-bit kernel pointer first:
mov x10, xsrc
cbz x10, 1f
sub w10, w10, w11
1:
str x10, [sp, #slot]
The 32-bit subtraction is sufficient since (u32)(kaddr - base) ==
(u32)kaddr - (u32)base, and it clears the upper half as the JITs require
of arena pointer registers. Stack-passed arguments already reload
through x10, so only the subtraction (and the NULL test) is inserted
there.
The register loop now walks arguments rather than registers so that the
per-argument flags line up with the slots a multi-slot argument occupies;
the sequence of stores is otherwise unchanged. bpf_tramp_arena_base()
returns a base only for a single-program struct_ops indirect trampoline,
so a tracing trampoline emits exactly what it did before and never
touches x11. The size probe reruns the same emission with the same model
and nodes, so the image size matches by construction.
Conversion must never reach the original function, which takes kernel
addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with
BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin
save_args() and assert the flag combination the same way x86 does,
rather than leaving the invariant to a comment.
With both the kfunc and struct_ops directions implemented, flip
bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only
qualifier from the kfunc documentation.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-5-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Implement arena argument rebasing for kfunc calls on arm64. x28 already
holds kern_vm_start whenever the prog has an arena, and the newly added
extended-register add zero-extends the 32-bit arena offset in place, so
an unconditional argument costs a single instruction emitted right
before the call:
add xN, x28, wN, uxtw
A nullable argument first truncates into wN so that a zero offset leaves
xN holding a real NULL, then tests it and jumps over the add:
mov wN, wN
cbz wN, 1f
add xN, x28, wN, uxtw
1:
The rebase is native code generated after constant blinding has run on
the BPF instruction stream, so blinding never sees it and needs no
special handling. The emitted count depends only on the kfunc model, so
it is identical across JIT passes.
bpf_jit_supports_arena_args() is not flipped yet; that happens when the
struct_ops trampoline side is in place as well.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-4-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The insn library encodes the immediate and shifted-register forms of
ADD/SUB but not the extended-register form. The BPF JIT wants it to
rebase a 32-bit arena offset onto the arena kernel base in a single
instruction, add xN, xBASE, wN, uxtw, instead of a separate zero-extend
followed by a plain add.
Add aarch64_insn_gen_add_sub_extended_reg(), modeled on the
shifted-register generator. The option and imm3 fields occupy the same
bits as the shifted form's shift amount, so they are encoded through the
existing IMM_6 field type. The opt field in bits 23:22 is part of the
opcode here rather than a shift type, and any value other than 00 is
unallocated, so the decode masks cover it.
Note that register 31 does not mean the same thing in the two forms: in
the extended-register encoding it is SP for Rn, and for Rd unless the
instruction sets the flags, while it stays XZR for Rm. Callers porting a
shifted-register site that passes A64_ZR need to be aware of that, so
say so above the function.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-3-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
save_args() reads stack-passed arguments relative to FP assuming the
trampoline is entered through the fentry call from a traced function, in
which case both the parent frame (FP/x9) and the traced function frame
(FP/LR) are saved before FP is set, so the arguments start at FP + 32.
An indirect trampoline for a struct_ops callback is entered through a
function pointer (blr), so only the FP/LR frame is pushed and the
arguments start at FP + 16, not FP + 32. Every stack-passed argument of
a struct_ops callback with more than eight argument slots is read two
slots off.
This went unnoticed because no struct_ops member passed arguments on the
stack until bpf_testmod_ops3::test_arena_stack, added by
commit 2d4de9a493 ("selftests/bpf: Test stack-passed struct_ops arena arguments").
That member covers this on arm64 once the JIT gains arena argument
support later in this series. Pass is_struct_ops into save_args() and
pick the offset accordingly, mirroring the x86 fix.
Fixes: 9014cf56f1 ("bpf, arm64: Support up to 12 function arguments")
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-2-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Eduard Zingerman says:
====================
selftests/bpf: fix for veristat file/prog filters processing
At the moment veristat filtering behaves unexpectedly for the
following filter expression:
-f !file/prog
The expression rejects all programs with name 'prog', and all programs
in a file with name 'file'. Fix the expression to exclude only a
program 'prog' from a file 'file', also add a set of tests to exercise
filtering logic.
Changelog:
v1 -> v2:
- added fixes tag for patch #1 (bot+bpf-ci);
- extended test cases for '!*foo*' and '*foo*' filters in patch #2
(bot+bpf-ci);
- added patch #3, replacing direct read() calls with calls to
read_output(), guaranteeing input buffer null termination
(bot+bpf-ci).
v1: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v1-0-b5b43c431550@gmail.com/
---
====================
Link: https://patch.msgid.link/20260811-veristat-filter-fix-v2-0-6c234c4cd6ef@gmail.com
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
map_dump() closes the map fd in its error path, and do_dump() then
closes the same fd again after a successful dump. Closing an already
closed fd leaves errno set to EBADF, which poisons later errno checks
such as the batch file read check in do_batch(). Let do_dump() own the
fd and remove the close from map_dump().
The same double-close pattern exists in do_show_subset(): both
show_map_close_json() and show_map_close_plain() already close the fd,
so drop the extra close() there as well.
Also propagate the error when bpf_map_get_info_by_fd() fails on a
subsequent map in do_dump(): set err = -1 before breaking out of the
loop, so a later failure is not silently hidden after an earlier
iteration succeeded.
Fixes: 99f9863a0c ("bpftool: Match maps by name")
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260810142224.2907373-2-chenyuan_fl@163.com
Leon Hwang says:
====================
bpf: Introduce global percpu data
This patch set introduces global percpu data, similar to commit
6316f78306 ("Merge branch 'support-global-data'"), to reduce restrictions
in C for BPF programs.
With this enhancement, it becomes possible to define and use global percpu
variables, like the DEFINE_PER_CPU() macro in the kernel
include/linux/percpu-defs.h.
The section name for global peurcpu data is ".percpu". Even though, a one-byte
percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash
with Clang 17 [1], users are expected to use such small variables as global
percpu data with newer Clang versions, which don't have the issue.
The idea stems from the bpfsnoop [2], which itself was inspired by
retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR
(Last Branch Record) entries were observed related to the
bpf_get_smp_processor_id() helper.
Since commit 1ae6921009 ("bpf: inline bpf_get_smp_processor_id() helper"),
the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing
the overhead and consequently minimizing these two LBR records.
However, the introduction of global percpu data offers a more robust
solution. By leveraging the percpu_array map and percpu instruction,
global percpu data can be implemented intrinsically.
This feature also facilitates sharing percpu information between tail
callers and callees or between freplace callers and callees through a
shared global percpu variable. Previously, this was achieved using a
1-entry percpu_array map, which this patch set aims to improve upon.
Links:
[1] https://lore.kernel.org/bpf/fd1b3f58-c27f-403d-ad99-644b7d06ecb3@linux.dev/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop
Changes:
v11 -> v12:
* Improve feature check in bpf_object__create_maps() in libbpf.
* Add percpu_array map support in bpf_map__set_value_size() in libbpf.
* Exercise bpf_map__set_value_size() in selftest.
* Drop dead warning in bpf_object__populate_internal_map() in libbpf.
(Sashiko)
* v11: https://lore.kernel.org/bpf/20260806163125.11172-1-leon.hwang@linux.dev/
v10 -> v11:
* Drop env->prog->jit_requested check when inlining insns for global
percpu data.
* Do not autocreate percpu_array map when kernel does not have global
percpu data support in libbpf.
* Check map->btf_value_type_id in bpftool's is_skel_data().
* Exercise bpf_map__lookup_elem() in selftest.
* Collect Reviewed-by tags from Emil, thanks.
* Drop all duplicate blank lines in kernel/bpf/*.c. (Emil)
* Factor out check_map_mem_read() helper. (Emil)
* Check bpf_jit_supports_percpu_insn() first in
percpu_array_map_direct_value_addr/meta(). (Emil)
* Add comment for 'map->libbpf_type == LIBBPF_MAP_PERCPU' in libbpf's
map_is_mmapable(). (Emil)
* Init update_flags as a const var in libbpf's
bpf_object__populate_internal_map(). (Emil)
* Keep is_mmapable_map() beyond is_skel_data() in bpftool. (Emil)
* Add 'run' and 'cpu_id' in selftest. (Emil)
* Drop subskel test. Verify the generated subskel manually. (Emil)
* Add comment to the raw insns in selftest. (Emil)
* v10: https://lore.kernel.org/bpf/20260715153254.92010-1-leon.hwang@linux.dev/
v9 -> v10:
* Rebase latest bpf-next tree to resolve code conflict in verifier in
patch #1.
* v9: https://lore.kernel.org/bpf/20260713154024.30851-1-leon.hwang@linux.dev/
v8 -> v9:
* Use real name for percpu data maps in libbpf in patch #4.
* Add long map name test in patch #6.
* Move parse_cpu_mask_file() to test_percpu_data_on_cpus() in test in
patch #6.
* Validate map type in get_map_ident() for percpu data maps in patch #5.
* Update code comment in verifier in patch #2. (per Andrii)
* Pass 'type' to internal_map_name in libbpf in patch #4. (per Andrii)
* Factor out the helper is_skel_data() in bpftool in patch #5.
(per Quentin and Andrii)
* v8: https://lore.kernel.org/bpf/20260629152406.52582-1-leon.hwang@linux.dev/
v7 -> v8:
* Send patch #1 and #2 separately that fix interpreter fallback issues.
(Andrii)
* Use 'array->elem_size' to avoid 'range' local variable in
percpu_array_map_direct_value_meta(). (Andrii)
* Keep original map name for percpu data's map in libbpf. (Andrii)
* Factor out helper bpf_map_is_skel_data() in bpftool. (Andrii)
* Update commit message of direct access read-only percpu_array map.
(Andrii)
* Add test to verify that it is disallowed to directly write data of
read-only percpu_array map. (Andrii)
* Drop unused 'num_cpus' in test. (bot+bpf-ci)
* Factor out helper test_percpu_data_on_cpus() in test. (bot+bpf-ci)
* v7: https://lore.kernel.org/bpf/20260622143557.22955-1-leon.hwang@linux.dev/
v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/20260615152646.27639-1-leon.hwang@linux.dev/
v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in
interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling
bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton.
(Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without
test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/20260608145113.65857-1-leon.hwang@linux.dev/
v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in
interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read()
in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to
bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the
off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped
has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/20260414132421.63409-1-leon.hwang@linux.dev/
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag
is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf
(per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/20250526162146.24429-1-leon.hwang@linux.dev/
v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/20250213161931.46399-1-leon.hwang@linux.dev/
v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/20250127162158.84906-1-leon.hwang@linux.dev/
rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(),
and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and
PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/20250113152437.67196-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260813152324.97937-1-leon.hwang@linux.dev
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
If the arch, like s390x, does not support percpu insn, these cases won't
test global percpu data by checking FEAT_PERCPU_DATA support.
The following APIs have been tested for global percpu data:
1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. bpf_map__set_value_size()
4. generated percpu struct pointer pointing to internal map's mmaped data
5. bpf_map__lookup_elem() for global percpu data map
6. bpf_map_lookup_elem_flags() for global percpu data map
At the same time, the case is also tested with 'bpftool gen skeleton -L'.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-8-leon.hwang@linux.dev
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev
Add support for global percpu data in libbpf by adding a new ".percpu"
section, similar to ".data". It enables efficient handling of percpu
global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array
map used for global percpu data using BPF_F_ALL_CPUS, in order to update
values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked
as read-only after populating the percpu_array map. Thereafter, users can
read those initialized percpu data after loading prog. If they want to
update the percpu data after loading prog, they have to update the
percpu_array map using key=0 instead.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev
Introduce global percpu data, inspired by the commit
6316f78306 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
The chained_global_func_calls_success() test hardcodes the instruction
counts reported by the verifier's per-subprog stats:
subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack
processed 14 insns
global_good() does 'return arr[0]', where arr[] is an int array and the
return type is long. Without cpu v4 this is a zero-extending load
followed by a <<32/s>>32 sign-extension pair. With -mcpu=v4 llvm emits a
single sign-extending load instead:
18: (18) r1 = 0xffa00000008eb000
20: (81) r0 = *(s32 *)(r1 +0)
21: (95) exit
so the subprog is 3 insns rather than 5, and the whole program is
12 processed insns rather than 14. test_progs-cpuv4 fails with:
EXPECTED REGEX: 'subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack'
#606/1 verifier_global_subprogs/chained_global_func_calls_success:FAIL
Select the expected counts based on __BPF_CPU_VERSION__.
Fixes: c2e6c7de88 ("bpf: Show more useful info in stack depth stats")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260813150641.3347662-1-yonghong.song@linux.dev
The commit 7619a0ee93 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Add resolve_btfids support for __arena kfunc suffix
Use __arena/__arena__nullable suffixes to emit address_space(1)
annotations on kfunc definitions in vmlinux.h. See commits for details.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/20260809085155.3305519-1-memxor@gmail.com
* Avoid enumerating all the ways resolve_btfids can emit the
"address_space(1)" attribute in its header comment and in kfuncs.rst.
(Ihor)
* Drop the kfunc_has_arena_arg() helper: add_arena_tagged_proto()
returns the original prototype when nothing needs tagging, so it can
be invoked unconditionally for every kfunc. (Ihor)
* Add resolve_btfids selftest cases with mixed tagged and untagged
arguments, and a kfunc that combines the KF_ARENA_RET flag with
suffixed arena arguments. (Ihor)
====================
Link: https://patch.msgid.link/20260812193842.2879226-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a suffix-only kfunc declaration with arena annotations on all five
arguments. Verify that resolve_btfids emits address_space(1) type tags
for every position without KF_ARENA_ARG flags in the BTF ID sets.
Represent expected arena arguments as a per-parameter bitmap so the
test covers suffixes beyond the two positions expressible by flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Kfunc declarations can identify arena arguments through parameter name
suffixes without repeating KF_ARENA_ARG flags in their BTF ID sets.
resolve_btfids currently misses those arguments when synthesizing the
address_space(1) attributes used by generated vmlinux.h files.
Teach the arena prototype rewrite to recognize __arena and
__arena__nullable directly on each parameter. Keep KF_ARENA_ARG1 and
KF_ARENA_ARG2 handling for explicitly flagged kfuncs, while allowing
suffixes on any argument without synthesizing kfunc flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
