Add a verifier test that a BPF_LOAD_ACQ from a rdonly_untrusted_mem pointer
(PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED, obtained via bpf_rdonly_cast())
is rejected. Such a source requires BPF_PROBE_MEM fault protection which
is not applied to atomic loads; without the verifier fix the load is accepted
and would crash the kernel on a fault.
# LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_load_acquire
[...]
#621/1 verifier_load_acquire/load-acquire, 8-bit:OK
#621/2 verifier_load_acquire/load-acquire, 8-bit @unpriv:OK
#621/3 verifier_load_acquire/load-acquire, 16-bit:OK
#621/4 verifier_load_acquire/load-acquire, 16-bit @unpriv:OK
#621/5 verifier_load_acquire/load-acquire, 32-bit:OK
#621/6 verifier_load_acquire/load-acquire, 32-bit @unpriv:OK
#621/7 verifier_load_acquire/load-acquire, 64-bit:OK
#621/8 verifier_load_acquire/load-acquire, 64-bit @unpriv:OK
[...]
#621/19 verifier_load_acquire/load-acquire from rdonly_untrusted_mem pointer:OK
#621/20 verifier_load_acquire/load-acquire with invalid register R15:OK
#621/21 verifier_load_acquire/load-acquire with invalid register R15 @unpriv:OK
#621/22 verifier_load_acquire/load-acquire from pkt pointer:OK
#621/23 verifier_load_acquire/load-acquire from flow_keys pointer:OK
#621/24 verifier_load_acquire/load-acquire from sock pointer:OK
#621 verifier_load_acquire:OK
Summary: 1/24 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-6-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add stream_arena_load_acquire_fault, which performs a load-acquire from an
unmapped arena address, next to the existing read and write fault tests.
The test covers both halves of the JIT bug that treated a load-acquire as
a store when populating its exception table entry:
- the fault has to be reported as a READ, and at the address held by
the source register, which __stderr() and test_address() check, and
- the destination register has to be cleared by the fault handler,
which the program checks by poisoning it before the load-acquire
and returning it, so __retval(0) fails if it is left untouched
Note, load-acquire is open coded since linux/filter.h cannot be included
alongside vmlinux.h.
# LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream_arena_fault_address
[...]
#462/1 stream_arena_fault_address/read_fault:OK
#462/2 stream_arena_fault_address/write_fault:OK
#462/3 stream_arena_fault_address/load_acquire_fault:OK
#462 stream_arena_fault_address:OK
Summary: 1/3 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-5-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Same problem as on x86-64: add_exception_handler() decides whether an
instruction is a load by its class, and a load-acquire is of BPF_STX
class even though it reads from src_reg into dst_reg. As a result ...
if (BPF_CLASS(insn->code) != BPF_LDX)
dst_reg = DONT_CLEAR;
... drops the register to clear, and ...
if (BPF_CLASS(insn->code) == BPF_LDX)
arena_reg = bpf2a64[insn->src_reg];
else
arena_reg = bpf2a64[insn->dst_reg];
... hands ex_handler_bpf() the value register instead of the address
register. A load-acquire from an arena pointer that faults on an
unmapped page is therefore reported as a WRITE at a bogus address,
and dst_reg keeps its previous value instead of being cleared to 0.
Note that emit_atomic_ld_st() already picks src_reg as the address
for BPF_LOAD_ACQ, so only the exception table metadata was out of sync
with the emitted access.
Same as on x86-64, use bpf_atomic_is_load_acq() so a load-acquire takes
the load path.
Fixes: 9bb12368d5 ("bpf, arm64: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Link: https://lore.kernel.org/bpf/20260806201047.333389-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and
gets an exception table entry, but the entry is filled in as if it were a
store, since populate_extable() decides based on instruction class alone
and a load-acquire is of BPF_STX class:
if (BPF_CLASS(insn->code) == BPF_LDX) {
arena_reg = reg2pt_regs[src_reg];
fixup_reg = reg2pt_regs[dst_reg];
} else {
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
}
For a load-acquire dst_reg holds the loaded value and src_reg holds the
address, so both assignments in the else branch are wrong. On a fault
over an unmapped arena page ex_handler_bpf() then:
- computes the reported address from the value register instead
of the address register
- reports the access as a WRITE, since it derives the direction
from fixup_reg == DONT_CLEAR
- leaves dst_reg untouched, so the program continues with a stale
value instead of the 0 that BPF_PROBE_* loads deliver
The access itself is emitted correctly, emit_atomic_ld_st_index() uses
src_reg as the address, so this is a broken probe contract and a wrong
diagnostic rather than a memory safety issue.
Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path.
Fixes: 5341c9a4d8 ("bpf, x86: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-3-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A load-acquire is the only BPF_STX class instruction that reads from
src_reg into dst_reg, that is, it has the operand roles of a BPF_LDX.
JIT code which tells loads from stores apart by instruction class alone
has to special case it, for example when deciding which register holds
the faulting address and which one to clear from an exception handler.
riscv64 already does so, open coded as a bare insn->imm test. Add a
bpf_atomic_is_load_acq() helper and convert riscv64 over to it, so that
the x86-64 and arm64 JITs can use the same helper in subsequent patches.
Unlike bpf_atomic_is_load_store(), which presumes that its argument is
already known to be a BPF_ATOMIC instruction, the new helper is called
from code which still sees all instruction classes, so it checks class
and mode itself.
Also, move bpf_atomic_is_load_store() to filter.h next to BPF_ATOMIC_OP,
so that both helpers stay together. No functional change intended.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-2-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier,
unlike a regular BPF_LDX, so the JIT emits a plain load with no exception
table entry and a fault panics the kernel instead of being handled.
Reject the source pointer types that a BPF_LDX would have had that fault
protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns
into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED,
PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY |
PTR_UNTRUSTED.
This is reachable e.g. by loading ->mm out of a trusted task_struct
yields an untrusted pointer to mm_struct, and it is NULL for a kernel
thread:
[...]
SEC("tp_btf/sched_switch")
int BPF_PROG(demo, bool preempt, struct task_struct *prev,
struct task_struct *next)
{
struct mm_struct *mm = next->mm; /* untrusted */
out_ldx = (__u64)mm->pgd; /* BPF_LDX */
out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */
return 0;
}
[...]
Both dereference the same pointer, but only the BPF_LDX is protected
(x86-64 JIT, jump targets shown prog-relative):
[...]
; out_ldx = (__u64)mm->pgd;
17: movq $-10485760, %r10
1e: movq %rsi, %r11
21: addq $184, %r11
28: subq %r10, %r11
2b: movabsq $140737498841088, %r10
35: cmpq %r10, %r11
38: ja 0x3e <-- kernel addr?
3a: xorl %edi, %edi <-- no: dst = 0, skip the load
3c: jmp 0x45
3e: movq 184(%rsi), %rdi <-- yes: load + extable entry
[...]
; load_acquire(&mm->pgd)
53: movq %rsi, %rdi
56: movq 184(%rdi), %rax <-- no check, no extable entry
[...]
Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as
bpf_insn_prepare_dump() rewrites it back to BPF_MEM.
A PTR_TRUSTED pointer is deliberately not on the list. Such a load is
not converted either, but it does not need to be, since the pointer is
guaranteed live, so load-acquire from it stays allowed.
The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release
error messages are unchanged; writes (RMW / store-release) to such
pointers are already rejected elsewhere, so only load-acquire needs this.
Fixes: 880442305a ("bpf: Introduce load-acquire and store-release instructions")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260806201047.333389-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Ihor Solodrai says:
====================
resolve_btfids: Implement BTF tags emission for kfuncs
BTF data for the kernel is generated through the following pipeline:
* DWARF is emitted by the compiler
* pahole reads in DWARF and produces BTF
* resolve_btfids makes kernel-specific btf2btf transformation and
patches .BTF_ids section
This is orchestrated by link-vmlinux.sh, gen-btf.sh and Makefile.btf
in ./scripts directory.
Historically kernel-specific BTF features were implemented in pahole,
and controlled by the feature flags. This requires kernel build
process to be aware of pahole version used for the build to set
correct runtime arguments for BTF encoding [1].
This is a burden which can be alleviated by splitting kernel/module
BTF generation in two stages:
1. Generic BTF generation from the kernel source code.
2. Kernel-specific BTF modifications to support various BPF features.
So far both stages were fused in pahole's BTF encoding. By moving
stage (2) in-tree, the dependency of kernel build on pahole can become
much more loose.
resolve_btfids is already responsible for a few kernel-specific BTF
modifications:
* .BTF.base generation for modules [2]
* BTF sorting [3]
* KF_IMPLICIT_ARGS support [4]
This series completes the migration by emitting BTF kfunc annotations
in-tree: the "bpf_kfunc" and "bpf_fastcall" decl tags and the arena
"address_space(1)" type attribute, dropping the corresponding pahole
feature flags.
The three annotations depend on two pahole feature flags:
"decl_tag_kfuncs" and "attributes". Since emission is unconditional,
each flag has to be dropped in the same commit as the emission that
replaces it.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/Makefile.btf?h=v7.1-rc5
[2] https://docs.kernel.org/bpf/btf.html#btf-base-section
[3] https://lore.kernel.org/bpf/20260109130003.3313716-4-dolinux.peng@gmail.com/
[4] https://lore.kernel.org/bpf/20260120222638.3976562-1-ihor.solodrai@linux.dev/
[5] https://lore.kernel.org/bpf/20260722233518.778854-1-ihor.solodrai@linux.dev/
[6] https://lore.kernel.org/bpf/20260617210619.1562858-1-ihor.solodrai@linux.dev/
---
v2->v3:
* Refactoring in patch #2 (Eduard)
* restructure add_arena_tagged_proto() such that first we copy the
func proto and then update param types in place
* push error messages down to arena_tag_ptr()
* introduce is_arena_arg() helper
* Docs cleanup in patch #6 (Eduard)
* Add stats in commit message for patch #1
v2: https://lore.kernel.org/bpf/20260805230648.2354989-1-ihor.solodrai@linux.dev/
v1->v2:
* The bottom part of v1 has already been landed [5][6].
* New patch #1: run btf__dedup() in finalize_btf().
* Drop the "ensure" pattern. Emission is unconditional; kbuild owns
the pahole flags, so assume input BTF is not already tagged.
* Each pahole flag is now dropped in the same commit as the emission
that replaces it.
* Fail hard with an error on invalid kfunc declarations such as an
arena flag naming a missing argument or a non-pointer type.
* Various cleanups and nits (Andrii, Emil, Jiri, Sashiko).
v1: https://lore.kernel.org/bpf/20260601221805.821394-1-ihor.solodrai@linux.dev/
---
====================
Link: https://patch.msgid.link/20260807032029.78092-1-ihor.solodrai@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
For kfuncs flagged with KF_ARENA_RET, KF_ARENA_ARG1 or KF_ARENA_ARG2,
the address_space(1) attribute (a type tag with kflag=1) must be
emitted for the corresponding type in BTF. This was previously done by
pahole via the "attributes" BTF feature [1].
Implement the emission of the arena attributes in resolve_btfids: for
flagged kfuncs create a new function prototype with updated BTF types.
The original proto may be shared with sibling FUNCs, so it is not
modified in place.
Emission is unconditional: kbuild controls the pahole flags, so the
input BTF is expected to not have these attributes. Invalid
declarations are reported as errors.
Drop the "attributes" pahole feature from scripts/Makefile.btf
resolve_btfids now emits them for all supported pahole versions.
[1] https://lore.kernel.org/dwarves/20250228194654.1022535-1-ihor.solodrai@linux.dev/
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://patch.msgid.link/20260807032029.78092-3-ihor.solodrai@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
btf2btf() adds new types to the BTF: the KF_IMPLICIT_ARGS transform
synthesizes an _impl FUNC together with its FUNC_PROTO and copies of the
kfunc's decl tags. Nothing deduplicates them afterwards. pahole runs
btf__dedup() on its own output, but that happens before resolve_btfids
sees the BTF, so any type the tool itself creates is emitted as-is, even
when a structurally identical type is already present.
Call btf__dedup() at the start of finalize_btf(), so that base
distillation and the by-name sort both operate on the canonical set of
types.
On an x86_64 build with the BPF selftests config this removes 17
duplicate FUNC_PROTOs from vmlinux BTF.
The dedup call increases runtime of resolve_btfids on vmlinux by 30-40%.
The performance hit is an acceptable cost to keep kernel BTF deduped [1].
[1] https://lore.kernel.org/bpf/986e6f4e-4b51-4440-a37c-9624906d7370@linux.dev/
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://patch.msgid.link/20260807032029.78092-2-ihor.solodrai@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Ning Ding says:
====================
bpf: Track overlapping RCU protection
Preemption-disabled and IRQ-disabled regions provide RCU protection, but
the verifier does not account for them. Current implementation can invalidate
a task kptr while another RCU source remains active, or keep it valid
after the final source ends.
Track these regions and invalidate RCU-protected pointers only after the
last protection ends. Add task kptr tests for overlapping protection and
final-exit rejection.
This follows review of the applied spin-unlock fix series [1].
Tested in QEMU/KVM:
./test_progs -t task_kfunc
./test_progs -t preempt_lock
./test_progs -t irq
[1] https://lore.kernel.org/r/20260803112615.3362122-1-dingning04@gmail.com
====================
Link: https://patch.msgid.link/20260805233940.3966981-1-dingning04@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add task kptr tests that keep RCU protection active after a spin or RCU
unlock when preemption or IRQs remain disabled.
Also test the reverse order with explicit RCU. Verify that task kptrs are
rejected after leaving the final preemption-disabled or IRQ-disabled
region.
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260805233940.3966981-3-dingning04@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Disabling preemption or local IRQs keeps the current CPU in an RCU
read-side critical section, but in_rcu_cs() does not account for either
state. The verifier therefore rejects safe kptr accesses and invalidates
pointers when another RCU source ends.
Include preemption-disabled and IRQ-disabled state in in_rcu_cs().
Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ
restore only after the final protection ends.
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260805233940.3966981-2-dingning04@gmail.com
[ kkd: Simplify was_in_rcu_cs on spin unlock and adjust the selftest. ]
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kaitao Cheng says:
====================
bpf: Allow selected kfuncs under bpf_spin_lock
The verifier currently has a hard-coded allowlist of kfuncs that may be
called while a BPF program holds a bpf_spin_lock. This works for the
small set of built-in kfuncs known to the verifier, but it does not give
kfunc providers a registration-time way to declare that a kfunc is safe
in such a region. In particular, module kfuncs cannot be added to that
allowlist without changing verifier code.
This series adds a new KF_SPINLOCK_SAFE kfunc flag and teaches the
verifier to use kfunc registration metadata when deciding whether a
kfunc call is allowed while a bpf_spin_lock is held.
The built-in kfuncs that are currently accepted by the verifier's
lock-held allowlist are annotated with the new flag. This preserves the
existing behavior while removing the verifier-side category checks and
uses the same mechanism for built-in and module kfuncs.
The selftest coverage marks one bpf_testmod kfunc as KF_SPINLOCK_SAFE
and verifies that it can be called under a bpf_spin_lock. It also calls
another registered but unmarked bpf_testmod kfunc under the lock and
checks that the verifier rejects it.
Changes in v2:
- Rename KF_SPIN_LOCK to KF_SPINLOCK_SAFE. (Kumar Kartikeya Dwivedi,
Leon Hwang)
- Deprecate the verifier's lock-held allowlist mechanism and annotate the
relevant kfuncs uniformly with KF_SPINLOCK_SAFE (Kumar Kartikeya Dwivedi)
- Add selftests. (Leon Hwang)
Link to v1:
https://lore.kernel.org/bpf/DKG0YUDSTBUY.1X220287HT9V3@gmail.com/
====================
Link: https://patch.msgid.link/20260805153340.34776-1-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The verifier uses kfunc registration flags to decide whether a kfunc may
be called while a BPF program holds a bpf_spin_lock.
Mark bpf_testmod_test_mod_kfunc() as KF_SPINLOCK_SAFE and verify that it
can be called while holding a bpf_spin_lock. Also attempt to call the
unmarked bpf_kfunc_trigger_ctx_check() under the lock and verify that the
program is rejected.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-4-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The verifier currently keeps a hard-coded list of kfuncs that may be
called while holding a bpf_spin_lock. With KF_SPINLOCK_SAFE available,
retaining this list creates two sources of truth and requires verifier
changes whenever another lock-safe kfunc is added.
Mark every kfunc currently accepted by kfunc_spin_allowed() with
KF_SPINLOCK_SAFE. This covers the graph, numeric iterator, resource
spin lock, arena, and stream kfuncs.
Remove the obsolete category checks and make kfunc_spin_allowed() rely
solely on the kfunc registration metadata. This preserves the behavior
of existing kfuncs while using the same mechanism for built-in and
module kfuncs.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260805153340.34776-3-kaitao.cheng@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bloom_map_alloc() has two 32-bit-specific problems when the computed
bitmap reaches the U32_MAX fallback case.
First, BITS_TO_BYTES(U32_MAX) is evaluated with 32-bit arithmetic. The
addition performed by DIV_ROUND_UP wraps, so the map allocates only the
fixed-size bloom filter object while keeping bitset_mask == U32_MAX.
Subsequent updates can then write past the allocated object.
Second, fixing only the allocation size is not sufficient. The bloom hash
is a u32, but set_bit() takes a signed long bit number and x86 test_bit()
eventually feeds the index to variable_test_bit(long, ...). On 32-bit
kernels, hashes in [0x80000000, U32_MAX] therefore become negative bit
offsets. x86 bt/bts with a memory operand interpret those offsets relative
to the supplied base, so a map with bitset_mask == U32_MAX can read or
write before bloom->bitset even after allocating the full 512 MiB bitmap.
Keep the U32_MAX fallback, but split each hash into a word pointer and an
in-word bit number before calling test_bit() or set_bit(). The bitops
argument is then always in [0, BITS_PER_LONG - 1], while BIT_WORD(h) still
selects the intended word in the full bitmap.
Compute the bitset size from (u64)bitset_mask + 1 before passing the final
size to bpf_map_area_alloc(). This fixes the original under-allocation and
keeps the allocated storage consistent with the addressable bitset.
Exploitation note: local privilege escalation is possible on a 32-bit x86
kernel using the under-allocation bug from a binary with CAP_BPF.
Fixes: 9330986c03 ("bpf: Add bloom filter map implementation")
Signed-off-by: Jérémy Jean <Jeremy.Jean@oss.cyber.gouv.fr>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260805060228.2703051-1-Jeremy.Jean@oss.cyber.gouv.fr
Assisted-by: Codex:gpt-5
Leon Hwang says:
====================
bpf: Fix sleepable check for tracing/lsm prog
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog
is allowed to attach to '__x64_'-alike prefix symbols.
It is because the verifier does not verify whether the symbol is a kernel
function or a bpf prog. That said, a sleepable tracing prog is allowed to
attach to a bpf prog target whose name has '__x64_'-alike prefix.
For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP
prog, and copies buffer from a user pointer with bpf_copy_from_user()
helper. After attaching the XDP prog to lo interface, the kernel BUG
could be triggered by 'ping -c 1 -W 1 127.0.0.1':
[ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324
Fix it by disallowing sleepable prog always when its target
btf is not kernel's btf.
Changes:
v3 -> v4:
* Move btf check outside of 'switch (prog->type)'. (per Andrii)
* v3: https://lore.kernel.org/bpf/20260804145710.43062-1-leon.hwang@linux.dev/
v2 -> v3:
* Use btf_is_kernel() instead of passing 'tgt_prog'. (per Andrii)
* v2: https://lore.kernel.org/bpf/20260725132624.78373-1-leon.hwang@linux.dev/
v1 -> v2:
* Drop redundant 'prog->sleepable' check. (per Viktor)
* Collect Acked-by from Viktor, Thanks.
* v1: https://lore.kernel.org/bpf/20260724141422.10463-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260805150810.34907-1-leon.hwang@linux.dev
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Add a test to verify that the sleepable tracing prog cannot attach to a
'__x64_sys' prefix prog target.
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, without the fix, the
test would trigger the BUG:
[ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260805150810.34907-3-leon.hwang@linux.dev
When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog
is allowed to attach to '__x64_'-alike prefix symbols.
It is because the verifier does not verify whether the symbol is a kernel
function or a bpf prog. That said, a sleepable tracing prog is allowed to
attach to a bpf prog target whose name has '__x64_'-alike prefix.
For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP
prog, and copies buffer from a user pointer with bpf_copy_from_user()
helper. After attaching the XDP prog to lo interface, the kernel BUG
could be triggered by 'ping -c 1 -W 1 127.0.0.1':
[ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324
Fix it by disallowing sleepable prog always when its target
btf is not a kernel's btf.
Fixes: 16d9c56606 ("bpf: Always allow sleepable programs on syscalls")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/20260805150810.34907-2-leon.hwang@linux.dev
There's no need to modify the trace object bpf_get_stackid_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-13-jolsa@kernel.org
There's no need to modify the trace object bpf_get_stack_pe, we just
need to pass the needed callchain length in separate argument.
This way we can have callchain pointers const and remove the trace->nr
modification and restoration.
Assisted-by: Codex:GPT-5.5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-12-jolsa@kernel.org
get_perf_callchain() returns a per-CPU perf_callchain_entry buffer and
releases its recursion slot via put_callchain_entry() before returning,
so nothing keeps the entry reserved while __bpf_get_stack() consumes
it below.
A preemptible BPF program (e.g. a non-sleepable raw tracepoint program
on a PREEMPT kernel, which runs under migrate_disable() but not
preempt_disable()) can be scheduled out between obtaining the entry
and the copy. Another task scheduled on the same CPU then reuses the
same per-CPU buffer and overwrites trace->nr with a larger value.
copy_len is then computed from the inflated trace->nr and can exceed
the caller's buffer, causing an out-of-bounds write in the memcpy()
and in the build_id path.
The rcu_read_lock() taken here alone does not prevent this. It is
only taken on the may_fault path, and under CONFIG_PREEMPT_RCU it does
not disable preemption; it merely keeps perf's callchain buffer array
alive (freed via call_rcu()) and does nothing to stop another task
from reusing the entry.
Disable preemption around obtaining the callchain entry and copying
it into the caller's buffer, so the entry cannot be reused underneath
us and trace->nr stays bounded by max_depth. Build ID resolution may
fault and is therefore deferred until after preemption is re-enabled;
by then the instruction pointers have already been copied into buf,
so it operates only on that private copy. Note, preempt_disable() also
subsumes the buffer-lifetime guarantee the rcu_read_lock() provided,
since a preempt-disabled section is an RCU read-side critical section
for the callchain buffers' call_rcu() reclaim.
Fixes: c195651e56 ("bpf: add bpf_get_stack helper")
Reported-by: Tao Chen <chen.dylane@linux.dev>
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <borkmann@iogearbox.net>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/bpf/20260803210149.296496-11-jolsa@kernel.org
Closes: https://lore.kernel.org/bpf/20260206090653.1336687-1-chen.dylane@linux.dev/
[ changed Fixes: commit ]
Both bpf_get_task_stack and bpf_get_task_stack_sleepable helpers that
use __bpf_get_task_stack have buf defined as ARG_PTR_TO_UNINIT_MEM
argument and we should initialize the buf on every return path.
Adding missing buf memset for __bpf_get_task_stack fail paths. This
provides deterministic buffer contents, which is useful when the buffer
is used directly as a map key.
Fixes: 06ab134ce8 ("bpf: Refcount task stack in bpf_get_task_stack")
Fixes: b992f01e66 ("bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-10-jolsa@kernel.org
Replacing __bpf_get_stackid calls with sequence of following functions:
stackid_fastpath
stackid_new_bucket
stackid_install
This makes code more structured and allows us to easily disable
preemption only in bpf_get_stackid in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260803210149.296496-5-jolsa@kernel.org
Puranjay Mohan says:
====================
bpf: Inline the numeric open-coded iterator kfuncs
The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
tear it down. The verifier emits these as ordinary kfunc calls, so a
bpf_for() loop pays function-call overhead on setup, teardown, and -- most
importantly -- on every single iteration via bpf_iter_num_next().
All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
(struct bpf_iter_num_kern { int cur; int end; }). That makes them good
candidates for inlining, the same way several other special kfuncs are
already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
the three calls with an equivalent inline BPF instruction sequence:
- bpf_iter_num_new(): the end - start range check is done with 32-bit
arithmetic (start <= end is checked first, so the distance fits in a
u32) and range-checked against BPF_MAX_LOOPS as unsigned. This avoids
the cpuv4 sign-extension insns that some JITs do not implement. Returns
the same -EINVAL / -E2BIG / 0 as the kfunc.
- bpf_iter_num_next(): the hot path. cur and end are int, so the kfunc's
s->cur + 1 >= s->end test is an ordinary signed 32-bit compare and the
inlined code needs no sign extension.
- bpf_iter_num_destroy(): the stack slot is no longer tracked as iterator
state once destroy() returns, so nothing needs to be written to it.
Both the kfunc and the inlined form become a no-op, which just drops the
call.
The emitted instructions are plain BPF and remain valid for the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.
Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):
+--------+---------------------+---------------------+---------------------+
| arch | bpf_loop | bpf_for non-inlined | bpf_for inlined |
+--------+---------------------+---------------------+---------------------+
| x86-64 | 4252 M/s (0.24 ns) | 3608 M/s (0.28 ns) | 7417 M/s (0.13 ns) |
+--------+---------------------+---------------------+---------------------+
| arm64 | 649 M/s (1.54 ns) | 548 M/s (1.82 ns) | 546 M/s (1.83 ns) |
+--------+---------------------+---------------------+---------------------+
On x86-64 removing the per-iteration call roughly doubles bpf_for()
throughput. On arm64 it is neutral, and rather than guess why this was
checked with perf: inlining removes ~28% of the executed instructions (the
call) but leaves the cycle count unchanged -- IPC drops from ~4.2 to ~3.0
and backend stalls rise from ~50% to ~66%. The loop is bound by the latency
of the iterator's on-stack counter, not by call overhead:
bpf_iter_num_next() loads s->cur from the stack, increments it and stores it
back each iteration, and the next iteration's load depends on that store.
The removed call instructions were executing in the shadow of that
store->load stall and were never on the critical path.
A small userspace microbenchmark isolates the effect: a same-address
store->load->add round-trip (the shape of the on-stack counter) costs
~6 cycles/iteration on the tested arm64 core but ~1 cycle on x86-64, where
the core collapses the same-address round-trip into a register move (memory
renaming / store-to-load-forwarding elimination). So on x86-64 the loop is
not latency-bound and the per-iteration call dominates -- removing it is the
~2x win -- whereas on arm64 the call fits entirely inside the store->load
stall the loop already has, so adding or removing it changes nothing.
bpf_loop() is shown for reference only; it is a different construct (a
callback invoked per iteration) and this series does not change it. Its
counter lives in a register rather than on the stack, so on arm64 it avoids
the store->load latency above and is faster than bpf_for() there.
Changelog:
v4: https://lore.kernel.org/all/20260729203633.213973-1-puranjay@kernel.org/
Changes in v5:
- Inline the new()/next()/destroy() sequences directly in
bpf_fixup_kfunc_call() instead of via helper functions (Andrii Nakryiko)
- Trim the code comments; keep the explanation in the bpf_iter.c kfuncs and
leave only brief comments at the inline sites, and shorten the
bpf_iter_num_destroy() kfunc to /* no-op */ (Andrii Nakryiko)
- Reword the patch 1 comment so it no longer forward-references the inlined
bpf_iter_num_next(), which is only added later in the series (bpf-ci)
- Switch the bpf_iter_num_next() comment to the networking multi-line style
v3: https://lore.kernel.org/all/20260722132424.450230-1-puranjay@kernel.org/
Changes in v4:
- Drop the "elide range checks for constant bounds" patch (Andrii Nakryiko)
- bpf_iter_num_new(): range-check the distance against BPF_MAX_LOOPS with an
unsigned compare (Andrii Nakryiko)
- bpf_iter_num_destroy(): make it a no-op in both the kfunc and the inlined
form instead of zeroing the iterator state (Andrii Nakryiko)
- New patch: fix the misleading overflow comment in bpf_iter_num_next() and
drop the redundant (s64) cast; the int wraparound is intentional and
load-bearing (Andrii Nakryiko)
- bpf_for benchmark: nr_loops is int, matching what bpf_for() expects
(Andrii Nakryiko)
- Corroborate the arm64/x86 benchmark difference with perf counters and a
store-to-load-forwarding microbenchmark (Kumar Kartikeya Dwivedi,
Andrii Nakryiko)
v2: https://lore.kernel.org/bpf/20260717120215.2171057-1-puranjay@kernel.org/
Changes in v3:
- Elide the range checks in bpf_iter_num_new() when start and end are
constant, marking the registers precise so paths reaching the call with
different constants are not pruned (Eduard Zingerman)
- Add __xlated selftests pinning the inlined new()/next()/destroy() shapes
(Eduard Zingerman)
- Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
- Pick up Acked-by on patch 3
v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
Changes in v2:
- Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32,
sparc64) decode them as a plain move and would miscompile the range check
====================
Link: https://patch.msgid.link/20260804134601.2305303-1-puranjay@kernel.org
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Add a bpf_for() benchmark modelled on bench_bpf_loop so the per-iteration
iterator cost can be measured and compared against bpf_loop. It runs an
empty bpf_for(i, 0, nr_loops) loop 1000 times per trigger and accounts
nr_loops hits per outer iteration:
$ ./bench -p 1 --nr_loops 1000 bpf-for
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-7-puranjay@kernel.org
Add an __xlated test pinning the inlined bpf_iter_num_{new,next,destroy}()
shapes. The program is __naked, so there is no compiler glue and the whole
sequence is matched instruction for instruction.
Gate it to x86_64 and arm64 (bpf_jit_needs_zext() == false); elsewhere the
verifier interleaves "wN = wN" zero-extensions that would not match. The
inlining is arch independent, so these two are enough.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-6-puranjay@kernel.org
bpf_iter_num_next() runs on every bpf_for() iteration, so inlining it
drops a call from the loop body. R1 points to the iterator; the returned
pointer to s->cur is R1 itself, since s->cur is first.
s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end is a
signed 32-bit compare and the inlined code needs no sign extension.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-4-puranjay@kernel.org
bpf_for() expands to the bpf_iter_num_{new,next,destroy}() kfuncs, which
the verifier emits as regular calls. They are tiny and only touch the
8-byte on-stack iterator state, so open-code them in bpf_fixup_kfunc_call()
like the other special kfuncs there.
Start with bpf_iter_num_new(): R1 points to the iterator, R2/R3 hold
start/end. The inlined sequence mirrors the kfunc and returns the same
-EINVAL / -E2BIG / 0.
start > end is rejected first, so end - start fits in a u32; range-check
it as u32 on both sides ((u32)(end - start) in the kfunc). A movsx-based
check would emit a cpuv4 instruction that some JITs (x86-32, mips32,
sparc64) decode as a plain move and get wrong.
The emitted instructions are plain BPF, so the interpreter path stays
correct and no jit_required marking is needed.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-3-puranjay@kernel.org
The comment on the s->cur + 1 >= s->end check claims the (s64) cast is
needed to avoid overflow when s->cur == s->end == INT_MAX. It isn't:
s->cur + 1 is computed in int and wraps before the cast, so the cast
changes nothing (INT_MAX + 1 compares the same either way).
The wraparound is the point. bpf_iter_num_new() sets s->cur = start - 1,
which wraps to INT_MAX for start == INT_MIN, and the wrapping s->cur + 1
brings it back to start. (s64)s->cur + 1 would instead break iterators
starting at INT_MIN.
Drop the cast and reword the comment. No functional change; the wrap is
well-defined under -fno-strict-overflow.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260804134601.2305303-2-puranjay@kernel.org
check_atomic_load() calls check_load_mem() before atomic_ptr_type_ok().
For a load-acquire that fetches into its own source register (dst_reg ==
src_reg), check_load_mem() overwrites src_reg's type with the type of the
loaded value, so the subsequent atomic_ptr_type_ok() no longer sees the
source pointer and fails to reject the disallowed types (ctx, pkt,
flow_keys, sock).
Since bpf_convert_ctx_accesses() does not rewrite atomic loads, the raw
access to the underlying kernel object is left in place. The destination
type is taken from the ctx access itself, so a load-acquire of the sk
field of struct __sk_buff for example leaves the register typed as
PTR_TO_SOCK_COMMON_OR_NULL, which type_is_sk_pointer() does not match
either, while it actually holds unconverted struct sk_buff bytes. Once
the NULL check has passed this is a type confusion, not just a leak of
kernel data.
Validate src_reg with check_reg_arg() and check the source pointer type
with atomic_ptr_type_ok() before the load again, mirroring
check_atomic_rmw(). Out-of-range register numbers are already rejected
earlier by check_and_resolve_insns() (commit 503d21ef8e ("bpf: Do
register range validation early")), and the only exemption there,
is_stack_arg_ldx(), requires BPF_LDX | BPF_MEM | BPF_DW and thus never
matches a BPF_ATOMIC insn. atomic_ptr_type_ok() can therefore not
dereference register state out of bounds, that is, the out-of-bounds
read addressed by the Fixes commit below does not reappear (as proven
also via selftest).
Fixes: c03bb2fa32 ("bpf: Fix out-of-bounds read in check_atomic_load/store()")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260804201917.253491-1-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A potential invalid storage access issue can occur after replacing a
cgroup bpf prog.
This occurs in the following scenario:
1. prog1 with storage is attached to a cgroup in multi-attach mode.
2. prog1 is replaced with prog2 using BPF_F_REPLACE in multi-attach
mode, but fails midway (e.g. in bpf_trampoline_link_cgroup_shim or
update_effective_progs).
3. A new prog3 is attached to the cgroup in multi-attach mode.
The reason is that __cgroup_bpf_attach overwrites pl->storage with the
new storage prior to attachment completion. When attachment fails
midway, the cleanup path calls bpf_cgroup_storages_free(new_storage) to
free the newly allocated storage, but fails to restore pl->storage back
to old_storage.
Consequently, the still-active prog1 holds invalid or dangling storage
pointers, leading to an invalid memory access when prog1 executes and
calls bpf_get_local_storage. Additionally, original pl->flags and
cgrp->bpf.flags[atype] are left unrestored.
Fix this by saving old_pl_flags, old_storage, and old_flags prior to the
update, and properly restoring all of them in the cleanup path on error.
Fixes: 7d9c342789 ("bpf: Make cgroup storages shared between programs on the same cgroup")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260803013934.4036646-1-pulehui@huaweicloud.com
Build xskxceiver, xdp_hw_metadata, and xdp_features from explicit source
lists instead of reusing helper objects produced by test_progs rules.
Reusing shared objects such as network_helpers.o and xsk.o can pull in
test_progs-only dependency chains and trigger unrelated libarena builds
when invoking a single target.
Keep these standalone binaries self-contained so each target builds only
its own required sources and BPF skeleton dependencies.
Signed-off-by: Tushar Vyavahare <tushar.vyavahare@intel.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Tested-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Reviewed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Link: https://lore.kernel.org/bpf/20260728115036.2049536-1-tushar.vyavahare@intel.com
Ning Ding says:
====================
bpf: Invalidate RCU pointers after final spin unlock
In a sleepable BPF program, a spin lock can provide the only RCU protection
for a kptr. The final spin unlock ends that protection, but the verifier
leaves the pointer valid. Another CPU can then free the object before the
pointer is used. A capability-limited runtime PoC triggered a
KASAN-confirmed task_struct use-after-free.
Patch 1 invalidates RCU-protected pointers only when an unlock leaves the
final RCU-protected context. Patch 2 adds a negative sleepable test and
positive controls for non-sleepable and explicit-RCU contexts.
Testing used fresh QEMU/KVM guests with KASAN enabled. The patched focused
test passed all three expected outcomes. The full task_kfunc test passed
all 39 subtests, and the selected RCU, refcount, and spin-lock group had no
failures.
---
v2:
- Rebase onto bpf-next commit 60781269e2.
- Target bpf-next and split the fix from its selftests, as requested.
- Add positive controls for RCU contexts that remain valid after unlock.
v1: https://lore.kernel.org/r/20260802231248.2781334-1-dingning04@gmail.com
====================
Link: https://patch.msgid.link/20260803112615.3362122-1-dingning04@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The verifier previously accepted a task kptr after the final spin unlock
ended its RCU protection in a sleepable program. The pointer could then be
used after the task was freed.
Add a negative test for that case. Add positive controls showing that the
pointer remains valid in a non-sleepable program and while an explicit RCU
read-side section is still active.
Assisted-by: Codex:gpt-5.6-sol
Assisted-by: ChatGPT:GPT-5.6-Pro
Signed-off-by: Ning Ding <dingning04@gmail.com>
Link: https://lore.kernel.org/bpf/20260803112615.3362122-3-dingning04@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
