Reported by the Sashiko AI review.
arena_vm_fault() returns VM_FAULT_RETRY when it can't take
arena->spinlock, but it never took mmap_lock. The fault path assumes a
VM_FAULT_RETRY handler already dropped mmap_lock and re-takes it on the
retry, so mmap_lock gets taken twice and can deadlock:
do_user_addr_fault()
{
fault = handle_mm_fault(...); // calls arena_vm_fault()
if (fault & VM_FAULT_RETRY)
goto retry; // re-locks mmap_lock
mmap_read_unlock(mm);
}
Return VM_FAULT_SIGBUS instead, for two reasons:
1. We could keep VM_FAULT_RETRY, but then we'd have to drop the fault
lock first and cap the retry ourselves, the way __folio_lock_or_retry()
does.
2. A failed raw_res_spin_lock_irqsave() already means a possible deadlock
was detected, so retrying just hits the same lock again.
So returning VM_FAULT_RETRY here is overkill.
Fixes: b8467290ed ("bpf: arena: make arena kfuncs any context safe")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260728060517.95183-1-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kfunc argument checking re-derives each argument's kfunc_ptr_arg_type
from BTF on every verification of a call in check_kfunc_args(). Now that
get_kfunc_arg_type() is a function of the kfunc's BTF alone, it no
longer inspects register state. The classification can be computed once
when the call is added and cached. This is a step toward describing
kfuncs with a bpf_func_proto and sharing the helper argument-checking
path.
Generate the classification at bpf_add_kfunc_call() time:
- Extend struct bpf_func_proto to be able to describe a kfunc: widen
arg_type[] and the arg_btf_id[]/arg_size[] union from 5 to
MAX_BPF_FUNC_ARGS, since a kfunc may take up to 12 arguments (5 in
registers, 7 on the stack).
- Embed a bpf_func_proto in struct bpf_kfunc_desc, populated by
gen_kfunc_arg_proto() which runs get_kfunc_arg_type() for each
argument and stores the result in proto.arg_type[]. Grow the
descriptor table's descs[] as a flexible array to not waste memory.
- check_kfunc_args() reads the cached classification from meta->fn
The KF_ARG_PTR_TO_CTX classification depends on the resolved program type,
and for BPF_PROG_TYPE_EXT that is the target program's type, which
resolve_prog_type() reads from prog->aux->saved_dst_prog_type. That field
is normally recorded later during verification in check_attach_btf_id(),
after bpf_add_kfunc_call() has run.
Record saved_dst_prog_type and saved_dst_attach_type from dst_prog at
program load time in bpf_prog_load() so the resolved type is available
at add-call time without reordering check_attach_btf_id(). This keeps
e.g. an freplace of an XDP program calling bpf_xdp_metadata_rx_hash()
classifying its struct xdp_md * argument as context.
The classification result is unchanged; it is only computed earlier and
cached.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-19-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Add kfunc scalar argument types, classify them in get_kfunc_arg_type()
along side with pointer arguments and move scalar type verification
into the main switch in check_kfunc_args(). This keeps BTF-based
classification separate from register validation for every argument,
paving the way for generating the kfunc argument prototype at add-call
time. No functional change intended.
KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE now are reachable. Therefore,
remove the fallthrough from KF_ARG_PTR_TO_MEM case and adjust the
register indexing.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-18-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Now that get_kfunc_ptr_arg_type() classifies a kfunc pointer argument
from its BTF alone, express a nullable argument by OR-ing PTR_MAYBE_NULL
into the classified type, and resolve a NULL register after
classification instead of before it.
Previously check_kfunc_args() short-circuited a nullable argument passed
a NULL register with a continue placed before get_kfunc_ptr_arg_type(),
so the NULL never reached classification. That kept a register-state
decision (bpf_register_is_null()) ahead of the BTF-based classification.
This mirrors how helper arguments carry PTR_MAYBE_NULL in their
bpf_arg_type and is a step toward describing kfuncs with a bpf_func_proto:
the nullability now travels with the per-argument classification, so it is
captured when the prototype is generated at add-call time.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-17-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
get_kfunc_ptr_arg_type() decided part of a kfunc pointer argument's type
from the caller's register: a PTR_TO_BTF_ID (or reg2btf_ids) register made
the argument KF_ARG_PTR_TO_BTF_ID, otherwise it fell through to a memory
buffer. Folding register state into argument classification prevents
describing a kfunc's arguments from its BTF alone, which is a prerequisite
for generating a helper-like prototype and eventually sharing the argument
checking (check_func_arg()) between helpers and kfuncs.
Classify pointer arguments from BTF only, and resolve them against the
register in check_kfunc_args():
- A pointer to a struct that is not paired with a __sz/__szk size
argument is classified KF_ARG_PTR_TO_BTF_ID and then checked against
the register. A register carrying a BTF ID (PTR_TO_BTF_ID or a
reg2btf_ids type) must be referenced or trusted and is matched against
the expected type. The only relaxation is when the struct is composed
of scalars, the register may be verified as a fixed-size memory buffer
sized from the BTF type; anything else is rejected.
- A pointer paired with a size argument is always a memory buffer and is
never classified as BTF_ID, so the __sz/__szk case no longer detours
through BTF_ID.
The new design now accepts one previously rejected case: passing
PTR_TO_BTF_ID to a pointer to scalar w/o a following __sz/__szk. The
argument will be classified as KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The
PTR_TO_BTF_ID register will go through check_mem_reg() ->
check_helper_mem_access() -> check_ptr_to_btf_access(). For a pointer to
scalar arg, a kernel btf id will be rejected unless explicitly granted
by btf_struct_access(); a program allocated btf id will be allowed.
The referenced-or-trusted check thus moves into the KF_ARG_PTR_TO_BTF_ID
resolution, alongside the type match.
get_kfunc_ptr_arg_type() no longer needs the register, so drop its regs
and reg parameters; it is now a pure function of the kfunc's BTF.
When a register cannot satisfy a BTF_ID argument, report the register type
passed and, when the expected struct has a reg2btf_ids mapping, the
register type that would be accepted, instead of a confusing "socket".
Update the affected selftest messages accordingly.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-16-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A kfunc memory-pointer argument comes in two flavors: a fixed-size buffer
whose access size is derived from the pointed-to BTF type, and a
variable-size buffer paired with a following __sz/__szk size argument.
Both were represented by separate kfunc_ptr_arg_type values
(KF_ARG_PTR_TO_MEM vs KF_ARG_PTR_TO_MEM_SIZE) with the pointer classified
as the latter when a size argument followed.
Mirror how helpers describe the same distinction: classify both as
KF_ARG_PTR_TO_MEM and OR in MEM_FIXED_SIZE for the fixed-size case, just
as helpers use ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The switches now key on
base_type(kf_arg_type) so the flag rides along, and the KF_ARG_PTR_TO_MEM
handler either resolves the size from BTF (MEM_FIXED_SIZE) or falls
through to the mem/size-pair check, which validates the buffer against the
following size register and skips it. No functional change.
Currently, KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE are only reachable
from ARG_PTR_TO_MEM fallthrough. A patch later will merge scalar
checking into the same switch and remove the fallthrough.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-15-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
get_kfunc_ptr_arg_type() returned KF_ARG_PTR_TO_NULL when a nullable
pointer argument was passed a NULL register. This folded a register-state
decision (bpf_register_is_null()) into what is otherwise BTF-based
argument classification, and it short-circuited before the BTF_ID/MEM
resolution.
Drop KF_ARG_PTR_TO_NULL and handle the NULL case in check_kfunc_args()
instead: a nullable argument that is actually NULL is skipped. Note that
it is okay to skip even when it is a mem+size pair because the size
argument check has been moved to the scalar section. The skip is done
before get_kfunc_ptr_arg_type() so that a NULL passed to a nullable
non-scalar-struct argument is not newly rejected by the BTF_ID/MEM
resolution.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-14-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
check_kfunc_args() already makes sure a scalar value is passed to a
scalar kfunc argument. Drop the check in is_kfunc_arg_mem_size() and
is_kfunc_arg_const_mem_size() to further decouple
get_kfunc_ptr_arg_type() from register state (a prerequisite for
generating a helper-like prototype from kfunc's BTF).
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-13-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
To align helper and kfunc pointer to memory argument handling, move
kfunc constant memorry size argument handling to the kfunc scalar
section. In addition, factor out constant scalar argument handling.
The constant size argument (__szk) of a kfunc memory/size pair was
recorded into meta->arg_constant by a dedicated block in the
KF_ARG_PTR_TO_MEM_SIZE case, duplicating the "only one constant
argument" and "must be a known constant" checks already in the generic
scalar argument handling. That block also did an explicit i++ to skip
the size argument.
This also fixes a precision gap: the old dedicated block did not mark
the size register precise, relying on check_mem_size_reg() for that. But
check_mem_size_reg() is skipped when the buffer is a nullable arg passed
as NULL (e.g. bpf_dynptr_slice(_rdwr) with a NULL buffer), so in that
case the __szk value was recorded and used for regs[R0].mem_size without
marking it precise. Routing the size through the scalar path marks it
precise in all cases.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-11-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
ARG_CONST_SIZE does not require a constant: check_mem_size_reg() accepts
any bounded scalar and verifies the memory access against its maximum
(reg_umax). Rename ARG_CONST_SIZE and ARG_CONST_SIZE_OR_ZERO to
ARG_MEM_SIZE and ARG_MEM_SIZE_OR_ZERO to reflect that. ARG_CONST_ALLOC_
SIZE_OR_ZERO, which does require a constant, is left unchanged.
Pure rename, no functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-10-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Fixed-size memory arguments went through two paths: helpers called
check_helper_mem_access() directly, while kfuncs and global subprogs
used check_mem_reg(). Route the helper MEM_FIXED_SIZE case through
check_mem_reg() too so all three share the same check.
This also fixes a bug in the helper path. When passing a NULL to
PTR_MAYBE_NULL | ARG_PTR_TO_FIXED_SIZE_MEM argument, the program would
be falsely rejected by check_helper_mem_access(). This is not
triggerable since there is no such kind of helper. Also, note that
check_reg_type() still make sure NULL cannot be passed to an argument
not marked with PTR_MAYBE_NULL.
It also tightens the poisoned-stack-slot check. check_mem_reg() encoded
"a STACK_POISON slot may be read" as a negative access size for any
PTR_TO_STACK argument, but that is only sound for global subprogs, where
static stack liveness proved the callee body does not read those slots
(2cb27158ad ("bpf: poison dead stack slots")). Since check_mem_reg() is
also used for kfuncs, kfuncs accidentally inherited it and could read a
poisoned (dead, possibly uninitialized) stack slot. Restrict the negative
size to global subprogs (meta == NULL) so kfuncs, like helpers, require
the whole argument initialized.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-9-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Helper ARG_CONST_SIZE and kfunc KF_ARG_PTR_TO_MEM_SIZE memory arguments
already share check_mem_size_reg(), but the kfunc path reached it
through a thin wrapper, check_kfunc_mem_size_reg(). The wrapper existed
only to invoke check_mem_size_reg() twice. Once for BPF_READ and once for
BPF_WRITE because a kfunc mem argument may be both read and written,
whereas a helper argument carries a single access direction.
Let check_mem_size_reg() take a bitmask of access directions (widening
access_type to u32) and perform each requested access, then pass
BPF_READ | BPF_WRITE from the kfunc call site. This removes the
check_kfunc_mem_size_reg() wrapper so helper and kfunc mem+size arguments
run through exactly the same code.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-7-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_map_lookup_elem() is typed to return PTR_TO_MAP_VALUE for every
map, but for some map kinds the looked up value is actually a
different object: an inner map, a socket or an xsk socket.
Until now this reinterpretation happened once the pointer was
converted from its NULL-able form to a concrete value.
Such reinterpretation logic placement led to mark_ptr_not_null_reg()
being called for a temporary register copy in check_mem_reg() and
check_kfunc_mem_size_reg() (check_mem_size_reg() was buggy because of
not calling it). The temporary copy was necessary to pass
reinterpreted parameters as nullable helper and kfunc arguments.
Avoid this complication by refining map lookup result type right away.
The test case verifier_map_in_map/on_the_inner_map_pointer needs an
update because the verifier now prints a concrete NULL-able type for
the lookup.
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-6-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kfuncs used a single '__map' suffix (KF_ARG_PTR_TO_MAP) for two different
things: a verifier-known map matched by map_uid against a bound
timer/wq/task_work object (bpf_wq_init, bpf_task_work_schedule*), and an
opaque 'struct bpf_map *' used only at runtime (bpf_arena_*), which may be a
map fd or a PTR_TO_BTF_ID struct bpf_map (e.g. a bpf_map iterator's ctx->map).
That combined path only accepted the btf map form due to type confusion. The
'if (!reg->map_ptr)' check reads reg->map_ptr, which aliases reg->btf in the
bpf_reg_state union. A PTR_TO_BTF_ID register always has a non-NULL reg->btf,
so the guard silently passed and validation fell through to
process_kf_arg_ptr_to_btf_id(). It also recorded PTR_TO_BTF_ID info in
meta->map, which would be meaningless.
Split the annotation to avoid such type confusion and to align with
helper:
- '__const_map' -> KF_ARG_CONST_MAP_PTR: verifier-known map, handled by
process_map_ptr_arg() like helper ARG_CONST_MAP_PTR.
- '__map' -> KF_ARG_PTR_TO_BTF_ID: opaque struct bpf_map, validated by
process_kf_arg_ptr_to_btf_id(). A map fd still matches via
reg2btf_ids[CONST_PTR_TO_MAP], so bpf_arena_alloc_pages(&map) keeps
working.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-4-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Both the helper ARG_CONST_MAP_PTR and the kfunc KF_ARG_PTR_TO_MAP
recorded the map pointer in meta->map and, when a map was already
bound by a preceding timer/workqueue/task_work argument, rejected a
mismatching map.
Factor the logic into a single process_map_ptr_arg() used by both
paths. The bound-object name (timer, workqueue, or bpf_task_work) is
derived from the bound map's btf_record, and the register numbers in
the message are computed from the map argument position instead of
being hard-coded.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260801074633.1595644-3-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
btf_distill_func_proto() builds the function model used for the
fentry/fexit/fmod_ret/fsession trampolines and struct_ops. It has
accepted a 16-byte __int128 return value since the trampoline was
introduced: __get_type_size() returns the integer's type size, and the
return-type check only rejected ret < 0.
But the BPF trampoline preserves only 8 bytes of the return value (RAX on
x86, i.e. R0). For an attach type that reads the target's return value the
second half (RDX / R3) is neither saved nor restored, so a program
attached to a function returning a 16-byte value corrupts the value seen
by the real caller and itself observes only half of it. struct_ops
trampolines have the same limitation.
This affects the attach types that read the target's return value: fexit,
fmod_ret and fsession (plus the _multi variants of fexit and fsession),
and struct_ops. fentry/fentry_multi run before the target returns and are
unaffected.
Reject a >8 byte return value for these attach types in
bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
struct_ops in bpf_struct_ops_desc_init().
Fixes: fec56f5890 ("bpf: Introduce BPF trampoline")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729050159.2585809-1-yonghong.song@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Replace silent WARN_ON_ONCE with WARN_ONCE that prints the actual error
code from bpf_trampoline_unlink_prog(). This aids debugging of race
conditions during link teardown, while keeping the warning rate limited
to avoid log flooding.
This will be very helpful for speeding up trouble-shooting of some crash
UAF due to bpf_trampoline_unlink_prog failures.
No change to unlink behavior.
Signed-off-by: Xu Xin <xu.xin16@zte.com.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260729141159128mEJmS_aujBKr-cBu1p_UI@zte.com.cn
In bpf_netns_link_update_prog, the checks for old_prog and prog type
are currently performed locklessly before acquiring netns_bpf_mutex.
This creates a race condition that can lead to a UAF issue.
If two threads concurrently execute BPF_LINK_UPDATE on the same netns
link, the following execution path can trigger a UAF:
CPU0 CPU1
bpf_netns_link_update_prog
if (old_prog && old_prog != link->prog)
return -EPERM;
bpf_netns_link_update_prog
if (old_prog && old_prog != link->prog)
...
old_prog = xchg(&link->prog, new_prog);
bpf_prog_put(old_prog);
if (new_prog->type != link->prog->type) <-- trigger UAF
Fix this by moving the old_prog and prog->type checks inside the
netns_bpf_mutex critical section. Meanwhile, use guard() to simplify
lock management and avoid all the goto jumping.
Fixes: 7f045a49fe ("bpf: Add link-based BPF program attachment to network namespace")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0]
Link: https://lore.kernel.org/bpf/20260728023259.2813482-1-pulehui@huaweicloud.com
There is no particular reason to keep bpf_res_spin_lock() disabled in
tracing programs, since it is safe against reentrancy and deadlocks.
Remove the restriction for tracing programs covered by the predicate
is_tracing_prog_type().
This is a prerequisite before the definition of is_tracing_prog_type()
is updated to include raw_tp, fentry, fexit, and fmod_ret. Existing
tracing programs will be updated to use bpf_res_spin_lock() instead when
it is available.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260719113551.1294284-2-memxor@gmail.com
The trampoline could be corrupted by the blindly
'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier.
1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became
'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the
trampoline would poke the target prog's nop insn using jmp insn instead
of call insn.
2. Another fexit loaded with the same tail_call_reachable prog target.
'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'.
3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in
'tr->flags', the trampoline will fail to restore the prog's nop insn
using call insn.
[ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98
...
[ 3.428793] bpf_link_free+0x58/0x130
[ 3.429293] bpf_link_release+0x23/0x30
Fix the warning by updating 'tr->flags' with '|=' and lock.
Fixes: 2b5dcb31a1 ("bpf, x64: Fix tailcall infinite loop")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
A global subprogram argument tagged __arg_nonnull is set up as a
non-nullable PTR_TO_MEM. However the verifier does not check against a
scalar NULL, leading to real NULL pointer dereference. Reject it as
well.
Fixes: 94e1c70a34 ("bpf: support 'arg:xxx' btf_decl_tag-based hints for global subprog args")
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260723221815.367797-1-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
execmem_alloc() can return ROX memory that is already filled with
architecture defined trapping instructions.
In preparation for enabling this mode for BPF on x86, make sure that there
is no redundant management of the ROX memory.
There is no need to fill allocated memory with trapping instructions, to
request permissions reset on free and to set ROX permissions as this all
is handled by execmem_alloc().
Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if
execmem_alloc() returns ROX memory and skip the redundant steps in that
case.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the
corresponding execmem APIs.
Architectures define the properties of the memory range needed by BPF in
their initialization of execmem and don't need to override neither of
them.
Drop the __weak qualifier from bpf_jit_alloc_exec() and
bpf_jit_free_exec().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_dispatcher->rw_image is a temporary writable buffer that
arch_prepare_bpf_dispatcher() fills and then copies into
bpf_dispatcher->image using bpf_arch_text_copy().
The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against
->image, so ->rw_image does not need to live in the module address range.
Allocate ->rw_image with vzalloc() to avoid permissions dance when
EXECMEM_BPF will be backed by ROX caches.
Using vzalloc() rather than vmalloc() ensures that the memory that
bpf_dispatcher_update() unconditionally copies into the executable buffer
is zeroed, which is not ideal but still better than random memory returned
by the existing bpf_jit_alloc_exec() or plain vmalloc().
Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of
space in the more scarce module address space.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-1-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
btf_find_struct_field() initializes a fresh seen mask for every recursive
descent. Unique special fields in different levels of the same aggregate
therefore do not see one another. The duplicate fields can reach
btf_parse_fields(), where they trigger an invariant WARN_ON_ONCE(). A
crafted user BTF can consequently trigger the warning before map creation
checks capabilities.
Initialize the seen mask once in btf_find_field() and pass the same pointer
through struct, datasec, and nested-struct walks. This gives the entire field
traversal one shared uniqueness state.
Fixes: 64e8ee8148 ("bpf: look into the types of the fields of a struct type recursively.")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Queue and stack pop/peek helpers accept an uninitialized output buffer
because the verifier expects the helper to initialize it. The empty-map
error path clears the buffer, but a failed lock acquisition returns
-EBUSY without writing it.
Clear the output before returning -EBUSY so BPF programs cannot observe
uninitialized stack contents after a failed helper call.
Fixes: a34a9f1a19 ("bpf: Avoid deadlock when using queue and stack maps from NMI")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260719125419.1782196-1-memxor@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The BPF_MOV64_PERCPU_REG insn requires JIT to emit native code to for
'dst_reg = src_reg + <percpu_base_off>'.
However, the interpreter ignores the 'off' at its ALU64_MOV_X label.
The 'off' indicates the insn is BPF_MOV64_PERCPU_REG insn. Then, when
the interpreter loads memory from the register, it will hit a page
fault.
[ 2.545572] BUG: unable to handle page fault for address: ffffffffacaaf034
[ 2.546485] #PF: supervisor read access in kernel mode
[ 2.547167] #PF: error_code(0x0000) - not-present page
[ 2.547850] PGD 134e63067 P4D 134e63067 PUD 134e64063 PMD 10021c063 PTE 800ffffeca550062
[ 2.548912] Oops: Oops: 0000 [#1] SMP PTI
Set jit_required as true in order to disallow interpreter fallback in
core.c::__bpf_prog_select_runtime(), if any BPF_ADDR_PERCPU insn is
patched to the prog.
BTW, rename the helper bpf_map_supports_cpu_flags() to
bpf_map_is_percpu_map().
Fixes: 7bdbf74463 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-4-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The interpreter does not recognize the BPF_JMP|BPF_JA|BPF_X insn, which
is used for insn_array map. Thereafter, it would hit the BUG_ON() in
___bpf_prog_run() at run time.
[ 2.563726] BPF interpreter: unknown opcode 0d (imm: 0x0)
[ 2.564557] ------------[ cut here ]------------
[ 2.565206] kernel BUG at kernel/bpf/core.c:2349!
[ 2.565882] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when insn_array map is used in the prog in
order to disallow interpreter fallback for gotox insn in
core.c::__bpf_prog_select_runtime().
Fixes: 493d9e0d60 ("bpf, x86: add support for indirect jumps")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-3-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Since the interpreter does not support the arena-related insns,
interpreter fallback should not be allowed for these insns in
core.c::__bpf_prog_select_runtime().
Currently, when the interpreter executes the arena ST/LDX/STX insns,
it would hit the BUG_ON() in ___bpf_prog_run() at run time.
[ 2.579196] BPF interpreter: unknown opcode a2 (imm: 0x0)
[ 2.579998] ------------[ cut here ]------------
[ 2.580652] kernel BUG at kernel/bpf/core.c:2349!
[ 2.581314] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when arena map is used in the prog to disallow
interpreter fallback for arena-related insns.
Fixes: 6082b6c328 ("bpf: Recognize addr_space_cast instruction in the verifier.")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_arena_free_pages() accepts scalar arena addresses. The runtime
masks the address to the low 32 bits and reconstructs a full user
address from the arena base before returning the range to the arena
free tree.
When the scalar value is below the low 32 bits of the arena base,
full_uaddr falls below user_vm_start. The existing upper-end clipping
then turns this into an out-of-range free-tree offset. A later
allocation can reuse that offset and return an address below the arena
mapping.
Reject such frees before computing the clipped range.
Fixes: 317460317a ("bpf: Introduce bpf_arena.")
Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260717-c10-031-public-bpf-next-v2-b4-v2-1-54b555443a7c@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arena is the only map type whose map_mem_usage() still returns 0, so
"bpftool map show" and fdinfo always showed 0 memlock for an arena no
matter how many pages it had.
Count the pages that are actually mapped into the arena: bump a counter in
apply_range_set_cb() when a page goes in and drop it in
apply_range_clear_cb() when a page goes out, both under the arena spinlock.
map_mem_usage() then just returns nr_pages << PAGE_SHIFT.
Only real data pages are counted, not the scratch page.
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260717114117.350851-3-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
check_kfunc_call() reads meta.func_name when bpf_fetch_kfunc_arg_meta()
returns -EACCES, but that error can come from fetch_kfunc_meta() (e.g.
fd_array_get_btf() rejecting BTF binding for a signed program) before
meta is memset(), leaving it uninitialized and risking a garbage deref
in verbose().
Move the memset() to the start of bpf_fetch_kfunc_arg_meta() so meta is
zeroed on every error return. The intended "not allowed" -EACCES path
still sets func_name first, so its message is unchanged.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260715172127.2416388-3-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
For a bloom filter, the value argument of bpf_map_peek_elem() is always
an input. Therefore, the verifier should not allow passing uninitialized
stack memory to it to avoid information leak.
bpf_map_peek_elem() tags its value argument ARG_PTR_TO_MAP_VALUE |
MEM_UNINIT, telling the verifier the callee fills the buffer. This holds
for queue/stack maps, but not for a bloom filter, which reads the buffer
as an input to test set membership and never writes it.
As a result, a program can pass an uninitialized stack buffer to
bpf_map_peek_elem() on a bloom filter. The verifier accepts it and marks
the buffer initialized on return, letting the program read back leftover
kernel stack memory. Bloom maps require CAP_BPF to create, so this is a
CAP_BPF-gated stack infoleak that bypasses the boundary CAP_BPF is meant
to enforce (arbitrary kernel reads are gated behind CAP_PERFMON).
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260715172127.2416388-2-ameryhung@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Some JIT compilers, such as x86_64, rely on a register to pass the TCC.
When subprograms of synchronous callback invoke tailcall, C helpers
invoking bpf callback clobber this register, and the corrupted TCC may
bypass the TCC limit, leading to infinite tailcall.
Fix this by rejecting tailcall inside all subprogs of sync callback.
This also cleanly consolidates the existing async and exception callback
checks into a single unified `is_cb` check.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Reported-by: Björn Töpel <bjorn@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260716120157.835937-3-pulehui@huaweicloud.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Currently in check_max_stack_depth_subprog, when the verifier enters a
new callee branch, the local tail_call_reachable is not properly
synchronized with the callee's state.
Consider a main prog branching into multiple subprogs:
subprog0 -> tailcall
main <
subprog1 -> subprog2
When the verifier finishes checking subprog0 and backtracks to main
prog, the local tail_call_reachable state is left as true. As it
proceeds to subprog1, this uncleared state leaks into the new branch,
falsely marking subprog1 and subprog2 as tailcall reachable.
Fix this by explicitly syncing tail_call_reachable with the callee's
has_tail_call state on entry. The caller's state is safely preserved and
restored via the existing backtracking logic.
Fixes: ebf7d1f508 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Link: https://patch.msgid.link/20260716120157.835937-2-pulehui@huaweicloud.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
A kfunc marked with KF_IMPLICIT_ARGS flag takes implicit arguments
(such as bpf_prog_aux) that the verifier injects at load time.
resolve_btfids strips those from the kfunc's BTF-visible prototype and
keeps the real kernel ABI in a counterpart _impl prototype [1].
fentry/fexit/fmod_ret/fsession programs may attach to the BPF kernel
functions, including those with implicit args. However
bpf_check_attach_target() and bpf_check_attach_btf_id_multi() extract
the struct btf_func_model from the wrong BTF prototype of the
kfunc. The btf_func_model is later read to construct the trampoline,
which then causes the injected implicit argument to be clobbered and
the kfunc dereferencing garbage.
Add btf_attach_func_proto() to resolve the real ABI prototype of the
kfunc the way the call site does: by looking up the _impl prototype
for a KF_IMPLICIT_ARGS kfunc. Use it at both attach-target model
construction sites.
To enable this, make two supporting changes:
* pass bpf_verifier_log instead of bpf_verifier_env to
find_kfunc_impl_proto(), so it can be reused from the attach path
* add btf_kfunc_check_flag() to test a flag across all of a kfunc's
hook sets, because a program attaching to a kfunc is not in the
kfunc's call-set
KF_IMPLICIT_ARGS must be consistent across the sets, so
btf_kfunc_check_flag() returns -EINVAL on inconsistency.
btf_kfunc_check_flag() reads the kfunc's flags from the target's
kfunc_set_tab. For a module BTF that table is stable only after the
module is live, so take a module reference around the read, mirroring
how the kfunc call path gates the same lookup with btf_try_get_module().
The remaining call sites of btf_distill_func_proto() are safe as
is. The BPF_TRACE_ITER case distills a registered iterator's
prototype, and bpf_struct_ops_desc_init() distills the
function-pointer members of a struct_ops type. Neither is a kfunc, and
so can't have implicit arguments.
[1] https://lore.kernel.org/all/20260120222638.3976562-1-ihor.solodrai@linux.dev/
Fixes: 64e1360524 ("bpf: Verifier support for KF_IMPLICIT_ARGS")
Reported-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://github.com/sched-ext/scx/issues/3687#issuecomment-4906694106
Link: https://patch.msgid.link/20260713235223.1639022-2-ihor.solodrai@linux.dev
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier rejects variable offsets for PTR_TO_TP_BUFFER and PTR_TO_BUF
accesses, but it currently accepts a constant negative offset produced by
pointer arithmetic.
Commit 022ac07508 ("bpf: use reg->var_off instead of reg->off for
pointers") moved constant pointer offsets from reg->off to reg->var_off.
However, __check_buffer_access() continued to check only the instruction
offset. An access with reg->var_off equal to -8 and an instruction offset
of zero therefore passes verification.
For writable raw tracepoints, the access end is also calculated from the
unsigned reg->var_off.value. An eight-byte access starting at -8 wraps
the calculated end to zero, allowing the program to load and attach
without increasing max_tp_access.
After ensuring that reg->var_off is constant, calculate the effective
access start using signed arithmetic and reject it when it is negative.
Use the validated start to calculate the access end for both
PTR_TO_TP_BUFFER and PTR_TO_BUF.
Fixes: 022ac07508 ("bpf: use reg->var_off instead of reg->off for pointers")
Signed-off-by: Sun Jian <sun.jian.kdev@gmail.com>
Acked-by: Shung-Hsi Yu <shung-hsi.yu@suse.com>
Cc: stable@vger.kernel.org # 5.2.0
Link: https://patch.msgid.link/20260714093846.18159-2-sun.jian.kdev@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Helper and kfunc argument checking carried two separate meta structs: the
verifier-local struct bpf_call_arg_meta and bpf_kfunc_call_arg_meta.
Merge them into a single struct bpf_call_arg_meta. This is groundwork for
sharing argument checking between helpers and kfuncs.
While merging, drop the btf_id field from the helper meta since it is
never used.
No functional change.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-8-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
meta->pkt_access is only ever a copy of fn->pkt_access, assigned once in
check_helper_call() and read back in may_access_direct_pkt_data(). Have
may_access_direct_pkt_data() take the bpf_func_proto and read
fn->pkt_access directly, and drop the meta field along with its
assignment.
The only non-NULL caller, check_func_arg(), already has fn in scope; the
remaining callers pass NULL and are unaffected.
No functional change intended.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-7-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The constant "size of the PTR_TO_MEM returned in R0" argument is handled by
both helpers (ARG_CONST_ALLOC_SIZE_OR_ZERO) and kfuncs (__rdonly_buf_size /
__rdwr_buf_size), each with its own meta field (meta->mem_size,
meta->r0_size) and duplicated validation. Add struct arg_alloc_mem_desc
and a shared process_const_alloc_mem_size(), and replace both fields with
meta->arg_alloc_mem.
The desc records presence with a 'found' flag instead of using a non-zero
size as the sentinel. This also fixes a pre-existing bug on the kfunc
return path: "no size argument" was tested as r0_size == 0, so an explicit
__rdonly_buf_size/__rdwr_buf_size of 0 was treated as absent and fell
through to btf_resolve_size(), giving R0 the size of the pointed-to return
type instead of 0. With 'found', an explicit zero size is honored and
btf_resolve_size() is used only when no size argument was passed.
The size is stored in a u32, matching regs[R0].mem_size. The U32_MAX
check now apply to both helper and kfunc through
process_const_alloc_mem_size().
Fold bpf_session_cookie return size assignment into current kfunc return
size resolution path.
Note that verifier saves kfunc return size through r0_size instead of
mem_size. The later has no active readers so remove it.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260715064047.1793790-5-ameryhung@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
