The trampoline could be corrupted by the blindly
'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier.
1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became
'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the
trampoline would poke the target prog's nop insn using jmp insn instead
of call insn.
2. Another fexit loaded with the same tail_call_reachable prog target.
'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'.
3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in
'tr->flags', the trampoline will fail to restore the prog's nop insn
using call insn.
[ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98
...
[ 3.428793] bpf_link_free+0x58/0x130
[ 3.429293] bpf_link_release+0x23/0x30
Fix the warning by updating 'tr->flags' with '|=' and lock.
Fixes: 2b5dcb31a1 ("bpf, x64: Fix tailcall infinite loop")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Feng Jiang says:
====================
bpf, riscv: add timed may_goto support
This series adds RISC-V JIT support for the timed may_goto loop bound.
Patch 1 implements arch_bpf_timed_may_goto() and enables
bpf_jit_supports_timed_may_goto() so the verifier uses the timed
expansion path.
Patch 2 adds a test that checks R0-R5 are preserved across
arch_bpf_timed_may_goto() calls.
Patch 3 enables the verifier_may_goto_1, stream_cond_break, and
may_goto_interaction fastcall tests on riscv64.
Tested on riscv64 QEMU (rva23s64): may_goto programs load and JIT
correctly, and the 250ms timeout path works as expected.
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Tested-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
---
Changes in v5:
- Use REG_S/REG_L/SZREG in arch_bpf_timed_may_goto assembly. (Pu Lehui)
- Add __arch_s390x to the timed_may_goto_preserves_regs test. (Pu Lehui)
- Switch the preserves-regs test to SEC("syscall") to fix
bpf_prog_test_run() EINVAL.
- Link to v4: https://lore.kernel.org/r/20260722-riscv-bpf-timed-may-goto-v4-0-e117e6337bc7@kylinos.cn
Changes in v4:
- Add 'bpf-next' prefix to match the BPF kernel tree workflow.
- Add a test checking that R0-R5 are preserved across
arch_bpf_timed_may_goto() calls. Use bpf_get_prandom_u32() to
prevent the verifier from removing the checks via DCE.
- Rename may_goto_interaction_arm64() to may_goto_interaction().
- Wrap the arch_bpf_timed_may_goto address check to a single line.
- Link to v3: https://lore.kernel.org/r/20260715-riscv-bpf-timed-may-goto-v3-0-cf2a9c3d843f@kylinos.cn
Changes in v3:
- Set up the frame pointer in arch_bpf_timed_may_goto() so the function
does not break stack unwinding under CONFIG_FRAME_POINTER.
Changes in v2:
- Fix BPF_REG_0 being clobbered after arch_bpf_timed_may_goto() calls.
- Enable the may_goto_interaction fastcall test on riscv64.
---
====================
Link: https://patch.msgid.link/20260723-riscv-bpf-timed-may-goto-v5-0-86acb54e5642@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Teach libarena's BPF atomic primitives to use compiler builtins for
load-acquire and store-release when Clang advertises
__BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the
existing barrier-based fallback.
Notably, as BPF programs begin running on arm64, it is better to use the
more appropriate variants since we can no longer rely on x86 TSO ordering.
Commit 880442305a ("bpf: Introduce load-acquire and store-release instructions")
introduced support, hence kernels from 6.15 onwards are needed when
compiling with compilers supporting these instructions. We have
relatively recent kernel version requirements in libarena anyway, and
have not cut first release, hence declare such a dependency.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260722141003.2841007-1-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Mike Rapoport says:
====================
bpf, x86: enable EXECMEM_ROX_CACHE for BPF allocations
Hi,
BPF allocations of executable memory on x86 are essentially read-only. Most
paths that call bpf_jit_alloc_exec() immediately make it ROX with
set_memory_rox().
The code generation, at least on x86, uses separately allocated writable
buffers and then updates the actual text memory with text_poke().
These patches do several small adjustments to how BPF allocates executable
memory and enable EXECMEM_ROX_CACHE for BPF allocations on x86.
Acked-by: Song Liu <song@kernel.org>
---
v3 changes:
* replace vmalloc() with vzalloc() in bpf_dispatcher_change_prog()
* rebase on the current bpf-next
v2: https://patch.msgid.link/20260711-execmem-x86-rox-bpf-v0-v2-0-bfd956d35119@kernel.org
* rebase on the current bpf-next
v1: https://patch.msgid.link/20260626-execmem-x86-rox-bpf-v0-v1-0-45a0b0ed4fe9@kernel.org
---
---
Mike Rapoport (Microsoft) (5):
bpf: dispatcher: allocate bpf_dispatcher->rw_image with vzalloc()
bpf: drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec()
bpf: alloc_prog_pack(): skip ROX management for already ROX memory
bpf, x86: make sure allocation in arch_bpf_trampoline_size() is writable
x86/bpf: enable EXECMEM_ROX_CACHE for BPF allocations
arch/x86/mm/init.c | 4 ++--
arch/x86/net/bpf_jit_comp.c | 5 ++---
include/linux/filter.h | 1 +
kernel/bpf/core.c | 30 +++++++++++++++++++++---------
kernel/bpf/dispatcher.c | 5 ++++-
5 files changed, 30 insertions(+), 15 deletions(-)
---
base-commit: d1f4b56417
change-id: 20260626-execmem-x86-rox-bpf-v0-b4241ade80df
--
Sincerely yours,
Mike.
====================
Link: https://patch.msgid.link/20260716-execmem-x86-rox-bpf-v0-v3-0-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
arch_bpf_trampoline_size() allocates a buffer to get actual size required
for a trampoline.
This buffer must be in the module address space because
__arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to
that buffer.
In preparation for enabling ROX mode for EXECMEM_BPF make sure that the
allocated memory is writable.
Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for
buffer allocation in arch_bpf_trampoline_size().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
execmem_alloc() can return ROX memory that is already filled with
architecture defined trapping instructions.
In preparation for enabling this mode for BPF on x86, make sure that there
is no redundant management of the ROX memory.
There is no need to fill allocated memory with trapping instructions, to
request permissions reset on free and to set ROX permissions as this all
is handled by execmem_alloc().
Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if
execmem_alloc() returns ROX memory and skip the redundant steps in that
case.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the
corresponding execmem APIs.
Architectures define the properties of the memory range needed by BPF in
their initialization of execmem and don't need to override neither of
them.
Drop the __weak qualifier from bpf_jit_alloc_exec() and
bpf_jit_free_exec().
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_dispatcher->rw_image is a temporary writable buffer that
arch_prepare_bpf_dispatcher() fills and then copies into
bpf_dispatcher->image using bpf_arch_text_copy().
The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against
->image, so ->rw_image does not need to live in the module address range.
Allocate ->rw_image with vzalloc() to avoid permissions dance when
EXECMEM_BPF will be backed by ROX caches.
Using vzalloc() rather than vmalloc() ensures that the memory that
bpf_dispatcher_update() unconditionally copies into the executable buffer
is zeroed, which is not ideal but still better than random memory returned
by the existing bpf_jit_alloc_exec() or plain vmalloc().
Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of
space in the more scarce module address space.
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-1-4e76158c01c5@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Fix unique field logic in BTF
Fix constraint of certain fields that have to be unique when nested
structs are present, BPF_REFCOUNT which should be marked as unique.
See commit logs for details. While at it, fix improper offset check.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/20260719142401.2420111-1-memxor@gmail.com
* Add fix for BPF_REFCOUNT not being unique. (Sashiko)
* Roll improper offset warning check into series. (Sashiko)
====================
Link: https://patch.msgid.link/20260719153634.2908692-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
btf_find_struct_field() initializes a fresh seen mask for every recursive
descent. Unique special fields in different levels of the same aggregate
therefore do not see one another. The duplicate fields can reach
btf_parse_fields(), where they trigger an invariant WARN_ON_ONCE(). A
crafted user BTF can consequently trigger the warning before map creation
checks capabilities.
Initialize the seen mask once in btf_find_field() and pass the same pointer
through struct, datasec, and nested-struct walks. This gives the entire field
traversal one shared uniqueness state.
Fixes: 64e8ee8148 ("bpf: look into the types of the fields of a struct type recursively.")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260719153634.2908692-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Kuan-Wei Chiu says:
====================
riscv, bpf: Add support for signed operations and 32-bit atomics
Add support for missing signed operations and 32-bit atomics in the
RV32 BPF JIT compiler.
The current implementation lacks support for BPF_SDIV, BPF_SMOD, and
BPF_MOVSX, ignoring the instruction offset field and treating them as
unsigned or zero-extended. Introduce support for these operations by
checking the offset field and emitting the corresponding instructions.
Additionally, we leverage the mandatory A extension to natively support
32-bit BPF atomics (and, or, xor, xchg) by mapping them directly to
amo*.w instructions. BPF_CMPXCHG continues to fall back to the
interpreter.
As a result, test_bpf.ko now runs with 0 failures, and the total number
of successfully JIT'ed test cases increases from 843 to 902.
Tested-by: Pu Lehui <pulehui@huawei.com>
---
Changes in v4:
- Add a 'dst' param to emit_bpf_atomic().
Changes in v3:
- Pass insn directly to emit_alu_r32().
- Remove the Fixes: tag and update title.
- Consolidate size, mode, and off into insn for emit_store_r64().
Changes in v2:
- Add missing Fixes tags.
- Fix memory ordering by emitting aq=1, rl=1
Kuan-Wei Chiu (3):
riscv, bpf: Add support for BPF_SDIV and BPF_SMOD in RV32 JIT
riscv, bpf: Add support for BPF_MOVSX in RV32 JIT
riscv, bpf: Add 32 bit atomic operations to RV32 JIT
arch/riscv/net/bpf_jit_comp32.c | 111 +++++++++++++++++++++++++++-----
1 file changed, 95 insertions(+), 16 deletions(-)
====================
Link: https://patch.msgid.link/20260714002451.4091139-1-visitorckw@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The RV32 BPF JIT compiler currently only supports the BPF_ADD atomic
operation. Other 32 bit atomic operations (and, or, xor, xchg) and
their BPF_FETCH variants are not supported and gracefully fall back to
the interpreter.
Since the RISC-V A extension is required for Linux on RV32, we can
natively support these 32-bit BPF atomic operations by mapping them
directly to the corresponding RISC-V amo*.w instructions.
Implement BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, and BPF_XCHG with and
without BPF_FETCH. BPF_CMPXCHG requires a more complex lr.w/sc.w
loop and is left to fall back to the interpreter.
Before this patch:
[ 138.862161] test_bpf: Summary: 1054 PASSED, 0 FAILED, [843/1042 JIT'ed]
After this patch:
[ 157.024124] test_bpf: Summary: 1054 PASSED, 0 FAILED, [902/1042 JIT'ed]
Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20260714002451.4091139-4-visitorckw@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
emit_atomic_ld_st() returns 1 to have build_body() skip the zext after
a sub-word load_acquire. The caller does "ret = ret ?:
add_exception_handler(...)", which skips add_exception_handler() on any
non-zero ret, so the extable entry is missing and a faulting
PROBE_ATOMIC load_acquire oopses.
REG_DONT_CLEAR_MARKER leaves rd stale on fault, and the verifier still
thinks the load overwrote it, so a program can leak it through a map.
Check ret >= 0 before calling add_exception_handler(), and pass rd for
LOAD_ACQ so the fault zeroes rd like a PROBE_MEM load. Return ret
unchanged for the zext skip.
Fixes: fb7cefabae ("riscv, bpf: Add support arena atomics for RV64")
Suggested-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Feng Jiang <jiangfeng@kylinos.cn>
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260720-bpf-riscv-fix-extable-v4-1-165c0b3b07d5@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Pu Lehui says:
====================
Mixing bpf2bpf and tailcalls for RV64
In the current RV64 JIT, if we just don't initialize the TCC in subprog,
the TCC can be propagated from the parent process to the subprocess, but
the updated TCC of the parent process cannot be restored when the
subprocess exits. Since the RV64 TCC is initialized before saving the
callee saved registers into the stack, we cannot use the callee saved
register to pass the TCC, otherwise the original value of the callee
saved register will be destroyed. So we implemented mixing bpf2bpf and
tailcalls similar to x86_64, i.e. using a non-callee saved register to
transfer the TCC between functions, and saving that register to the
stack to protect the TCC value. As for the tailcall hierarchy issue,
inspired by the s390's low-overhead approach, we store TCC from
RV_REG_TCC back to stack after calling bpf2bpf call or calling orig bpf
func in bpf trampoline.
In addition, some code cleans are also attached to this patchset.
Tests test_bpf.ko and test_verifier have passed, as well as the relative
testcases of test_progs*.
v6:
- Add new patch to fix memory leak in bpf_jit_free. (Sashiko)
v5: https://lore.kernel.org/bpf/20260707142219.2871758-1-pulehui@huaweicloud.com
- Fix TCC value was not restored to RV_REG_TCC upon trampoline exit,
resulting in an infinite tail call. (Sashiko)
- Fix epilogue not restored TCC value to RV_REG_TCC, resulting in
leaf tailcall callee may clobbered RV_REG_TCC. (Sashiko)
- Remove patch of `Remove ctx->offset initialization` as it would hit
some corner case. (Sashiko)
- Fix double-count kcfi insn when tailcall to target. (Sashiko)
- Add detail for selftest commit message. (BPF BOT)
v4: https://lore.kernel.org/bpf/20260629140048.733346-1-pulehui@huaweicloud.com
- Fix tailcall hierarchy issue.
- use is_struct_ops_tramp helper in bpf trampoline
v3: https://lore.kernel.org/bpf/20240201083351.943121-1-pulehui@huaweicloud.com
- Remove duplicate RV_REG_TCC load in epiloguei. (Björn Töpel)
v2: https://lore.kernel.org/bpf/20240130040958.230673-1-pulehui@huaweicloud.com
- Fix emit restore RV_REG_TCC double times when `flags &
BPF_TRAMP_F_CALL_ORIG`
- Use bpf_is_subprog helper
v1: https://lore.kernel.org/bpf/20230919035711.3297256-1-pulehui@huaweicloud.com
====================
Link: https://patch.msgid.link/20260708064436.2971933-1-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
In the current RV64 JIT, if we just don't initialize the TCC in subprog,
the TCC can be propagated from the parent process to the subprocess, but
the updated TCC of the parent process cannot be restored when the
subprocess exits. Since the RV64 TCC is initialized before saving the
callee saved registers into the stack, we cannot use the callee saved
register to pass the TCC, otherwise the original value of the callee
saved register will be destroyed. So we implemented mixing bpf2bpf and
tailcalls similar to x86_64, i.e. using a non-callee saved register to
transfer the TCC between functions, and saving that register to the
stack to protect the TCC value. As for the tailcall hierarchy issue,
inspired by the s390's low-overhead approach, we store TCC from
RV_REG_TCC back to stack after calling bpf2bpf call or calling orig bpf
func in bpf trampoline.
Tests test_bpf.ko and test_verifier have passed, as well as the relative
testcases of test_progs*.
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-7-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When bpf_int_jit_compile() is called for subprograms, it returns early
during the first pass (!prog->is_func || extra_pass is false), keeping
ctx->offset alive for the subsequent extra pass.
If JIT compilation fails for a later subprogram, the BPF core aborts
and calls bpf_jit_free() to clean up the first subprogram. However,
bpf_jit_free() fails to free jit_data->ctx.offset, which causes a
memory leak of the JIT context offsets array.
Fix this by adding the missing kfree(jit_data->ctx.offset) in
bpf_jit_free().
Fixes: 48a8f78c50 ("bpf, riscv: use prog pack allocator in the BPF JIT")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Reviewed-by: Björn Töpel <bjorn@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20260708064436.2971933-3-pulehui@huaweicloud.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
attach_probe/uprobe-lib and uprobe_autoattach selftests fail with "failed
to resolve full path for libc.so.6" on older non-usrmerged distros, where
libc.so.6 lives under a top-level /lib64 or /lib rather than /usr/lib64 or
/usr/lib. Add /lib64:/lib to the search paths, alongside the existing
/usr/lib64:/usr/lib and Debian multiarch entries.
Fixes: 1ce3a60e3c ("libbpf: auto-resolve programs/libraries when necessary for uprobes")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-3-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
progs/lsm_bdev.c keys its verity_devices hashmap with the raw kernel dev_t
read straight off bdev->bd_dev, i.e. MKDEV(major, minor) = (major << 20) |
minor. prog_tests/lsm_bdev.c instead builds its lookup key with dev_key =
(__u32)st.st_rdev from stat(2), but the stat(2) syscall fills st_rdev via
the kernel's new_encode_dev(), a different bit layout: (minor & 0xff) |
(major << 8) | ((minor & ~0xff) << 12).
For any device with a non-trivial major these two values differ, so the
lookup can never find what the BPF program stored, and test_lsm_bdev()
always fails with:
test_lsm_bdev:FAIL:map lookup unexpected error: -2 (errno 2)
Reconstruct the raw kernel dev_t from the decoded major/minor instead of
casting st_rdev directly, restoring the layout the BPF program actually
reads.
Fixes: 96f4c251a0 ("selftests/bpf: add block device management selftests")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-2-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
After "make install", test_progs fails because two files end up in the
wrong place:
- bpftool: TEST_GEN_PROGS_EXTENDED flattens it into $(INSTALL_PATH), losing
the tools/sbin/ prefix that detect_bpftool_path() expects. Remove it from
TEST_GEN_PROGS_EXTENDED and install it explicitly under tools/sbin/
instead.
- *.BTF: resolve_btfids writes resolve_btfids.test.o.BTF as a side-effect
of the build but INSTALL_RULE never copies it over. Install all *.BTF
files alongside the rest of the per-flavor output.
Fixes: f21fae5774 ("selftests/bpf: Add a few helpers for bpftool testing")
Fixes: 522397d05e ("resolve_btfids: Change in-place update with raw binary output")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-1-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Queue and stack pop/peek helpers accept an uninitialized output buffer
because the verifier expects the helper to initialize it. The empty-map
error path clears the buffer, but a failed lock acquisition returns
-EBUSY without writing it.
Clear the output before returning -EBUSY so BPF programs cannot observe
uninitialized stack contents after a failed helper call.
Fixes: a34a9f1a19 ("bpf: Avoid deadlock when using queue and stack maps from NMI")
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260719125419.1782196-1-memxor@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
cast_user converts an arena offset into a user address by combining the
low 32 bits of the pointer with the upper 32 bits of user_vm_start, while
keeping a NULL pointer NULL. The current sequence always emits six
instructions: it materializes user_vm_start >> 32 into a register, shifts
it into place, and ORs in the offset.
The upper half of user_vm_start is a constant, so it can be written
directly onto the offset with MOVK. Move the 32-bit offset into dst
(which also zeroes the upper 32 bits), then MOVK the non-zero halfwords
of the upper address, branching over the MOVKs when the offset is zero so
NULL is preserved.
This emits at most four instructions, and only one when the upper half of
user_vm_start is zero. The generated code is equivalent.
Before:
; bpf_addr_space_cast(page1, 1, 0);
7c: mov w10, w8
80: mov w8, #1
84: lsl x8, x8, #32
88: cbz x10, 0xffff800087b80c20
8c: orr x10, x8, x10
90: mov x8, x10
After:
; bpf_addr_space_cast(page1, 1, 0);
7c: mov w8, w8
80: cbz w8, 0xffff800087b80c28
84: movk x8, #1, lsl #32
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260721105921.1070501-1-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Viktor Malik says:
====================
selftests/bpf: Fix compilation with RELEASE=1
When compiling BPF selftests with RELEASE=1 (which most notably adds
-O2), GCC reports several warnings. These are by default treated as
errors and abort the compilation.
This series fixes all of the issues such that selftests can be compiled
with RELEASE=1.
====================
Link: https://patch.msgid.link/cover.1784112948.git.vmalik@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When compiling BPF selftests with -O2, GCC reports a maybe-uninitialized
warning in libarena code:
In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:11:
In function ‘libarena_asan_init’,
inlined from ‘run_test’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:59:8,
inlined from ‘test_libarena_asan’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:91:2:
/bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h:126:14: error: ‘globals_pages’ may be used uninitialized [-Werror=maybe-uninitialized]
126 | args = (struct asan_init_args){
| ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
127 | .arena_all_pages = arena_all_pages,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
128 | .arena_globals_pages = globals_pages,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129 | };
| ~
/bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h: In function ‘test_libarena_asan’:
/bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h:118:13: note: ‘globals_pages’ was declared here
118 | u64 globals_pages;
| ^~~~~~~~~~~~~
Silence the warning by initializing globals_pages to 0.
Fixes: cfc00618b9 ("selftests/bpf: Add ASAN support for libarena selftests")
Signed-off-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/9f77a5c05c3c731ab2655fd66716ab9de4478b15.1784112948.git.vmalik@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
When compiling BPF selftests with -O2, GCC reports an array bounds
violation warning in global_map_resize test:
In function ‘global_map_resize_bss_subtest’,
inlined from ‘test_global_map_resize’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:228:3:
/bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:64:33: error: array subscript 1 is above array bounds of ‘int[1]’ [-Werror=array-bounds=]
64 | skel->bss->array[i] = 1;
| ~~~~~~~~~~~~~~~~^~~
In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:6:
./test_global_map_resize.skel.h: In function ‘test_global_map_resize’:
./test_global_map_resize.skel.h:44:21: note: while referencing ‘array’
44 | int array[1];
| ^~~~~
This is a false positive because `array` (a BPF map) has been resized
from within the BPF program. GCC doesn't know that so let us silence the
warning by accessing the array via a plain pointer.
Fixes: 08b0895675 ("libbpf: Selftests for resizing datasec maps")
Signed-off-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/57765bc465a27923c3c093eba222cc24d08d8c40.1784112948.git.vmalik@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Replace ASSERT_OK_PTR by ASSERT_NEQ(res, NULL, ...) when checking the
result of malloc. It is more accurate since malloc returns NULL, not an
error code, on failure and it also prevents the following false GCC
warning when compiling BPF selftests with -O2:
In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c:4:
/bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c: In function ‘test_sha256’:
./test_progs.h:393:22: error: ‘data’ may be used uninitialized [-Werror=maybe-uninitialized]
393 | int ___err = libbpf_get_error(___res); \
| ^~~~~~~~~~~~~~~~~~~~~~~~
/bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c:28:14: note: in expansion of macro ‘ASSERT_OK_PTR’
28 | if (!ASSERT_OK_PTR(data, "malloc"))
| ^~~~~~~~~~~~~
In file included from /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/bpf.h:32,
from ./test_progs.h:37:
/bpf-next/tools/testing/selftests/bpf/tools/include/bpf/libbpf_legacy.h:113:17: note: by argument 1 of type ‘const void *’ to ‘libbpf_get_error’ declared here
113 | LIBBPF_API long libbpf_get_error(const void *ptr);
| ^~~~~~~~~~~~~~~~
Fixes: f09f57c746 ("selftests/bpf: Add test for libbpf_sha256()")
Signed-off-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/f9dec09cca0c2aa5eeb4fdcd400a13aa19e2c073.1784112948.git.vmalik@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Replace ASSERT_OK_PTR by ASSERT_NEQ(res, NULL, ...) when checking the
result of malloc. It is more accurate since malloc returns NULL, not an
error code, on failure and it also prevents the following false GCC
warning when compiling BPF selftests with -O2:
In file included from test_loader.c:6:
test_loader.c: In function ‘verify_stderr’:
/bpf-next/tools/testing/selftests/bpf/test_progs.h:393:22: error: ‘buf’ may be used uninitialized [-Werror=maybe-uninitialized]
393 | int ___err = libbpf_get_error(___res); \
| ^~~~~~~~~~~~~~~~~~~~~~~~
test_loader.c:810:14: note: in expansion of macro ‘ASSERT_OK_PTR’
810 | if (!ASSERT_OK_PTR(buf, "malloc"))
| ^~~~~~~~~~~~~
In file included from /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/bpf.h:32,
from /bpf-next/tools/testing/selftests/bpf/test_progs.h:37:
/bpf-next/tools/testing/selftests/bpf/tools/include/bpf/libbpf_legacy.h:113:17: note: by argument 1 of type ‘const void *’ to ‘libbpf_get_error’ declared here
113 | LIBBPF_API long libbpf_get_error(const void *ptr);
| ^~~~~~~~~~~~~~~~
Fixes: 554e4eb9e4 ("selftests/bpf: Reuse stderr parsing for libarena ASAN tests")
Signed-off-by: Viktor Malik <vmalik@redhat.com>
Link: https://lore.kernel.org/bpf/e25d50805fbcb3632f24b488568ab5ba49b82094.1784112948.git.vmalik@redhat.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Looking up a prog or map by name walks the whole id space. There is a
window between bpf_prog_get_next_id()/bpf_map_get_next_id() and getting
an fd for that id in which an unrelated object can be freed, and the
lookup then fails with ENOENT and aborts the whole command.
Skip such ids and keep walking, the same way do_show() already does.
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/bpf/20260720071520.396363-1-jiayuan.chen@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Leon Hwang says:
====================
bpf: Disallow interpreter fallback for interpreter-unsupported insns
Sashiko reported two potential issues about interpreter fallback [1]
[2].
After verifying them by patch #7 of v1, I think they are real issues. With
LLM assistance, the interpreter does not support the internal
BPF_PROBE_ATOMIC insn and the gotox insn (used for indirect jumps),
either.
1) the user BPF_ADDR_SPACE_CAST insn
the interpreter just ignores it.
2) the arena ST/STX/LDX insn
the interpreter could hit the BUG_ON() in ___bpf_prog_run().
3) the BPF_MOV64_PERCPU_REG insn
the interpreter could hit page fault, due to loading memory from
invalid __percpu pointer.
4) the internal BPF_PROBE_ATOMIC insn
the interpreter could hit the BUG_ON() in ___bpf_prog_run().
5) the gotox insn used for indirect jumps
the interpreter could hit the BUG_ON() in ___bpf_prog_run(), too.
Reject these insns on interpreter fallback path in
__bpf_prog_select_runtime() by setting 'jit_required = true'.
Link:
[1] https://lore.kernel.org/bpf/20260608151347.2C77D1F00893@smtp.kernel.org/
[2] https://lore.kernel.org/bpf/20260622150759.EC9071F000E9@smtp.kernel.org/
Changes:
v1 -> v2:
* Drop RFC.
* Change target tree to bpf-next to utilize the 'jit_required' bit.
* Set jit_required as true if there's arena map, then all arena-related
insns will be rejected if JIT is not available.
* Set jit_required as true if there's insn_array map, then the gotox
insns will be rejected if JIT is not available.
* Drop the issues-proven patch.
* v1: https://lore.kernel.org/bpf/20260626154330.33619-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260715141122.15783-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The BPF_MOV64_PERCPU_REG insn requires JIT to emit native code to for
'dst_reg = src_reg + <percpu_base_off>'.
However, the interpreter ignores the 'off' at its ALU64_MOV_X label.
The 'off' indicates the insn is BPF_MOV64_PERCPU_REG insn. Then, when
the interpreter loads memory from the register, it will hit a page
fault.
[ 2.545572] BUG: unable to handle page fault for address: ffffffffacaaf034
[ 2.546485] #PF: supervisor read access in kernel mode
[ 2.547167] #PF: error_code(0x0000) - not-present page
[ 2.547850] PGD 134e63067 P4D 134e63067 PUD 134e64063 PMD 10021c063 PTE 800ffffeca550062
[ 2.548912] Oops: Oops: 0000 [#1] SMP PTI
Set jit_required as true in order to disallow interpreter fallback in
core.c::__bpf_prog_select_runtime(), if any BPF_ADDR_PERCPU insn is
patched to the prog.
BTW, rename the helper bpf_map_supports_cpu_flags() to
bpf_map_is_percpu_map().
Fixes: 7bdbf74463 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-4-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The interpreter does not recognize the BPF_JMP|BPF_JA|BPF_X insn, which
is used for insn_array map. Thereafter, it would hit the BUG_ON() in
___bpf_prog_run() at run time.
[ 2.563726] BPF interpreter: unknown opcode 0d (imm: 0x0)
[ 2.564557] ------------[ cut here ]------------
[ 2.565206] kernel BUG at kernel/bpf/core.c:2349!
[ 2.565882] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when insn_array map is used in the prog in
order to disallow interpreter fallback for gotox insn in
core.c::__bpf_prog_select_runtime().
Fixes: 493d9e0d60 ("bpf, x86: add support for indirect jumps")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-3-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Since the interpreter does not support the arena-related insns,
interpreter fallback should not be allowed for these insns in
core.c::__bpf_prog_select_runtime().
Currently, when the interpreter executes the arena ST/LDX/STX insns,
it would hit the BUG_ON() in ___bpf_prog_run() at run time.
[ 2.579196] BPF interpreter: unknown opcode a2 (imm: 0x0)
[ 2.579998] ------------[ cut here ]------------
[ 2.580652] kernel BUG at kernel/bpf/core.c:2349!
[ 2.581314] Oops: invalid opcode: 0000 [#1] SMP PTI
Set jit_required as true when arena map is used in the prog to disallow
interpreter fallback for arena-related insns.
Fixes: 6082b6c328 ("bpf: Recognize addr_space_cast instruction in the verifier.")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260715141122.15783-2-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Yiyang Chen says:
====================
bpf: Reject arena frees below the arena base
bpf_arena_free_pages() can be called with a scalar arena address. The
runtime reconstructs a full user address from the arena base and the low
32 bits before returning the range to the arena free tree. A scalar one
page below the arena base can otherwise produce an out-of-domain free-tree
offset and make a later allocation return an address below the arena
mapping.
Patch 1 rejects frees whose reconstructed full user address is below
user_vm_start. Patch 2 adds verifier_arena coverage for the
scalar-below-base case.
Changes in v2:
- Add Reviewed-by tags from Emil Tsalapatis.
- Remove the empty inline asm from the selftest.
v1: https://lore.kernel.org/bpf/cover.1782813442.git.chenyy23@mails.tsinghua.edu.cn/
====================
Link: https://patch.msgid.link/20260717-c10-031-public-bpf-next-v2-b4-v2-0-54b555443a7c@mails.tsinghua.edu.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
