From 509ca545d425512f83ca70093f6d836ec8ab5bd1 Mon Sep 17 00:00:00 2001 From: Jordan Rife Date: Thu, 18 Jun 2026 11:20:32 -0700 Subject: [PATCH 001/373] bpf: Support BPF_F_EGRESS with bpf_redirect_peer We have several use cases where a pod injects traffic into the datapath of another so that the traffic appears to have originated from that pod. One such use case is a synthetic flow generator which injects synthetic traffic into a pod's datapath to enable dynamic probing and debugging. Another is a transparent proxy where connections originating from one pod are redirected towards another which proxies that connection. The new connection is bound to the IP of the original pod using IP_TRANSPARENT and its traffic is injected into that pod's datapath and handled as if it had originated there. This can be used for mTLS, etc. We use bpf_redirect(BPF_F_INGRESS) to direct traffic leaving the proxy, flow generator, etc. towards the target pod, ensuring that eBPF programs that are meant to intercept traffic leaving that pod are executed. However, this doesn't work with netkit. With netkit, an ingress redirection from proxy to workload skips eBPF programs that are meant to intercept traffic leaving the pod, since they reside on the netkit peer device. One workaround is to attach the same program to both the netkit peer device and the TCX ingress hook for the netkit pair's primary interface, but a) This seems hacky and we need to be careful not to run the same program twice for the same skb in cases where we want to pass that traffic to the host stack. b) We're trying to keep the proxy redirection / traffic injection systems as modular and separated from Cilium as possible, the system that manages netkit setup and core eBPF programming. It would be handy if instead we could redirect traffic directly from one netkit peer device to another. This patch proposes an extension to bpf_redirect_peer to allow us to do just that. With this patch, the BPF_F_EGRESS flag tells bpf_redirect_peer to emit the skb in the egress direction of the target interface's peer device While the main use case is netkit, I suppose you could also use this mode with veth as well if, e.g., there were some eBPF programs attached to that side of the veth pair that needed to intercept traffic. +---------------------------------------------------------------------+ | +-------------------------+ 6. bpf_redirect_neigh(eth0) | | | pod (10.244.0.10) | ------------------------ | | | | | | | | | +--------+ | | +---------+ | | | | 1. packet -->| | | | | | | | | | leaves ^ | netkit |<===========|======| netkit | | | | | | | peer |=======(eBPF)=====>| primary | | | | | | | | | | | | | | | | | +--------+ | | +---------+ | | | | | | | 2. bpf_redirect v | | +-----------|-------------+ |___________________ +-------| | | | | eth0 | | | 5. bpf_redirect_peer(BPF_F_EGRESS) | +-------| | |________________________ | | | +-------------------------+ | | | | | proxy (10.244.0.11) | | | | | | IP_TRANSPARENT | | | | | | +--------+ | | +---------+ | | | | 3. packet <--| | | | | |<-- | | | enters | netkit |<===========|======| netkit | | | | [proxy] | peer |=======(eBPF)=====>| primary | | | | 4. packet -->| | | | | | | | leaves +--------+ | +---------+ | | | sip=10.244.0.10 | | | +-------------------------+ | +---------------------------------------------------------------------+ Using the proxy use case as an example, in step 5 we would redirect traffic leaving the proxy towards the pod's peer device using bpf_redirect_peer(BPF_F_EGRESS). As a bonus, since the skb doesn't have to go through the backlog queue it can take full advantage of netkit's performance benefits. I set up a test where outgoing iperf3 traffic is injected into the datapath of another pod using either bpf_redirect_peer(BPF_F_EGRESS) or bpf_redirect(BPF_F_INGRESS). I used Cilium's eBPF host routing mode which skips the host stack and uses BPF redirect helpers to do all the routing. (net.ipv4.tcp_congestion_control=cubic,mtu=1500,100GiB link,Cilium eBPF host routing mode) BASELINE [bpf_redirect(BPF_F_INGRESS)] 1. [iperf pod] ==bpf_redirect([pod b], BPF_F_INGRESS)==> [pod b] 2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0 3. eth0 ==over network==> [host b] [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-60.00 sec 231 GBytes 33.0 Gbits/sec 12060 sender [ 5] 0.00-60.00 sec 230 GBytes 33.0 Gbits/sec receiver TEST [bpf_redirect_peer(BPF_F_EGRESS)] 1. [iperf pod] ==bpf_redirect_peer([pod b], BPF_F_EGRESS)==> [pod b] 2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0 3. eth0 ==over network==> [host b] [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec 0 sender [ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec receiver In this test, using bpf_redirect_peer(BPF_F_EGRESS) for the hop from [iperf pod] to [pod b] led to ~18% more throughput compared to bpf_redirect(BPF_F_INGRESS). Signed-off-by: Jordan Rife Acked-by: Daniel Borkmann Acked-by: Paul Chaignon Reviewed-by: Jiayuan Chen Link: https://lore.kernel.org/r/20260618182035.43811-2-jordan@jrife.io Signed-off-by: Alexei Starovoitov --- include/uapi/linux/bpf.h | 19 +++++++++++-------- net/core/filter.c | 12 +++++++----- tools/include/uapi/linux/bpf.h | 19 +++++++++++-------- 3 files changed, 29 insertions(+), 21 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 89b36de5fdbb..c91b5a4bda03 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -5079,17 +5079,19 @@ union bpf_attr { * Description * Redirect the packet to another net device of index *ifindex*. * This helper is somewhat similar to **bpf_redirect**\ (), except - * that the redirection happens to the *ifindex*' peer device and - * the netns switch takes place from ingress to ingress without - * going through the CPU's backlog queue. + * that the redirection happens to the *ifindex*' peer device. If + * *flags* is 0, the netns switch takes place from ingress to + * ingress without going through the CPU's backlog queue. If the + * **BPF_F_EGRESS** flag is provided then redirection happens in + * the egress direction of the peer device. * * *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during * the netns switch. * - * The *flags* argument is reserved and must be 0. The helper is - * currently only supported for tc BPF program types at the - * ingress hook and for veth and netkit target device types. The - * peer device must reside in a different network namespace. + * If the *flags* argument is 0, the helper is currently only + * supported for tc BPF program types at the ingress hook and for + * veth and netkit target device types. The peer device must reside + * in a different network namespace. * Return * The helper returns **TC_ACT_REDIRECT** on success or * **TC_ACT_SHOT** on error. @@ -6336,9 +6338,10 @@ enum { /* Flags for bpf_redirect and bpf_redirect_map helpers */ enum { BPF_F_INGRESS = (1ULL << 0), /* used for skb path */ + BPF_F_EGRESS = (1ULL << 1), /* used for skb path */ BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */ BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */ -#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) +#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_EGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) }; #define __bpf_md_ptr(type, name) \ diff --git a/net/core/filter.c b/net/core/filter.c index b446aa8be5c3..4f5cbcac3e78 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -2529,16 +2529,18 @@ int skb_do_redirect(struct sk_buff *skb) if (unlikely(!dev)) goto out_drop; if (flags & BPF_F_PEER) { - if (unlikely(!skb_at_tc_ingress(skb))) - goto out_drop; dev = skb_get_peer_dev(dev); if (unlikely(!dev || !(dev->flags & IFF_UP) || net_eq(net, dev_net(dev)))) goto out_drop; + skb_scrub_packet(skb, false); + if (flags & BPF_F_EGRESS) + return __bpf_redirect(skb, dev, 0); + if (unlikely(!skb_at_tc_ingress(skb))) + goto out_drop; skb->dev = dev; dev_sw_netstats_rx_add(dev, skb->len); - skb_scrub_packet(skb, false); return -EAGAIN; } return flags & BPF_F_NEIGH ? @@ -2575,10 +2577,10 @@ BPF_CALL_2(bpf_redirect_peer, u32, ifindex, u64, flags) { struct bpf_redirect_info *ri = bpf_net_ctx_get_ri(); - if (unlikely(flags)) + if (unlikely(flags & ~BPF_F_EGRESS)) return TC_ACT_SHOT; - ri->flags = BPF_F_PEER; + ri->flags = BPF_F_PEER | flags; ri->tgt_index = ifindex; return TC_ACT_REDIRECT; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 89b36de5fdbb..c91b5a4bda03 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -5079,17 +5079,19 @@ union bpf_attr { * Description * Redirect the packet to another net device of index *ifindex*. * This helper is somewhat similar to **bpf_redirect**\ (), except - * that the redirection happens to the *ifindex*' peer device and - * the netns switch takes place from ingress to ingress without - * going through the CPU's backlog queue. + * that the redirection happens to the *ifindex*' peer device. If + * *flags* is 0, the netns switch takes place from ingress to + * ingress without going through the CPU's backlog queue. If the + * **BPF_F_EGRESS** flag is provided then redirection happens in + * the egress direction of the peer device. * * *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during * the netns switch. * - * The *flags* argument is reserved and must be 0. The helper is - * currently only supported for tc BPF program types at the - * ingress hook and for veth and netkit target device types. The - * peer device must reside in a different network namespace. + * If the *flags* argument is 0, the helper is currently only + * supported for tc BPF program types at the ingress hook and for + * veth and netkit target device types. The peer device must reside + * in a different network namespace. * Return * The helper returns **TC_ACT_REDIRECT** on success or * **TC_ACT_SHOT** on error. @@ -6336,9 +6338,10 @@ enum { /* Flags for bpf_redirect and bpf_redirect_map helpers */ enum { BPF_F_INGRESS = (1ULL << 0), /* used for skb path */ + BPF_F_EGRESS = (1ULL << 1), /* used for skb path */ BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */ BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */ -#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) +#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_EGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) }; #define __bpf_md_ptr(type, name) \ From 006b9456e9f4e8a5cada19409d26bec8193acc13 Mon Sep 17 00:00:00 2001 From: Jordan Rife Date: Thu, 18 Jun 2026 11:20:33 -0700 Subject: [PATCH 002/373] selftests/bpf: Add tests for bpf_redirect_peer with BPF_F_EGRESS Extend redirect tests to cover bpf_redirect_peer(BPF_F_EGRESS). SRC redirects to DST using bpf_redirect_peer(BPF_F_EGRESS) then traffic is hairpinned into DST using bpf_redirect. Signed-off-by: Jordan Rife Acked-by: Daniel Borkmann Acked-by: Paul Chaignon Link: https://lore.kernel.org/r/20260618182035.43811-3-jordan@jrife.io Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/tc_redirect.c | 68 +++++++++++++++++++ .../selftests/bpf/progs/test_tc_peer.c | 22 ++++++ 2 files changed, 90 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/tc_redirect.c b/tools/testing/selftests/bpf/prog_tests/tc_redirect.c index 64fbda082309..af8968b89ad7 100644 --- a/tools/testing/selftests/bpf/prog_tests/tc_redirect.c +++ b/tools/testing/selftests/bpf/prog_tests/tc_redirect.c @@ -192,6 +192,8 @@ static int create_netkit(int mode, char *prim, char *peer) req.n.nlmsg_len += sizeof(struct ifinfomsg); addattr_l(&req.n, sizeof(req), IFLA_IFNAME, peer, strlen(peer)); addattr_nest_end(&req.n, peer_info); + addattr32(&req.n, sizeof(req), IFLA_NETKIT_SCRUB, + NETKIT_SCRUB_NONE); addattr_nest_end(&req.n, data); addattr_nest_end(&req.n, linkinfo); @@ -405,6 +407,24 @@ static int netns_load_bpf(const struct bpf_program *src_prog, return -1; } +static struct bpf_link *netns_attach_nk(const char *ns, int ifindex, + struct bpf_program *prog) +{ + LIBBPF_OPTS(bpf_netkit_opts, optl); + struct nstoken *nstoken = NULL; + struct bpf_link *link = NULL; + + nstoken = open_netns(ns); + if (!ASSERT_OK_PTR(nstoken, "setns")) + goto cleanup; + + link = bpf_program__attach_netkit(prog, ifindex, &optl); +cleanup: + if (nstoken) + close_netns(nstoken); + return link; +} + static void test_tcp(int family, const char *addr, __u16 port) { int listen_fd = -1, accept_fd = -1, client_fd = -1; @@ -1082,6 +1102,53 @@ static void test_tc_redirect_peer(struct netns_setup_result *setup_result) close_netns(nstoken); } +static void test_tc_redirect_peer_ing(struct netns_setup_result *setup_result) +{ + struct test_tc_peer *skel; + struct nstoken *nstoken; + int err; + + nstoken = open_netns(NS_FWD); + if (!ASSERT_OK_PTR(nstoken, "setns fwd")) + return; + + skel = test_tc_peer__open(); + if (!ASSERT_OK_PTR(skel, "test_tc_peer__open")) + goto done; + + skel->rodata->IFINDEX_SRC = setup_result->ifindex_src_fwd; + skel->rodata->IFINDEX_DST = setup_result->ifindex_dst_fwd; + ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc_src_ing, + BPF_NETKIT_PRIMARY), 0, "src_prog_attach_type"); + ASSERT_EQ(bpf_program__set_expected_attach_type(skel->progs.tc_dst_ing, + BPF_NETKIT_PRIMARY), 0, "dst_prog_attach_type"); + + err = test_tc_peer__load(skel); + if (!ASSERT_OK(err, "test_tc_peer__load")) + goto done; + + skel->links.tc_src_ing = netns_attach_nk(NS_SRC, + setup_result->ifindex_src, + skel->progs.tc_src_ing); + if (!ASSERT_OK_PTR(skel->links.tc_src_ing, "attach_src")) + goto done; + skel->links.tc_dst_ing = netns_attach_nk(NS_DST, + setup_result->ifindex_dst, + skel->progs.tc_dst_ing); + if (!ASSERT_OK_PTR(skel->links.tc_dst_ing, "attach_dst")) + goto done; + + if (!ASSERT_OK(set_forwarding(false), "disable forwarding")) + goto done; + + test_connectivity(); + +done: + if (skel) + test_tc_peer__destroy(skel); + close_netns(nstoken); +} + static int tun_open(char *name) { struct ifreq ifr; @@ -1280,6 +1347,7 @@ static void *test_tc_redirect_run_tests(void *arg) RUN_TEST(tc_redirect_peer, MODE_VETH); RUN_TEST(tc_redirect_peer, MODE_NETKIT); + RUN_TEST(tc_redirect_peer_ing, MODE_NETKIT); RUN_TEST(tc_redirect_peer_l3, MODE_VETH); RUN_TEST(tc_redirect_peer_l3, MODE_NETKIT); RUN_TEST(tc_redirect_neigh, MODE_VETH); diff --git a/tools/testing/selftests/bpf/progs/test_tc_peer.c b/tools/testing/selftests/bpf/progs/test_tc_peer.c index 365eacb5dc34..cfb9ef7f467c 100644 --- a/tools/testing/selftests/bpf/progs/test_tc_peer.c +++ b/tools/testing/selftests/bpf/progs/test_tc_peer.c @@ -34,6 +34,28 @@ int tc_src(struct __sk_buff *skb) return bpf_redirect_peer(IFINDEX_DST, 0); } +SEC("tc") +int tc_dst_ing(struct __sk_buff *skb) +{ + if (!skb->mark) { + skb->mark = 0x1; + return bpf_redirect_peer(IFINDEX_SRC, BPF_F_EGRESS); + } + + return bpf_redirect(IFINDEX_DST, 0); +} + +SEC("tc") +int tc_src_ing(struct __sk_buff *skb) +{ + if (!skb->mark) { + skb->mark = 0x1; + return bpf_redirect_peer(IFINDEX_DST, BPF_F_EGRESS); + } + + return bpf_redirect(IFINDEX_SRC, 0); +} + SEC("tc") int tc_dst_l3(struct __sk_buff *skb) { From 69fdbe63e16919a885a8f9441e248ce0ddf15b25 Mon Sep 17 00:00:00 2001 From: Woojin Ji Date: Thu, 25 Jun 2026 19:25:37 +0900 Subject: [PATCH 003/373] bpf: Preserve scalar zero spills for stack reads Stack reads can read back bytes that belong to a previously spilled scalar constant zero. Today mark_reg_stack_read() only treats STACK_ZERO bytes as known zero bytes, so the destination register can become unknown even though every byte in the read range is known to be zero. This can lead to rejecting otherwise valid programs once the loaded byte is used as a pointer offset. The original reproducer uses a variable-offset stack byte read emitted by clang 22.1.6 at -O2/-O3 from a small helper-based BPF C program. Fixed offset reads have a related mixed case as well: pure scalar-zero spill reads are already handled, but a fixed read spanning both STACK_ZERO and scalar const-zero STACK_SPILL bytes still falls back to unknown. Teach mark_reg_stack_read() to also consider STACK_SPILL bytes backed by a spilled scalar constant zero as zero bytes, and use that path for both variable-offset stack reads and fixed-offset mixed reads. Keep the existing pure register-fill behavior unchanged. When a zero result depends on such a spill, mark the contributing stack slots precise before accepting the const-zero result so pruning cannot reuse a zero-spill state for a later non-zero spill state. No deployed-program regression is currently known, so target bpf-next. Assisted-by: opencode:gpt-5.5 Signed-off-by: Woojin Ji Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260625-bpf-stack-var-off-zero-v1-v3-1-a068210a761b@gmail.com Signed-off-by: Alexei Starovoitov --- include/linux/bpf_verifier.h | 5 ++++ kernel/bpf/verifier.c | 53 ++++++++++++++++++++++++++++-------- 2 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 39a851e690ec..76b8b7627a10 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1243,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, bt->stack_masks[frame] |= 1ull << slot; } +static inline void bpf_bt_set_frame_slot_mask(struct backtrack_state *bt, u32 frame, u64 mask) +{ + bt->stack_masks[frame] |= mask; +} + static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot) { bt->stack_arg_masks[frame] |= 1 << slot; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 21a365d436a5..25aea4271cd0 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3702,14 +3702,21 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, * SCALAR. This function does not deal with register filling; the caller must * ensure that all spilled registers in the stack range have been marked as * read. + * + * STACK_SPILL bytes backed by spilled scalar const zeroes are also considered + * zero bytes. In that case, mark the contributing stack slots precise so + * pruning cannot reuse a zero-spill state for a later non-zero spill state. + * + * Returns an error if precision backtracking fails. */ -static void mark_reg_stack_read(struct bpf_verifier_env *env, - /* func where src register points to */ - struct bpf_func_state *ptr_state, - int min_off, int max_off, int dst_regno) +static int mark_reg_stack_read(struct bpf_verifier_env *env, + /* func where src register points to */ + struct bpf_func_state *ptr_state, + int min_off, int max_off, int dst_regno) { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; + u64 zero_spill_mask = 0; int i, slot, spi; u8 *stype; int zeros = 0; @@ -3719,19 +3726,33 @@ static void mark_reg_stack_read(struct bpf_verifier_env *env, spi = slot / BPF_REG_SIZE; mark_stack_slot_scratched(env, spi); stype = ptr_state->stack[spi].slot_type; - if (stype[slot % BPF_REG_SIZE] != STACK_ZERO) - break; - zeros++; + if (stype[slot % BPF_REG_SIZE] == STACK_ZERO) { + zeros++; + continue; + } + if (stype[slot % BPF_REG_SIZE] == STACK_SPILL && + bpf_register_is_null(&ptr_state->stack[spi].spilled_ptr)) { + zero_spill_mask |= 1ull << spi; + zeros++; + continue; + } + break; } if (zeros == max_off - min_off) { /* Any access_size read into register is zero extended, * so the whole register == const_zero. */ __mark_reg_const_zero(env, &state->regs[dst_regno]); + if (zero_spill_mask) { + bpf_bt_set_frame_slot_mask(&env->bt, ptr_state->frameno, zero_spill_mask); + return mark_chain_precision_batch(env, env->cur_state); + } } else { /* have read misc data from the stack */ mark_reg_unknown(env, state->regs, dst_regno); } + + return 0; } /* Read the stack at 'off' and put the results into the register indicated by @@ -3753,6 +3774,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, int i, slot = -off - 1, spi = slot / BPF_REG_SIZE; struct bpf_reg_state *reg; u8 *stype, type; + int err; int insn_flags = INSN_F_STACK_ACCESS; int hist_spi = spi, hist_frame = reg_state->frameno; @@ -3835,7 +3857,10 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, __mark_reg_const_zero(env, &state->regs[dst_regno]); insn_flags = 0; /* not restoring original register state */ } else { - mark_reg_unknown(env, state->regs, dst_regno); + err = mark_reg_stack_read(env, reg_state, off, off + size, + dst_regno); + if (err) + return err; insn_flags = 0; /* not restoring original register state */ } } @@ -3880,8 +3905,11 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, } return -EACCES; } - if (dst_regno >= 0) - mark_reg_stack_read(env, reg_state, off, off + size, dst_regno); + if (dst_regno >= 0) { + err = mark_reg_stack_read(env, reg_state, off, off + size, dst_regno); + if (err) + return err; + } insn_flags = 0; /* we are not restoring spilled register */ } if (insn_flags) @@ -3935,7 +3963,10 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg min_off = reg_smin(reg) + off; max_off = reg_smax(reg) + off; - mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); + err = mark_reg_stack_read(env, ptr_state, min_off, max_off + size, + dst_regno); + if (err) + return err; check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off); return 0; } From e693da913d2cdd69c1612fbf71bdf2b6771f6dba Mon Sep 17 00:00:00 2001 From: Woojin Ji Date: Thu, 25 Jun 2026 19:25:38 +0900 Subject: [PATCH 004/373] selftests/bpf: Cover stack reads from zero spills Add verifier_var_off coverage for variable-offset stack reads from spilled scalar constant zero values. Cover single-slot and cross-slot spilled zero reads, a sub-8-byte spill with neighbouring STACK_ZERO bytes, and a sub-8-byte spill with neighbouring STACK_MISC bytes that must not be treated as zero. Add verifier_spill_fill coverage for a fixed-offset stack read spanning both STACK_ZERO bytes and scalar const-zero STACK_SPILL bytes. Use verifier log assertions to check both the zero result and the precision backtracking trail. Assisted-by: opencode:gpt-5.5 Signed-off-by: Woojin Ji Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260625-bpf-stack-var-off-zero-v1-v3-2-a068210a761b@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_spill_fill.c | 26 +++++ .../selftests/bpf/progs/verifier_var_off.c | 110 ++++++++++++++++++ 2 files changed, 136 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c index 0174887e28f5..72c691333703 100644 --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c @@ -634,6 +634,32 @@ __naked void partial_stack_load_preserves_partial_zeros(void) : __clobber_common); } +SEC("raw_tp") +__log_level(2) +__success +__msg("mark_precise: frame0: regs= stack=-8") +__msg("R2=0") +__naked void stack_load_preserves_mixed_zero_and_zero_spill(void) +{ + asm volatile ( + /* fp-8 has scalar const-zero spill bytes and STACK_ZERO bytes. */ + ".8byte %[fp4_st_zero];" /* LLVM-18+: *(u32 *)(r10 -4) = 0; */ + "r0 = 0;" + "*(u32 *)(r10 -8) = r0;" + + "r1 = %[single_byte_buf];" + "r2 = *(u64 *)(r10 -8);" + "r1 += r2;" + "*(u8 *)(r1 + 0) = r2;" /* this should be fine */ + + "r0 = 0;" + "exit;" + : + : __imm_ptr(single_byte_buf), + __imm_insn(fp4_st_zero, BPF_ST_MEM(BPF_W, BPF_REG_FP, -4, 0)) + : __clobber_common); +} + char two_byte_buf[2] SEC(".data.two_byte_buf"); SEC("raw_tp") diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c index f345466bca68..24cd0a763673 100644 --- a/tools/testing/selftests/bpf/progs/verifier_var_off.c +++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c @@ -59,6 +59,116 @@ __naked void stack_read_priv_vs_unpriv(void) " ::: __clobber_all); } +SEC("cgroup/skb") +__description("variable-offset stack read preserves spilled zero") +__success +__log_level(2) +__msg("mark_precise: frame0: regs= stack=-8") +__msg("R3=0") +__retval(0) +__naked void stack_read_var_off_preserves_spilled_zero(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u64*)(r10 - 8) = r0; \ + r2 = *(u32*)(r1 + 0); \ + r2 &= 7; \ + r2 -= 8; \ + r2 += r10; \ + r3 = *(u8*)(r2 + 0); \ + r1 = r10; \ + r1 += -1; \ + r1 += r3; \ + *(u8*)(r1 + 0) = r3; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("cgroup/skb") +__description("variable-offset stack read preserves spilled zero across slots") +__success +__log_level(2) +__msg("mark_precise: frame0: regs= stack=-8,-16") +__msg("R3=0") +__retval(0) +__naked void stack_read_var_off_preserves_spilled_zero_across_slots(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u64*)(r10 - 8) = r0; \ + *(u64*)(r10 - 16) = r0; \ + r2 = *(u32*)(r1 + 0); \ + r2 &= 15; \ + r2 -= 16; \ + r2 += r10; \ + r3 = *(u8*)(r2 + 0); \ + r1 = r10; \ + r1 += -1; \ + r1 += r3; \ + *(u8*)(r1 + 0) = r3; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("cgroup/skb") +__description("variable-offset stack read preserves partial spilled zero") +__success +__log_level(2) +__msg("mark_precise: frame0: regs= stack=-8") +__msg("R3=0") +__retval(0) +__naked void stack_read_var_off_preserves_partial_spilled_zero(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u8*)(r10 - 9) = r0; \ + *(u8*)(r10 - 10) = r0; \ + *(u8*)(r10 - 11) = r0; \ + *(u8*)(r10 - 12) = r0; \ + *(u8*)(r10 - 13) = r0; \ + *(u8*)(r10 - 14) = r0; \ + *(u8*)(r10 - 15) = r0; \ + *(u32*)(r10 - 8) = r0; \ + r2 = *(u32*)(r1 + 0); \ + r2 &= 15; \ + if r2 > 10 goto l0_%=; \ + r2 -= 15; \ + r2 += r10; \ + r3 = *(u8*)(r2 + 0); \ + r1 = r10; \ + r1 += -1; \ + r1 += r3; \ + *(u8*)(r1 + 0) = r3; \ +l0_%=: r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("cgroup/skb") +__description("variable-offset stack read partial spill with misc data") +__failure +__msg("invalid variable-offset write to stack R1") +__naked void stack_read_var_off_partial_spill_with_misc_data(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u32*)(r10 - 8) = r0; \ + r2 = *(u32*)(r1 + 0); \ + r2 &= 7; \ + r2 -= 8; \ + r2 += r10; \ + r3 = *(u8*)(r2 + 0); \ + r1 = r10; \ + r1 += -1; \ + r1 += r3; \ + *(u8*)(r1 + 0) = 0; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + SEC("cgroup/skb") __description("variable-offset stack read, uninitialized") __success From df3153758ddba58b546f9fc85e5274bfcaa0bf51 Mon Sep 17 00:00:00 2001 From: Andrii Nakryiko Date: Wed, 24 Jun 2026 13:49:46 -0700 Subject: [PATCH 005/373] libbpf: fix -Wformat warnings from format/argument type mismatches Building libbpf with -Wall (as happens via bpftool's bootstrap build) surfaces ~120 -Wformat warnings where pr_warn/pr_debug format specifiers don't match their argument types: %d for __u32/Elf64_Word, %u for signed ints, %zd for size_t, %ld for unsigned long, and %x/%lx/%llx applied to signed values. Match each specifier to its argument's type where a correctly-signed specifier exists (%d<->%u, %ld->%lu, %zd->%zu). For hex conversions, which have no signed form, cast the argument instead (%x->(unsigned), %lx->(unsigned long), %llx->(unsigned long long)). No functional change. Note, the fdinfo map_flags sscanf used %i into a __u32 *, which warns. The kernel prints map_flags as hex ("map_flags:\t%#x\n" in bpf_map_show_fdinfo(), unchanged since the field was added to fdinfo), so switch the conversion to %x: it parses the 0x-prefixed value and expects unsigned int *, matching the destination, so the warning is gone with no cast. Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/r/20260624204946.2901178-1-andrii@kernel.org Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/btf.c | 10 ++--- tools/lib/bpf/btf_dump.c | 4 +- tools/lib/bpf/btf_relocate.c | 8 ++-- tools/lib/bpf/elf.c | 2 +- tools/lib/bpf/gen_loader.c | 16 ++++---- tools/lib/bpf/libbpf.c | 78 ++++++++++++++++++------------------ tools/lib/bpf/nlattr.c | 2 +- tools/lib/bpf/relo_core.c | 22 +++++----- tools/lib/bpf/usdt.c | 33 ++++++++------- 9 files changed, 89 insertions(+), 86 deletions(-) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index 823bce895178..bf6a68118405 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -589,7 +589,7 @@ static int btf_parse_type_sec(struct btf *btf) if (type_size < 0) return type_size; if (next_type + type_size > end_type) { - pr_warn("BTF type [%d] is malformed\n", btf->start_id + btf->nr_types); + pr_warn("BTF type [%u] is malformed\n", btf->start_id + btf->nr_types); return -EINVAL; } @@ -1424,7 +1424,7 @@ static int btf_find_elf_sections(Elf *elf, const char *path, struct btf_elf_secs continue; if (sh.sh_type != SHT_PROGBITS) { - pr_warn("unexpected section type (%d) of section(%d, %s) from %s\n", + pr_warn("unexpected section type (%u) of section(%d, %s) from %s\n", sh.sh_type, idx, name, path); goto err; } @@ -4854,7 +4854,7 @@ static bool btf_dedup_identical_types(struct btf_dedup *d, __u32 id1, __u32 id2, continue; if (!btf_dedup_identical_types(d, m1->type, m2->type, depth - 1)) { if (t1->name_off) { - pr_debug("%s '%s' size=%d vlen=%d id1[%u] id2[%u] shallow-equal but not identical for field#%d '%s'\n", + pr_debug("%s '%s' size=%u vlen=%u id1[%u] id2[%u] shallow-equal but not identical for field#%d '%s'\n", k1 == BTF_KIND_STRUCT ? "STRUCT" : "UNION", btf__name_by_offset(d->btf, t1->name_off), t1->size, btf_vlen(t1), id1, id2, i, @@ -5104,7 +5104,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, eq = btf_dedup_is_equiv(d, cand_m->type, canon_m->type); if (eq <= 0) { if (cand_type->name_off) { - pr_debug("%s '%s' size=%d vlen=%d cand_id[%u] canon_id[%u] shallow-equal but not equiv for field#%d '%s': %d\n", + pr_debug("%s '%s' size=%u vlen=%u cand_id[%u] canon_id[%u] shallow-equal but not equiv for field#%d '%s': %d\n", cand_kind == BTF_KIND_STRUCT ? "STRUCT" : "UNION", btf__name_by_offset(d->btf, cand_type->name_off), cand_type->size, vlen, cand_id, canon_id, i, @@ -6069,7 +6069,7 @@ static int btf_add_distilled_types(struct btf_distill *dist) err = btf_add_type(&dist->pipe, t); break; default: - pr_warn("unexpected kind when adding base type '%s'[%u] of kind [%u] to distilled base BTF.\n", + pr_warn("unexpected kind when adding base type '%s'[%d] of kind [%d] to distilled base BTF.\n", name, i, kind); return -EINVAL; diff --git a/tools/lib/bpf/btf_dump.c b/tools/lib/bpf/btf_dump.c index cc1ba65bb6c5..123c448f20c7 100644 --- a/tools/lib/bpf/btf_dump.c +++ b/tools/lib/bpf/btf_dump.c @@ -1776,7 +1776,7 @@ static int btf_dump_get_bitfield_value(struct btf_dump *d, /* Maximum supported bitfield size is 64 bits */ if (t->size > 8) { - pr_warn("unexpected bitfield size %d\n", t->size); + pr_warn("unexpected bitfield size %u\n", t->size); return -EINVAL; } @@ -2251,7 +2251,7 @@ static int btf_dump_get_enum_value(struct btf_dump *d, *value = is_signed ? *(__s8 *)data : *(__u8 *)data; return 0; default: - pr_warn("unexpected size %d for enum, id:[%u]\n", t->size, id); + pr_warn("unexpected size %u for enum, id:[%u]\n", t->size, id); return -EINVAL; } } diff --git a/tools/lib/bpf/btf_relocate.c b/tools/lib/bpf/btf_relocate.c index 53d1f3541bce..df5fa4bd87d6 100644 --- a/tools/lib/bpf/btf_relocate.c +++ b/tools/lib/bpf/btf_relocate.c @@ -280,7 +280,7 @@ static int btf_relocate_map_distilled_base(struct btf_relocate *r) cmp_btf_name_size(&base_info, dist_info) == 0; dist_info++) { if (!dist_info->id || dist_info->id >= r->nr_dist_base_types) { - pr_warn("base BTF id [%d] maps to invalid distilled base BTF id [%d]\n", + pr_warn("base BTF id [%u] maps to invalid distilled base BTF id [%u]\n", id, dist_info->id); err = -EINVAL; goto done; @@ -368,7 +368,7 @@ static int btf_relocate_map_distilled_base(struct btf_relocate *r) continue; dist_t = btf_type_by_id(r->dist_base_btf, id); name = btf__name_by_offset(r->dist_base_btf, dist_t->name_off); - pr_warn("distilled base BTF type '%s' [%d] is not mapped to base BTF id\n", + pr_warn("distilled base BTF type '%s' [%u] is not mapped to base BTF id\n", name, id); err = -EINVAL; break; @@ -397,11 +397,11 @@ static int btf_relocate_validate_distilled_base(struct btf_relocate *r) case BTF_KIND_FWD: if (t->name_off) break; - pr_warn("type [%d], kind [%d] is invalid for distilled base BTF; it is anonymous\n", + pr_warn("type [%u], kind [%d] is invalid for distilled base BTF; it is anonymous\n", i, kind); return -EINVAL; default: - pr_warn("type [%d] in distilled based BTF has unexpected kind [%d]\n", + pr_warn("type [%u] in distilled based BTF has unexpected kind [%d]\n", i, kind); return -EINVAL; } diff --git a/tools/lib/bpf/elf.c b/tools/lib/bpf/elf.c index 295dbda24580..fe136d025967 100644 --- a/tools/lib/bpf/elf.c +++ b/tools/lib/bpf/elf.c @@ -354,7 +354,7 @@ long elf_find_func_offset(Elf *elf, const char *binary_path, const char *name) if (ret > 0) { pr_debug("elf: symbol address match for '%s' in '%s': 0x%lx\n", name, binary_path, - ret); + (unsigned long)ret); } else { if (ret == 0) { pr_warn("elf: '%s' is 0 in symtab for '%s': %s\n", name, binary_path, diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index d79695f01c87..c7f2d2ac7bb3 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -384,7 +384,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) int i; if (nr_progs < gen->nr_progs || nr_maps != gen->nr_maps) { - pr_warn("nr_progs %d/%d nr_maps %d/%d mismatch\n", + pr_warn("nr_progs %d/%u nr_maps %d/%u mismatch\n", nr_progs, gen->nr_progs, nr_maps, gen->nr_maps); gen->error = -EFAULT; return gen->error; @@ -488,7 +488,7 @@ void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data, attr.btf_size = tgt_endian(btf_raw_size); btf_load_attr = add_data(gen, &attr, attr_size); - pr_debug("gen: load_btf: off %d size %d, attr: off %d size %d\n", + pr_debug("gen: load_btf: off %d size %u, attr: off %d size %d\n", btf_data, btf_raw_size, btf_load_attr, attr_size); /* populate union bpf_attr with user provided log details */ @@ -534,7 +534,7 @@ void bpf_gen__map_create(struct bpf_gen *gen, attr.btf_value_type_id = tgt_endian(map_attr->btf_value_type_id); map_create_attr = add_data(gen, &attr, attr_size); - pr_debug("gen: map_create: %s idx %d type %d value_type_id %d, attr: off %d size %d\n", + pr_debug("gen: map_create: %s idx %d type %u value_type_id %u, attr: off %d size %d\n", map_name, map_idx, map_type, map_attr->btf_value_type_id, map_create_attr, attr_size); @@ -1082,7 +1082,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, license_off = add_data(gen, license, strlen(license) + 1); /* add insns to blob of bytes */ insns_off = add_data(gen, insns, insn_cnt * sizeof(struct bpf_insn)); - pr_debug("gen: prog_load: prog_idx %d type %d insn off %d insns_cnt %zd license off %d\n", + pr_debug("gen: prog_load: prog_idx %d type %u insn off %d insns_cnt %zu license off %d\n", prog_idx, prog_type, insns_off, insn_cnt, license_off); /* convert blob insns to target endianness */ @@ -1105,21 +1105,21 @@ void bpf_gen__prog_load(struct bpf_gen *gen, attr.func_info_rec_size = tgt_endian(load_attr->func_info_rec_size); attr.func_info_cnt = tgt_endian(load_attr->func_info_cnt); func_info = add_data(gen, load_attr->func_info, func_info_tot_sz); - pr_debug("gen: prog_load: func_info: off %d cnt %d rec size %d\n", + pr_debug("gen: prog_load: func_info: off %d cnt %u rec size %u\n", func_info, load_attr->func_info_cnt, load_attr->func_info_rec_size); attr.line_info_rec_size = tgt_endian(load_attr->line_info_rec_size); attr.line_info_cnt = tgt_endian(load_attr->line_info_cnt); line_info = add_data(gen, load_attr->line_info, line_info_tot_sz); - pr_debug("gen: prog_load: line_info: off %d cnt %d rec size %d\n", + pr_debug("gen: prog_load: line_info: off %d cnt %u rec size %u\n", line_info, load_attr->line_info_cnt, load_attr->line_info_rec_size); attr.core_relo_rec_size = tgt_endian((__u32)sizeof(struct bpf_core_relo)); attr.core_relo_cnt = tgt_endian(gen->core_relo_cnt); core_relos = add_data(gen, gen->core_relos, core_relo_tot_sz); - pr_debug("gen: prog_load: core_relos: off %d cnt %d rec size %zd\n", + pr_debug("gen: prog_load: core_relos: off %d cnt %d rec size %zu\n", core_relos, gen->core_relo_cnt, sizeof(struct bpf_core_relo)); @@ -1234,7 +1234,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, } map_update_attr = add_data(gen, &attr, attr_size); - pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n", + pr_debug("gen: map_update_elem: idx %d, value: off %d size %u, attr: off %d size %d\n", map_idx, value, value_size, map_update_attr, attr_size); move_blob2blob(gen, attr_field(map_update_attr, map_fd), 4, blob_fd_array_off(gen, map_idx)); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 1368752aa13c..7162146280a8 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -1486,7 +1486,7 @@ static int init_struct_ops_maps(struct bpf_object *obj, const char *sec_name, type->size); st_ops->type_id = type_id; - pr_debug("struct_ops init: struct %s(type_id=%u) %s found at offset %u\n", + pr_debug("struct_ops init: struct %s(type_id=%d) %s found at offset %u\n", tname, type_id, var_name, vsi->offset); } @@ -2626,7 +2626,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf, t = btf__type_by_id(btf, m->type); if (!t) { - pr_warn("map '%s': key type [%d] not found.\n", + pr_warn("map '%s': key type [%u] not found.\n", map_name, m->type); return -EINVAL; } @@ -2666,7 +2666,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf, t = btf__type_by_id(btf, m->type); if (!t) { - pr_warn("map '%s': value type [%d] not found.\n", + pr_warn("map '%s': value type [%u] not found.\n", map_name, m->type); return -EINVAL; } @@ -2720,7 +2720,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf, map_def->value_size = 4; t = btf__type_by_id(btf, m->type); if (!t) { - pr_warn("map '%s': %s type [%d] not found.\n", + pr_warn("map '%s': %s type [%u] not found.\n", map_name, desc, m->type); return -EINVAL; } @@ -3476,7 +3476,7 @@ static int btf_fixup_datasec(struct bpf_object *obj, struct btf *btf, var_name = btf__name_by_offset(btf, t_var->name_off); if (!var_name) { - pr_debug("sec '%s': failed to find name of DATASEC's member #%d\n", + pr_debug("sec '%s': failed to find name of DATASEC's member #%u\n", sec_name, i); return -ENOENT; } @@ -3971,7 +3971,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj) if (!data) return -LIBBPF_ERRNO__FORMAT; - pr_debug("elf: section(%d) %s, size %ld, link %d, flags %lx, type=%d\n", + pr_debug("elf: section(%d) %s, size %lu, link %d, flags %lx, type=%d\n", idx, name, (unsigned long)data->d_size, (int)sh->sh_link, (unsigned long)sh->sh_flags, (int)sh->sh_type); @@ -4494,7 +4494,7 @@ static int bpf_object__collect_externs(struct bpf_object *obj) ext->kcfg.data_off = roundup(off, ext->kcfg.align); off = ext->kcfg.data_off + ext->kcfg.sz; - pr_debug("extern (kcfg) #%d: symbol %d, off %u, name %s\n", + pr_debug("extern (kcfg) #%d: symbol %d, off %d, name %s\n", i, ext->sym_idx, ext->kcfg.data_off, ext->name); } sec->size = off; @@ -4626,7 +4626,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, struct bpf_map *map; if (!is_call_insn(insn) && !is_ldimm64_insn(insn)) { - pr_warn("prog '%s': invalid relo against '%s' for insns[%d].code 0x%x\n", + pr_warn("prog '%s': invalid relo against '%s' for insns[%u].code 0x%x\n", prog->name, sym_name, insn_idx, insn->code); return -LIBBPF_ERRNO__RELOC; } @@ -4749,7 +4749,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, map->sec_idx != sym->st_shndx || map->sec_offset != sym->st_value) continue; - pr_debug("prog '%s': found map %zd (%s, sec %d, off %zu) for insn #%u\n", + pr_debug("prog '%s': found map %zu (%s, sec %d, off %zu) for insn #%u\n", prog->name, map_idx, map->name, map->sec_idx, map->sec_offset, insn_idx); break; @@ -4776,7 +4776,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, map = &obj->maps[map_idx]; if (map->libbpf_type != type || map->sec_idx != sym->st_shndx) continue; - pr_debug("prog '%s': found data map %zd (%s, sec %d, off %zu) for insn %u\n", + pr_debug("prog '%s': found data map %zu (%s, sec %d, off %zu) for insn %u\n", prog->name, map_idx, map->name, map->sec_idx, map->sec_offset, insn_idx); break; @@ -4985,7 +4985,7 @@ static int bpf_get_map_info_from_fdinfo(int fd, struct bpf_map_info *info) info->value_size = val; else if (sscanf(buff, "max_entries:\t%u", &val) == 1) info->max_entries = val; - else if (sscanf(buff, "map_flags:\t%i", &val) == 1) + else if (sscanf(buff, "map_flags:\t%x", &val) == 1) info->map_flags = val; } @@ -5521,11 +5521,11 @@ static int init_map_in_map_slots(struct bpf_object *obj, struct bpf_map *map) } if (err) { err = -errno; - pr_warn("map '%s': failed to initialize slot [%d] to map '%s' fd=%d: %s\n", + pr_warn("map '%s': failed to initialize slot [%u] to map '%s' fd=%d: %s\n", map->name, i, targ_map->name, fd, errstr(err)); return err; } - pr_debug("map '%s': slot [%d] set to map '%s' fd=%d\n", + pr_debug("map '%s': slot [%u] set to map '%s' fd=%d\n", map->name, i, targ_map->name, fd); } @@ -5554,11 +5554,11 @@ static int init_prog_array_slots(struct bpf_object *obj, struct bpf_map *map) err = bpf_map_update_elem(map->fd, &i, &fd, 0); if (err) { err = -errno; - pr_warn("map '%s': failed to initialize slot [%d] to prog '%s' fd=%d: %s\n", + pr_warn("map '%s': failed to initialize slot [%u] to prog '%s' fd=%d: %s\n", map->name, i, targ_prog->name, fd, errstr(err)); return err; } - pr_debug("map '%s': slot [%d] set to prog '%s' fd=%d\n", + pr_debug("map '%s': slot [%u] set to prog '%s' fd=%d\n", map->name, i, targ_prog->name, fd); } @@ -5788,7 +5788,7 @@ int bpf_core_add_cands(struct bpf_core_cand *local_cand, if (strncmp(local_name, targ_name, local_essent_len) != 0) continue; - pr_debug("CO-RE relocating [%d] %s %s: found target candidate [%d] %s %s in [%s]\n", + pr_debug("CO-RE relocating [%u] %s %s: found target candidate [%d] %s %s in [%s]\n", local_cand->id, btf_kind_str(local_t), local_name, i, btf_kind_str(t), targ_name, targ_btf_name); @@ -5848,7 +5848,7 @@ static int load_module_btfs(struct bpf_object *obj) if (errno == ENOENT) continue; /* expected race: BTF was unloaded */ err = -errno; - pr_warn("failed to get BTF object #%d FD: %s\n", id, errstr(err)); + pr_warn("failed to get BTF object #%u FD: %s\n", id, errstr(err)); return err; } @@ -5861,7 +5861,7 @@ static int load_module_btfs(struct bpf_object *obj) err = bpf_btf_get_info_by_fd(fd, &info, &len); if (err) { err = -errno; - pr_warn("failed to get BTF object #%d info: %s\n", id, errstr(err)); + pr_warn("failed to get BTF object #%u info: %s\n", id, errstr(err)); break; } @@ -5874,7 +5874,7 @@ static int load_module_btfs(struct bpf_object *obj) btf = btf_get_from_fd(fd, obj->btf_vmlinux); err = libbpf_get_error(btf); if (err) { - pr_warn("failed to load module [%s]'s BTF object #%d: %s\n", + pr_warn("failed to load module [%s]'s BTF object #%u: %s\n", name, id, errstr(err)); break; } @@ -6067,7 +6067,7 @@ static int bpf_core_resolve_relo(struct bpf_program *prog, !hashmap__find(cand_cache, local_id, &cands)) { cands = bpf_core_find_cands(prog->obj, local_btf, local_id); if (IS_ERR(cands)) { - pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s %s: %ld\n", + pr_warn("prog '%s': relo #%d: target candidate search failed for [%u] %s %s: %ld\n", prog_name, relo_idx, local_id, btf_kind_str(local_type), local_name, PTR_ERR(cands)); return PTR_ERR(cands); @@ -6127,7 +6127,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path) goto out; } - pr_debug("sec '%s': found %d CO-RE relocations\n", sec_name, sec->num_info); + pr_debug("sec '%s': found %u CO-RE relocations\n", sec_name, sec->num_info); for_each_btf_ext_rec(seg, sec, i, rec) { if (rec->insn_off % BPF_INSN_SZ) @@ -6181,7 +6181,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path) err = bpf_core_patch_insn(prog->name, insn, insn_idx, rec, i, &targ_res); if (err) { - pr_warn("prog '%s': relo #%d: failed to patch insn #%u: %s\n", + pr_warn("prog '%s': relo #%d: failed to patch insn #%d: %s\n", prog->name, i, insn_idx, errstr(err)); goto out; } @@ -6346,7 +6346,7 @@ static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struc goto err_close; } if (sym_off + jt_size > obj->jumptables_data_sz) { - pr_warn("map '.jumptables': jumptables_data size is %zd, trying to access %d\n", + pr_warn("map '.jumptables': jumptables_data size is %zu, trying to access %u\n", obj->jumptables_data_sz, sym_off + jt_size); err = -EINVAL; goto err_close; @@ -6381,7 +6381,7 @@ static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struc */ if (insn_off > UINT32_MAX) { pr_warn("map '.jumptables': invalid jump table value 0x%llx at offset %u\n", - (long long)jt[i], sym_off + i * jt_entry_size); + (unsigned long long)jt[i], sym_off + i * jt_entry_size); err = -EINVAL; goto err_close; } @@ -6517,7 +6517,7 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog) } break; default: - pr_warn("prog '%s': relo #%d: bad relo type %d\n", + pr_warn("prog '%s': relo #%d: bad relo type %u\n", prog->name, i, relo->type); return -EINVAL; } @@ -6797,7 +6797,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, */ continue; if (relo && relo->type != RELO_CALL && relo->type != RELO_SUBPROG_ADDR) { - pr_warn("prog '%s': unexpected relo for insn #%zu, type %d\n", + pr_warn("prog '%s': unexpected relo for insn #%zu, type %u\n", prog->name, insn_idx, relo->type); return -LIBBPF_ERRNO__RELOC; } @@ -7587,7 +7587,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, } name = elf_sym_str(obj, sym->st_name) ?: ""; - pr_debug(".maps relo #%d: for %zd value %zd rel->r_offset %zu name %d ('%s')\n", + pr_debug(".maps relo #%d: for %zd value %zu rel->r_offset %zu name %u ('%s')\n", i, (ssize_t)(rel->r_info >> 32), (size_t)sym->st_value, (size_t)rel->r_offset, sym->st_name, name); @@ -7678,7 +7678,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj, } map->init_slots[moff] = is_map_in_map ? (void *)targ_map : (void *)targ_prog; - pr_debug(".maps relo #%d: map '%s' slot [%d] points to %s '%s'\n", + pr_debug(".maps relo #%d: map '%s' slot [%u] points to %s '%s'\n", i, map->name, moff, type, name); } @@ -8738,7 +8738,7 @@ static int bpf_object__resolve_ksym_var_btf_id(struct bpf_object *obj, local_name = btf__name_by_offset(obj->btf, local_type->name_off); targ_name = btf__name_by_offset(btf, targ_type->name_off); - pr_warn("extern (var ksym) '%s': incompatible types, expected [%d] %s %s, but kernel has [%d] %s %s\n", + pr_warn("extern (var ksym) '%s': incompatible types, expected [%u] %s %s, but kernel has [%u] %s %s\n", ext->name, local_type_id, btf_kind_str(local_type), local_name, targ_type_id, btf_kind_str(targ_type), targ_name); @@ -8915,7 +8915,7 @@ static int bpf_object__resolve_externs(struct bpf_object *obj, if (err) return err; pr_debug("extern (kcfg) '%s': set to 0x%llx\n", - ext->name, (long long)value); + ext->name, (unsigned long long)value); } else { pr_warn("extern '%s': unrecognized extern kind\n", ext->name); return -EINVAL; @@ -10494,7 +10494,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj, moff = rel->r_offset - map->sec_offset; shdr_idx = sym->st_shndx; st_ops = map->st_ops; - pr_debug("struct_ops reloc %s: for %lld value %lld shdr_idx %u rel->r_offset %zu map->sec_offset %zu name %d (\'%s\')\n", + pr_debug("struct_ops reloc %s: for %lld value %lld shdr_idx %u rel->r_offset %zu map->sec_offset %zu name %u (\'%s\')\n", map->name, (long long)(rel->r_info >> 32), (long long)sym->st_value, @@ -10643,7 +10643,7 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd, int t memset(&info, 0, info_len); err = bpf_prog_get_info_by_fd(attach_prog_fd, &info, &info_len); if (err) { - pr_warn("failed bpf_prog_get_info_by_fd for FD %d: %s\n", + pr_warn("failed bpf_prog_get_info_by_fd for FD %u: %s\n", attach_prog_fd, errstr(err)); return err; } @@ -10656,7 +10656,7 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd, int t btf = btf_load_from_kernel(info.btf_id, NULL, token_fd); err = libbpf_get_error(btf); if (err) { - pr_warn("Failed to get BTF %d of the program: %s\n", info.btf_id, errstr(err)); + pr_warn("Failed to get BTF %u of the program: %s\n", info.btf_id, errstr(err)); goto out; } err = btf__find_by_name_kind(btf, name, BTF_KIND_FUNC); @@ -10738,7 +10738,7 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attac } err = libbpf_find_prog_btf_id(attach_name, attach_prog_fd, prog->obj->token_fd); if (err < 0) { - pr_warn("prog '%s': failed to find BPF program (FD %d) BTF ID for '%s': %s\n", + pr_warn("prog '%s': failed to find BPF program (FD %u) BTF ID for '%s': %s\n", prog->name, attach_prog_fd, attach_name, errstr(err)); return err; } @@ -11233,7 +11233,7 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz, } if (value_sz != num_cpu * elem_sz) { - pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n", + pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zu\n", map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz); return -EINVAL; } @@ -11774,7 +11774,7 @@ static void gen_probe_legacy_event_name(char *buf, size_t buf_sz, static int index = 0; int i; - snprintf(buf, buf_sz, "libbpf_%u_%d_%s_0x%zx", getpid(), + snprintf(buf, buf_sz, "libbpf_%d_%d_%s_0x%zx", getpid(), __sync_fetch_and_add(&index, 1), name, offset); /* sanitize name in the probe name */ @@ -12924,8 +12924,8 @@ static long elf_find_func_offset_from_archive(const char *archive_path, const ch ret = elf_find_func_offset(elf, file_name, func_name); if (ret > 0) { pr_debug("elf: symbol address match for %s of %s in %s: 0x%x + 0x%lx = 0x%lx\n", - func_name, file_name, archive_path, entry.data_offset, ret, - ret + entry.data_offset); + func_name, file_name, archive_path, entry.data_offset, (unsigned long)ret, + (unsigned long)(ret + entry.data_offset)); ret += entry.data_offset; } elf_end(elf); @@ -14570,7 +14570,7 @@ perf_buffer__process_record(struct perf_event_header *e, void *ctx) break; } default: - pr_warn("unknown perf sample type %d\n", e->type); + pr_warn("unknown perf sample type %u\n", e->type); return LIBBPF_PERF_EVENT_ERROR; } return LIBBPF_PERF_EVENT_CONT; diff --git a/tools/lib/bpf/nlattr.c b/tools/lib/bpf/nlattr.c index 06663f9ea581..007fe17d17b4 100644 --- a/tools/lib/bpf/nlattr.c +++ b/tools/lib/bpf/nlattr.c @@ -123,7 +123,7 @@ int libbpf_nla_parse(struct nlattr *tb[], int maxtype, struct nlattr *head, if (tb[type]) { pr_warn("Attribute of type %#x found multiple times in message, " - "previous attribute is being ignored.\n", type); + "previous attribute is being ignored.\n", (unsigned)type); } tb[type] = nla; diff --git a/tools/lib/bpf/relo_core.c b/tools/lib/bpf/relo_core.c index 6ae3f2a15ad0..8ad2715721cf 100644 --- a/tools/lib/bpf/relo_core.c +++ b/tools/lib/bpf/relo_core.c @@ -216,7 +216,7 @@ int __bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id, goto recur; } default: - pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n", + pr_warn("unexpected kind %s relocated, local [%u], target [%u]\n", btf_kind_str(local_type), local_id, targ_id); return 0; } @@ -384,7 +384,7 @@ int bpf_core_parse_spec(const char *prog_name, const struct btf *btf, return sz; spec->bit_offset += access_idx * sz * 8; } else { - pr_warn("prog '%s': relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %s\n", + pr_warn("prog '%s': relo for [%u] %s (at idx %d) captures type [%u] of unexpected kind %s\n", prog_name, relo->type_id, spec_str, i, id, btf_kind_str(t)); return -EINVAL; } @@ -725,7 +725,7 @@ static int bpf_core_calc_field_relo(const char *prog_name, return -EINVAL; *val = sz; } else { - pr_warn("prog '%s': relo %d at insn #%d can't be applied to array access\n", + pr_warn("prog '%s': relo %u at insn #%u can't be applied to array access\n", prog_name, relo->kind, relo->insn_off / 8); return -EINVAL; } @@ -747,7 +747,7 @@ static int bpf_core_calc_field_relo(const char *prog_name, while (bit_off + bit_sz - byte_off * 8 > byte_sz * 8) { if (byte_sz >= 8) { /* bitfield can't be read with 64-bit read */ - pr_warn("prog '%s': relo %d at insn #%d can't be satisfied for bitfield\n", + pr_warn("prog '%s': relo %u at insn #%u can't be satisfied for bitfield\n", prog_name, relo->kind, relo->insn_off / 8); return -E2BIG; } @@ -971,7 +971,7 @@ static int bpf_core_calc_relo(const char *prog_name, err = 0; } else if (err == -EOPNOTSUPP) { /* EOPNOTSUPP means unknown/unsupported relocation */ - pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%d) at insn #%d\n", + pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%u) at insn #%u\n", prog_name, relo_idx, core_relo_kind_str(relo->kind), relo->kind, relo->insn_off / 8); } @@ -1067,7 +1067,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn, if (BPF_SRC(insn->code) != BPF_K) return -EINVAL; if (res->validate && insn->imm != orig_val) { - pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %llu -> %llu\n", + pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %d, exp %llu -> %llu\n", prog_name, relo_idx, insn_idx, insn->imm, (unsigned long long)orig_val, (unsigned long long)new_val); @@ -1083,7 +1083,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn, case BPF_ST: case BPF_STX: if (res->validate && insn->off != orig_val) { - pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %llu -> %llu\n", + pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %d, exp %llu -> %llu\n", prog_name, relo_idx, insn_idx, insn->off, (unsigned long long)orig_val, (unsigned long long)new_val); return -EINVAL; @@ -1159,7 +1159,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn, default: pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n", prog_name, relo_idx, insn_idx, insn->code, - insn->src_reg, insn->dst_reg, insn->off, insn->imm); + (unsigned)insn->src_reg, (unsigned)insn->dst_reg, (unsigned)insn->off, (unsigned)insn->imm); return -EINVAL; } @@ -1323,7 +1323,7 @@ int bpf_core_calc_relo_insn(const char *prog_name, const char *spec_str; spec_str = btf__name_by_offset(local_btf, relo->access_str_off); - pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n", + pr_warn("prog '%s': relo #%d: parsing [%u] %s %s + %s failed: %d\n", prog_name, relo_idx, local_id, btf_kind_str(local_type), str_is_empty(local_name) ? "" : local_name, spec_str ?: "", err); @@ -1346,7 +1346,7 @@ int bpf_core_calc_relo_insn(const char *prog_name, /* libbpf doesn't support candidate search for anonymous types */ if (str_is_empty(local_name)) { - pr_warn("prog '%s': relo #%d: <%s> (%d) relocation doesn't support anonymous types\n", + pr_warn("prog '%s': relo #%d: <%s> (%u) relocation doesn't support anonymous types\n", prog_name, relo_idx, core_relo_kind_str(relo->kind), relo->kind); return -EOPNOTSUPP; } @@ -1697,7 +1697,7 @@ int __bpf_core_types_match(const struct btf *local_btf, __u32 local_id, const st goto recur; } default: - pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n", + pr_warn("unexpected kind %s relocated, local [%u], target [%u]\n", btf_kind_str(local_t), local_id, targ_id); return 0; } diff --git a/tools/lib/bpf/usdt.c b/tools/lib/bpf/usdt.c index 57fb82bb81b5..db9432adb967 100644 --- a/tools/lib/bpf/usdt.c +++ b/tools/lib/bpf/usdt.c @@ -327,7 +327,7 @@ static int sanity_check_usdt_elf(Elf *elf, const char *path) int endianness; if (elf_kind(elf) != ELF_K_ELF) { - pr_warn("usdt: unrecognized ELF kind %d for '%s'\n", elf_kind(elf), path); + pr_warn("usdt: unrecognized ELF kind %u for '%s'\n", elf_kind(elf), path); return -EBADF; } @@ -438,8 +438,9 @@ static int parse_elf_segs(Elf *elf, const char *path, struct elf_seg **segs, siz } pr_debug("usdt: discovered PHDR #%d in '%s': vaddr 0x%lx memsz 0x%lx offset 0x%lx type 0x%lx flags 0x%lx\n", - i, path, (long)phdr.p_vaddr, (long)phdr.p_memsz, (long)phdr.p_offset, - (long)phdr.p_type, (long)phdr.p_flags); + i, path, + (unsigned long)phdr.p_vaddr, (unsigned long)phdr.p_memsz, (unsigned long)phdr.p_offset, + (unsigned long)phdr.p_type, (unsigned long)phdr.p_flags); if (phdr.p_type != PT_LOAD) continue; @@ -719,14 +720,14 @@ static int collect_usdt_targets(struct usdt_manager *man, struct elf_fd *elf_fd, if (!seg) { err = -ESRCH; pr_warn("usdt: failed to find ELF program segment for '%s:%s' in '%s' at IP 0x%lx\n", - usdt_provider, usdt_name, path, usdt_abs_ip); + usdt_provider, usdt_name, path, (unsigned long)usdt_abs_ip); goto err_out; } if (!seg->is_exec) { err = -ESRCH; pr_warn("usdt: matched ELF binary '%s' segment [0x%lx, 0x%lx) for '%s:%s' at IP 0x%lx is not executable\n", - path, seg->start, seg->end, usdt_provider, usdt_name, - usdt_abs_ip); + path, (unsigned long)seg->start, (unsigned long)seg->end, usdt_provider, usdt_name, + (unsigned long)usdt_abs_ip); goto err_out; } /* translate from virtual address to file offset */ @@ -766,7 +767,7 @@ static int collect_usdt_targets(struct usdt_manager *man, struct elf_fd *elf_fd, if (!seg) { err = -ESRCH; pr_warn("usdt: failed to find shared lib memory segment for '%s:%s' in '%s' at relative IP 0x%lx\n", - usdt_provider, usdt_name, path, usdt_rel_ip); + usdt_provider, usdt_name, path, (unsigned long)usdt_rel_ip); goto err_out; } @@ -775,8 +776,10 @@ static int collect_usdt_targets(struct usdt_manager *man, struct elf_fd *elf_fd, pr_debug("usdt: probe for '%s:%s' in %s '%s': addr 0x%lx base 0x%lx (resolved abs_ip 0x%lx rel_ip 0x%lx) args '%s' in segment [0x%lx, 0x%lx) at offset 0x%lx\n", usdt_provider, usdt_name, ehdr.e_type == ET_EXEC ? "exec" : "lib ", path, - note.loc_addr, note.base_addr, usdt_abs_ip, usdt_rel_ip, note.args, - seg ? seg->start : 0, seg ? seg->end : 0, seg ? seg->offset : 0); + (unsigned long)note.loc_addr, (unsigned long)note.base_addr, + (unsigned long)usdt_abs_ip, (unsigned long)usdt_rel_ip, note.args, + (unsigned long)(seg ? seg->start : 0), (unsigned long)(seg ? seg->end : 0), + (unsigned long)(seg ? seg->offset : 0)); /* Adjust semaphore address to be a file offset */ if (note.sema_addr) { @@ -791,14 +794,14 @@ static int collect_usdt_targets(struct usdt_manager *man, struct elf_fd *elf_fd, if (!seg) { err = -ESRCH; pr_warn("usdt: failed to find ELF loadable segment with semaphore of '%s:%s' in '%s' at 0x%lx\n", - usdt_provider, usdt_name, path, note.sema_addr); + usdt_provider, usdt_name, path, (unsigned long)note.sema_addr); goto err_out; } if (seg->is_exec) { err = -ESRCH; pr_warn("usdt: matched ELF binary '%s' segment [0x%lx, 0x%lx] for semaphore of '%s:%s' at 0x%lx is executable\n", - path, seg->start, seg->end, usdt_provider, usdt_name, - note.sema_addr); + path, (unsigned long)seg->start, (unsigned long)seg->end, usdt_provider, usdt_name, + (unsigned long)note.sema_addr); goto err_out; } @@ -806,8 +809,8 @@ static int collect_usdt_targets(struct usdt_manager *man, struct elf_fd *elf_fd, pr_debug("usdt: sema for '%s:%s' in %s '%s': addr 0x%lx base 0x%lx (resolved 0x%lx) in segment [0x%lx, 0x%lx] at offset 0x%lx\n", usdt_provider, usdt_name, ehdr.e_type == ET_EXEC ? "exec" : "lib ", - path, note.sema_addr, note.base_addr, usdt_sema_off, - seg->start, seg->end, seg->offset); + path, (unsigned long)note.sema_addr, (unsigned long)note.base_addr, (unsigned long)usdt_sema_off, + (unsigned long)seg->start, (unsigned long)seg->end, (unsigned long)seg->offset); } /* Record adjusted addresses and offsets and parse USDT spec */ @@ -1117,7 +1120,7 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct spec_id, usdt_provider, usdt_name, path); } else { pr_warn("usdt: failed to map IP 0x%lx to spec #%d for '%s:%s' in '%s': %s\n", - target->abs_ip, spec_id, usdt_provider, usdt_name, + (unsigned long)target->abs_ip, spec_id, usdt_provider, usdt_name, path, errstr(err)); } goto err_out; From 1dbf26eca0a4fba9bdf4fe6ab72c2678e2e5851e Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Tue, 23 Jun 2026 17:55:43 -0700 Subject: [PATCH 006/373] tools/bpf: Sync btf_ids.h to tools Sync tools/include/linux/btf_ids.h with include/linux/btf_ids.h so tools-side code can use BTF_ID_FLAGS(), BTF_SET8_START(), and BTF_KFUNCS_START(). Keep the tools copy's existing compiler header dependency: tools/include/linux/compiler.h already provides __maybe_unused and tools/include/linux/compiler_attributes.h does not exist. Reviewed-by: Emil Tsalapatis Acked-by: Eduard Zingerman Acked-by: Jiri Olsa Signed-off-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260624005546.1818483-2-ihor.solodrai@linux.dev Signed-off-by: Alexei Starovoitov --- tools/include/linux/btf_ids.h | 78 ++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 5 deletions(-) diff --git a/tools/include/linux/btf_ids.h b/tools/include/linux/btf_ids.h index 72ea363d434d..4fe5c5f1558c 100644 --- a/tools/include/linux/btf_ids.h +++ b/tools/include/linux/btf_ids.h @@ -10,6 +10,9 @@ struct btf_id_set { u32 ids[]; }; +/* This flag implies BTF_SET8 holds kfunc(s) */ +#define BTF_SET8_KFUNCS (1 << 0) + struct btf_id_set8 { u32 cnt; u32 flags; @@ -22,6 +25,7 @@ struct btf_id_set8 { #ifdef CONFIG_DEBUG_INFO_BTF #include /* for __PASTE */ +#include /* * Following macros help to define lists of BTF IDs placed @@ -35,7 +39,7 @@ struct btf_id_set8 { #define BTF_IDS_SECTION ".BTF_ids" -#define ____BTF_ID(symbol) \ +#define ____BTF_ID(symbol, word) \ asm( \ ".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \ ".local " #symbol " ; \n" \ @@ -43,10 +47,11 @@ asm( \ ".size " #symbol ", 4; \n" \ #symbol ": \n" \ ".zero 4 \n" \ +word \ ".popsection; \n"); -#define __BTF_ID(symbol) \ - ____BTF_ID(symbol) +#define __BTF_ID(symbol, word) \ + ____BTF_ID(symbol, word) #define __ID(prefix) \ __PASTE(__PASTE(prefix, __COUNTER__), __LINE__) @@ -56,7 +61,14 @@ asm( \ * to 4 zero bytes. */ #define BTF_ID(prefix, name) \ - __BTF_ID(__ID(__BTF_ID__##prefix##__##name##__)) + __BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), "") + +#define ____BTF_ID_FLAGS(prefix, name, flags) \ + __BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), ".long " #flags "\n") +#define __BTF_ID_FLAGS(prefix, name, flags, ...) \ + ____BTF_ID_FLAGS(prefix, name, flags) +#define BTF_ID_FLAGS(prefix, name, ...) \ + __BTF_ID_FLAGS(prefix, name, ##__VA_ARGS__, 0) /* * The BTF_ID_LIST macro defines pure (unsorted) list @@ -155,10 +167,58 @@ asm( \ ".popsection; \n"); \ extern struct btf_id_set name; +/* + * The BTF_SET8_START/END macros pair defines sorted list of + * BTF IDs and their flags plus its members count, with the + * following layout: + * + * BTF_SET8_START(list) + * BTF_ID_FLAGS(type1, name1, flags) + * BTF_ID_FLAGS(type2, name2, flags) + * BTF_SET8_END(list) + * + * __BTF_ID__set8__list: + * .zero 8 + * list: + * __BTF_ID__type1__name1__3: + * .zero 4 + * .word (1 << 0) | (1 << 2) + * __BTF_ID__type2__name2__5: + * .zero 4 + * .word (1 << 3) | (1 << 1) | (1 << 2) + * + */ +#define __BTF_SET8_START(name, scope, flags) \ +__BTF_ID_LIST(name, local) \ +asm( \ +".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \ +"." #scope " __BTF_ID__set8__" #name "; \n" \ +"__BTF_ID__set8__" #name ":; \n" \ +".zero 4 \n" \ +".long " __stringify(flags) "\n" \ +".popsection; \n"); + +#define BTF_SET8_START(name) \ +__BTF_SET8_START(name, local, 0) + +#define BTF_SET8_END(name) \ +asm( \ +".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \ +".size __BTF_ID__set8__" #name ", .-" #name " \n" \ +".popsection; \n"); \ +extern struct btf_id_set8 name; + +#define BTF_KFUNCS_START(name) \ +__BTF_SET8_START(name, local, BTF_SET8_KFUNCS) + +#define BTF_KFUNCS_END(name) \ +BTF_SET8_END(name) + #else -#define BTF_ID_LIST(name) static u32 __maybe_unused name[5]; +#define BTF_ID_LIST(name) static u32 __maybe_unused name[128]; #define BTF_ID(prefix, name) +#define BTF_ID_FLAGS(prefix, name, ...) #define BTF_ID_UNUSED #define BTF_ID_LIST_GLOBAL(name, n) u32 __maybe_unused name[n]; #define BTF_ID_LIST_SINGLE(name, prefix, typename) static u32 __maybe_unused name[1]; @@ -166,6 +226,10 @@ extern struct btf_id_set name; #define BTF_SET_START(name) static struct btf_id_set __maybe_unused name = { 0 }; #define BTF_SET_START_GLOBAL(name) static struct btf_id_set __maybe_unused name = { 0 }; #define BTF_SET_END(name) +#define BTF_SET8_START(name) static struct btf_id_set8 __maybe_unused name = { 0 }; +#define BTF_SET8_END(name) +#define BTF_KFUNCS_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_KFUNCS }; +#define BTF_KFUNCS_END(name) #endif /* CONFIG_DEBUG_INFO_BTF */ @@ -215,5 +279,9 @@ MAX_BTF_TRACING_TYPE, }; extern u32 btf_tracing_ids[]; +extern u32 bpf_cgroup_btf_id[]; +extern u32 bpf_local_storage_map_btf_id[]; +extern u32 btf_bpf_map_id[]; +extern u32 bpf_kmem_cache_btf_id[]; #endif From 07b181a084bc0a08300a84d0dbaf856970020eef Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Tue, 23 Jun 2026 17:55:44 -0700 Subject: [PATCH 007/373] selftests/bpf: Modernize resolve_btfids test scaffolding Refactor resolve_btfids test in order to: * use newer ASSERT_* macros instead of CHECK * extend the lifetime of loaded BTF to enable additional checks * cleanup unused/unnecessary code Reviewed-by: Emil Tsalapatis Acked-by: Eduard Zingerman Acked-by: Jiri Olsa Signed-off-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260624005546.1818483-3-ihor.solodrai@linux.dev Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/resolve_btfids.c | 59 +++++++------------ 1 file changed, 20 insertions(+), 39 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index 41dfaaabb73f..a26adf404d7e 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -10,7 +10,7 @@ #include #include "test_progs.h" -static int duration; +#define BTF_DATA_FILE "resolve_btfids.test.o.BTF" struct symbol { const char *name; @@ -70,10 +70,8 @@ __resolve_symbol(struct btf *btf, int type_id) unsigned int i; type = btf__type_by_id(btf, type_id); - if (!type) { - PRINT_FAIL("Failed to get type for ID %d\n", type_id); + if (!ASSERT_OK_PTR(type, "btf__type_by_id")) return -1; - } for (i = 0; i < ARRAY_SIZE(test_symbols); i++) { if (test_symbols[i].id >= 0) @@ -83,10 +81,8 @@ __resolve_symbol(struct btf *btf, int type_id) continue; str = btf__name_by_offset(btf, type->name_off); - if (!str) { - PRINT_FAIL("Failed to get name for BTF ID %d\n", type_id); + if (!ASSERT_OK_PTR(str, "btf__name_by_offset")) return -1; - } if (!strcmp(str, test_symbols[i].name)) test_symbols[i].id = type_id; @@ -95,25 +91,15 @@ __resolve_symbol(struct btf *btf, int type_id) return 0; } -static int resolve_symbols(void) +static int resolve_symbols(struct btf *btf) { - struct btf *btf; + __u32 nr = btf__type_cnt(btf); int type_id; - __u32 nr; - - btf = btf__parse_raw("resolve_btfids.test.o.BTF"); - if (CHECK(libbpf_get_error(btf), "resolve", - "Failed to load BTF from resolve_btfids.test.o.BTF\n")) - return -1; - - nr = btf__type_cnt(btf); for (type_id = 1; type_id < nr; type_id++) { if (__resolve_symbol(btf, type_id)) - break; + return -1; } - - btf__free(btf); return 0; } @@ -121,25 +107,22 @@ void test_resolve_btfids(void) { __u32 *test_list, *test_lists[] = { test_list_local, test_list_global }; unsigned int i, j; - int ret = 0; + struct btf *btf; - if (resolve_symbols()) + btf = btf__parse_raw(BTF_DATA_FILE); + if (!ASSERT_OK_PTR(btf, "btf_parse")) return; + if (resolve_symbols(btf)) + goto out; + /* Check BTF_ID_LIST(test_list_local) and * BTF_ID_LIST_GLOBAL(test_list_global) IDs */ for (j = 0; j < ARRAY_SIZE(test_lists); j++) { test_list = test_lists[j]; - for (i = 0; i < ARRAY_SIZE(test_symbols); i++) { - ret = CHECK(test_list[i] != test_symbols[i].id, - "id_check", - "wrong ID for %s (%d != %d)\n", - test_symbols[i].name, - test_list[i], test_symbols[i].id); - if (ret) - return; - } + for (i = 0; i < ARRAY_SIZE(test_symbols); i++) + ASSERT_EQ(test_list[i], test_symbols[i].id, test_symbols[i].name); } /* Check BTF_SET_START(test_set) IDs */ @@ -153,15 +136,13 @@ void test_resolve_btfids(void) break; } - ret = CHECK(!found, "id_check", - "ID %d not found in test_symbols\n", - test_set.ids[i]); - if (ret) + if (!ASSERT_TRUE(found, "id_in_test_symbols")) break; - if (i > 0) { - if (!ASSERT_LE(test_set.ids[i - 1], test_set.ids[i], "sort_check")) - return; - } + if (i > 0) + ASSERT_LE(test_set.ids[i - 1], test_set.ids[i], "sort_check"); } + +out: + btf__free(btf); } From 6f10765ecd465174ad73b16e988401331ec8c742 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Tue, 23 Jun 2026 17:55:45 -0700 Subject: [PATCH 008/373] selftests/bpf: Fix resolve_btfids test reads of BTF ID sets in PIE builds TL;DR On aarch64 with gcc toolchain, when test_progs is linked as a PIE, reads of BTF ID array by C name return garbage because the GNU assembler on aarch64 unconditionally folds .local symbol references into section+addend form, and GOT slots cannot carry an addend per the AArch64 ELF spec. Fix by marking the test's BTF ID objects with hidden visibility, which makes gcc emit a direct access that bypasses the GOT entirely. Details below. The subsequent patches adding kfunc checks to resolve_btfids test may cause test failures on aarch64 / gcc-15: test_resolve_btfids:FAIL:kfunc_set_flags actual 13 != expected 1 test_resolve_btfids:FAIL:kfunc_set_cnt actual 0 != expected 4 The test defines its BTF ID sets with the same macros as the kernel and reads them back directly by C name (in the same way as the kernel code does). test_kfunc_set is a .local symbol emitted into .BTF_ids by inline asm and declared to the compiler as a plain default-visibility extern, that is: extern struct btf_id_set8 test_kfunc_set; Depending on the build environment, test_progs may be linked as a position-independent executable (for example, gcc defaults to -fpie [1]). In a PIE, taking the address of a default-visibility extern is routed through the GOT (Global Offset Table) [2]. The GNU assembler's adjust_reloc_syms() pass (gas/write.c [3]) replaces references to local symbols with the corresponding section symbol, folding the symbol's offset into the relocation addend. On aarch64 this conversion is unconditional: tc_fix_adjustable() is defined to 1 for all fixups (gas/config/tc-aarch64.h [4]), so even GOT-generating relocations are subject to it. The resulting object file therefore contains: R_AARCH64_ADR_GOT_PAGE .BTF_ids + 0x54 R_AARCH64_LD64_GOT_LO12_NC .BTF_ids + 0x54 However, the AArch64 ELF specification mandates that GOT-generating relocations must have a zero addend [5]. The +0x54 is therefore not honored: the linker creates a GOT slot pointing at the .BTF_ids base, and every access through that slot reads offset 0 instead of 0x54. This is purely a read-side problem, specific to the PIE test binary on aarch64 with gcc toolchain. resolve_btfids patches the set header correctly and the .BTF_ids bytes in test_progs are correct. vmlinux is unaffected because it is built with -fno-PIE [6] and reaches .BTF_ids with direct, addend-preserving relocations rather than the GOT. clang is unaffected because LLVM's assembler retains the original symbol for GOT relocations instead of converting to section+addend [7]. To mitigate this issue, mark the test's .local BTF ID objects (test_list_local and test_set) hidden with a visibility pragma so that gcc treats them as non-interposable and emits a direct access instead of a GOT load. test_list_global is .globl, which the assembler does not fold into section+addend, so it is left at default visibility. This keeps the natural by-name access, works in both PIE and non-PIE builds, and needs no change to the BTF_ID macros or resolve_btfids. [1] https://gcc.gnu.org/onlinedocs/gnat_ugn/Position-Independent-Executable-PIE-Enabled-by-Default-on-Linux.html [2] https://gcc.gnu.org/wiki/Visibility [3] https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gas/write.c#l922 [4] https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gas/config/tc-aarch64.h#l279 [5] https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#5733relocation-operations [6] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Makefile?h=v7.1-rc6#n593 [7] https://github.com/llvm/llvm-project/blob/4b3bc46d1d794b8ed78b75ccd35a6cc30235bf31/llvm/lib/MC/ELFObjectWriter.cpp#L1213-L1224 Acked-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Acked-by: Jiri Olsa Signed-off-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260624005546.1818483-4-ihor.solodrai@linux.dev Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/resolve_btfids.c | 39 ++++++++++++++----- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index a26adf404d7e..98ce8fca2007 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -34,6 +34,24 @@ asm ( ".balign 4, 0; \n" ".popsection; \n"); +/* + * test_list_local and test_set are .local symbols placed in .BTF_ids by + * inline asm, and are read here directly by C name. To the compiler they + * are plain, default-visibility extern objects. + * + * When test_progs is linked as a position-independent executable (PIE), + * taking the address of such an extern is routed through the GOT. The + * GNU assembler on aarch64 unconditionally converts references to .local + * symbols into section + addend form (".BTF_ids + "), but a GOT + * slot cannot carry an addend (the AArch64 ELF spec mandates zero), so + * the linker resolves it to the .BTF_ids base. + * + * Mark them hidden so the compiler treats them as non-interposable and + * emits a direct, addend-preserving PC-relative access instead of a GOT + * load, in both PIE and non-PIE builds. test_list_global is .globl and + * not affected, so it is left at default visibility. + */ +#pragma GCC visibility push(hidden) BTF_ID_LIST(test_list_local) BTF_ID_UNUSED BTF_ID(typedef, S) @@ -43,16 +61,6 @@ BTF_ID(struct, S) BTF_ID(union, U) BTF_ID(func, func) -extern __u32 test_list_global[]; -BTF_ID_LIST_GLOBAL(test_list_global, 1) -BTF_ID_UNUSED -BTF_ID(typedef, S) -BTF_ID(typedef, T) -BTF_ID(typedef, U) -BTF_ID(struct, S) -BTF_ID(union, U) -BTF_ID(func, func) - BTF_SET_START(test_set) BTF_ID(typedef, S) BTF_ID(typedef, T) @@ -61,6 +69,17 @@ BTF_ID(struct, S) BTF_ID(union, U) BTF_ID(func, func) BTF_SET_END(test_set) +#pragma GCC visibility pop + +extern __u32 test_list_global[]; +BTF_ID_LIST_GLOBAL(test_list_global, 1) +BTF_ID_UNUSED +BTF_ID(typedef, S) +BTF_ID(typedef, T) +BTF_ID(typedef, U) +BTF_ID(struct, S) +BTF_ID(union, U) +BTF_ID(func, func) static int __resolve_symbol(struct btf *btf, int type_id) From 795638829476fe74db0d4e244212faef6674b883 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Tue, 23 Jun 2026 17:55:46 -0700 Subject: [PATCH 009/373] selftests/bpf: Add kfunc set test to resolve_btfids Extend the resolve_btfids selftest to cover kfunc sets defined with BTF_KFUNCS_START/BTF_KFUNCS_END. The test verifies that resolve_btfids correctly processes BTF_ID_FLAGS, resolves function IDs, and checks the kfunc set is sorted. Reviewed-by: Emil Tsalapatis Acked-by: Jiri Olsa Signed-off-by: Ihor Solodrai Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260624005546.1818483-5-ihor.solodrai@linux.dev Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/resolve_btfids.c | 79 +++++++++++++++++-- tools/testing/selftests/bpf/progs/btf_data.c | 10 +++ 2 files changed, 82 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index 98ce8fca2007..ac51fd454821 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -12,6 +12,10 @@ #define BTF_DATA_FILE "resolve_btfids.test.o.BTF" +#ifndef KF_FASTCALL +#define KF_FASTCALL (1 << 12) +#endif + struct symbol { const char *name; int type; @@ -28,6 +32,17 @@ struct symbol test_symbols[] = { { "func", BTF_KIND_FUNC, -1 }, }; +struct kfunc_symbol { + const char *name; + s32 id; + u32 flags; +}; + +static struct kfunc_symbol kfunc_symbols[] = { + { "kfunc_a", -1, 0 }, + { "kfunc_b", -1, KF_FASTCALL }, +}; + /* Align the .BTF_ids section to 4 bytes */ asm ( ".pushsection " BTF_IDS_SECTION " ,\"a\"; \n" @@ -35,9 +50,9 @@ asm ( ".popsection; \n"); /* - * test_list_local and test_set are .local symbols placed in .BTF_ids by - * inline asm, and are read here directly by C name. To the compiler they - * are plain, default-visibility extern objects. + * test_list_local, test_set and test_kfunc_set are .local symbols placed + * in .BTF_ids by inline asm, and are read here directly by C name. To the + * compiler they are plain, default-visibility extern objects. * * When test_progs is linked as a position-independent executable (PIE), * taking the address of such an extern is routed through the GOT. The @@ -69,6 +84,20 @@ BTF_ID(struct, S) BTF_ID(union, U) BTF_ID(func, func) BTF_SET_END(test_set) + +BTF_KFUNCS_START(test_kfunc_set) +BTF_ID_FLAGS(func, kfunc_a) +BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) +BTF_KFUNCS_END(test_kfunc_set) + +/* + * Same kfuncs in reverse declaration order, so resolve_btfids has to + * actually sort at least one of the two sets. + */ +BTF_KFUNCS_START(test_kfunc_set_rev) +BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) +BTF_ID_FLAGS(func, kfunc_a) +BTF_KFUNCS_END(test_kfunc_set_rev) #pragma GCC visibility pop extern __u32 test_list_global[]; @@ -92,6 +121,8 @@ __resolve_symbol(struct btf *btf, int type_id) if (!ASSERT_OK_PTR(type, "btf__type_by_id")) return -1; + str = btf__name_by_offset(btf, type->name_off); + for (i = 0; i < ARRAY_SIZE(test_symbols); i++) { if (test_symbols[i].id >= 0) continue; @@ -99,14 +130,20 @@ __resolve_symbol(struct btf *btf, int type_id) if (BTF_INFO_KIND(type->info) != test_symbols[i].type) continue; - str = btf__name_by_offset(btf, type->name_off); - if (!ASSERT_OK_PTR(str, "btf__name_by_offset")) - return -1; - if (!strcmp(str, test_symbols[i].name)) test_symbols[i].id = type_id; } + if (!btf_is_func(type)) + return 0; + + for (i = 0; i < ARRAY_SIZE(kfunc_symbols); i++) { + if (kfunc_symbols[i].id >= 0) + continue; + if (!strcmp(str, kfunc_symbols[i].name)) + kfunc_symbols[i].id = type_id; + } + return 0; } @@ -122,6 +159,31 @@ static int resolve_symbols(struct btf *btf) return 0; } +static void check_kfunc_set(struct btf_id_set8 *set) +{ + unsigned int i, j; + + ASSERT_EQ(set->flags, BTF_SET8_KFUNCS, "kfunc_set_flags"); + ASSERT_EQ(set->cnt, ARRAY_SIZE(kfunc_symbols), "kfunc_set_cnt"); + + for (i = 0; i < set->cnt; i++) { + for (j = 0; j < ARRAY_SIZE(kfunc_symbols); j++) { + if (kfunc_symbols[j].id == (s32)set->pairs[i].id) { + ASSERT_EQ(set->pairs[i].flags, + kfunc_symbols[j].flags, "kfunc_flags_check"); + break; + } + } + + ASSERT_TRUE(j < ARRAY_SIZE(kfunc_symbols), "kfunc_id_found"); + + if (i > 0) { + ASSERT_LE(set->pairs[i - 1].id, + set->pairs[i].id, "kfunc_sort_check"); + } + } +} + void test_resolve_btfids(void) { __u32 *test_list, *test_lists[] = { test_list_local, test_list_global }; @@ -162,6 +224,9 @@ void test_resolve_btfids(void) ASSERT_LE(test_set.ids[i - 1], test_set.ids[i], "sort_check"); } + check_kfunc_set(&test_kfunc_set); + check_kfunc_set(&test_kfunc_set_rev); + out: btf__free(btf); } diff --git a/tools/testing/selftests/bpf/progs/btf_data.c b/tools/testing/selftests/bpf/progs/btf_data.c index baa525275bde..8587658012c3 100644 --- a/tools/testing/selftests/bpf/progs/btf_data.c +++ b/tools/testing/selftests/bpf/progs/btf_data.c @@ -48,3 +48,13 @@ int func(struct root_struct *root) { return 0; } + +int kfunc_a(struct root_struct *root) +{ + return 0; +} + +int kfunc_b(struct root_struct *root) +{ + return 0; +} From a954c9e3168cdf0c3cad07b43dfc8ca2945d773a Mon Sep 17 00:00:00 2001 From: Andrii Nakryiko Date: Tue, 30 Jun 2026 13:54:18 -0700 Subject: [PATCH 010/373] bpftool: Strip all -Wformat* flags from bootstrap libbpf build Commit 9080b97689db ("bpftool: Pass host flags to bootstrap libbpf") started building the bootstrap libbpf with HOST_CFLAGS, stripping the warning options that are unsuitable for that build by filtering out -W -Wall -Wextra -Wformat -Wformat-signedness. HOST_CFLAGS inherits EXTRA_WARNINGS, which includes -Wformat-security and -Wformat-y2k. The filter drops -Wall and -Wformat (the latter being what actually enables -Wformat), but leaves those two -Wformat-* children in LIBBPF_BOOTSTRAP_CFLAGS. Building the bootstrap libbpf with it then warns: cc1: warning: '-Wformat-y2k' ignored without '-Wformat' cc1: warning: '-Wformat-security' ignored without '-Wformat' The warning is easy to miss in an in-tree build: tools/lib/bpf/Makefile re-adds -Wall via "override CFLAGS += -Wall", which re-enables -Wformat for the libbpf objects, so only libbpf's feature-detection probe (which uses the passed CFLAGS verbatim) leaks the two warnings. The standalone libbpf Makefile (github.com/libbpf/libbpf, used by the bpftool mirror) instead uses "CFLAGS ?= ... -Wall", which the passed-in CFLAGS overrides, so -Wall is never re-added and every bootstrap object warns. Use a -Wformat% wildcard in the filter-out so the orphaned children are removed together with the parent. Fixes: 9080b97689db ("bpftool: Pass host flags to bootstrap libbpf") Signed-off-by: Andrii Nakryiko Acked-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20260630205418.3483969-1-andrii@kernel.org --- tools/bpf/bpftool/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile index 271a7dc77273..b0f7168e7943 100644 --- a/tools/bpf/bpftool/Makefile +++ b/tools/bpf/bpftool/Makefile @@ -99,7 +99,7 @@ endif HOST_LDFLAGS := $(LDFLAGS) # Remove warnings for libbpf bootstrap build -LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS)) +LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat%,$(HOST_CFLAGS)) INSTALL ?= install RM ?= rm -f From 66d7e39e49b0dd57610c9b63afc65b4d5690983b Mon Sep 17 00:00:00 2001 From: Yichong Chen Date: Wed, 24 Jun 2026 10:50:54 +0800 Subject: [PATCH 011/373] tools/bpf/bpftool: Reset vmlinux BTF after map commands get_map_kv_btf() caches the vmlinux BTF object when a map uses btf_vmlinux_value_type_id. map dump released that object when the command completed, but left the global pointer stale. The same cached object can also be returned to print_key_value(), which freed it directly. That leaves btf_vmlinux dangling before the command cleanup path runs. Use free_map_kv_btf() for per-entry cleanup, and reset the cached btf_vmlinux pointer when the map command releases the object. This keeps batch mode from reusing a freed BTF object. Fixes: 4e1ea33292ff ("bpftool: Support dumping a map with btf_vmlinux_value_type_id") Signed-off-by: Yichong Chen Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/9072F43B3F74DF91+20260624025055.1574875-2-chenyichong@uniontech.com --- tools/bpf/bpftool/map.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index 71a45d96617e..6b9649294ca1 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -790,6 +790,12 @@ static int maps_have_btf(int *fds, int nb_fds) static struct btf *btf_vmlinux; +static void free_btf_vmlinux(void) +{ + btf__free(btf_vmlinux); + btf_vmlinux = NULL; +} + static int get_map_kv_btf(const struct bpf_map_info *info, struct btf **btf) { int err = 0; @@ -958,7 +964,7 @@ static int do_dump(int argc, char **argv) close(fds[i]); exit_free: free(fds); - btf__free(btf_vmlinux); + free_btf_vmlinux(); return err; } @@ -1049,7 +1055,7 @@ static void print_key_value(struct bpf_map_info *info, void *key, btf_wtr = get_btf_writer(); if (!btf_wtr) { p_info("failed to create json writer for btf. falling back to plain output"); - btf__free(btf); + free_map_kv_btf(btf); btf = NULL; print_entry_plain(info, key, value); } else { @@ -1065,7 +1071,7 @@ static void print_key_value(struct bpf_map_info *info, void *key, } else { print_entry_plain(info, key, value); } - btf__free(btf); + free_map_kv_btf(btf); } static int do_lookup(int argc, char **argv) From f7f540e19751face50c68bb9ce58460fcb46c293 Mon Sep 17 00:00:00 2001 From: Yichong Chen Date: Wed, 24 Jun 2026 10:50:55 +0800 Subject: [PATCH 012/373] tools/bpf/bpftool: Reset vmlinux BTF after struct_ops commands struct_ops frees the global btf_vmlinux object. In batch mode, a later struct_ops command can reuse stale state. Reset the BTF pointer and cached map info state. Fixes: 65c93628599d ("bpftool: Add struct_ops support") Signed-off-by: Yichong Chen Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/9F9017160ABE125F+20260624025055.1574875-3-chenyichong@uniontech.com --- tools/bpf/bpftool/struct_ops.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/bpf/bpftool/struct_ops.c b/tools/bpf/bpftool/struct_ops.c index aa43dead249c..835e5e561f7f 100644 --- a/tools/bpf/bpftool/struct_ops.c +++ b/tools/bpf/bpftool/struct_ops.c @@ -643,6 +643,10 @@ int do_struct_ops(int argc, char **argv) err = cmd_select(cmds, argc, argv, do_help); btf__free(btf_vmlinux); + btf_vmlinux = NULL; + map_info_type = NULL; + map_info_alloc_len = 0; + map_info_type_id = 0; return err; } From 7cf9cd98cf6f0df3befc167ca6b54c07014d71de Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 24 Jun 2026 23:51:14 +0800 Subject: [PATCH 013/373] bpf: Copy per-CPU map value padding in copy_map_value_long() In kernel, per-CPU map elements are stored with round_up(map->value_size, 8) bytes. On UAPI lookup paths, it copies the rounded size for each CPU into a temporary buffer. However, copy_map_value_long() passes 'map->value_size' to bpf_obj_memcpy(). When the map has special fields, bpf_obj_memcpy() copies around those fields with memcpy(), and does not copy the tail padding between 'map->value_size' and round_up(map->value_size, 8). The temporary UAPI lookup buffers are allocated without __GFP_ZERO. As a result, when the per-CPU map's value size is not equal to round_up(map->value_size, 8), UAPI LOOKUP_ELEM and its variants can return stale heap contents from that padding to user space. The same issue applies to bpf_iter for per-CPU maps. Pass round_up(map->value_size, 8) to bpf_obj_memcpy() from copy_map_value_long(), so per-CPU maps both with and without special fields copy the entire per-CPU slot. Remove the now redundant round_up() from bpf_obj_memcpy()'s long_memcpy path. Fixes: 448325199f57 ("bpf: Add copy_map_value_long to copy to remote percpu memory") Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260624155115.85196-2-leon.hwang@linux.dev --- include/linux/bpf.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7719f6528445..ba09795e0bfd 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -570,7 +570,7 @@ static inline void bpf_obj_memcpy(struct btf_record *rec, if (IS_ERR_OR_NULL(rec)) { if (long_memcpy) - bpf_long_memcpy(dst, src, round_up(size, 8)); + bpf_long_memcpy(dst, src, size); else memcpy(dst, src, size); return; @@ -593,7 +593,7 @@ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src) static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src) { - bpf_obj_memcpy(map->record, dst, src, map->value_size, true); + bpf_obj_memcpy(map->record, dst, src, round_up(map->value_size, 8), true); } static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src) From 163944262f8646bf3a1eec557b6aff1f38582a9f Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 24 Jun 2026 23:51:15 +0800 Subject: [PATCH 014/373] selftests/bpf: Verify no non-zeroed kernel heap memory exposure When lookup element from those per-CPU maps, which have special field in their values and their value size is not equal to roundup(value_sz, 8), the padding size of temporary non-zeroed kernel heap memory allocated by kvmalloc should not be exposed to user space. Without the fix: test_map_uninit_mem_exposure:FAIL:zeroed tail bytes unexpected memory mismatch actual: 2B 2B 2B 2B expected: 00 00 00 00 Assisted-by: Codex:gpt-5.5 Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260624155115.85196-3-leon.hwang@linux.dev --- .../bpf/prog_tests/test_map_uninit.c | 68 +++++++++++++++++++ tools/testing/selftests/bpf/progs/map_kptr.c | 12 ++++ 2 files changed, 80 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_map_uninit.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_map_uninit.c b/tools/testing/selftests/bpf/prog_tests/test_map_uninit.c new file mode 100644 index 000000000000..d0ba2ca587b0 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_map_uninit.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +#include "map_kptr.skel.h" + +void test_map_uninit_mem_exposure(void) +{ + size_t value_sz, slot_sz, lookup_sz, tail_sz; + int err, key, nr_cpus, cpu, map_fd; + __u8 *value = NULL, *zero = NULL; + struct bpf_program *prog; + struct map_kptr *skel; + + nr_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GT(nr_cpus, 0, "libbpf_num_possible_cpus")) + return; + + skel = map_kptr__open(); + if (!ASSERT_OK_PTR(skel, "map_kptr__open")) + return; + + bpf_object__for_each_program(prog, skel->obj) { + err = bpf_program__set_autoload(prog, false); + if (!ASSERT_OK(err, "bpf_program__set_autoload")) + goto out; + } + + err = map_kptr__load(skel); + if (!ASSERT_OK(err, "map_kptr__load")) + goto out; + + value_sz = bpf_map__value_size((skel)->maps.pcpu_array); + slot_sz = roundup(value_sz, 8); + tail_sz = slot_sz - value_sz; + if (!ASSERT_NEQ(tail_sz, 0, "tail_sz")) + goto out; + + lookup_sz = slot_sz * nr_cpus; + map_fd = bpf_map__fd(skel->maps.pcpu_array); + + value = malloc(lookup_sz); + zero = calloc(1, tail_sz); + if (!ASSERT_OK_PTR(value, "malloc value") || !ASSERT_OK_PTR(zero, "calloc zero")) + goto out; + + key = 0; + memset(value, 0x2B, lookup_sz); + err = bpf_map_update_elem(map_fd, &key, value, BPF_ANY); + if (!ASSERT_OK(err, "bpf_map_update_elem")) + goto out; + + memset(value, 0xFF, lookup_sz); + err = bpf_map_lookup_elem(map_fd, &key, value); + if (!ASSERT_OK(err, "bpf_map_lookup_elem")) + goto out; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + __u8 *tail = value + cpu * slot_sz + value_sz; + + if (!ASSERT_MEMEQ(tail, zero, tail_sz, "zeroed tail bytes")) + goto out; + } + +out: + free(zero); + free(value); + map_kptr__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/map_kptr.c b/tools/testing/selftests/bpf/progs/map_kptr.c index 3fbefc568e0a..0d87c97dac99 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr.c +++ b/tools/testing/selftests/bpf/progs/map_kptr.c @@ -4,6 +4,18 @@ #include #include "../test_kmods/bpf_testmod_kfunc.h" +struct map_uninit_value { + struct prog_test_ref_kfunc __kptr_untrusted *unref_ptr; + __u32 data; +} __attribute__((packed)); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __type(key, int); + __type(value, struct map_uninit_value); + __uint(max_entries, 1); +} pcpu_array SEC(".maps"); + struct map_value { struct prog_test_ref_kfunc __kptr_untrusted *unref_ptr; struct prog_test_ref_kfunc __kptr *ref_ptr; From 859055e07697c46f6964109981aa1cd23d6bde47 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 29 Jun 2026 23:22:06 +0200 Subject: [PATCH 015/373] bpf: Add tracing_multi link info support Adding BPF_OBJ_GET_INFO_BY_FD support for tracing_multi links. We expose following tracing_multi link data: - attach_type of the program - number of ids - array of BTF ids - array of its related kernel addresses - array of cookies The change follows the kprobe_multi and uprobe_multi link-info convention of optional output arrays with an in/out count, On top of standard tracing link data we also expose addresses, because they are useful info for user (especially when the attachment was done via pattern). This data is hidden when kallsyms does not allow exposing kernel pointer values. Assisted-by: Codex:GPT-5 Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Acked-by: Leon Hwang Acked-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260629212208.895962-2-jolsa@kernel.org --- include/uapi/linux/bpf.h | 9 ++++++ kernel/trace/bpf_trace.c | 55 ++++++++++++++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 9 ++++++ 3 files changed, 73 insertions(+) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index c91b5a4bda03..2f1d24fef857 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6842,6 +6842,15 @@ struct bpf_link_info { __u32 flags; __u32 pid; } uprobe_multi; + struct { + __u32 attach_type; + __u32 count; /* in/out: tracing_multi target count */ + __u32 btf_obj_id; + __u32 :32; + __aligned_u64 ids; + __aligned_u64 addrs; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 type; /* enum bpf_perf_event_type */ __u32 :32; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 75495a5c3507..76ab51deaa6b 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -3700,6 +3700,60 @@ static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) kvfree(tr_link); } +static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link, + struct bpf_link_info *info) +{ + u64 __user *ucookies = u64_to_user_ptr(info->tracing_multi.cookies); + u64 __user *uaddrs = u64_to_user_ptr(info->tracing_multi.addrs); + u32 __user *uids = u64_to_user_ptr(info->tracing_multi.ids); + struct bpf_tracing_multi_link *tr_link; + u32 ucount = info->tracing_multi.count; + bool has_cookies, show_addrs; + int err = 0; + + if ((uids || ucookies || uaddrs) && !ucount) + return -EINVAL; + + tr_link = container_of(link, struct bpf_tracing_multi_link, link); + + info->tracing_multi.attach_type = tr_link->link.attach_type; + info->tracing_multi.count = tr_link->nodes_cnt; + info->tracing_multi.btf_obj_id = btf_obj_id(tr_link->link.prog->aux->attach_btf); + + if (!uids && !ucookies && !uaddrs) + return 0; + + if (ucount < tr_link->nodes_cnt) + err = -ENOSPC; + else + ucount = tr_link->nodes_cnt; + + has_cookies = !!tr_link->cookies; + show_addrs = kallsyms_show_value(current_cred()); + + for (int i = 0; i < ucount; i++) { + struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i]; + u64 addr, cookie; + u32 id; + + bpf_trampoline_unpack_key(mnode->trampoline->key, NULL, &id); + + addr = show_addrs ? mnode->trampoline->ip : 0; + cookie = has_cookies ? tr_link->cookies[i] : 0; + + if (uids && put_user(id, uids + i)) + return -EFAULT; + if (uaddrs && put_user(addr, uaddrs + i)) + return -EFAULT; + if (ucookies && put_user(cookie, ucookies + i)) + return -EFAULT; + + cond_resched(); + } + + return err; +} + #ifdef CONFIG_PROC_FS static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link, struct seq_file *seq) @@ -3730,6 +3784,7 @@ static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link, static const struct bpf_link_ops bpf_tracing_multi_link_lops = { .release = bpf_tracing_multi_link_release, .dealloc_deferred = bpf_tracing_multi_link_dealloc, + .fill_link_info = bpf_tracing_multi_link_fill_link_info, #ifdef CONFIG_PROC_FS .show_fdinfo = bpf_tracing_multi_show_fdinfo, #endif diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c91b5a4bda03..2f1d24fef857 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6842,6 +6842,15 @@ struct bpf_link_info { __u32 flags; __u32 pid; } uprobe_multi; + struct { + __u32 attach_type; + __u32 count; /* in/out: tracing_multi target count */ + __u32 btf_obj_id; + __u32 :32; + __aligned_u64 ids; + __aligned_u64 addrs; + __aligned_u64 cookies; + } tracing_multi; struct { __u32 type; /* enum bpf_perf_event_type */ __u32 :32; From d36e4dd547bb061eb78c7aca54f4575395598540 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 29 Jun 2026 23:22:07 +0200 Subject: [PATCH 016/373] selftests/bpf: Add tracing_multi link info tests Adding tracing_multi link info tests that follow the kprobe_multi and uprobe_multi tests logic. Assisted-by: Codex:GPT-5 Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260629212208.895962-3-jolsa@kernel.org --- .../selftests/bpf/prog_tests/fill_link_info.c | 242 ++++++++++++++++++ .../selftests/bpf/progs/test_fill_link_info.c | 6 + 2 files changed, 248 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/fill_link_info.c b/tools/testing/selftests/bpf/prog_tests/fill_link_info.c index f589eefbf9fb..0918321c8e63 100644 --- a/tools/testing/selftests/bpf/prog_tests/fill_link_info.c +++ b/tools/testing/selftests/bpf/prog_tests/fill_link_info.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include "trace_helpers.h" #include "test_fill_link_info.skel.h" @@ -24,6 +25,22 @@ static __u64 kmulti_cookies[] = { 3, 1, 2 }; #define KPROBE_FUNC "bpf_fentry_test1" static __u64 kprobe_addr; +static const char * const tmulti_syms[] = { + "bpf_fentry_test2", + "bpf_fentry_test1", + "bpf_fentry_test3", +}; + +static __u64 tmulti_cookies[] = { 30, 10, 20 }; +#define TRACING_MULTI_CNT ARRAY_SIZE(tmulti_syms) + +struct tmulti_target { + const char *name; + __u64 addr; + __u64 cookie; + __u32 id; +}; + #define UPROBE_FILE "/proc/self/exe" static ssize_t uprobe_offset; /* uprobe attach point */ @@ -396,6 +413,224 @@ static void test_kprobe_multi_fill_link_info(struct test_fill_link_info *skel, bpf_link__destroy(link); } +static int tmulti_target_cmp(const void *a, const void *b) +{ + const struct tmulti_target *ta = a; + const struct tmulti_target *tb = b; + + return (ta->id > tb->id) - (ta->id < tb->id); +} + +static int setup_tmulti_targets(const struct bpf_program *prog, + struct tmulti_target *targets, + __u32 *btf_obj_id) +{ + struct bpf_prog_info prog_info; + __u32 len = sizeof(prog_info); + struct btf *btf; + int err, i; + __s32 id; + + btf = btf__load_vmlinux_btf(); + if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf")) + return -1; + + for (i = 0; i < TRACING_MULTI_CNT; i++) { + id = btf__find_by_name_kind(btf, tmulti_syms[i], BTF_KIND_FUNC); + if (!ASSERT_GT(id, 0, "btf__find_by_name_kind")) + goto error; + + targets[i].name = tmulti_syms[i]; + targets[i].addr = ksym_get_addr(tmulti_syms[i]); + targets[i].cookie = tmulti_cookies[i]; + targets[i].id = id; + } + + memset(&prog_info, 0, len); + err = bpf_prog_get_info_by_fd(bpf_program__fd(prog), &prog_info, &len); + if (!ASSERT_OK(err, "bpf_prog_get_info_by_fd")) + goto error; + if (!ASSERT_GT(prog_info.attach_btf_obj_id, 0, "attach_btf_obj_id")) + goto error; + *btf_obj_id = prog_info.attach_btf_obj_id; + + /* + * The kernel tracing multi attach sorts ids. We sort as well, + * so we can easily compare ids and cookies later. + */ + qsort(targets, TRACING_MULTI_CNT, sizeof(targets[0]), tmulti_target_cmp); + btf__free(btf); + return 0; + +error: + btf__free(btf); + return -1; +} + +static int verify_tracing_multi_link_info(int fd, const struct bpf_program *prog, + const struct tmulti_target *targets, + __u32 btf_obj_id, bool has_cookies) +{ + enum bpf_attach_type attach_type = bpf_program__expected_attach_type(prog); + __u64 addrs[TRACING_MULTI_CNT], cookies[TRACING_MULTI_CNT]; + __u32 ids[TRACING_MULTI_CNT]; + struct bpf_link_info info; + __u32 len = sizeof(info); + int err, i; + + memset(&info, 0, sizeof(info)); + err = bpf_link_get_info_by_fd(fd, &info, &len); + if (!ASSERT_OK(err, "bpf_link_get_info_by_fd")) + return -1; + + if (!ASSERT_EQ(info.type, BPF_LINK_TYPE_TRACING_MULTI, "info.type")) + return -1; + + ASSERT_EQ(info.tracing_multi.attach_type, attach_type, "info.tracing_multi.attach_type"); + ASSERT_EQ(info.tracing_multi.count, TRACING_MULTI_CNT, "info.tracing_multi.count"); + + memset(ids, 0, sizeof(ids)); + memset(cookies, 0, sizeof(cookies)); + memset(addrs, 0, sizeof(addrs)); + + info.tracing_multi.ids = ptr_to_u64(ids); + info.tracing_multi.addrs = ptr_to_u64(addrs); + info.tracing_multi.cookies = has_cookies ? ptr_to_u64(cookies) : 0; + info.tracing_multi.count = TRACING_MULTI_CNT; + + err = bpf_link_get_info_by_fd(fd, &info, &len); + if (!ASSERT_OK(err, "bpf_link_get_info_by_fd")) + return -1; + + if (!ASSERT_EQ(info.type, BPF_LINK_TYPE_TRACING_MULTI, "info.type")) + return -1; + + ASSERT_EQ(info.tracing_multi.attach_type, attach_type, "info.tracing_multi.attach_type"); + ASSERT_EQ(info.tracing_multi.count, TRACING_MULTI_CNT, "info.tracing_multi.count"); + ASSERT_EQ(info.tracing_multi.btf_obj_id, btf_obj_id, "tracing_multi.btf_obj_id"); + + for (i = 0; i < TRACING_MULTI_CNT; i++) { + ASSERT_EQ(ids[i], targets[i].id, "tracing_multi.ids"); + ASSERT_EQ(cookies[i], has_cookies ? targets[i].cookie : 0, "tracing_multi.cookies"); + + if (targets[i].addr) { + struct ksym *ksym; + + if (!ASSERT_NEQ(addrs[i], 0, "tracing_multi.addrs")) + return -1; + ksym = ksym_search(addrs[i]); + if (!ASSERT_OK_PTR(ksym, "ksym_search")) + return -1; + ASSERT_STREQ(ksym->name, targets[i].name, "tracing_multi.addr_name"); + } else { + ASSERT_EQ(addrs[i], 0, "tracing_multi.addrs"); + } + } + + return 0; +} + +static void verify_tracing_multi_invalid_user_buffer(int fd, const struct tmulti_target *targets) +{ + __u32 ids[TRACING_MULTI_CNT] = {}; + struct bpf_link_info info; + __u32 len = sizeof(info); + int err, i; + + /* Wrong info setup (ids != NULL and cnt == 0) -> EINVAL */ + memset(&info, 0, sizeof(info)); + info.tracing_multi.ids = ptr_to_u64(ids); + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_EQ(err, -EINVAL, "tracing_multi.invalid_count"); + + /* Smaller than actual count provided -> ENOSPC */ + memset(ids, 0, sizeof(ids)); + memset(&info, 0, sizeof(info)); + info.tracing_multi.ids = ptr_to_u64(ids); + info.tracing_multi.count = TRACING_MULTI_CNT - 1; + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_EQ(err, -ENOSPC, "tracing_multi.small_count"); + for (i = 0; i < TRACING_MULTI_CNT - 1; i++) + ASSERT_EQ(ids[i], targets[i].id, "tracing_multi.partial_ids"); + /* check that the last entry is not populated */ + ASSERT_EQ(ids[i], 0, "tracing_multi.partial_ids"); + + /* Bigger than actual count provided -> OK */ + memset(ids, 0, sizeof(ids)); + memset(&info, 0, sizeof(info)); + info.tracing_multi.ids = ptr_to_u64(ids); + info.tracing_multi.count = TRACING_MULTI_CNT + 1; + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_OK(err, "tracing_multi.big_count"); + for (i = 0; i < TRACING_MULTI_CNT; i++) + ASSERT_EQ(ids[i], targets[i].id, "tracing_multi.ids"); + + /* Invalid ids pointer -> EFAULT */ + memset(&info, 0, sizeof(info)); + info.tracing_multi.ids = 0x1; + info.tracing_multi.count = TRACING_MULTI_CNT; + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_EQ(err, -EFAULT, "tracing_multi.bad_btf_ids"); + + /* Invalid cookies pointer -> EFAULT */ + memset(&info, 0, sizeof(info)); + info.tracing_multi.cookies = 0x1; + info.tracing_multi.count = TRACING_MULTI_CNT; + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_EQ(err, -EFAULT, "tracing_multi.bad_cookies"); + + /* Invalid addrs pointer -> EFAULT */ + memset(&info, 0, sizeof(info)); + info.tracing_multi.addrs = 0x1; + info.tracing_multi.count = TRACING_MULTI_CNT; + err = bpf_link_get_info_by_fd(fd, &info, &len); + ASSERT_EQ(err, -EFAULT, "tracing_multi.bad_addrs"); +} + +static void test_tracing_multi_fill_link_info(struct test_fill_link_info *skel, + bool has_cookies, bool invalid) +{ + LIBBPF_OPTS(bpf_tracing_multi_opts, opts); + struct tmulti_target targets[TRACING_MULTI_CNT]; + __u32 ids[TRACING_MULTI_CNT], btf_obj_id; + __u64 cookies[TRACING_MULTI_CNT]; + struct bpf_link *link; + int link_fd, err, i; + +#ifndef __x86_64__ + test__skip(); + return; +#endif + + if (setup_tmulti_targets(skel->progs.tmulti_run, targets, &btf_obj_id)) + return; + + for (i = 0; i < TRACING_MULTI_CNT; i++) { + ids[i] = targets[i].id; + cookies[i] = targets[i].cookie; + } + + opts.ids = ids; + opts.cnt = TRACING_MULTI_CNT; + if (has_cookies) + opts.cookies = cookies; + + link = bpf_program__attach_tracing_multi(skel->progs.tmulti_run, NULL, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_tracing_multi")) + return; + + link_fd = bpf_link__fd(link); + if (invalid) { + verify_tracing_multi_invalid_user_buffer(link_fd, targets); + } else { + err = verify_tracing_multi_link_info(link_fd, skel->progs.tmulti_run, + targets, btf_obj_id, has_cookies); + ASSERT_OK(err, "verify_tracing_multi_link_info"); + } + + bpf_link__destroy(link); +} + #define SEC(name) __attribute__((section(name), used)) static short uprobe_link_info_sema_1 SEC(".probes"); @@ -640,6 +875,13 @@ void test_fill_link_info(void) if (test__start_subtest("kprobe_multi_invalid_ubuff")) test_kprobe_multi_fill_link_info(skel, true, true, true); + if (test__start_subtest("tracing_multi_link_info")) { + test_tracing_multi_fill_link_info(skel, false, false); + test_tracing_multi_fill_link_info(skel, true, false); + } + if (test__start_subtest("tracing_multi_invalid_ubuff")) + test_tracing_multi_fill_link_info(skel, true, true); + if (test__start_subtest("uprobe_multi_link_info")) test_uprobe_multi_fill_link_info(skel, false, false); if (test__start_subtest("uretprobe_multi_link_info")) diff --git a/tools/testing/selftests/bpf/progs/test_fill_link_info.c b/tools/testing/selftests/bpf/progs/test_fill_link_info.c index 137bd6292163..c85081538e93 100644 --- a/tools/testing/selftests/bpf/progs/test_fill_link_info.c +++ b/tools/testing/selftests/bpf/progs/test_fill_link_info.c @@ -58,4 +58,10 @@ int BPF_PROG(umulti_run) return 0; } +SEC("fentry.multi") +int BPF_PROG(tmulti_run) +{ + return 0; +} + char _license[] SEC("license") = "GPL"; From 37c1e353c9fd68eb27fa2103bfed22835936ec3e Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 29 Jun 2026 23:22:08 +0200 Subject: [PATCH 017/373] bpftool: Add tracing_multi link info output Adding bpftool support to show tracing_multi link details, the new output looks like: # bpftool link ... 61: tracing_multi prog 167 attach_type trace_fentry_multi btf_obj_id 1 count 3 btf_id addr cookie func [module] 92598 ffffffff825017c4 10 bpf_fentry_test1 92600 ffffffff82503814 30 bpf_fentry_test2 92601 ffffffff82503824 20 bpf_fentry_test3 pids test_progs(1540) Assisted-by: Codex:GPT-5 Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260629212208.895962-4-jolsa@kernel.org --- tools/bpf/bpftool/link.c | 133 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) diff --git a/tools/bpf/bpftool/link.c b/tools/bpf/bpftool/link.c index bdcd717b0348..088d1d206065 100644 --- a/tools/bpf/bpftool/link.c +++ b/tools/bpf/bpftool/link.c @@ -377,6 +377,25 @@ static __u64 *u64_to_arr(__u64 val) return (__u64 *) u64_to_ptr(val); } +static __u32 *u64_to_u32_arr(__u64 val) +{ + return (__u32 *)u64_to_ptr(val); +} + +static struct kernel_sym *find_kernel_sym_by_addr(__u64 addr, bool is_ibt_enabled) +{ + struct kernel_sym *sym; + + if (!addr) + return NULL; + + sym = kernel_syms_search(&dd, addr); + if (!sym && is_ibt_enabled && addr >= 4) + sym = kernel_syms_search(&dd, addr - 4); + + return sym; +} + static void show_uprobe_multi_json(struct bpf_link_info *info, json_writer_t *wtr) { @@ -403,6 +422,52 @@ show_uprobe_multi_json(struct bpf_link_info *info, json_writer_t *wtr) jsonw_end_array(json_wtr); } +static void +show_tracing_multi_json(struct bpf_link_info *info, json_writer_t *wtr) +{ + bool is_ibt_enabled = is_x86_ibt_enabled(), show_symbol; + __u64 *addrs, *cookies; + __u32 i, *ids; + + if (!dd.sym_count) + kernel_syms_load(&dd); + show_symbol = !!dd.sym_count; + + show_link_attach_type_json(info->tracing_multi.attach_type, wtr); + jsonw_uint_field(wtr, "func_cnt", info->tracing_multi.count); + jsonw_uint_field(wtr, "btf_obj_id", info->tracing_multi.btf_obj_id); + jsonw_name(wtr, "funcs"); + + jsonw_start_array(wtr); + + ids = u64_to_u32_arr(info->tracing_multi.ids); + addrs = u64_to_arr(info->tracing_multi.addrs); + cookies = u64_to_arr(info->tracing_multi.cookies); + + for (i = 0; i < info->tracing_multi.count; i++) { + struct kernel_sym *sym; + __u64 addr = addrs[i]; + + sym = show_symbol ? find_kernel_sym_by_addr(addr, is_ibt_enabled) : NULL; + + jsonw_start_object(wtr); + jsonw_uint_field(wtr, "id", ids[i]); + jsonw_uint_field(wtr, "addr", addr); + if (sym) { + jsonw_string_field(wtr, "func", sym->name); + if (sym->module[0] == '\0') { + jsonw_name(wtr, "module"); + jsonw_null(wtr); + } else { + jsonw_string_field(wtr, "module", sym->module); + } + } + jsonw_uint_field(wtr, "cookie", cookies[i]); + jsonw_end_object(wtr); + } + jsonw_end_array(wtr); +} + static void show_perf_event_kprobe_json(struct bpf_link_info *info, json_writer_t *wtr) { @@ -589,6 +654,9 @@ static int show_link_close_json(int fd, struct bpf_link_info *info) case BPF_LINK_TYPE_UPROBE_MULTI: show_uprobe_multi_json(info, json_wtr); break; + case BPF_LINK_TYPE_TRACING_MULTI: + show_tracing_multi_json(info, json_wtr); + break; case BPF_LINK_TYPE_PERF_EVENT: switch (info->perf_event.type) { case BPF_PERF_EVENT_EVENT: @@ -833,6 +901,46 @@ static void show_uprobe_multi_plain(struct bpf_link_info *info) } } +static void show_tracing_multi_plain(struct bpf_link_info *info) +{ + bool is_ibt_enabled = is_x86_ibt_enabled(), show_symbol; + __u64 *addrs, *cookies; + __u32 i, *ids; + + if (!info->tracing_multi.count) + return; + + if (!dd.sym_count) + kernel_syms_load(&dd); + show_symbol = !!dd.sym_count; + + printf("\n\t"); + show_link_attach_type_plain(info->tracing_multi.attach_type); + printf("btf_obj_id %u ", info->tracing_multi.btf_obj_id); + printf("count %u ", info->tracing_multi.count); + + printf("\n\t%-16s %-16s %-16s %s", + "btf_id", "addr", "cookie", "func [module]"); + + ids = u64_to_u32_arr(info->tracing_multi.ids); + addrs = u64_to_arr(info->tracing_multi.addrs); + cookies = u64_to_arr(info->tracing_multi.cookies); + + for (i = 0; i < info->tracing_multi.count; i++) { + __u64 addr = addrs[i]; + struct kernel_sym *sym; + + sym = show_symbol ? find_kernel_sym_by_addr(addr, is_ibt_enabled) : NULL; + + printf("\n\t%-16u %016llx %-16llu", ids[i], addr, cookies[i]); + if (sym) { + printf(" %s", sym->name); + if (sym->module[0] != '\0') + printf(" [%s]", sym->module); + } + } +} + static void show_perf_event_kprobe_plain(struct bpf_link_info *info) { const char *buf; @@ -989,6 +1097,9 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info) case BPF_LINK_TYPE_UPROBE_MULTI: show_uprobe_multi_plain(info); break; + case BPF_LINK_TYPE_TRACING_MULTI: + show_tracing_multi_plain(info); + break; case BPF_LINK_TYPE_PERF_EVENT: switch (info->perf_event.type) { case BPF_PERF_EVENT_EVENT: @@ -1029,6 +1140,7 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info) static int do_show_link(int fd) { __u64 *ref_ctr_offsets = NULL, *offsets = NULL, *cookies = NULL; + __u32 *ids = NULL; struct bpf_link_info info; __u32 len = sizeof(info); char path_buf[PATH_MAX]; @@ -1114,6 +1226,26 @@ static int do_show_link(int fd) goto again; } } + if (info.type == BPF_LINK_TYPE_TRACING_MULTI && !info.tracing_multi.ids) { + count = info.tracing_multi.count; + if (count) { + ids = calloc(count, sizeof(__u32)); + addrs = calloc(count, sizeof(__u64)); + cookies = calloc(count, sizeof(__u64)); + if (!ids || !addrs || !cookies) { + p_err("mem alloc failed"); + close(fd); + free(cookies); + free(addrs); + free(ids); + return -ENOMEM; + } + info.tracing_multi.ids = ptr_to_u64(ids); + info.tracing_multi.addrs = ptr_to_u64(addrs); + info.tracing_multi.cookies = ptr_to_u64(cookies); + goto again; + } + } if (info.type == BPF_LINK_TYPE_PERF_EVENT) { switch (info.perf_event.type) { case BPF_PERF_EVENT_TRACEPOINT: @@ -1153,6 +1285,7 @@ static int do_show_link(int fd) free(cookies); free(offsets); free(addrs); + free(ids); close(fd); return 0; } From b4b8b334f6b535a86ab83f18d3d241fe01270bc3 Mon Sep 17 00:00:00 2001 From: Guillaume Maudoux Date: Tue, 30 Jun 2026 11:57:23 +0200 Subject: [PATCH 018/373] selftests/bpf: Mask socket type flags in mptcpify prog The mptcpify BPF prog upgrades eligible TCP sockets to MPTCP, but only when the socket type is exactly SOCK_STREAM. Its update_socket_protocol() hook runs on the raw type from userspace, before the socket core masks it with SOCK_TYPE_MASK, so the type may still carry SOCK_CLOEXEC or SOCK_NONBLOCK in its upper bits and the equality check fails. As a result, a socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0) -- what common libraries do by default -- is silently left as plain TCP. This was hit in practice with curl. Since mptcpify.c is referenced as example code for enabling MPTCP transparently, the same mistake is likely to be copied into real deployments where it fails the same way and is hard to diagnose. Mask the type before comparing, mirroring the socket core. Extend the test to also create the server with SOCK_CLOEXEC set; the same masking is applied to start_server_addr() so a flagged type still listens. Fixes: ddba122428a7 ("selftests/bpf: Add mptcpify test") Signed-off-by: Guillaume Maudoux Signed-off-by: Andrii Nakryiko Reviewed-by: Matthieu Baerts (NGI0) Link: https://lore.kernel.org/bpf/20260630095723.564392-1-layus.on@gmail.com --- tools/testing/selftests/bpf/network_helpers.c | 4 ++-- tools/testing/selftests/bpf/network_helpers.h | 5 +++++ tools/testing/selftests/bpf/prog_tests/mptcp.c | 13 ++++++++++--- tools/testing/selftests/bpf/progs/bpf_tracing_net.h | 3 +++ tools/testing/selftests/bpf/progs/mptcpify.c | 2 +- 5 files changed, 21 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/network_helpers.c b/tools/testing/selftests/bpf/network_helpers.c index b82f572641b7..db935a9d9fc1 100644 --- a/tools/testing/selftests/bpf/network_helpers.c +++ b/tools/testing/selftests/bpf/network_helpers.c @@ -111,7 +111,7 @@ int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t a if (settimeo(fd, opts->timeout_ms)) goto error_close; - if (type == SOCK_STREAM && + if ((type & SOCK_TYPE_MASK) == SOCK_STREAM && setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on))) { log_err("Failed to enable SO_REUSEADDR"); goto error_close; @@ -128,7 +128,7 @@ int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t a goto error_close; } - if (type == SOCK_STREAM) { + if ((type & SOCK_TYPE_MASK) == SOCK_STREAM) { if (listen(fd, opts->backlog ? MAX(opts->backlog, 0) : 1) < 0) { log_err("Failed to listed on socket"); goto error_close; diff --git a/tools/testing/selftests/bpf/network_helpers.h b/tools/testing/selftests/bpf/network_helpers.h index 79a010c88e11..75133119c04a 100644 --- a/tools/testing/selftests/bpf/network_helpers.h +++ b/tools/testing/selftests/bpf/network_helpers.h @@ -25,6 +25,11 @@ typedef __u16 __sum16; #define VIP_NUM 5 #define MAGIC_BYTES 123 +/* include/linux/net.h */ +#ifndef SOCK_TYPE_MASK +#define SOCK_TYPE_MASK 0xf +#endif + struct network_helper_opts { int timeout_ms; int proto; diff --git a/tools/testing/selftests/bpf/prog_tests/mptcp.c b/tools/testing/selftests/bpf/prog_tests/mptcp.c index 8fade8bdc451..32dfc1c511af 100644 --- a/tools/testing/selftests/bpf/prog_tests/mptcp.c +++ b/tools/testing/selftests/bpf/prog_tests/mptcp.c @@ -264,7 +264,7 @@ static int verify_mptcpify(int server_fd, int client_fd) return err; } -static int run_mptcpify(int cgroup_fd) +static int run_mptcpify(int cgroup_fd, int type) { int server_fd, client_fd, err = 0; struct mptcpify *mptcpify_skel; @@ -280,7 +280,7 @@ static int run_mptcpify(int cgroup_fd) goto out; /* without MPTCP */ - server_fd = start_server(AF_INET, SOCK_STREAM, NULL, 0, 0); + server_fd = start_server(AF_INET, type, NULL, 0, 0); if (!ASSERT_GE(server_fd, 0, "start_server")) { err = -EIO; goto out; @@ -317,7 +317,14 @@ static void test_mptcpify(void) if (!ASSERT_OK_PTR(netns, "netns_new")) goto fail; - ASSERT_OK(run_mptcpify(cgroup_fd), "run_mptcpify"); + ASSERT_OK(run_mptcpify(cgroup_fd, SOCK_STREAM), "run_mptcpify"); + /* userspace sets flags such as SOCK_CLOEXEC together with the type; + * the BPF prog must still upgrade the socket to MPTCP. See + * update_socket_protocol() in net/socket.c, which runs before the + * type is masked with SOCK_TYPE_MASK. + */ + ASSERT_OK(run_mptcpify(cgroup_fd, SOCK_STREAM | SOCK_CLOEXEC), + "run_mptcpify_cloexec"); fail: netns_free(netns); diff --git a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h index d8dacef37c16..c4b438854565 100644 --- a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h +++ b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h @@ -8,6 +8,9 @@ #define AF_INET 2 #define AF_INET6 10 +/* include/linux/net.h */ +#define SOCK_TYPE_MASK 0xf + #define SOL_SOCKET 1 #define SO_REUSEADDR 2 #define SO_SNDBUF 7 diff --git a/tools/testing/selftests/bpf/progs/mptcpify.c b/tools/testing/selftests/bpf/progs/mptcpify.c index cbdc730c3a47..e3f8cb54dbe9 100644 --- a/tools/testing/selftests/bpf/progs/mptcpify.c +++ b/tools/testing/selftests/bpf/progs/mptcpify.c @@ -15,7 +15,7 @@ int BPF_PROG(mptcpify, int family, int type, int protocol) return protocol; if ((family == AF_INET || family == AF_INET6) && - type == SOCK_STREAM && + (type & SOCK_TYPE_MASK) == SOCK_STREAM && (!protocol || protocol == IPPROTO_TCP)) { return IPPROTO_MPTCP; } From 475b59db3bd5c7717b5441481ac06f226815cb0a Mon Sep 17 00:00:00 2001 From: Tiezhu Yang Date: Wed, 1 Jul 2026 11:51:07 +0800 Subject: [PATCH 019/373] bpf: Use BPF_CALL_IMM macro consistently in bpf_do_misc_fixups In bpf_do_misc_fixups(), the conversion from a function address to a BPF immediate value is handled using the BPF_CALL_IMM macro inside the 'patch_map_ops_generic' label block. However, immediately following it in the 'patch_call_imm' label block, the immediate value is calculated manually by subtracting __bpf_call_base from fn->func. Inspired by KaFai Wan's review comments on fixing helper call offsets, use the BPF_CALL_IMM macro in 'patch_call_imm' as well to clean this up. This removes the redundant manual pointer arithmetic and ensures coding style consistency across adjacent label blocks within the same function. Signed-off-by: Tiezhu Yang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260701035107.8069-1-yangtiezhu@loongson.cn --- kernel/bpf/fixups.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 3cf2cc6e3ab6..12a8a4eb757f 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -2338,7 +2338,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) func_id_name(insn->imm), insn->imm); return -EFAULT; } - insn->imm = fn->func - __bpf_call_base; + insn->imm = BPF_CALL_IMM(fn->func); next_insn: if (subprogs[cur_subprog + 1].start == i + delta + 1) { subprogs[cur_subprog].stack_depth += stack_depth_extra; From 27a0f3635d862919dd7e5e93e19f3f5d1b240e57 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Wed, 1 Jul 2026 15:14:22 +0800 Subject: [PATCH 020/373] selftests/bpf: Fix test_maps sockmap failure test_maps fails in the sockmap test because sockmap_verdict_prog.c drops the packet when the first 8 bytes are not directly accessible: if (data + 8 > data_end) return SK_DROP; The blamed commit removed bpf_skb_pull_data() from the stream parser program so that the parser no longer modifies the skb. That was needed, but it also removed an implicit side effect: bpf_skb_pull_data() linearized enough of the skb for later direct packet access. In this test, the send side goes through the sockmap SK_MSG path. The skb can have skb->len == 20 while its linear area is empty, so the verdict program sees data == data_end and drops the packet even though the payload length is sufficient. Keep the parser read-only, and pull the first 8 bytes in the verdict program before reading or writing them. Reload data/data_end after bpf_skb_pull_data() as required. Fixes: 22a0cc10dacb ("selftests/bpf: don't modify the skb in the strparser parser prog") Reported-by: Ihor Solodrai Signed-off-by: Jiayuan Chen Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260701071501.39628-1-jiayuan.chen@linux.dev Closes: https://lore.kernel.org/bpf/e3a91acd-2b4d-4e93-a3bb-a0e9ee5ede0f@linux.dev/ --- .../selftests/bpf/progs/sockmap_verdict_prog.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c b/tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c index 0660f29dca95..3177bc5b733a 100644 --- a/tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c +++ b/tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c @@ -44,8 +44,18 @@ int bpf_prog2(struct __sk_buff *skb) __sink(lport); __sink(rport); - if (data + 8 > data_end) - return SK_DROP; + if (data + 8 > data_end) { + if (bpf_skb_pull_data(skb, 8)) + return SK_DROP; + + data = (void *)(long)skb->data; + data_end = (void *)(long)skb->data_end; + + if (data + 8 > data_end) + return SK_DROP; + + d = data; + } map = d[0]; sk = d[1]; From 2ce3f548cfc6a1fe4c53479cf8a21931cdfd51d8 Mon Sep 17 00:00:00 2001 From: Sechang Lim Date: Wed, 1 Jul 2026 08:07:51 +0000 Subject: [PATCH 021/373] bpf,lsm: Drop bpf_prog_free from sleepable_lsm_hooks __bpf_prog_put_rcu() is the call_rcu() callback for non-sleepable programs. security_bpf_prog_free() called from there fires bpf_prog_free in softirq; if a sleepable LSM prog is attached to that hook, might_fault() BUGs: BUG: sleeping function called from invalid context in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 5038 preempt_count: 101, expected: 0 Call Trace: __bpf_prog_enter_sleepable+0x1cd/0x320 kernel/bpf/trampoline.c:1255 bpf_trampoline_6442549705+0x53/0xd7 security_bpf_prog_free+0xde/0x130 security/security.c:5465 __bpf_prog_put_rcu+0xab/0xd0 kernel/bpf/syscall.c:2365 rcu_do_batch kernel/rcu/tree.c:2617 [inline] handle_softirqs+0x236/0x800 kernel/softirq.c:622 The call_rcu/call_rcu_tasks_trace split reflects the freed program's sleepability, not that of any attached observer. security_bpf_prog_free() also frees prog->aux->security, which has to stay after the grace period, so drop bpf_prog_free from sleepable_lsm_hooks rather than move the call. Non-sleepable observers still run there. Fixes: 1b67772e4e3f ("bpf,lsm: Refactor bpf_prog_alloc/bpf_prog_free LSM hooks") Signed-off-by: Sechang Lim Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260701080757.1394144-1-rhkrqnwk98@gmail.com --- kernel/bpf/bpf_lsm.c | 1 - 1 file changed, 1 deletion(-) diff --git a/kernel/bpf/bpf_lsm.c b/kernel/bpf/bpf_lsm.c index 1433809bb166..3983b4ce73c8 100644 --- a/kernel/bpf/bpf_lsm.c +++ b/kernel/bpf/bpf_lsm.c @@ -295,7 +295,6 @@ BTF_ID(func, bpf_lsm_bpf_map_create) BTF_ID(func, bpf_lsm_bpf_map_free) BTF_ID(func, bpf_lsm_bpf_prog) BTF_ID(func, bpf_lsm_bpf_prog_load) -BTF_ID(func, bpf_lsm_bpf_prog_free) BTF_ID(func, bpf_lsm_bpf_token_create) BTF_ID(func, bpf_lsm_bpf_token_free) BTF_ID(func, bpf_lsm_bpf_token_cmd) From d262a25d8b58b697b2a1a2fe4b0078446cc00e11 Mon Sep 17 00:00:00 2001 From: Yuan Chen Date: Wed, 10 Jun 2026 22:50:53 +0800 Subject: [PATCH 022/373] libbpf: Skip bpf_object__probe_loading() when BPF token is in use bpf_object__probe_loading() tries to load trivial SOCKET_FILTER and TRACEPOINT programs to verify the BPF environment works. When a BPF token is in use with restricted program type permissions, these probe loads may fail because the token does not allow the specific program types, even though BPF loading is perfectly functional. Fix by skipping the probe when a token FD is present: BPF token creation itself proves the kernel has a working BPF subsystem. Real BPF issues will be caught during actual program and map loading. Signed-off-by: Yuan Chen Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260610145059.113412-2-chenyuan_fl@163.com --- tools/lib/bpf/libbpf.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 7162146280a8..f88b7c8de304 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -5172,12 +5172,8 @@ bpf_object__probe_loading(struct bpf_object *obj) BPF_EXIT_INSN(), }; int ret, insn_cnt = ARRAY_SIZE(insns); - LIBBPF_OPTS(bpf_prog_load_opts, opts, - .token_fd = obj->token_fd, - .prog_flags = obj->token_fd ? BPF_F_TOKEN_FD : 0, - ); - if (obj->gen_loader) + if (obj->gen_loader || obj->token_fd) return 0; ret = bump_rlimit_memlock(); @@ -5186,9 +5182,9 @@ bpf_object__probe_loading(struct bpf_object *obj) errstr(ret)); /* make sure basic loading works */ - ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &opts); + ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, NULL); if (ret < 0) - ret = bpf_prog_load(BPF_PROG_TYPE_TRACEPOINT, NULL, "GPL", insns, insn_cnt, &opts); + ret = bpf_prog_load(BPF_PROG_TYPE_TRACEPOINT, NULL, "GPL", insns, insn_cnt, NULL); if (ret < 0) { ret = errno; pr_warn("Error in %s(): %s. Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.\n", From 0b58988cacfc91604c4b5be2c3295f8aac0ee3d0 Mon Sep 17 00:00:00 2001 From: JP Kobryn Date: Fri, 26 Jun 2026 10:20:26 -0700 Subject: [PATCH 023/373] Documentation/bpf: make it clear that kfuncs should be non-static The kfunc documentation mentions how the macro __bpf_kfunc prevents inlining for static functions. This makes it sound like static kfuncs are acceptable. Although static kfuncs may happen to work, it is by chance that the compiler chose not to rename these functions and BTF resolution still succeeds. Make it clear in the documentation why kfuncs should not be declared static. First, remove wording that makes it sound like static is ok. Then point out the external naming needed for BTF resolution. Finally point out that sparse may warn on unreferenced kfuncs and that this warning can be ignored. Signed-off-by: JP Kobryn Acked-by: Roman Gushchin Acked-by: Yonghong Song Link: https://lore.kernel.org/r/20260626172026.7327-1-jp.kobryn@linux.dev Signed-off-by: Alexei Starovoitov --- Documentation/bpf/kfuncs.rst | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 4c814ff6061e..c801a330aece 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -273,22 +273,29 @@ flags on a set of kfuncs as follows:: BTF_KFUNCS_END(bpf_task_set) This set encodes the BTF ID of each kfunc listed above, and encodes the flags -along with it. Ofcourse, it is also allowed to specify no flags. +along with it. It is also allowed to specify no flags. kfunc definitions should also always be annotated with the ``__bpf_kfunc`` -macro. This prevents issues such as the compiler inlining the kfunc if it's a -static kernel function, or the function being elided in an LTO build as it's -not used in the rest of the kernel. Developers should not manually add -annotations to their kfunc to prevent these issues. If an annotation is -required to prevent such an issue with your kfunc, it is a bug and should be -added to the definition of the macro so that other kfuncs are similarly -protected. An example is given below:: +macro. This prevents issues such as the compiler inlining the kfunc, or the +function being elided in an LTO build as it's not used in the rest of the +kernel. Developers should not manually add annotations to their kfunc to prevent +these issues. If an annotation is required to prevent such an issue with your +kfunc, it is a bug and should be added to the definition of the macro so that +other kfuncs are similarly protected. An example is given below:: __bpf_kfunc struct task_struct *bpf_get_task_pid(s32 pid) { ... } +Note that kfuncs must not be declared ``static``. A kfunc can be called from a +BPF program ``*.c`` file outside the compilation unit that defines it, so its +externally visible name must remain available for BTF ID lookup. ``static`` +linkage allows the compiler to rename the function, which can break this +BTF-based kfunc resolution. Further note that sparse may warn that an otherwise +unreferenced kfunc should be static. Such warnings should be ignored for kfunc +definitions. + 2.5.1 KF_ACQUIRE flag --------------------- From bb4e90e91ba19b598cbdd9a2161b893b86a3f637 Mon Sep 17 00:00:00 2001 From: luoliang Date: Thu, 2 Jul 2026 09:23:11 +0800 Subject: [PATCH 024/373] bpftool: Use btf_vlen()/btf_kind()/btf_kflag() helpers consistently The btf_vlen(), btf_kind() and btf_kflag() inline helpers defined in tools/lib/bpf/btf.h are thin wrappers around the BTF_INFO_VLEN(), BTF_INFO_KIND() and BTF_INFO_KFLAG() UAPI macros - each one simply returns the corresponding macro applied to t->info. bpftool already uses these helpers in most places, but 13 call sites in btf.c and btf_dumper.c still open-code the raw macros. Use the helpers consistently, matching the rest of bpftool as well as libbpf. No functional change. Signed-off-by: Liang Luo Signed-off-by: Andrii Nakryiko Reviewed-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20260702012311.2265001-1-luoliang@kylinos.cn --- tools/bpf/bpftool/btf.c | 13 ++++++------- tools/bpf/bpftool/btf_dumper.c | 14 +++++++------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/tools/bpf/bpftool/btf.c b/tools/bpf/bpftool/btf.c index 6ef908adf3a4..c9589026da8d 100644 --- a/tools/bpf/bpftool/btf.c +++ b/tools/bpf/bpftool/btf.c @@ -179,7 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, case BTF_KIND_STRUCT: case BTF_KIND_UNION: { const struct btf_member *m = (const void *)(t + 1); - __u32 i, vlen = BTF_INFO_VLEN(t->info); + __u32 i, vlen = btf_vlen(t); if (json_output) { jsonw_uint_field(w, "size", t->size); @@ -193,7 +193,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, const char *name = btf_str(btf, m->name_off); __u32 bit_off, bit_sz; - if (BTF_INFO_KFLAG(t->info)) { + if (btf_kflag(t)) { bit_off = BTF_MEMBER_BIT_OFFSET(m->offset); bit_sz = BTF_MEMBER_BITFIELD_SIZE(m->offset); } else { @@ -224,7 +224,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, } case BTF_KIND_ENUM: { const struct btf_enum *v = (const void *)(t + 1); - __u32 i, vlen = BTF_INFO_VLEN(t->info); + __u32 i, vlen = btf_vlen(t); const char *encoding; encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; @@ -300,8 +300,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, break; } case BTF_KIND_FWD: { - const char *fwd_kind = BTF_INFO_KFLAG(t->info) ? "union" - : "struct"; + const char *fwd_kind = btf_kflag(t) ? "union" : "struct"; if (json_output) jsonw_string_field(w, "fwd_kind", fwd_kind); @@ -322,7 +321,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, } case BTF_KIND_FUNC_PROTO: { const struct btf_param *p = (const void *)(t + 1); - __u32 i, vlen = BTF_INFO_VLEN(t->info); + __u32 i, vlen = btf_vlen(t); if (json_output) { jsonw_uint_field(w, "ret_type_id", t->type); @@ -365,7 +364,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, case BTF_KIND_DATASEC: { const struct btf_var_secinfo *v = (const void *)(t + 1); const struct btf_type *vt; - __u32 i, vlen = BTF_INFO_VLEN(t->info); + __u32 i, vlen = btf_vlen(t); if (json_output) { jsonw_uint_field(w, "size", t->size); diff --git a/tools/bpf/bpftool/btf_dumper.c b/tools/bpf/bpftool/btf_dumper.c index 9dc8425b1789..e4075824343f 100644 --- a/tools/bpf/bpftool/btf_dumper.c +++ b/tools/bpf/bpftool/btf_dumper.c @@ -476,8 +476,8 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id, if (!t) return -EINVAL; - kind_flag = BTF_INFO_KFLAG(t->info); - vlen = BTF_INFO_VLEN(t->info); + kind_flag = btf_kflag(t); + vlen = btf_vlen(t); jsonw_start_object(d->jw); m = (struct btf_member *)(t + 1); @@ -535,7 +535,7 @@ static int btf_dumper_datasec(const struct btf_dumper *d, __u32 type_id, if (!t) return -EINVAL; - vlen = BTF_INFO_VLEN(t->info); + vlen = btf_vlen(t); vsi = (struct btf_var_secinfo *)(t + 1); jsonw_start_object(d->jw); @@ -557,7 +557,7 @@ static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id, { const struct btf_type *t = btf__type_by_id(d->btf, type_id); - switch (BTF_INFO_KIND(t->info)) { + switch (btf_kind(t)) { case BTF_KIND_INT: return btf_dumper_int(t, bit_offset, data, d->jw, d->is_plain_text); @@ -631,7 +631,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id, t = btf__type_by_id(btf, type_id); - switch (BTF_INFO_KIND(t->info)) { + switch (btf_kind(t)) { case BTF_KIND_INT: case BTF_KIND_TYPEDEF: case BTF_KIND_FLOAT: @@ -661,7 +661,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id, break; case BTF_KIND_FWD: BTF_PRINT_ARG("%s %s ", - BTF_INFO_KFLAG(t->info) ? "union" : "struct", + btf_kflag(t) ? "union" : "struct", btf__name_by_offset(btf, t->name_off)); break; case BTF_KIND_VOLATILE: @@ -718,7 +718,7 @@ static int btf_dump_func(const struct btf *btf, char *func_sig, BTF_PRINT_ARG("%s(", btf__name_by_offset(btf, func->name_off)); else BTF_PRINT_ARG("("); - vlen = BTF_INFO_VLEN(func_proto->info); + vlen = btf_vlen(func_proto); for (i = 0; i < vlen; i++) { struct btf_param *arg = &((struct btf_param *)(func_proto + 1))[i]; From d7123afeecc86026fdab830af977a79f65a77ab8 Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:25 -0400 Subject: [PATCH 025/373] selftests/bpf: libarena: Replace leftover st_ prefix with test_ The st_ (selftests_) prefix is confusing and has been replaced with the more descriptive test_. However, last patch did not properly move all files to the new prefix. Rename the existing files to complete the move. Acked-by: Ihor Solodrai Signed-off-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260706181730.21731-2-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/{st_asan_buddy.bpf.c => test_asan_buddy.bpf.c} | 2 +- .../libarena/selftests/{st_asan_common.h => test_asan_common.h} | 0 .../bpf/libarena/selftests/{st_buddy.bpf.c => test_buddy.bpf.c} | 0 3 files changed, 1 insertion(+), 1 deletion(-) rename tools/testing/selftests/bpf/libarena/selftests/{st_asan_buddy.bpf.c => test_asan_buddy.bpf.c} (99%) rename tools/testing/selftests/bpf/libarena/selftests/{st_asan_common.h => test_asan_common.h} (100%) rename tools/testing/selftests/bpf/libarena/selftests/{st_buddy.bpf.c => test_buddy.bpf.c} (100%) diff --git a/tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c similarity index 99% rename from tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c rename to tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c index 686caba2c643..256d62a03ce7 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/st_asan_buddy.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c @@ -12,7 +12,7 @@ extern struct buddy __arena buddy; #ifdef BPF_ARENA_ASAN -#include "st_asan_common.h" +#include "test_asan_common.h" static __always_inline int asan_test_buddy_oob_single(size_t alloc_size) { diff --git a/tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h b/tools/testing/selftests/bpf/libarena/selftests/test_asan_common.h similarity index 100% rename from tools/testing/selftests/bpf/libarena/selftests/st_asan_common.h rename to tools/testing/selftests/bpf/libarena/selftests/test_asan_common.h diff --git a/tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c similarity index 100% rename from tools/testing/selftests/bpf/libarena/selftests/st_buddy.bpf.c rename to tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c From 857071efc362d4641673c1fbf7e8a2d7a2408e75 Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:26 -0400 Subject: [PATCH 026/373] selftests/bpf: libarena: Fix can-loop zero variable definition BPF can_loop based loops require the index variable to stay imprecise. This means we must initialize them from a currently imprecise variable instead of directly assigning 0 to them, like so: static volatile u32 zero = 0; for (i = zero; i < NUM_LOOPS; i++) { /* loop body */ } The libarena implementation of this technique is currently faulty. For the technique to work, the variable must not be in a map. This includes the .rodata DATASEC map used for const variables. However, libarena still defines the zero variable as constant. Modify the zero variable definition into a volatile variable. This change adds a complication caused by the compiler optimizing array derefences from for (i = zero; i < NUM_LOOPS; i++) { val = *(ptr + i); } into for (i = zero; i < NUM_LOOPS; i++) { val = *ptr++; } and causing verification failures. Use the barrier_var() clobber macro to prevent this optimization from taking place. Using barrier_var() is the only way to break the optimization, as annotating the index as volatile does not suffice. After that, remove the bpf_for() invocations introduced in libarena for parallel spmc testing. Reported-by: Eduard Zingerman Signed-off-by: Emil Tsalapatis Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260706181730.21731-3-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/libarena/include/libarena/common.h | 2 +- .../bpf/libarena/selftests/test_asan_buddy.bpf.c | 8 ++++++-- .../selftests/bpf/libarena/selftests/test_buddy.bpf.c | 8 ++++++-- .../bpf/libarena/selftests/test_parallel_spmc.bpf.c | 9 ++++----- tools/testing/selftests/bpf/libarena/src/common.bpf.c | 3 +-- 5 files changed, 18 insertions(+), 12 deletions(-) diff --git a/tools/testing/selftests/bpf/libarena/include/libarena/common.h b/tools/testing/selftests/bpf/libarena/include/libarena/common.h index a3eb1641ac36..931ace9a49e2 100644 --- a/tools/testing/selftests/bpf/libarena/include/libarena/common.h +++ b/tools/testing/selftests/bpf/libarena/include/libarena/common.h @@ -43,7 +43,7 @@ struct { * imprecise. To force the variable to be imprecise, initialize it with * the opaque volatile variable 0 instead of the constant 0. */ -extern const volatile u32 zero; +volatile u32 zero __weak; extern volatile u64 asan_violated; int arena_fls(__u64 word); diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c index 256d62a03ce7..3266a28f53d7 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_asan_buddy.bpf.c @@ -154,7 +154,8 @@ __weak int asan_test_buddy_oob(void) size_t sizes[] = { 7, 8, 17, 18, 64, 256, 317, 512, 1024, }; - int ret, i; + int ret; + u32 i; ret = buddy_init(&buddy); if (ret) { @@ -163,6 +164,7 @@ __weak int asan_test_buddy_oob(void) } for (i = zero; i < sizeof(sizes) / sizeof(sizes[0]) && can_loop; i++) { + barrier_var(i); ret = asan_test_buddy_oob_single(sizes[i]); if (ret) { arena_stdout("%s:%d Failed for size %lu", __func__, @@ -190,7 +192,8 @@ __stderr("Call trace:\n" __weak int asan_test_buddy_uaf(void) { size_t sizes[] = { 16, 32, 64, 128, 256, 512, 1024, 16384 }; - int ret, i; + int ret; + u32 i; ret = buddy_init(&buddy); if (ret) { @@ -199,6 +202,7 @@ __weak int asan_test_buddy_uaf(void) } for (i = zero; i < sizeof(sizes) / sizeof(sizes[0]) && can_loop; i++) { + barrier_var(i); ret = asan_test_buddy_uaf_single(sizes[i]); if (ret) { arena_stdout("%s:%d Failed for size %lu", __func__, diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c index b45a306816c0..5628f0987012 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_buddy.bpf.c @@ -171,7 +171,8 @@ __weak int test_buddy_alloc_multiple(void) SEC("syscall") __weak int test_buddy_alignment(void) { - int ret, i; + int ret; + u32 i; ret = buddy_init(&buddy); if (ret) @@ -179,6 +180,7 @@ __weak int test_buddy_alignment(void) /* Allocate various sizes and check alignment */ for (i = zero; i < 17 && can_loop; i++) { + barrier_var(i); ptrs[i] = buddy_alloc(&buddy, alignment_sizes[i]); if (!ptrs[i]) { arena_stdout("alignment test: alloc failed for size %lu", @@ -198,8 +200,10 @@ __weak int test_buddy_alignment(void) } /* Free all allocations */ - for (i = zero; i < 17 && can_loop; i++) + for (i = zero; i < 17 && can_loop; i++) { + barrier_var(i); buddy_free(&buddy, ptrs[i]); + } buddy_destroy(&buddy); diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c index f08f2a92e194..5fa96eb74095 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_spmc.bpf.c @@ -155,7 +155,7 @@ int spmc_quiesce_on_owner(u64 epoch) { u64 i; - bpf_for(i, 0, TEST_SPMC_SYNC_SPINS) { + for (i = zero; i < TEST_SPMC_SYNC_SPINS && can_loop; i++) { if (test_abort) return -EINTR; if (smp_load_acquire(&owner_epoch) >= epoch) @@ -175,8 +175,7 @@ int spmc_quiesce_on_stealer(u64 epoch) int err = -ETIMEDOUT; target = STEALER_EPOCH(epoch); - bpf_for(i, 0, TEST_SPMC_SYNC_SPINS) { - + for (i = zero; i < TEST_SPMC_SYNC_SPINS && can_loop; i++) { if (test_abort) { err = -EINTR; break; @@ -391,7 +390,7 @@ int spmc_wait_for_stealers_to_start(u64 target) { u64 i; - bpf_for(i, 0, TEST_SPMC_SYNC_SPINS) { + for (i = zero; i < TEST_SPMC_SYNC_SPINS && can_loop; i++) { if (test_abort) return -EINTR; if (READ_ONCE(stealers_started) >= target) @@ -537,7 +536,7 @@ static int spmc_wait_for_round_steals(u64 target) arena_subprog_init(); - bpf_for(i, 0, TEST_SPMC_SYNC_SPINS) { + for (i = zero; i < TEST_SPMC_SYNC_SPINS && can_loop; i++) { if (test_abort) return -EINTR; if (round_steals >= target) diff --git a/tools/testing/selftests/bpf/libarena/src/common.bpf.c b/tools/testing/selftests/bpf/libarena/src/common.bpf.c index 50be57213dfb..1b4bb19b3c52 100644 --- a/tools/testing/selftests/bpf/libarena/src/common.bpf.c +++ b/tools/testing/selftests/bpf/libarena/src/common.bpf.c @@ -4,9 +4,8 @@ #include #include -const volatile u32 zero = 0; - struct buddy __arena buddy; +volatile u32 zero = 0; int arena_fls(__u64 word) { From 14c2b770d15d5b0d814cdef114de55e01a281e00 Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:27 -0400 Subject: [PATCH 027/373] selftests/bpf: libarena: Clean up allocation state before buddy tests Summary: The buddy allocator requires the global BPF buddy allocator to not be already initialized. However, the test currently merely resets the allocator before the buddy tests instead of destroying it, and the test worked because the buddy test happened to run first. Properly destroy the allocator instead of resetting it. Fixes: b1487dc1b181 ("selftests/bpf: Add selftests for libarena buddy allocator") Signed-off-by: Emil Tsalapatis Acked-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260706181730.21731-4-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/libarena/src/common.bpf.c | 6 ++++++ tools/testing/selftests/bpf/prog_tests/libarena.c | 8 ++++++-- tools/testing/selftests/bpf/prog_tests/libarena_asan.c | 8 ++++++-- 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/libarena/src/common.bpf.c b/tools/testing/selftests/bpf/libarena/src/common.bpf.c index 1b4bb19b3c52..569f0f64d518 100644 --- a/tools/testing/selftests/bpf/libarena/src/common.bpf.c +++ b/tools/testing/selftests/bpf/libarena/src/common.bpf.c @@ -37,6 +37,12 @@ __weak int arena_buddy_reset(void) return buddy_init(&buddy); } +SEC("syscall") +__weak int arena_buddy_destroy(void) +{ + return buddy_destroy(&buddy); +} + __weak void __arena *arena_malloc(size_t size) { return buddy_alloc(&buddy, size); diff --git a/tools/testing/selftests/bpf/prog_tests/libarena.c b/tools/testing/selftests/bpf/prog_tests/libarena.c index 61ea68dce410..ba5a5a50f7c0 100644 --- a/tools/testing/selftests/bpf/prog_tests/libarena.c +++ b/tools/testing/selftests/bpf/prog_tests/libarena.c @@ -15,7 +15,12 @@ static void run_libarena_test(struct libarena *skel, struct bpf_program *prog, { int ret; - if (!strstr(name, "test_buddy")) { + if (strstr(name, "test_buddy")) { + /* Buddy tests initialize the allocator directly. */ + ret = libarena_run_prog(bpf_program__fd(skel->progs.arena_buddy_destroy)); + if (!ASSERT_OK(ret, "arena_buddy_destroy")) + return; + } else { ret = libarena_run_prog(bpf_program__fd(skel->progs.arena_buddy_reset)); if (!ASSERT_OK(ret, "arena_buddy_reset")) return; @@ -24,7 +29,6 @@ static void run_libarena_test(struct libarena *skel, struct bpf_program *prog, ret = libarena_run_prog(bpf_program__fd(prog)); ASSERT_OK(ret, name); - } static void *run_libarena_parallel_prog(void *arg) diff --git a/tools/testing/selftests/bpf/prog_tests/libarena_asan.c b/tools/testing/selftests/bpf/prog_tests/libarena_asan.c index d59d9dd12ef2..f897405f701d 100644 --- a/tools/testing/selftests/bpf/prog_tests/libarena_asan.c +++ b/tools/testing/selftests/bpf/prog_tests/libarena_asan.c @@ -17,7 +17,12 @@ static void run_libarena_asan_test(struct libarena_asan *skel, { int ret; - if (!strstr(name, "test_buddy")) { + if (strstr(name, "test_buddy")) { + /* Buddy tests initialize the allocator directly. */ + ret = libarena_run_prog(bpf_program__fd(skel->progs.arena_buddy_destroy)); + if (!ASSERT_OK(ret, "arena_buddy_destroy")) + return; + } else { ret = libarena_run_prog(bpf_program__fd(skel->progs.arena_buddy_reset)); if (!ASSERT_OK(ret, "arena_buddy_reset")) return; @@ -90,4 +95,3 @@ void test_libarena_asan(void) return; } - From 5a93b10f6c47a55d6b578983ca76d98b8ae3d268 Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:28 -0400 Subject: [PATCH 028/373] selftests/bpf: Add arena-based bitmap data structure Add an arena-based word-aligned bitmap data struture. The structure is useful as a building block, e.g., sched-ext uses it to represent cpumask structures. Signed-off-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260706181730.21731-5-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/libarena/include/libarena/bitmap.h | 34 +++ .../selftests/bpf/libarena/src/bitmap.bpf.c | 245 ++++++++++++++++++ 2 files changed, 279 insertions(+) create mode 100644 tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h create mode 100644 tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c diff --git a/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h b/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h new file mode 100644 index 000000000000..cf6b63f5d9a4 --- /dev/null +++ b/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h @@ -0,0 +1,34 @@ +#pragma once + +#define BITS_PER_BYTE 8 +#define BYTES_TO_BITS(nb) ((nb) * BITS_PER_BYTE) + +#define BITS_PER_LONG_LONG (sizeof(long long) * BITS_PER_BYTE) +#define BITS_TO_LONG_LONGS(nr) (((nr) + BITS_PER_LONG_LONG - 1) / BITS_PER_LONG_LONG) +#define BIT_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG)) +#define BIT_WORD(nr) ((nr) / BITS_PER_LONG_LONG) + +struct bitmap { + u64 bits[0]; +}; + +struct bitmap __arena *bmp_alloc(size_t bits); +void bmp_free(struct bitmap __arena *bmp); + +void __bmp_set_bit(u32 bit, struct bitmap __arena *bmp); +void __bmp_clear_bit(u32 bit, struct bitmap __arena *bmp); +void bmp_set_bit(u32 bit, struct bitmap __arena *bmp); +void bmp_clear_bit(u32 bit, struct bitmap __arena *bmp); +bool bmp_test_bit(u32 bit, struct bitmap __arena *bmp); +bool bmp_test_and_clear_bit(u32 bit, struct bitmap __arena *bmp); +bool bmp_test_and_set_bit(u32 bit, struct bitmap __arena *bmp); + +void bmp_clear(size_t bits, struct bitmap __arena *bmp); +void bmp_and(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2); +void bmp_or(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2); +bool bmp_empty(size_t bits, struct bitmap __arena *bmp); +void bmp_copy(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src); + +bool bmp_intersects(size_t bits, struct bitmap __arena *arg1, struct bitmap __arena *arg2); +bool bmp_subset(size_t bits, struct bitmap __arena *big, struct bitmap __arena *small); +void bmp_print(size_t bits, struct bitmap __arena *bmp); diff --git a/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c new file mode 100644 index 000000000000..80e814401fb9 --- /dev/null +++ b/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c @@ -0,0 +1,245 @@ +// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause +/* + * Copyright (c) 2025-2026 Meta Platforms, Inc. and affiliates. + * Copyright (c) 2025-2026 Emil Tsalapatis + */ + +#include + +#include +#include + +__weak +struct bitmap __arena *bmp_alloc(size_t bits) +{ + struct bitmap __arena *bmp; + size_t size = BITS_TO_LONG_LONGS(bits) * sizeof(bmp->bits[0]); + + /* Assume long-aligned masks. */ + if (bits % BITS_PER_LONG_LONG) + return NULL; + + bmp = (struct bitmap __arena *)arena_malloc(size); + if (!bmp) + return NULL; + + bmp_clear(bits, bmp); + + return bmp; +} + +__weak +void bmp_free(struct bitmap __arena *bmp) +{ + arena_free(bmp); +} + +__weak +void __bmp_set_bit(u32 bit, struct bitmap __arena *bmp) +{ + bmp->bits[BIT_WORD(bit)] |= BIT_MASK(bit); +} + +__weak +void __bmp_clear_bit(u32 bit, struct bitmap __arena *bmp) +{ + bmp->bits[BIT_WORD(bit)] &= ~BIT_MASK(bit); +} + +__weak +bool bmp_test_bit(u32 bit, struct bitmap __arena *bmp) +{ + return bmp->bits[BIT_WORD(bit)] & BIT_MASK(bit); +} + +__weak +bool bmp_test_and_clear_bit(u32 bit, struct bitmap __arena *bmp) +{ + u64 val = BIT_MASK(bit); + u32 idx = BIT_WORD(bit); + u64 old, new, actual; + + do { + old = bmp->bits[idx]; + + if (!(old & val)) + return false; + + new = old & ~val; + actual = cmpxchg(&bmp->bits[idx], old, new); + + if (actual == old) + return true; + + } while (can_loop); + + return false; +} + +__weak +bool bmp_test_and_set_bit(u32 bit, struct bitmap __arena *bmp) +{ + u64 val = BIT_MASK(bit); + u32 idx = BIT_WORD(bit); + u64 old, new, actual; + + do { + old = bmp->bits[idx]; + + if ((old & val)) + return true; + + new = old | val; + actual = cmpxchg(&bmp->bits[idx], old, new); + + if (actual == old) + return false; + + } while (can_loop); + + return false; +} + +__weak +void bmp_clear_bit(u32 bit, struct bitmap __arena *bmp) +{ + u64 val = BIT_MASK(bit); + u32 idx = BIT_WORD(bit); + u64 old, new, actual; + + do { + old = bmp->bits[idx]; + new = old & ~val; + actual = cmpxchg(&bmp->bits[idx], old, new); + + } while (actual != old && can_loop); +} + +__weak +void bmp_set_bit(u32 bit, struct bitmap __arena *bmp) +{ + u64 val = BIT_MASK(bit); + u32 idx = BIT_WORD(bit); + u64 old, new, actual; + + do { + old = bmp->bits[idx]; + new = old | val; + actual = cmpxchg(&bmp->bits[idx], old, new); + + } while (actual != old && can_loop); +} + +__weak +void bmp_clear(size_t bits, struct bitmap __arena *bmp) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) + bmp->bits[i] = 0; +} + +static __always_inline u64 bmp_last_word_mask(size_t bits) +{ + u32 rem = bits % BITS_PER_LONG_LONG; + + return rem ? (1ULL << rem) - 1 : ~0ULL; +} + +__weak +void bmp_and(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) + dst->bits[i] = src1->bits[i] & src2->bits[i]; + + if (nwords && bits % BITS_PER_LONG_LONG) + dst->bits[nwords - 1] &= bmp_last_word_mask(bits); +} + +__weak +void bmp_or(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) + dst->bits[i] = src1->bits[i] | src2->bits[i]; + + if (nwords && bits % BITS_PER_LONG_LONG) + dst->bits[nwords - 1] &= bmp_last_word_mask(bits); +} + +__weak +bool bmp_empty(size_t bits, struct bitmap __arena *bmp) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) { + u64 mask = (i == nwords - 1) ? bmp_last_word_mask(bits) : ~0ULL; + + if (bmp->bits[i] & mask) + return false; + } + + return true; +} + +__weak +void bmp_copy(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) + dst->bits[i] = src->bits[i]; + + if (nwords && bits % BITS_PER_LONG_LONG) + dst->bits[nwords - 1] &= bmp_last_word_mask(bits); +} + +__weak +bool bmp_subset(size_t bits, struct bitmap __arena *big, struct bitmap __arena *small) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) { + u64 mask = (i == nwords - 1) ? bmp_last_word_mask(bits) : ~0ULL; + + if (~big->bits[i] & small->bits[i] & mask) + return false; + } + + return true; +} + +__weak +bool bmp_intersects(size_t bits, struct bitmap __arena *arg1, struct bitmap __arena *arg2) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) { + u64 mask = (i == nwords - 1) ? bmp_last_word_mask(bits) : ~0ULL; + + if (arg1->bits[i] & arg2->bits[i] & mask) + return true; + } + + return false; +} + +__weak +void bmp_print(size_t bits, struct bitmap __arena *bmp) +{ + size_t nwords = BITS_TO_LONG_LONGS(bits); + volatile u32 i; + + for (i = zero; i < nwords && can_loop; i++) + arena_stderr("%016llx ", bmp->bits[i]); +} From 5a903a4f73eec60804a702d624fe0a72d823266d Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:29 -0400 Subject: [PATCH 029/373] selftests/bpf: libarena: Add bitmap selftests Add testing for the new arena bitmap data structure. Signed-off-by: Emil Tsalapatis Acked-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260706181730.21731-6-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/libarena/selftests/test_bitmap.bpf.c | 394 ++++++++++++++++++ 1 file changed, 394 insertions(+) create mode 100644 tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c new file mode 100644 index 000000000000..e7d32f44d687 --- /dev/null +++ b/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c @@ -0,0 +1,394 @@ +#include + +#include +#include + +#define TEST_BITS (2 * BITS_PER_LONG_LONG) +#define TEST_WORDS BITS_TO_LONG_LONGS(TEST_BITS) +#define MID_BIT (BITS_PER_LONG_LONG + 1) +#define LAST_BIT (TEST_BITS - 1) + +static void test_bmp_setall(struct bitmap __arena *bmp) +{ + volatile u32 i; + + for (i = zero; i < TEST_WORDS && can_loop; i++) + bmp->bits[i] = ~0ULL; +} + +SEC("syscall") +__weak int test_bitmap_alloc_free(void) +{ + struct bitmap __arena *bmp; + + bmp = bmp_alloc(TEST_BITS); + if (!bmp) + return -ENOMEM; + + if (!bmp_empty(TEST_BITS, bmp)) + goto err; + + __bmp_set_bit(LAST_BIT, bmp); + if (!bmp_test_bit(LAST_BIT, bmp)) + goto err; + + __bmp_clear_bit(LAST_BIT, bmp); + if (bmp_test_bit(LAST_BIT, bmp)) + goto err; + + bmp_free(bmp); + return 0; + +err: + bmp_free(bmp); + return -EINVAL; +} + +SEC("syscall") +__weak int test_bitmap_bit_ops(void) +{ + struct bitmap __arena *bmp; + + bmp = bmp_alloc(TEST_BITS); + if (!bmp) + return -ENOMEM; + + __bmp_set_bit(0, bmp); + if (!bmp_test_bit(0, bmp)) + goto err; + + __bmp_set_bit(MID_BIT, bmp); + if (!bmp_test_bit(MID_BIT, bmp)) + goto err; + + __bmp_set_bit(LAST_BIT, bmp); + if (!bmp_test_bit(LAST_BIT, bmp)) + goto err; + + if (bmp_test_bit(MID_BIT - 1, bmp)) + goto err; + + __bmp_clear_bit(MID_BIT, bmp); + if (bmp_test_bit(MID_BIT, bmp)) + goto err; + + if (!bmp_test_bit(0, bmp)) + goto err; + + if (!bmp_test_bit(LAST_BIT, bmp)) + goto err; + + __bmp_clear_bit(0, bmp); + __bmp_clear_bit(LAST_BIT, bmp); + if (!bmp_empty(TEST_BITS, bmp)) + goto err; + + if (bmp->bits[0]) + goto err; + + if (bmp->bits[1]) + goto err; + + bmp_free(bmp); + return 0; + +err: + bmp_free(bmp); + return -EINVAL; +} + +static bool test_bitmap_test_and_clear_single(struct bitmap __arena *bmp, size_t ind) +{ + if (bmp_test_and_clear_bit(ind, bmp)) + return false; + + __bmp_set_bit(ind, bmp); + + if (!bmp_test_and_clear_bit(ind, bmp)) + return false; + + if (bmp_test_bit(ind, bmp)) + return false; + + if (bmp_test_and_clear_bit(ind, bmp)) + return false; + + return true; +} + +static bool test_bitmap_test_and_set_single(struct bitmap __arena *bmp, size_t ind) +{ + if (bmp_test_and_set_bit(ind, bmp)) + return false; + + if (!bmp_test_and_set_bit(ind, bmp)) + return false; + + if (!bmp_test_bit(ind, bmp)) + return false; + + __bmp_clear_bit(ind, bmp); + + if (bmp_test_and_set_bit(ind, bmp)) + return false; + + return true; +} + +SEC("syscall") +__weak int test_bitmap_test_and_clear_bit(void) +{ + struct bitmap __arena *bmp; + + bmp = bmp_alloc(TEST_BITS); + if (!bmp) + return -ENOMEM; + + if (!test_bitmap_test_and_clear_single(bmp, 0)) + goto err; + + if (!test_bitmap_test_and_clear_single(bmp, MID_BIT)) + goto err; + + if (!test_bitmap_test_and_clear_single(bmp, LAST_BIT)) + goto err; + + if (!bmp_empty(TEST_BITS, bmp)) + goto err; + + bmp_free(bmp); + return 0; + +err: + bmp_free(bmp); + return -EINVAL; +} + +SEC("syscall") +__weak int test_bitmap_test_and_set_bit(void) +{ + struct bitmap __arena *bmp; + + bmp = bmp_alloc(TEST_BITS); + if (!bmp) + return -ENOMEM; + + if (!test_bitmap_test_and_set_single(bmp, 0)) + goto err; + + if (!test_bitmap_test_and_set_single(bmp, MID_BIT)) + goto err; + + if (!test_bitmap_test_and_set_single(bmp, LAST_BIT)) + goto err; + + bmp_free(bmp); + return 0; + +err: + bmp_free(bmp); + return -EINVAL; +} + + +SEC("syscall") +__weak int test_bitmap_and(void) +{ + struct bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; + + src1 = bmp_alloc(TEST_BITS); + src2 = bmp_alloc(TEST_BITS); + dst = bmp_alloc(TEST_BITS); + if (!src1 || !src2 || !dst) + goto err; + + test_bmp_setall(dst); + + __bmp_set_bit(0, src1); + __bmp_set_bit(MID_BIT, src1); + __bmp_set_bit(LAST_BIT, src1); + + __bmp_set_bit(MID_BIT, src2); + __bmp_set_bit(LAST_BIT, src2); + + bmp_and(TEST_BITS, dst, src1, src2); + + if (bmp_test_bit(0, dst)) + goto err; + if (!bmp_test_bit(MID_BIT, dst)) + goto err; + if (!bmp_test_bit(LAST_BIT, dst)) + goto err; + + if (dst->bits[0]) + goto err; + if (dst->bits[1] != (BIT_MASK(MID_BIT) | BIT_MASK(LAST_BIT))) + goto err; + + bmp_free(src1); + bmp_free(src2); + bmp_free(dst); + return 0; + +err: + bmp_free(src1); + bmp_free(src2); + bmp_free(dst); + return -EINVAL; +} + +SEC("syscall") +__weak int test_bitmap_or(void) +{ + struct bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; + + src1 = bmp_alloc(TEST_BITS); + src2 = bmp_alloc(TEST_BITS); + dst = bmp_alloc(TEST_BITS); + if (!src1 || !src2 || !dst) + goto err; + + test_bmp_setall(dst); + + __bmp_set_bit(0, src1); + __bmp_set_bit(LAST_BIT, src1); + + __bmp_set_bit(MID_BIT, src2); + __bmp_set_bit(LAST_BIT, src2); + + bmp_or(TEST_BITS, dst, src1, src2); + + if (!bmp_test_bit(0, dst)) + goto err; + if (!bmp_test_bit(MID_BIT, dst)) + goto err; + if (!bmp_test_bit(LAST_BIT, dst)) + goto err; + + if (dst->bits[0] != BIT_MASK(0)) + goto err; + if (dst->bits[1] != (BIT_MASK(MID_BIT) | BIT_MASK(LAST_BIT))) + goto err; + + bmp_free(src1); + bmp_free(src2); + bmp_free(dst); + return 0; + +err: + bmp_free(src1); + bmp_free(src2); + bmp_free(dst); + return -EINVAL; +} + +SEC("syscall") +__weak int test_bitmap_subset(void) +{ + struct bitmap __arena *big = NULL, *small = NULL; + + big = bmp_alloc(TEST_BITS); + small = bmp_alloc(TEST_BITS); + if (!big || !small) + goto err; + + if (!bmp_subset(TEST_BITS, big, small)) + goto err; + + __bmp_set_bit(0, small); + if (bmp_subset(TEST_BITS, big, small)) + goto err; + + __bmp_set_bit(0, big); + if (!bmp_subset(TEST_BITS, big, small)) + goto err; + + __bmp_set_bit(LAST_BIT, small); + if (bmp_subset(TEST_BITS, big, small)) + goto err; + + __bmp_set_bit(LAST_BIT, big); + __bmp_set_bit(MID_BIT, big); + if (!bmp_subset(TEST_BITS, big, small)) + goto err; + + if (bmp_subset(TEST_BITS, small, big)) + goto err; + + bmp_free(big); + bmp_free(small); + return 0; + +err: + bmp_free(big); + bmp_free(small); + return -EINVAL; + +} + +SEC("syscall") +__weak int test_bitmap_intersects(void) +{ + struct bitmap __arena *arg1 = NULL, *arg2 = NULL; + + arg1 = bmp_alloc(TEST_BITS); + arg2 = bmp_alloc(TEST_BITS); + if (!arg1 || !arg2) + goto err; + + if (bmp_intersects(TEST_BITS, arg1, arg2)) + goto err; + + __bmp_set_bit(0, arg1); + __bmp_set_bit(MID_BIT, arg2); + if (bmp_intersects(TEST_BITS, arg1, arg2)) + goto err; + + __bmp_set_bit(LAST_BIT, arg1); + __bmp_set_bit(LAST_BIT, arg2); + if (!bmp_intersects(TEST_BITS, arg1, arg2)) + goto err; + + bmp_free(arg1); + bmp_free(arg2); + return 0; + +err: + bmp_free(arg1); + bmp_free(arg2); + return -EINVAL; +} + +SEC("syscall") +__weak int test_bitmap_copy(void) +{ + struct bitmap __arena *arg1 = NULL, *arg2 = NULL; + + arg1 = bmp_alloc(TEST_BITS); + arg2 = bmp_alloc(TEST_BITS); + if (!arg1 || !arg2) + goto err; + + __bmp_set_bit(0, arg1); + __bmp_set_bit(MID_BIT, arg1); + + /* Make sure those get overwritten. */ + __bmp_set_bit(1, arg2); + __bmp_set_bit(MID_BIT + 2, arg2); + + bmp_copy(TEST_BITS, arg2, arg1); + + /* Bitmaps are equal if a subset of each other. */ + if (!bmp_subset(TEST_BITS, arg1, arg2) || + !bmp_subset(TEST_BITS, arg2, arg1)) + goto err; + + bmp_free(arg1); + bmp_free(arg2); + return 0; + +err: + bmp_free(arg1); + bmp_free(arg2); + return -EINVAL; +} From df64aadc78c13419a3ca412ad0122b37e72f83fd Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Mon, 6 Jul 2026 14:17:30 -0400 Subject: [PATCH 030/373] selftests/bpf: libarena: Add parallel bitmap selftest Add a selftest for testing the atomic bitmap set/clear/ test_and_set/test_and_clear operations. The selftest checks atomicity by spawning two threads, each of which either only works on even bits or with odd bits. The test checks that threads do not affect each other's bits. Signed-off-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260706181730.21731-7-emil@etsalapatis.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/test_parallel_bitmap.bpf.c | 190 ++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c new file mode 100644 index 000000000000..eec2871e2b0d --- /dev/null +++ b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c @@ -0,0 +1,190 @@ +// SPDX-License-Identifier: LGPL-2.1 OR BSD-2-Clause + +#include + +#include + +#include +#include + +#define TEST_BITMAP_THREADS 2 +#define TEST_BITMAP_BITS (2 * BITS_PER_LONG_LONG) +#define TEST_BITMAP_SYNC_SPINS BPF_MAX_LOOPS +#define TEST_BITMAP_ITERS 10 * 1000 * 1000 + +static struct bitmap __arena *bitmap; +static volatile u64 started; +static volatile bool test_abort; + +/* + * The test needs cmpxchg atomics on arena memory. + */ +#if defined(ENABLE_ATOMICS_TESTS) && \ + (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ + defined(__TARGET_ARCH_s390) || \ + defined(__TARGET_ARCH_powerpc) || \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) +static bool bitmap_tests_enabled(void) +{ + return true; +} +#else +static bool bitmap_tests_enabled(void) +{ + return false; +} +#endif + +__weak +int bitmap_wait_for_start(void) +{ + u64 i; + + __sync_fetch_and_add(&started, 1); + + for (i = zero; i < TEST_BITMAP_SYNC_SPINS && can_loop; i++) { + if (test_abort) + return -EINTR; + if (smp_load_acquire(&started) >= TEST_BITMAP_THREADS) + return 0; + } + + test_abort = true; + return -ETIMEDOUT; +} + +/* + * The test makes sure writes don't clobber each other by overwriting + * the same word. One thread always writes on even bits, the other on + * odds. Both should be able to operate on the bitmap oblivious of the + * other's operations. + */ +__weak +int bitmap_test_bit_sequence(u32 bit) +{ + if (bmp_test_and_clear_bit(bit, bitmap)) + return -EINVAL; + + if (bmp_test_and_set_bit(bit, bitmap)) + return -EINVAL; + if (!bmp_test_bit(bit, bitmap)) + return -EINVAL; + + if (!bmp_test_and_set_bit(bit, bitmap)) + return -EINVAL; + if (!bmp_test_bit(bit, bitmap)) + return -EINVAL; + + if (!bmp_test_and_clear_bit(bit, bitmap)) + return -EINVAL; + if (bmp_test_bit(bit, bitmap)) + return -EINVAL; + + if (bmp_test_and_clear_bit(bit, bitmap)) + return -EINVAL; + + bmp_set_bit(bit, bitmap); + if (!bmp_test_bit(bit, bitmap)) + return -EINVAL; + + bmp_clear_bit(bit, bitmap); + if (bmp_test_bit(bit, bitmap)) + return -EINVAL; + + bmp_set_bit(bit, bitmap); + if (!bmp_test_bit(bit, bitmap)) + return -EINVAL; + + return 0; + +} + +static void bitmap_test_reset_single(int parity) +{ + u32 bit; + + for (bit = parity; bit < TEST_BITMAP_BITS && can_loop; bit += 2) + bmp_clear_bit(bit, bitmap); + +} + +static int bitmap_test_common_single(int parity) +{ + u32 bit; + int ret; + + for (bit = parity; bit < TEST_BITMAP_BITS && can_loop; bit += 2) { + if (test_abort) + return -EINTR; + + ret = bitmap_test_bit_sequence(bit); + if (ret) { + test_abort = true; + return ret; + } + } + + return 0; +} + +static int bitmap_test_common(int parity) +{ + int ret; + u32 i; + + arena_subprog_init(); + + ret = bitmap_wait_for_start(); + if (ret) + return ret; + + for (i = zero; i < TEST_BITMAP_ITERS && can_loop; i++) { + ret = bitmap_test_common_single(parity); + if (ret) + return ret; + + if (test_abort) + break; + + bitmap_test_reset_single(parity); + } + + return 0; +} + +SEC("syscall") int parallel_test_bitmap__enabled(void) +{ + return bitmap_tests_enabled() ? 0 : -EOPNOTSUPP; +} + +SEC("syscall") int parallel_test_bitmap__init(void) +{ + bitmap = bmp_alloc(TEST_BITMAP_BITS); + if (!bitmap) + return -ENOMEM; + + return 0; +} + +SEC("syscall") int parallel_test_bitmap__fini(void) +{ + int ret = 0; + + if (!bitmap) + return -EINVAL; + + bmp_free(bitmap); + bitmap = NULL; + + return ret; +} + +SEC("syscall") int parallel_test_bitmap__0(void) +{ + return bitmap_test_common(0); +} + +SEC("syscall") int parallel_test_bitmap__1(void) +{ + return bitmap_test_common(1); +} From 0bebfaa39deadec21638f6fba553eae12627a26d Mon Sep 17 00:00:00 2001 From: Malaya Kumar Rout Date: Sat, 4 Jul 2026 17:59:35 +0530 Subject: [PATCH 031/373] selftests/bpf: Fix memory leak in msg_alloc_iov error path In msg_alloc_iov(), when calloc() fails for an individual iov_base allocation, the error path frees all previously allocated iov_base entries but fails to free the iov array itself that was allocated with calloc() at the beginning of the function. This results in a memory leak of the iov array. Add free(iov) in the unwind_iov error path to ensure proper cleanup of all allocated memory. Fixes: 753fb2ee0934 ("bpf: sockmap, add msg_peek tests to test_sockmap") Signed-off-by: Malaya Kumar Rout Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260704122936.102394-1-malayarout91@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_sockmap.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/bpf/test_sockmap.c b/tools/testing/selftests/bpf/test_sockmap.c index ac814eb63edb..3e6be455d158 100644 --- a/tools/testing/selftests/bpf/test_sockmap.c +++ b/tools/testing/selftests/bpf/test_sockmap.c @@ -436,6 +436,7 @@ static int msg_alloc_iov(struct msghdr *msg, unwind_iov: for (i--; i >= 0 ; i--) free(msg->msg_iov[i].iov_base); + free(iov); return -ENOMEM; } From 9c9ee0324c774490ae953162aaaf4561d222bd93 Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Tue, 30 Jun 2026 08:41:26 +0000 Subject: [PATCH 032/373] bpf: Reject MEM_ALLOC BTF accesses past object bounds BTF struct walks relax the struct-size check for accesses through a trailing flexible array. That is valid for ordinary BTF type walking, but PTR_TO_BTF_ID | MEM_ALLOC values point to objects allocated with the static BTF type size. When walking a MEM_ALLOC object, reject the access before applying the flexible-array relaxation if the access range extends past the struct size. Apply the same policy to struct ID matching so kfunc and kptr type checks do not walk past the allocated object bounds either. Fixes: 958cf2e273f0 ("bpf: Introduce bpf_obj_new") Fixes: 36d8bdf75a93 ("bpf: Add alloc/xchg/direct_access support for local percpu kptr") Signed-off-by: Yiyang Chen Reviewed-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/4b8c8a81102ba4b595011434c881194f264ddc59.1782807039.git.chenyy23@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 2 +- kernel/bpf/btf.c | 17 +++++++++++------ kernel/bpf/verifier.c | 11 +++++++---- 3 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ba09795e0bfd..adf53f7edf28 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3146,7 +3146,7 @@ int btf_struct_access(struct bpf_verifier_log *log, bool btf_struct_ids_match(struct bpf_verifier_log *log, const struct btf *btf, u32 id, int off, const struct btf *need_btf, u32 need_type_id, - bool strict); + bool strict, bool walk_flex_arrays); int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 64572f85edc8..dff5c0d91641 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7108,7 +7108,7 @@ enum bpf_struct_walk_result { static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, const struct btf_type *t, int off, int size, u32 *next_btf_id, enum bpf_type_flag *flag, - const char **field_name) + const char **field_name, bool walk_flex_arrays) { u32 i, moff, mtrue_end, msize = 0, total_nelems = 0; const struct btf_type *mtype, *elem_type = NULL; @@ -7135,11 +7135,14 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, *flag |= PTR_UNTRUSTED; if (off + size > t->size) { + struct btf_array *array_elem; + + if (!walk_flex_arrays) + goto error; + /* If the last element is a variable size array, we may * need to relax the rule. */ - struct btf_array *array_elem; - if (vlen == 0) goto error; @@ -7404,7 +7407,8 @@ int btf_struct_access(struct bpf_verifier_log *log, t = btf_type_by_id(btf, id); do { - err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name); + err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, + field_name, !type_is_alloc(reg->type)); switch (err) { case WALK_PTR: @@ -7463,7 +7467,7 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1, bool btf_struct_ids_match(struct bpf_verifier_log *log, const struct btf *btf, u32 id, int off, const struct btf *need_btf, u32 need_type_id, - bool strict) + bool strict, bool walk_flex_arrays) { const struct btf_type *type; enum bpf_type_flag flag = 0; @@ -7482,7 +7486,8 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log, type = btf_type_by_id(btf, id); if (!type) return false; - err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL); + err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL, + walk_flex_arrays); if (err != WALK_STRUCT) return false; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d46f7db20d8f..a0f292635c59 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4379,7 +4379,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, */ if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->var_off.value, kptr_field->kptr.btf, kptr_field->kptr.btf_id, - kptr_field->type != BPF_KPTR_UNREF)) + kptr_field->type != BPF_KPTR_UNREF, + !type_is_alloc(reg->type))) goto bad_type; return 0; bad_type: @@ -7970,7 +7971,7 @@ static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *re if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->var_off.value, btf_vmlinux, *arg_btf_id, - strict_type_match)) { + strict_type_match, !type_is_alloc(reg->type))) { verbose(env, "%s is of type %s but %s is expected\n", reg_arg_name(env, argno), btf_type_name(reg->btf, reg->btf_id), @@ -11436,7 +11437,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, ®_ref_id); reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off); struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->var_off.value, - meta->btf, ref_id, strict_type_match); + meta->btf, ref_id, strict_type_match, + !type_is_alloc(reg->type)); /* If kfunc is accepting a projection type (ie. __sk_buff), it cannot * actually use it -- it must cast to the underlying type. So we allow * caller to pass in the underlying type. @@ -11883,7 +11885,8 @@ __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env, et = btf_type_by_id(field->graph_root.btf, field->graph_root.value_btf_id); t = btf_type_by_id(reg->btf, reg->btf_id); if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, 0, field->graph_root.btf, - field->graph_root.value_btf_id, true)) { + field->graph_root.value_btf_id, true, + !type_is_alloc(reg->type))) { verbose(env, "operation on %s expects arg#1 %s at offset=%d " "in struct %s, but arg is at offset=%d in struct %s\n", btf_field_type_name(head_field_type), From 4137bbd9af1f80f86419097d728e6af136e4fea8 Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Tue, 30 Jun 2026 08:41:27 +0000 Subject: [PATCH 033/373] selftests/bpf: Cover MEM_ALLOC access past object bounds Add a linked_list negative loader case for a program-BTF type whose last member is a zero-length flexible array. The program writes through the first flexible-array element of an object allocated by bpf_obj_new(). The verifier should reject the access when the BTF walk reaches beyond the static size of the allocated object. Signed-off-by: Yiyang Chen Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/e36fd5d2f4047809f0e5da46a7077083297e64db.1782807039.git.chenyy23@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/linked_list.c | 1 + .../selftests/bpf/progs/linked_list_fail.c | 23 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/linked_list.c b/tools/testing/selftests/bpf/prog_tests/linked_list.c index 8defea0253ed..c3d133c6a00d 100644 --- a/tools/testing/selftests/bpf/prog_tests/linked_list.c +++ b/tools/testing/selftests/bpf/prog_tests/linked_list.c @@ -68,6 +68,7 @@ static struct { { "obj_type_id_oor", "local type ID argument must be in range [0, U32_MAX]" }, { "obj_new_no_composite", "bpf_obj_new/bpf_percpu_obj_new type ID argument must be of a struct" }, { "obj_new_no_struct", "bpf_obj_new/bpf_percpu_obj_new type ID argument must be of a struct" }, + { "obj_new_flex_array", "access beyond struct obj_new_flex" }, { "obj_drop_non_zero_off", "R1 must have zero offset when passed to release func" }, { "new_null_ret", "R0 invalid mem access 'ptr_or_null_'" }, { "obj_new_acq", "Unreleased reference id=" }, diff --git a/tools/testing/selftests/bpf/progs/linked_list_fail.c b/tools/testing/selftests/bpf/progs/linked_list_fail.c index ddd26d1a083f..031e77a288ee 100644 --- a/tools/testing/selftests/bpf/progs/linked_list_fail.c +++ b/tools/testing/selftests/bpf/progs/linked_list_fail.c @@ -167,6 +167,16 @@ CHECK_OP(push_back); #undef CHECK_OP #undef INIT +struct obj_new_flex_elem { + int lo; + int hi; +}; + +struct obj_new_flex { + int hdr; + struct obj_new_flex_elem cells[]; +}; + SEC("?kprobe/xyz") int map_compat_kprobe(void *ctx) { @@ -230,6 +240,19 @@ int obj_new_no_struct(void *ctx) return 0; } +SEC("?tc") +int obj_new_flex_array(void *ctx) +{ + struct obj_new_flex *p; + + p = bpf_obj_new_impl(bpf_core_type_id_local(struct obj_new_flex), NULL); + if (!p) + return 0; + p->cells[0].hi = 42; + bpf_obj_drop_impl(p, NULL); + return 0; +} + SEC("?tc") int obj_drop_non_zero_off(void *ctx) { From f66d25468b59f1694eb9b531e765dcd548350b8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= Date: Mon, 6 Jul 2026 12:28:43 -0300 Subject: [PATCH 034/373] docs/bpf: Document BPF_STRICT_BUILD=0 to tolerate test build failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the kernel config does not fully match the BPF selftest config fragment, some tests may fail to compile. BPF_STRICT_BUILD (defaulting to 1) makes any such failure fatal. Mention the option so that developers are aware they can set it to 0 to skip broken tests and keep the build going, which is particularly useful during bringup or when testing on constrained (e.g. distribution) configurations. Signed-off-by: Ricardo B. Marlière Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260706-b4-bpf_strict_build_docs-v1-1-5324d605c7b0@suse.com Signed-off-by: Kumar Kartikeya Dwivedi --- Documentation/bpf/bpf_devel_QA.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/bpf/bpf_devel_QA.rst b/Documentation/bpf/bpf_devel_QA.rst index 45bc5c5cd793..edf8107a7beb 100644 --- a/Documentation/bpf/bpf_devel_QA.rst +++ b/Documentation/bpf/bpf_devel_QA.rst @@ -479,7 +479,10 @@ for details. To maximize the number of tests passing, the .config of the kernel under test should match the config file fragment in -tools/testing/selftests/bpf as closely as possible. +tools/testing/selftests/bpf as closely as possible. If not possible, +however, you can set ``BPF_STRICT_BUILD=0`` when invoking ``make`` +to tolerate individual compilation failures and continue building +the remaining tests rather than treating each failure as fatal. Finally to ensure support for latest BPF Type Format features - discussed in Documentation/bpf/btf.rst - pahole version 1.16 From 5f6cc299938b561cb01e343bab7042611fcee12a Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Fri, 3 Jul 2026 14:51:35 +0200 Subject: [PATCH 035/373] s390/bpf: Replace ly instruction with llgf cpu_nr is a 32 bit value and BPF_REG_0 is a 64 bit register, when ly loads the cpu_nr into BPF_REG_0 it does not zero the upper bits, but llgf does. Fixes: 9012cf2491e3 ("s390/bpf: Inline smp_processor_id and current_task") Signed-off-by: Maxim Khmelevskii Signed-off-by: Daniel Borkmann Reviewed-by: Ilya Leoshkevich Link: https://sashiko.dev/#/patchset/20260414142930.528751-1-max%40linux.ibm.com Link: https://lore.kernel.org/bpf/20260703125648.919196-5-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/s390/net/bpf_jit_comp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 31749c0362ca..9ddd89f71f28 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -1783,8 +1783,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, insn->imm == BPF_FUNC_get_smp_processor_id) { const u32 *cpu_nr = &get_lowcore()->cpu_nr; - /* ly %b0, cpu_nr */ - EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0, + /* llgf %b0, cpu_nr */ + EMIT6_DISP_LH(0xe3000000, 0x0016, BPF_REG_0, REG_0, REG_0, (unsigned long)cpu_nr); break; } From e7333034387d7cbc80de91d52b3427384902ada7 Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Fri, 3 Jul 2026 14:51:36 +0200 Subject: [PATCH 036/373] selftests/bpf: Add test for bpf_get_smp_processor_id Add a test which checks on each CPU that the bpf_get_smp_processor_id BPF helper is returning the correct CPU number. Signed-off-by: Maxim Khmelevskii Signed-off-by: Daniel Borkmann Reviewed-by: Ilya Leoshkevich Link: https://lore.kernel.org/bpf/20260703125648.919196-6-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/prog_tests/get_smp_processor_id.c | 45 +++++++++++++++++++ .../bpf/progs/get_smp_processor_id.c | 20 +++++++++ 2 files changed, 65 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/get_smp_processor_id.c create mode 100644 tools/testing/selftests/bpf/progs/get_smp_processor_id.c diff --git a/tools/testing/selftests/bpf/prog_tests/get_smp_processor_id.c b/tools/testing/selftests/bpf/prog_tests/get_smp_processor_id.c new file mode 100644 index 000000000000..1b5c738ab81f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/get_smp_processor_id.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include "bpf/libbpf_internal.h" +#include "get_smp_processor_id.skel.h" + +void test_get_smp_processor_id(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts, + .flags = BPF_F_TEST_RUN_ON_CPU, + .cpu = 0, + ); + struct get_smp_processor_id *skel; + int prog_fd, err, online_cpu_nr, i; + bool *online = NULL; + + err = parse_cpu_mask_file("/sys/devices/system/cpu/online", + &online, &online_cpu_nr); + if (!ASSERT_OK(err, "parse_cpu_mask_file")) + return; + + skel = get_smp_processor_id__open_and_load(); + if (!ASSERT_OK_PTR(skel, "get_smp_processor_id__open_and_load")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.call_bpf_get_smp_processor_id); + + for (i = 0; i < online_cpu_nr; i++) { + if (!online[i]) + continue; + + opts.cpu = i; + skel->bss->cpu_nr_result = -1; + + err = bpf_prog_test_run_opts(prog_fd, &opts); + if (!ASSERT_OK(err, "bpf_prog_test_run_opts")) + goto cleanup; + + ASSERT_EQ(skel->bss->cpu_nr_result, opts.cpu, "cpu_nr_result"); + } + +cleanup: + free(online); + get_smp_processor_id__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/get_smp_processor_id.c b/tools/testing/selftests/bpf/progs/get_smp_processor_id.c new file mode 100644 index 000000000000..cf4791a5cf07 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/get_smp_processor_id.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +__u64 cpu_nr_result; + +SEC("raw_tp") +void call_bpf_get_smp_processor_id(void) +{ + register __u64 r0 asm("r0") = -1; + asm volatile ("call %[bpf_get_smp_processor_id];" + : "+r"(r0) + : __imm(bpf_get_smp_processor_id) + : "r1", "r2", "r3", "r4", "r5", "memory"); + cpu_nr_result = r0; +} + +char _license[] SEC("license") = "GPL"; From be39165224d03d92a05f62b9ea10eec089365480 Mon Sep 17 00:00:00 2001 From: Sechang Lim Date: Tue, 30 Jun 2026 14:54:05 +0000 Subject: [PATCH 037/373] bpf, sockmap: Disallow update and delete from tc, xdp, socket_filter and flow_dissector sock_map_update_common() and __sock_map_delete() hold stab->lock and call sock_map_unref() -> sock_map_del_link(), which takes sk_callback_lock for write. That gives the order stab->lock -> sk_callback_lock. The reverse order comes from the SK_SKB stream parser. sk_psock_strp_data_ready() holds sk_callback_lock for read, and after the verdict tcp_bpf_strp_read_sock() acks the consumed data inline via __tcp_cleanup_rbuf(). The ACK goes out egress, where a sched_cls program deletes from the sockmap and takes stab->lock: WARNING: possible circular locking dependency detected ------------------------------------------------------ syz.9.8824 is trying to acquire lock: (&stab->lock){+.-.}-{3:3}, at: __sock_map_delete net/core/sock_map.c:421 but task is already holding lock: (clock-AF_INET){++.-}-{3:3}, at: sk_psock_strp_data_ready net/core/skmsg.c:1173 -> #1 (clock-AF_INET){++.-}-{3:3}: _raw_write_lock_bh sock_map_del_link net/core/sock_map.c:167 sock_map_unref net/core/sock_map.c:184 sock_map_update_common net/core/sock_map.c:509 sock_map_update_elem_sys net/core/sock_map.c:588 map_update_elem kernel/bpf/syscall.c:1805 -> #0 (&stab->lock){+.-.}-{3:3}: _raw_spin_lock_bh __sock_map_delete net/core/sock_map.c:421 sock_map_delete_elem net/core/sock_map.c:452 bpf_prog_06044d24140080b6 tcx_run net/core/dev.c:4451 sch_handle_egress net/core/dev.c:4541 __dev_queue_xmit net/core/dev.c:4808 ... tcp_bpf_strp_read_sock net/ipv4/tcp_bpf.c:701 strp_data_ready net/strparser/strparser.c:402 sk_psock_strp_data_ready net/core/skmsg.c:1174 tcp_data_queue net/ipv4/tcp_input.c:5661 Possible unsafe locking scenario: CPU0 CPU1 ---- ---- rlock(clock-AF_INET); lock(&stab->lock); lock(clock-AF_INET); lock(&stab->lock); *** DEADLOCK *** A tc, xdp, socket_filter or flow_dissector program has no reason to update or delete a sockmap, and redirect does not go through here. Drop them from may_update_sockmap() so the verifier rejects it. It also closes the matching sockhash inversion. Suggested-by: John Fastabend Signed-off-by: Sechang Lim Signed-off-by: Daniel Borkmann Reviewed-by: John Fastabend Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260630145410.3648099-2-rhkrqnwk98@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a0f292635c59..3193b473762b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8496,12 +8496,7 @@ static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id) if (func_id == BPF_FUNC_map_delete_elem) return true; break; - case BPF_PROG_TYPE_SOCKET_FILTER: - case BPF_PROG_TYPE_SCHED_CLS: - case BPF_PROG_TYPE_SCHED_ACT: - case BPF_PROG_TYPE_XDP: case BPF_PROG_TYPE_SK_REUSEPORT: - case BPF_PROG_TYPE_FLOW_DISSECTOR: case BPF_PROG_TYPE_SK_LOOKUP: return true; default: From 5de7a6eaed89b45b91ca1830a7ddda7d8f50673b Mon Sep 17 00:00:00 2001 From: Sechang Lim Date: Tue, 30 Jun 2026 14:54:06 +0000 Subject: [PATCH 038/373] selftests/bpf: Drop tc/xdp/flow_dissector/socket_filter sockmap mutation tests tc, xdp, socket_filter and flow_dissector programs can no longer update or delete a sockmap. Adjust the tests: - verifier_sockmap_mutate: the tc, xdp, socket_filter and flow_dissector cases now expect __failure with "cannot update sockmap in this context". - sockmap_basic: drop "sockmap update" / "sockhash update", which load a SEC("tc") program that copies a sock between maps. - fexit_bpf2bpf: drop "func_sockmap_update", whose freplace program updates a sockmap in the tc cls_redirect context. Remove the now-unused test_sockmap_update.c and freplace_cls_redirect.c. Signed-off-by: Sechang Lim Signed-off-by: Daniel Borkmann Reviewed-by: John Fastabend Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260630145410.3648099-3-rhkrqnwk98@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 14 ----- .../selftests/bpf/prog_tests/sockmap_basic.c | 52 ------------------- .../bpf/progs/freplace_cls_redirect.c | 34 ------------ .../selftests/bpf/progs/test_sockmap_update.c | 48 ----------------- .../bpf/progs/verifier_sockmap_mutate.c | 12 ++--- 5 files changed, 6 insertions(+), 154 deletions(-) delete mode 100644 tools/testing/selftests/bpf/progs/freplace_cls_redirect.c delete mode 100644 tools/testing/selftests/bpf/progs/test_sockmap_update.c diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 92c20803ea76..4a87d7163c8c 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -335,18 +335,6 @@ static void test_fmod_ret_freplace(void) bpf_object__close(pkt_obj); } - -static void test_func_sockmap_update(void) -{ - const char *prog_name[] = { - "freplace/cls_redirect", - }; - test_fexit_bpf2bpf_common("./freplace_cls_redirect.bpf.o", - "./test_cls_redirect.bpf.o", - ARRAY_SIZE(prog_name), - prog_name, false, NULL); -} - static void test_func_replace_void(void) { const char *prog_name[] = { @@ -599,8 +587,6 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace(); if (test__start_subtest("func_replace_verify")) test_func_replace_verify(); - if (test__start_subtest("func_sockmap_update")) - test_func_sockmap_update(); if (test__start_subtest("func_replace_return_code")) test_func_replace_return_code(); if (test__start_subtest("func_map_prog_compatibility")) diff --git a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c index cb3229711f93..33f788e2786d 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +++ b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c @@ -7,7 +7,6 @@ #include "test_progs.h" #include "test_skmsg_load_helpers.skel.h" -#include "test_sockmap_update.skel.h" #include "test_sockmap_invalid_update.skel.h" #include "test_sockmap_skb_verdict_attach.skel.h" #include "test_sockmap_progs_query.skel.h" @@ -235,53 +234,6 @@ static void test_skmsg_helpers_with_link(enum bpf_map_type map_type) test_skmsg_load_helpers__destroy(skel); } -static void test_sockmap_update(enum bpf_map_type map_type) -{ - int err, prog, src; - struct test_sockmap_update *skel; - struct bpf_map *dst_map; - const __u32 zero = 0; - char dummy[14] = {0}; - LIBBPF_OPTS(bpf_test_run_opts, topts, - .data_in = dummy, - .data_size_in = sizeof(dummy), - .repeat = 1, - ); - __s64 sk; - - sk = connected_socket_v4(); - if (!ASSERT_NEQ(sk, -1, "connected_socket_v4")) - return; - - skel = test_sockmap_update__open_and_load(); - if (!ASSERT_OK_PTR(skel, "open_and_load")) - goto close_sk; - - prog = bpf_program__fd(skel->progs.copy_sock_map); - src = bpf_map__fd(skel->maps.src); - if (map_type == BPF_MAP_TYPE_SOCKMAP) - dst_map = skel->maps.dst_sock_map; - else - dst_map = skel->maps.dst_sock_hash; - - err = bpf_map_update_elem(src, &zero, &sk, BPF_NOEXIST); - if (!ASSERT_OK(err, "update_elem(src)")) - goto out; - - err = bpf_prog_test_run_opts(prog, &topts); - if (!ASSERT_OK(err, "test_run")) - goto out; - if (!ASSERT_NEQ(topts.retval, 0, "test_run retval")) - goto out; - - compare_cookies(skel->maps.src, dst_map); - -out: - test_sockmap_update__destroy(skel); -close_sk: - close(sk); -} - static void test_sockmap_invalid_update(void) { struct test_sockmap_invalid_update *skel; @@ -1385,10 +1337,6 @@ void test_sockmap_basic(void) test_skmsg_helpers(BPF_MAP_TYPE_SOCKMAP); if (test__start_subtest("sockhash sk_msg load helpers")) test_skmsg_helpers(BPF_MAP_TYPE_SOCKHASH); - if (test__start_subtest("sockmap update")) - test_sockmap_update(BPF_MAP_TYPE_SOCKMAP); - if (test__start_subtest("sockhash update")) - test_sockmap_update(BPF_MAP_TYPE_SOCKHASH); if (test__start_subtest("sockmap update in unsafe context")) test_sockmap_invalid_update(); if (test__start_subtest("sockmap copy")) diff --git a/tools/testing/selftests/bpf/progs/freplace_cls_redirect.c b/tools/testing/selftests/bpf/progs/freplace_cls_redirect.c deleted file mode 100644 index 7e94412d47a5..000000000000 --- a/tools/testing/selftests/bpf/progs/freplace_cls_redirect.c +++ /dev/null @@ -1,34 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -// Copyright (c) 2020 Facebook - -#include -#include -#include -#include -#include - -struct { - __uint(type, BPF_MAP_TYPE_SOCKMAP); - __type(key, int); - __type(value, int); - __uint(max_entries, 2); -} sock_map SEC(".maps"); - -SEC("freplace/cls_redirect") -int freplace_cls_redirect_test(struct __sk_buff *skb) -{ - int ret = 0; - const int zero = 0; - struct bpf_sock *sk; - - sk = bpf_map_lookup_elem(&sock_map, &zero); - if (!sk) - return TC_ACT_SHOT; - - ret = bpf_map_update_elem(&sock_map, &zero, sk, 0); - bpf_sk_release(sk); - - return ret == 0 ? TC_ACT_OK : TC_ACT_SHOT; -} - -char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/test_sockmap_update.c b/tools/testing/selftests/bpf/progs/test_sockmap_update.c deleted file mode 100644 index 6d64ea536e3d..000000000000 --- a/tools/testing/selftests/bpf/progs/test_sockmap_update.c +++ /dev/null @@ -1,48 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -// Copyright (c) 2020 Cloudflare -#include "vmlinux.h" -#include - -struct { - __uint(type, BPF_MAP_TYPE_SOCKMAP); - __uint(max_entries, 1); - __type(key, __u32); - __type(value, __u64); -} src SEC(".maps"); - -struct { - __uint(type, BPF_MAP_TYPE_SOCKMAP); - __uint(max_entries, 1); - __type(key, __u32); - __type(value, __u64); -} dst_sock_map SEC(".maps"); - -struct { - __uint(type, BPF_MAP_TYPE_SOCKHASH); - __uint(max_entries, 1); - __type(key, __u32); - __type(value, __u64); -} dst_sock_hash SEC(".maps"); - -SEC("tc") -int copy_sock_map(void *ctx) -{ - struct bpf_sock *sk; - bool failed = false; - __u32 key = 0; - - sk = bpf_map_lookup_elem(&src, &key); - if (!sk) - return SK_DROP; - - if (bpf_map_update_elem(&dst_sock_map, &key, sk, 0)) - failed = true; - - if (bpf_map_update_elem(&dst_sock_hash, &key, sk, 0)) - failed = true; - - bpf_sk_release(sk); - return failed ? SK_DROP : SK_PASS; -} - -char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_sockmap_mutate.c b/tools/testing/selftests/bpf/progs/verifier_sockmap_mutate.c index fe4b123187b8..20332a731d4e 100644 --- a/tools/testing/selftests/bpf/progs/verifier_sockmap_mutate.c +++ b/tools/testing/selftests/bpf/progs/verifier_sockmap_mutate.c @@ -74,7 +74,7 @@ static __always_inline void test_sockmap_lookup_and_mutate(void) } SEC("action") -__success +__failure __msg("cannot update sockmap in this context") int test_sched_act(struct __sk_buff *skb) { test_sockmap_mutate(skb->sk); @@ -82,7 +82,7 @@ int test_sched_act(struct __sk_buff *skb) } SEC("classifier") -__success +__failure __msg("cannot update sockmap in this context") int test_sched_cls(struct __sk_buff *skb) { test_sockmap_mutate(skb->sk); @@ -90,7 +90,7 @@ int test_sched_cls(struct __sk_buff *skb) } SEC("flow_dissector") -__success +__failure __msg("cannot update sockmap in this context") int test_flow_dissector_delete(struct __sk_buff *skb __always_unused) { test_sockmap_delete(); @@ -98,7 +98,7 @@ int test_flow_dissector_delete(struct __sk_buff *skb __always_unused) } SEC("flow_dissector") -__failure __msg("program of this type cannot use helper bpf_sk_release") +__failure __msg("cannot update sockmap in this context") int test_flow_dissector_update(struct __sk_buff *skb __always_unused) { test_sockmap_lookup_and_update(); /* no access to skb->sk */ @@ -146,7 +146,7 @@ int test_sk_reuseport(struct sk_reuseport_md *ctx) } SEC("socket") -__success +__failure __msg("cannot update sockmap in this context") int test_socket_filter(struct __sk_buff *skb) { test_sockmap_mutate(skb->sk); @@ -179,7 +179,7 @@ int test_sockops_update_dedicated(struct bpf_sock_ops *ctx) } SEC("xdp") -__success +__failure __msg("cannot update sockmap in this context") int test_xdp(struct xdp_md *ctx __always_unused) { test_sockmap_lookup_and_mutate(); From 602701718649936eb287bf6c7ecf870ec54c6f71 Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Tue, 7 Jul 2026 16:14:34 +0800 Subject: [PATCH 039/373] selftests/bpf: Fix memory leak in msg_alloc_iov In the msg_alloc_iov function, the iov pointer is only assigned to msg->msg_iov after all memory allocations complete successfully. Therefore, when a calloc failure triggers the unwind_iov cleanup branch, we should use the local variable iov instead of msg->msg_iov. Fixes: 753fb2ee0934 ("bpf: sockmap, add msg_peek tests to test_sockmap") Signed-off-by: Feng Yang Reviewed-by: John Fastabend Link: https://lore.kernel.org/bpf/20260707081434.539327-1-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_sockmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_sockmap.c b/tools/testing/selftests/bpf/test_sockmap.c index 3e6be455d158..aaf2050e8845 100644 --- a/tools/testing/selftests/bpf/test_sockmap.c +++ b/tools/testing/selftests/bpf/test_sockmap.c @@ -435,7 +435,7 @@ static int msg_alloc_iov(struct msghdr *msg, return 0; unwind_iov: for (i--; i >= 0 ; i--) - free(msg->msg_iov[i].iov_base); + free(iov[i].iov_base); free(iov); return -ENOMEM; } From 1f737e46ca6a845e6e96982ad57a31026f3521fa Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Tue, 7 Jul 2026 17:44:28 -0700 Subject: [PATCH 040/373] bpf: Remove artificial limitations on pointer types eligible for spilling The verifier loses precision when simulating stack spills for the following register types: - PTR_TO_TP_BUFFER - PTR_TO_INSN - CONST_PTR_TO_DYNPTR These types are not allow-listed in the is_spillable_regtype(), because of that check_stack_write_fixed_off() takes the branch that marks the slots STACK_MISC. There are no technical reasons for this limitation. This commit replaces an explicit list of pointer types in is_spillable_regtype() with explicit list of non-pointer types. The function is renamed to is_pointer_regtype() for clarity. Reported-by: Andrii Nakryiko Suggested-by: Kumar Kartikeya Dwivedi Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260707-missing-spillable-types-v1-1-44a92121dc41@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 39 ++++++++------------------------------- 1 file changed, 8 insertions(+), 31 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3193b473762b..51f7965d42e3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3304,34 +3304,6 @@ static int mark_chain_precision_batch(struct bpf_verifier_env *env, return bpf_mark_chain_precision(env, starting_state, -1, NULL); } -static bool is_spillable_regtype(enum bpf_reg_type type) -{ - switch (base_type(type)) { - case PTR_TO_MAP_VALUE: - case PTR_TO_STACK: - case PTR_TO_CTX: - case PTR_TO_PACKET: - case PTR_TO_PACKET_META: - case PTR_TO_PACKET_END: - case PTR_TO_FLOW_KEYS: - case CONST_PTR_TO_MAP: - case PTR_TO_SOCKET: - case PTR_TO_SOCK_COMMON: - case PTR_TO_TCP_SOCK: - case PTR_TO_XDP_SOCK: - case PTR_TO_BTF_ID: - case PTR_TO_BUF: - case PTR_TO_MEM: - case PTR_TO_FUNC: - case PTR_TO_MAP_KEY: - case PTR_TO_ARENA: - return true; - default: - return false; - } -} - - /* check if register is a constant scalar value */ static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32) { @@ -3345,13 +3317,18 @@ static u64 reg_const_value(struct bpf_reg_state *reg, bool subreg32) return subreg32 ? tnum_subreg(reg->var_off).value : reg->var_off.value; } +static bool is_pointer_regtype(enum bpf_reg_type type) +{ + return type != SCALAR_VALUE && type != NOT_INIT; +} + static bool __is_pointer_value(bool allow_ptr_leaks, const struct bpf_reg_state *reg) { if (allow_ptr_leaks) return false; - return reg->type != SCALAR_VALUE; + return is_pointer_regtype(reg->type); } static void clear_scalar_id(struct bpf_reg_state *reg) @@ -3476,7 +3453,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, if (value_regno >= 0) reg = &cur->regs[value_regno]; if (!env->bypass_spec_v4) { - bool sanitize = reg && is_spillable_regtype(reg->type); + bool sanitize = reg && is_pointer_regtype(reg->type); for (i = 0; i < size; i++) { u8 type = state->stack[spi].slot_type[(slot - i) % @@ -3517,7 +3494,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, __mark_reg_known(tmp_reg, insn->imm); tmp_reg->type = SCALAR_VALUE; save_register_state(env, state, spi, tmp_reg, size); - } else if (reg && is_spillable_regtype(reg->type)) { + } else if (reg && is_pointer_regtype(reg->type)) { /* register containing pointer is being spilled into stack */ if (size != BPF_REG_SIZE) { verbose_linfo(env, insn_idx, "; "); From b6d29b9ba9d6c72558d0e30f3cc3f48990d08a54 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Tue, 7 Jul 2026 17:44:29 -0700 Subject: [PATCH 041/373] selftests/bpf: Test cases for missing spill types A few selftests checking that the verifier represents spills for the following pointer types w/o losing precision: - PTR_TO_INSN - PTR_TO_TP_BUFFER - CONST_PTR_TO_DYNPTR Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260707-missing-spillable-types-v1-2-44a92121dc41@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_gotox.c | 25 +++++++++++ .../selftests/bpf/progs/verifier_spill_fill.c | 42 +++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c index f88aa4cdb279..5b18c9a27717 100644 --- a/tools/testing/selftests/bpf/progs/verifier_gotox.c +++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c @@ -384,6 +384,31 @@ jt0_%=: \ : __clobber_all); } +/* check valid spill/fill, ptr to insn */ +SEC("socket") +__success +__naked void spill_fill_ptr_to_insn(void) +{ + asm volatile ( + ".pushsection .jumptables,\"\",@progbits;" + "jt0_%=:" + ".quad ret0_%= - socket;" + ".size jt0_%=, 8;" + ".global jt0_%=;" + ".popsection;" + "r0 = jt0_%= ll;" + "r0 = *(u64 *)(r0 + 0);" + "*(u64 *)(r10 - 8) = r0;" + "r0 = *(u64 *)(r10 - 8);" + ".8byte %[gotox_r0];" + "ret0_%=:" + "r0 = 0;" + "exit;" + : + : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0, 0)) + : __clobber_all); +} + #endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 || __TARGET_ARCH_powerpc*/ char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c index 72c691333703..8b166c42c4e0 100644 --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c @@ -1403,4 +1403,46 @@ __naked void partial_fill_from_cleaned_pointer_spill(void) ::: __clobber_all); } +/* check valid spill/fill, ptr to tp buffer */ +SEC("raw_tracepoint.w") +__success +__naked void spill_fill_ptr_to_tp_buffer(void) +{ + asm volatile ( + "r6 = *(u64*)(r1 + 0);" /* r6 is the writable tracepoint buffer */ + "*(u64*)(r10 - 8) = r6;" + "r7 = *(u64*)(r10 - 8);" + "r0 = 0;" + "*(u64*)(r7 + 0) = r0;" /* should be able to write through the buffer */ + "r0 = 0;" + "exit;" + ::: __clobber_all); +} + +__noinline int spill_fill_dynptr_subprog(struct bpf_dynptr *dptr) +{ + long *p; + + asm volatile ("*(u64 *)(r10 - 8) = %[dptr];" /* spill the CONST_PTR_TO_DYNPTR argument */ + "%[dptr] = *(u64 *)(r10 - 8);" + : [dptr] "+r"(dptr) :: "memory"); + p = bpf_dynptr_data(dptr, 0, sizeof(*p)); + if (!p) + return 0; + return 0; +} + +static char dptr_mem_buf[16]; + +/* check valid spill/fill, const ptr to dynptr */ +SEC("socket") +__success +int spill_fill_const_ptr_to_dynptr(void) +{ + struct bpf_dynptr ptr; + + bpf_dynptr_from_mem(dptr_mem_buf, sizeof(dptr_mem_buf), 0, &ptr); + return spill_fill_dynptr_subprog(&ptr); +} + char _license[] SEC("license") = "GPL"; From ee1bcf8271eb2e1d191bf97348ddf823c6071fa9 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Tue, 7 Jul 2026 15:01:36 -0700 Subject: [PATCH 042/373] selftests/bpf: Rename libarena struct bitmap to struct arena_bitmap When building bpf selftest with latest bpf-next, I got the following failure: In file included from /home/yhs/work/bpf-next/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c:8: /home/yhs/work/bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h:11:8: error: redefinition of 'bitmap' 11 | struct bitmap { | ^ /home/yhs/work/bpf-next/tools/testing/selftests/bpf/tools/include/vmlinux.h:51320:8: note: previous definition is here 51320 | struct bitmap { | ^ The vmlinux.h struct bitmap comes from drivers/md/md-bitmap.c: struct bitmap { struct bitmap_counts { ... } ... } To fix the issue, I renamed libarena struct bitmap to arena_bitmap to avoid the conflict. Signed-off-by: Yonghong Song Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260707220136.910374-1-yonghong.song@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/libarena/include/libarena/bitmap.h | 36 +++++++++--------- .../bpf/libarena/selftests/test_bitmap.bpf.c | 24 ++++++------ .../selftests/test_parallel_bitmap.bpf.c | 2 +- .../selftests/bpf/libarena/src/bitmap.bpf.c | 38 +++++++++---------- 4 files changed, 50 insertions(+), 50 deletions(-) diff --git a/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h b/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h index cf6b63f5d9a4..e2431ea6fdd6 100644 --- a/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h +++ b/tools/testing/selftests/bpf/libarena/include/libarena/bitmap.h @@ -8,27 +8,27 @@ #define BIT_MASK(nr) (1ULL << ((nr) % BITS_PER_LONG_LONG)) #define BIT_WORD(nr) ((nr) / BITS_PER_LONG_LONG) -struct bitmap { +struct arena_bitmap { u64 bits[0]; }; -struct bitmap __arena *bmp_alloc(size_t bits); -void bmp_free(struct bitmap __arena *bmp); +struct arena_bitmap __arena *bmp_alloc(size_t bits); +void bmp_free(struct arena_bitmap __arena *bmp); -void __bmp_set_bit(u32 bit, struct bitmap __arena *bmp); -void __bmp_clear_bit(u32 bit, struct bitmap __arena *bmp); -void bmp_set_bit(u32 bit, struct bitmap __arena *bmp); -void bmp_clear_bit(u32 bit, struct bitmap __arena *bmp); -bool bmp_test_bit(u32 bit, struct bitmap __arena *bmp); -bool bmp_test_and_clear_bit(u32 bit, struct bitmap __arena *bmp); -bool bmp_test_and_set_bit(u32 bit, struct bitmap __arena *bmp); +void __bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp); +void __bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp); +void bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp); +void bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp); +bool bmp_test_bit(u32 bit, struct arena_bitmap __arena *bmp); +bool bmp_test_and_clear_bit(u32 bit, struct arena_bitmap __arena *bmp); +bool bmp_test_and_set_bit(u32 bit, struct arena_bitmap __arena *bmp); -void bmp_clear(size_t bits, struct bitmap __arena *bmp); -void bmp_and(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2); -void bmp_or(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2); -bool bmp_empty(size_t bits, struct bitmap __arena *bmp); -void bmp_copy(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src); +void bmp_clear(size_t bits, struct arena_bitmap __arena *bmp); +void bmp_and(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src1, struct arena_bitmap __arena *src2); +void bmp_or(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src1, struct arena_bitmap __arena *src2); +bool bmp_empty(size_t bits, struct arena_bitmap __arena *bmp); +void bmp_copy(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src); -bool bmp_intersects(size_t bits, struct bitmap __arena *arg1, struct bitmap __arena *arg2); -bool bmp_subset(size_t bits, struct bitmap __arena *big, struct bitmap __arena *small); -void bmp_print(size_t bits, struct bitmap __arena *bmp); +bool bmp_intersects(size_t bits, struct arena_bitmap __arena *arg1, struct arena_bitmap __arena *arg2); +bool bmp_subset(size_t bits, struct arena_bitmap __arena *big, struct arena_bitmap __arena *small); +void bmp_print(size_t bits, struct arena_bitmap __arena *bmp); diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c index e7d32f44d687..76319a529f02 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_bitmap.bpf.c @@ -8,7 +8,7 @@ #define MID_BIT (BITS_PER_LONG_LONG + 1) #define LAST_BIT (TEST_BITS - 1) -static void test_bmp_setall(struct bitmap __arena *bmp) +static void test_bmp_setall(struct arena_bitmap __arena *bmp) { volatile u32 i; @@ -19,7 +19,7 @@ static void test_bmp_setall(struct bitmap __arena *bmp) SEC("syscall") __weak int test_bitmap_alloc_free(void) { - struct bitmap __arena *bmp; + struct arena_bitmap __arena *bmp; bmp = bmp_alloc(TEST_BITS); if (!bmp) @@ -47,7 +47,7 @@ __weak int test_bitmap_alloc_free(void) SEC("syscall") __weak int test_bitmap_bit_ops(void) { - struct bitmap __arena *bmp; + struct arena_bitmap __arena *bmp; bmp = bmp_alloc(TEST_BITS); if (!bmp) @@ -97,7 +97,7 @@ __weak int test_bitmap_bit_ops(void) return -EINVAL; } -static bool test_bitmap_test_and_clear_single(struct bitmap __arena *bmp, size_t ind) +static bool test_bitmap_test_and_clear_single(struct arena_bitmap __arena *bmp, size_t ind) { if (bmp_test_and_clear_bit(ind, bmp)) return false; @@ -116,7 +116,7 @@ static bool test_bitmap_test_and_clear_single(struct bitmap __arena *bmp, size_t return true; } -static bool test_bitmap_test_and_set_single(struct bitmap __arena *bmp, size_t ind) +static bool test_bitmap_test_and_set_single(struct arena_bitmap __arena *bmp, size_t ind) { if (bmp_test_and_set_bit(ind, bmp)) return false; @@ -138,7 +138,7 @@ static bool test_bitmap_test_and_set_single(struct bitmap __arena *bmp, size_t i SEC("syscall") __weak int test_bitmap_test_and_clear_bit(void) { - struct bitmap __arena *bmp; + struct arena_bitmap __arena *bmp; bmp = bmp_alloc(TEST_BITS); if (!bmp) @@ -167,7 +167,7 @@ __weak int test_bitmap_test_and_clear_bit(void) SEC("syscall") __weak int test_bitmap_test_and_set_bit(void) { - struct bitmap __arena *bmp; + struct arena_bitmap __arena *bmp; bmp = bmp_alloc(TEST_BITS); if (!bmp) @@ -194,7 +194,7 @@ __weak int test_bitmap_test_and_set_bit(void) SEC("syscall") __weak int test_bitmap_and(void) { - struct bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; + struct arena_bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; src1 = bmp_alloc(TEST_BITS); src2 = bmp_alloc(TEST_BITS); @@ -240,7 +240,7 @@ __weak int test_bitmap_and(void) SEC("syscall") __weak int test_bitmap_or(void) { - struct bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; + struct arena_bitmap __arena *src1 = NULL, *src2 = NULL, *dst = NULL; src1 = bmp_alloc(TEST_BITS); src2 = bmp_alloc(TEST_BITS); @@ -285,7 +285,7 @@ __weak int test_bitmap_or(void) SEC("syscall") __weak int test_bitmap_subset(void) { - struct bitmap __arena *big = NULL, *small = NULL; + struct arena_bitmap __arena *big = NULL, *small = NULL; big = bmp_alloc(TEST_BITS); small = bmp_alloc(TEST_BITS); @@ -329,7 +329,7 @@ __weak int test_bitmap_subset(void) SEC("syscall") __weak int test_bitmap_intersects(void) { - struct bitmap __arena *arg1 = NULL, *arg2 = NULL; + struct arena_bitmap __arena *arg1 = NULL, *arg2 = NULL; arg1 = bmp_alloc(TEST_BITS); arg2 = bmp_alloc(TEST_BITS); @@ -362,7 +362,7 @@ __weak int test_bitmap_intersects(void) SEC("syscall") __weak int test_bitmap_copy(void) { - struct bitmap __arena *arg1 = NULL, *arg2 = NULL; + struct arena_bitmap __arena *arg1 = NULL, *arg2 = NULL; arg1 = bmp_alloc(TEST_BITS); arg2 = bmp_alloc(TEST_BITS); diff --git a/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c index eec2871e2b0d..ea1fac95b461 100644 --- a/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c +++ b/tools/testing/selftests/bpf/libarena/selftests/test_parallel_bitmap.bpf.c @@ -12,7 +12,7 @@ #define TEST_BITMAP_SYNC_SPINS BPF_MAX_LOOPS #define TEST_BITMAP_ITERS 10 * 1000 * 1000 -static struct bitmap __arena *bitmap; +static struct arena_bitmap __arena *bitmap; static volatile u64 started; static volatile bool test_abort; diff --git a/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c b/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c index 80e814401fb9..5ff8e688ddc7 100644 --- a/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c +++ b/tools/testing/selftests/bpf/libarena/src/bitmap.bpf.c @@ -10,16 +10,16 @@ #include __weak -struct bitmap __arena *bmp_alloc(size_t bits) +struct arena_bitmap __arena *bmp_alloc(size_t bits) { - struct bitmap __arena *bmp; + struct arena_bitmap __arena *bmp; size_t size = BITS_TO_LONG_LONGS(bits) * sizeof(bmp->bits[0]); /* Assume long-aligned masks. */ if (bits % BITS_PER_LONG_LONG) return NULL; - bmp = (struct bitmap __arena *)arena_malloc(size); + bmp = (struct arena_bitmap __arena *)arena_malloc(size); if (!bmp) return NULL; @@ -29,31 +29,31 @@ struct bitmap __arena *bmp_alloc(size_t bits) } __weak -void bmp_free(struct bitmap __arena *bmp) +void bmp_free(struct arena_bitmap __arena *bmp) { arena_free(bmp); } __weak -void __bmp_set_bit(u32 bit, struct bitmap __arena *bmp) +void __bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp) { bmp->bits[BIT_WORD(bit)] |= BIT_MASK(bit); } __weak -void __bmp_clear_bit(u32 bit, struct bitmap __arena *bmp) +void __bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp) { bmp->bits[BIT_WORD(bit)] &= ~BIT_MASK(bit); } __weak -bool bmp_test_bit(u32 bit, struct bitmap __arena *bmp) +bool bmp_test_bit(u32 bit, struct arena_bitmap __arena *bmp) { return bmp->bits[BIT_WORD(bit)] & BIT_MASK(bit); } __weak -bool bmp_test_and_clear_bit(u32 bit, struct bitmap __arena *bmp) +bool bmp_test_and_clear_bit(u32 bit, struct arena_bitmap __arena *bmp) { u64 val = BIT_MASK(bit); u32 idx = BIT_WORD(bit); @@ -77,7 +77,7 @@ bool bmp_test_and_clear_bit(u32 bit, struct bitmap __arena *bmp) } __weak -bool bmp_test_and_set_bit(u32 bit, struct bitmap __arena *bmp) +bool bmp_test_and_set_bit(u32 bit, struct arena_bitmap __arena *bmp) { u64 val = BIT_MASK(bit); u32 idx = BIT_WORD(bit); @@ -101,7 +101,7 @@ bool bmp_test_and_set_bit(u32 bit, struct bitmap __arena *bmp) } __weak -void bmp_clear_bit(u32 bit, struct bitmap __arena *bmp) +void bmp_clear_bit(u32 bit, struct arena_bitmap __arena *bmp) { u64 val = BIT_MASK(bit); u32 idx = BIT_WORD(bit); @@ -116,7 +116,7 @@ void bmp_clear_bit(u32 bit, struct bitmap __arena *bmp) } __weak -void bmp_set_bit(u32 bit, struct bitmap __arena *bmp) +void bmp_set_bit(u32 bit, struct arena_bitmap __arena *bmp) { u64 val = BIT_MASK(bit); u32 idx = BIT_WORD(bit); @@ -131,7 +131,7 @@ void bmp_set_bit(u32 bit, struct bitmap __arena *bmp) } __weak -void bmp_clear(size_t bits, struct bitmap __arena *bmp) +void bmp_clear(size_t bits, struct arena_bitmap __arena *bmp) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -148,7 +148,7 @@ static __always_inline u64 bmp_last_word_mask(size_t bits) } __weak -void bmp_and(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2) +void bmp_and(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src1, struct arena_bitmap __arena *src2) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -161,7 +161,7 @@ void bmp_and(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src } __weak -void bmp_or(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1, struct bitmap __arena *src2) +void bmp_or(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src1, struct arena_bitmap __arena *src2) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -174,7 +174,7 @@ void bmp_or(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src1 } __weak -bool bmp_empty(size_t bits, struct bitmap __arena *bmp) +bool bmp_empty(size_t bits, struct arena_bitmap __arena *bmp) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -190,7 +190,7 @@ bool bmp_empty(size_t bits, struct bitmap __arena *bmp) } __weak -void bmp_copy(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *src) +void bmp_copy(size_t bits, struct arena_bitmap __arena *dst, struct arena_bitmap __arena *src) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -203,7 +203,7 @@ void bmp_copy(size_t bits, struct bitmap __arena *dst, struct bitmap __arena *sr } __weak -bool bmp_subset(size_t bits, struct bitmap __arena *big, struct bitmap __arena *small) +bool bmp_subset(size_t bits, struct arena_bitmap __arena *big, struct arena_bitmap __arena *small) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -219,7 +219,7 @@ bool bmp_subset(size_t bits, struct bitmap __arena *big, struct bitmap __arena * } __weak -bool bmp_intersects(size_t bits, struct bitmap __arena *arg1, struct bitmap __arena *arg2) +bool bmp_intersects(size_t bits, struct arena_bitmap __arena *arg1, struct arena_bitmap __arena *arg2) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; @@ -235,7 +235,7 @@ bool bmp_intersects(size_t bits, struct bitmap __arena *arg1, struct bitmap __ar } __weak -void bmp_print(size_t bits, struct bitmap __arena *bmp) +void bmp_print(size_t bits, struct arena_bitmap __arena *bmp) { size_t nwords = BITS_TO_LONG_LONGS(bits); volatile u32 i; From ac65c710cc643cbc52b899627577357867249530 Mon Sep 17 00:00:00 2001 From: Nicholas Dudar Date: Wed, 8 Jul 2026 05:07:50 +0200 Subject: [PATCH 043/373] bpf: Reject writes through untrusted BTF pointers check_ptr_to_btf_access() lets program-type btf_struct_access callbacks validate writes before the default BTF access path rejects non-read accesses. That bypasses the read-only policy for untrusted BTF pointers created by helpers such as bpf_rdonly_cast(). Reject non-read accesses through PTR_UNTRUSTED BTF pointers at the common entry point, before the callback branch to handle all cases. Fixes: 282de143ead9 ("bpf: Introduce allocated objects support") Signed-off-by: Nicholas Dudar Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Eduard Zingerman Reviewed-by: Amery Hung Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 51f7965d42e3..4f42b4e929ad 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5790,6 +5790,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, return -EACCES; } + if (atype != BPF_READ && (type_flag(reg->type) & PTR_UNTRUSTED)) { + verbose(env, "only read is supported\n"); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) { if (!btf_is_kernel(reg->btf)) { verifier_bug(env, "reg->btf must be kernel btf"); @@ -5802,8 +5807,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, reg_arg_name(env, argno), tname, off, size); } else { /* Writes are permitted with default btf_struct_access for - * program allocated objects (which always have id > 0), - * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC. + * program allocated objects (which always have id > 0). */ if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) { verbose(env, "only read is supported\n"); From 9eab4790f11f751de2da13436845a39c4ca54e54 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Wed, 8 Jul 2026 05:07:51 +0200 Subject: [PATCH 044/373] selftests/bpf: Add untrusted BTF write regression Add a TCP congestion-control struct_ops load test for a write through a BTF pointer produced by bpf_rdonly_cast(). The test expects the verifier to reject the program before the TCP CA btf_struct_access callback can whitelist the tcp_sock field write. Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Eduard Zingerman Reviewed-by: Amery Hung Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/bpf_tcp_ca.c | 12 +++++++++ .../bpf/progs/tcp_ca_untrusted_btf_write.c | 26 +++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/tcp_ca_untrusted_btf_write.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c index fe30181e6336..eb05fc82f81b 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c @@ -14,6 +14,7 @@ #include "tcp_ca_incompl_cong_ops.skel.h" #include "tcp_ca_unsupp_cong_op.skel.h" #include "tcp_ca_kfunc.skel.h" +#include "tcp_ca_untrusted_btf_write.skel.h" #include "bpf_cc_cubic.skel.h" static const unsigned int total_bytes = 10 * 1024 * 1024; @@ -579,6 +580,15 @@ static void test_tcp_ca_kfunc(void) tcp_ca_kfunc__destroy(skel); } +static void test_untrusted_btf_write(void) +{ + struct tcp_ca_untrusted_btf_write *skel; + + skel = tcp_ca_untrusted_btf_write__open_and_load(); + ASSERT_ERR_PTR(skel, "tcp_ca_untrusted_btf_write__open_and_load"); + tcp_ca_untrusted_btf_write__destroy(skel); +} + static void test_cc_cubic(void) { struct cb_opts cb_opts = { @@ -637,6 +647,8 @@ void test_bpf_tcp_ca(void) test_link_replace(); if (test__start_subtest("tcp_ca_kfunc")) test_tcp_ca_kfunc(); + if (test__start_subtest("untrusted_btf_write")) + test_untrusted_btf_write(); if (test__start_subtest("cc_cubic")) test_cc_cubic(); if (test__start_subtest("dctcp_autoattach_map")) diff --git a/tools/testing/selftests/bpf/progs/tcp_ca_untrusted_btf_write.c b/tools/testing/selftests/bpf/progs/tcp_ca_untrusted_btf_write.c new file mode 100644 index 000000000000..eda4697aac80 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tcp_ca_untrusted_btf_write.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "bpf_tracing_net.h" +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +SEC("struct_ops") +void BPF_PROG(untrusted_btf_write_init, struct sock *sk) +{ + struct tcp_sock *tp; + int v = 1; + void *p; + + p = bpf_rdonly_cast(&v, 0); + tp = bpf_rdonly_cast(p, bpf_core_type_id_kernel(struct tcp_sock)); + tp->snd_cwnd = 1; +} + +SEC(".struct_ops") +struct tcp_congestion_ops untrusted_btf_write = { + .init = (void *)untrusted_btf_write_init, + .name = "bpf_ro_btf", +}; From 0bdbed9133fd10a7dfec6d3e4b5b2a208f86f22a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= Date: Thu, 2 Jul 2026 10:26:30 +0200 Subject: [PATCH 045/373] tools/resolve_btfids: Include libsubcmd headers directly from source tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently each build with resolve_btfids enabled unnecessarily prints the line 'INSTALL libsubcmd_headers' from libsubcmd. Use the libcmd headers from source tree instead, without installation. The same was done for objtool in commit ac999926774a ("objtool: Include libsubcmd headers directly from source tree"), albeit for a different reason. Signed-off-by: Thomas Weißschuh Signed-off-by: Eduard Zingerman Tested-by: Ihor Solodrai Link: https://patch.msgid.link/20260702-libsubcmd-spam-v1-1-300ec142a62f@linutronix.de Signed-off-by: Eduard Zingerman --- tools/bpf/resolve_btfids/Makefile | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tools/bpf/resolve_btfids/Makefile b/tools/bpf/resolve_btfids/Makefile index 7672208f65e4..bb0463b380af 100644 --- a/tools/bpf/resolve_btfids/Makefile +++ b/tools/bpf/resolve_btfids/Makefile @@ -40,7 +40,6 @@ LIBBPF_DESTDIR := $(LIBBPF_OUT) LIBBPF_INCLUDE := $(LIBBPF_DESTDIR)include SUBCMD_DESTDIR := $(SUBCMD_OUT) -SUBCMD_INCLUDE := $(SUBCMD_DESTDIR)include BINARY := $(OUTPUT)/resolve_btfids BINARY_IN := $(BINARY)-in.o @@ -56,7 +55,7 @@ $(OUTPUT) $(OUTPUT)/libsubcmd $(LIBBPF_OUT): $(SUBCMDOBJ): fixdep FORCE | $(OUTPUT)/libsubcmd $(Q)$(MAKE) -C $(SUBCMD_SRC) OUTPUT=$(SUBCMD_OUT) \ DESTDIR=$(SUBCMD_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \ - $(abspath $@) install_headers + $(abspath $@) $(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(LIBBPF_OUT) $(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) OUTPUT=$(LIBBPF_OUT) \ @@ -77,7 +76,7 @@ HOSTCFLAGS_resolve_btfids += -g \ -I$(srctree)/tools/include \ -I$(srctree)/tools/include/uapi \ -I$(LIBBPF_INCLUDE) \ - -I$(SUBCMD_INCLUDE) \ + -I$(srctree)/tools/lib \ $(LIBELF_FLAGS) \ -Wall -Werror From d5a85392392c77b61a74e975a74da0e9c146f6d3 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:36 +0200 Subject: [PATCH 046/373] bpf: Resolve and cache fd_array objects at load time The fd_array passed to BPF_PROG_LOAD carries the map and module BTF file descriptors a program binds. The verifier reads it more than once during a load: process_fd_array() walks it to bind the maps and BTFs, and check_and_resolve_insns() and the kfunc BTF resolver later read it again to resolve the program's BPF_PSEUDO_MAP_IDX* and module kfunc refs. For signed BPF, we need these upfront in memory, thus resolve each fd to its object once and cache it by fd_array index, then bind that cached object for the rest of the load. env->fd_array becomes a small per-slot {map, btf} cache rather than a bpfptr_t; every later reference is then an in-bounds lookup of an already-resolved object, and an index outside the cache is rejected instead of read from user memory: - continuous (fd_array_cnt given): the caller declares the length and every entry is resolved and bound up front (used also by the BPF signed loader) - sparse (no fd_array_cnt): left as the legacy path with no fd_array cache; each reference reads its fd from the caller's fd_array and resolves it on the spot. Deduplication in used_maps and the kfunc BTF table keeps this correct, and only unsigned programs use this shape. Split these into separate helpers to make it easier to follow. Signed-off-by: Daniel Borkmann Acked-by: Anton Protopopov Link: https://lore.kernel.org/bpf/20260708075343.358712-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 22 +++- kernel/bpf/verifier.c | 223 +++++++++++++++++++++++++++-------- 2 files changed, 193 insertions(+), 52 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 76b8b7627a10..bb57773cde37 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -898,6 +898,14 @@ struct bpf_scc_info { struct bpf_liveness; +struct bpf_fd_array { + union { + struct bpf_map *map; + struct btf *btf; + unsigned long val; + }; +}; + /* single container for all structs * one verifier_env per bpf_check() call */ @@ -989,7 +997,19 @@ struct bpf_verifier_env { u32 free_list_size; u32 explored_states_size; u32 num_backedges; - bpfptr_t fd_array; + /* + * The program's fd_array comes in two shapes, told apart by whether + * the caller passed fd_array_cnt. They are mutually exclusive: + * - continuous (fd_array_cnt given): ->fd_array holds every entry + * resolved to its object up front, indexed by fd_array position, + * with ->fd_array_cnt slots; ->fd_array_raw is unused. + * - sparse (no fd_array_cnt): ->fd_array is NULL, and entries are + * read from ->fd_array_raw (the caller's fd_array) and resolved + * on the spot at each reference. + */ + struct bpf_fd_array *fd_array; + u32 fd_array_cnt; + bpfptr_t fd_array_raw; /* bit mask to keep track of whether a register has been accessed * since the last time the function state was printed diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4f42b4e929ad..e8e21d1a919a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2490,6 +2490,79 @@ int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id, return 0; } +#define BPF_FD_SLOT_BTF 1UL + +static void fd_slot_set_map(struct bpf_fd_array *slot, struct bpf_map *map) +{ + slot->val = (unsigned long)map; +} + +static void fd_slot_set_btf(struct bpf_fd_array *slot, struct btf *btf) +{ + slot->val = (unsigned long)btf | BPF_FD_SLOT_BTF; +} + +static struct bpf_map *fd_slot_map(struct bpf_fd_array slot) +{ + if (slot.val & BPF_FD_SLOT_BTF) + return NULL; + return (struct bpf_map *)slot.val; +} + +static struct btf *fd_slot_btf(struct bpf_fd_array slot) +{ + if (!(slot.val & BPF_FD_SLOT_BTF)) + return NULL; + return (struct btf *)(slot.val & ~BPF_FD_SLOT_BTF); +} + +static struct btf * +fd_array_get_btf_continuous(struct bpf_verifier_env *env, u32 idx) +{ + struct btf *btf; + + if (idx >= env->fd_array_cnt) { + verbose(env, "kfunc fd_idx %u out of bounds, fd_array_cnt %u\n", + idx, env->fd_array_cnt); + return ERR_PTR(-EINVAL); + } + btf = fd_slot_btf(env->fd_array[idx]); + if (!btf) { + verbose(env, "kfunc fd_idx %u is not a module BTF\n", idx); + return ERR_PTR(-EINVAL); + } + btf_get(btf); + return btf; +} + +static struct btf * +fd_array_get_btf_sparse(struct bpf_verifier_env *env, u32 idx) +{ + struct btf *btf; + int btf_fd; + + if (copy_from_bpfptr_offset(&btf_fd, env->fd_array_raw, + (size_t)idx * sizeof(btf_fd), sizeof(btf_fd))) + return ERR_PTR(-EFAULT); + btf = btf_get_by_fd(btf_fd); + if (IS_ERR(btf)) { + verbose(env, "invalid module BTF fd specified\n"); + return btf; + } + return btf; +} + +static struct btf *fd_array_get_btf(struct bpf_verifier_env *env, u32 idx) +{ + if (env->fd_array) + return fd_array_get_btf_continuous(env, idx); + if (!bpfptr_is_null(env->fd_array_raw)) + return fd_array_get_btf_sparse(env, idx); + + verbose(env, "kfunc offset > 0 without fd_array is invalid\n"); + return ERR_PTR(-EPROTO); +} + static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset) { @@ -2498,7 +2571,6 @@ static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env, struct bpf_kfunc_btf *b; struct module *mod; struct btf *btf; - int btf_fd; tab = env->prog->aux->kfunc_btf_tab; b = bsearch(&kf_btf, tab->descs, tab->nr_descs, @@ -2509,22 +2581,9 @@ static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env, return ERR_PTR(-E2BIG); } - if (bpfptr_is_null(env->fd_array)) { - verbose(env, "kfunc offset > 0 without fd_array is invalid\n"); - return ERR_PTR(-EPROTO); - } - - if (copy_from_bpfptr_offset(&btf_fd, env->fd_array, - offset * sizeof(btf_fd), - sizeof(btf_fd))) - return ERR_PTR(-EFAULT); - - btf = btf_get_by_fd(btf_fd); - if (IS_ERR(btf)) { - verbose(env, "invalid module BTF fd specified\n"); + btf = fd_array_get_btf(env, offset); + if (IS_ERR(btf)) return btf; - } - if (!btf_is_module(btf)) { verbose(env, "BTF fd for kfunc is not a module BTF\n"); btf_put(btf); @@ -17902,6 +17961,44 @@ static int add_used_map(struct bpf_verifier_env *env, int fd) return __add_used_map(env, map); } +static int fd_array_get_map_idx_continuous(struct bpf_verifier_env *env, u32 idx) +{ + struct bpf_map *map; + + if (idx >= env->fd_array_cnt) { + verbose(env, "fd_idx %u out of bounds, fd_array_cnt %u\n", + idx, env->fd_array_cnt); + return -EINVAL; + } + map = fd_slot_map(env->fd_array[idx]); + if (!map) { + verbose(env, "fd_idx %u is not a map\n", idx); + return -EINVAL; + } + return __add_used_map(env, map); +} + +static int fd_array_get_map_idx_sparse(struct bpf_verifier_env *env, u32 idx) +{ + int fd; + + if (copy_from_bpfptr_offset(&fd, env->fd_array_raw, + (size_t)idx * sizeof(fd), sizeof(fd))) + return -EFAULT; + return add_used_map(env, fd); +} + +static int fd_array_get_map_idx(struct bpf_verifier_env *env, u32 idx) +{ + if (env->fd_array) + return fd_array_get_map_idx_continuous(env, idx); + if (!bpfptr_is_null(env->fd_array_raw)) + return fd_array_get_map_idx_sparse(env, idx); + + verbose(env, "fd_idx without fd_array is invalid\n"); + return -EPROTO; +} + static int check_alu_fields(struct bpf_verifier_env *env, struct bpf_insn *insn) { u8 class = BPF_CLASS(insn->code); @@ -18119,7 +18216,6 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env) struct bpf_map *map; int map_idx; u64 addr; - u32 fd; if (i == insn_cnt - 1 || insn[1].code != 0 || insn[1].dst_reg != 0 || insn[1].src_reg != 0 || @@ -18171,21 +18267,13 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env) switch (insn[0].src_reg) { case BPF_PSEUDO_MAP_IDX_VALUE: case BPF_PSEUDO_MAP_IDX: - if (bpfptr_is_null(env->fd_array)) { - verbose(env, "fd_idx without fd_array is invalid\n"); - return -EPROTO; - } - if (copy_from_bpfptr_offset(&fd, env->fd_array, - insn[0].imm * sizeof(fd), - sizeof(fd))) - return -EFAULT; + map_idx = fd_array_get_map_idx(env, insn[0].imm); break; default: - fd = insn[0].imm; + map_idx = add_used_map(env, insn[0].imm); break; } - map_idx = add_used_map(env, fd); if (map_idx < 0) return map_idx; map = env->used_maps[map_idx]; @@ -19460,7 +19548,7 @@ struct btf *bpf_get_btf_vmlinux(void) * this case expect that every file descriptor in the array is either a map or * a BTF. Everything else is considered to be trash. */ -static int add_fd_from_fd_array(struct bpf_verifier_env *env, int fd) +static int add_fd_from_fd_array(struct bpf_verifier_env *env, u32 idx, int fd) { struct bpf_map *map; struct btf *btf; @@ -19472,51 +19560,83 @@ static int add_fd_from_fd_array(struct bpf_verifier_env *env, int fd) err = __add_used_map(env, map); if (err < 0) return err; + fd_slot_set_map(&env->fd_array[idx], map); return 0; } btf = __btf_get_by_fd(f); if (!IS_ERR(btf)) { btf_get(btf); - return __add_used_btf(env, btf); + err = __add_used_btf(env, btf); + if (err < 0) + return err; + fd_slot_set_btf(&env->fd_array[idx], btf); + return 0; } verbose(env, "fd %d is not pointing to valid bpf_map or btf\n", fd); return PTR_ERR(map); } -static int process_fd_array(struct bpf_verifier_env *env, union bpf_attr *attr, bpfptr_t uattr) +/* + * A continuous fd_array is resolved into an in-memory cache with one slot + * per entry. The bound here is deliberately generous and not derived from + * the per-program object limits: Duplicate entries /are/ permitted, and + * the number of distinct maps and BTFs a program can bind is enforced when + * each entry is resolved by __add_used_map() and __add_used_btf(). + */ +#define MAX_FD_ARRAY_CNT 4096 + +static int process_fd_array_continuous(struct bpf_verifier_env *env, + bpfptr_t fd_array, u32 cnt) { - size_t size = sizeof(int); - int ret; - int fd; + int fd, ret; u32 i; - env->fd_array = make_bpfptr(attr->fd_array, uattr.is_kernel); - - /* - * The only difference between old (no fd_array_cnt is given) and new - * APIs is that in the latter case the fd_array is expected to be - * continuous and is scanned for map fds right away - */ - if (!attr->fd_array_cnt) - return 0; - - /* Check for integer overflow */ - if (attr->fd_array_cnt >= (U32_MAX / size)) { - verbose(env, "fd_array_cnt is too big (%u)\n", attr->fd_array_cnt); - return -EINVAL; + if (cnt > MAX_FD_ARRAY_CNT) { + verbose(env, "fd_array has too many entries (%u, max %u)\n", + cnt, MAX_FD_ARRAY_CNT); + return -E2BIG; } - for (i = 0; i < attr->fd_array_cnt; i++) { - if (copy_from_bpfptr_offset(&fd, env->fd_array, i * size, size)) + env->fd_array = kvcalloc(cnt, sizeof(*env->fd_array), + GFP_KERNEL_ACCOUNT); + if (!env->fd_array) + return -ENOMEM; + env->fd_array_cnt = cnt; + for (i = 0; i < cnt; i++) { + if (copy_from_bpfptr_offset(&fd, fd_array, + (size_t)i * sizeof(fd), sizeof(fd))) return -EFAULT; - - ret = add_fd_from_fd_array(env, fd); + ret = add_fd_from_fd_array(env, i, fd); if (ret) return ret; } + return 0; +} +static int process_fd_array(struct bpf_verifier_env *env, + union bpf_attr *attr, bpfptr_t uattr) +{ + bpfptr_t fd_array = make_bpfptr(attr->fd_array, uattr.is_kernel); + + if (bpfptr_is_null(fd_array)) { + if (attr->fd_array_cnt) { + verbose(env, "fd_array_cnt %u without fd_array is invalid\n", + attr->fd_array_cnt); + return -EINVAL; + } + return 0; + } + /* + * New API: the caller passes fd_array_cnt and a continuous array that + * is resolved and bound up front. Legacy API (no fd_array_cnt): keep + * the caller's array and resolve entries on the spot at each reference. + */ + if (attr->fd_array_cnt) + return process_fd_array_continuous(env, fd_array, + attr->fd_array_cnt); + env->fd_array_raw = fd_array; return 0; } @@ -20017,6 +20137,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, mutex_unlock(&bpf_verifier_lock); bpf_clear_insn_aux_data(env, 0, env->prog->len); err_free_env: + kvfree(env->fd_array); bpf_stack_liveness_free(env); kvfree(env->cfg.insn_postorder); kvfree(env->scc_info); From b707068e0ed92b64bb66bae4f6f3a521f7017220 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:37 +0200 Subject: [PATCH 047/373] bpf: Verify signed loader metadata at load time A signed gen_loader program carries the programs, maps and relocations it installs in a metadata array map. The loader instructions are covered by the PKCS#7 signature, but the metadata map is not: Today the loader compares the map contents from within BPF against a hash baked into its (signed) instructions, using the kernel-cached map hash. The kernel itself never actually attests that the metadata the loader installs is the metadata that was signed. This split is the core of the long-standing objection to the BPF signing scheme from the LSM / integrity side: the integrity check of a light skeleton only completes once the loader program runs, that is, after the security_bpf_prog_load() hook, so at admission time an LSM observes a program whose payload has not yet been verified. Auditing the chain link is also not a purely cryptographic operation: whoever signs or reviews an lskel has to disassemble the loader's preamble to convince themselves that the embedded hash check is present and correct [0][1]. Two acceptable fixes were identified in those threads: Complete the integrity check before the admission hook fires, or add a second hook that collects the verification result after the loader ran [2]. Covering both the loader and its maps directly with the PKCS#7 signature is what Blaise Boscaccy's patchsets proposed in several forms. Let's implement the former, without growing the UAPI, and in particular as a single unified scheme where the signature spans the raw bytes rather than derived hashes. A signed loader binds its metadata map(s) through the existing fd_array, and an exclusive map is already bound to a program digest (excl_prog_hash). So when a signature is present, collect the exclusive maps from fd_array and append their frozen contents to the instructions before verification: The signature now covers insns || metadata_0 || metadata_1 || [...] in the fd_array order, and verification completes in bpf_check(), once the fd_array maps are resolved into used_maps, before the LSM admission hook and the rest of verification. A program is either BPF_SIG_UNSIGNED or BPF_SIG_VERIFIED, with nothing in between. While folding the fd_array maps, a non-exclusive map bound to a signed program is rejected, so every map folded into the signature is exclusive. A signed loader that fails to cover its metadata thus does not load, and BPF_SIG_VERIFIED always means the instructions and every exclusive map are authentic. The maps must be frozen so the hashed bytes cannot change before the loader runs; the map <-> program digest binding is enforced by the verifier for every used map. Binding maps through fd_array_cnt makes the verifier resolve and excl-check them (excl_prog_sha vs prog->digest) before it would otherwise compute the digest, so compute prog->digest up front in bpf_check(), over the unmodified instructions the signature covers, for a load that folds metadata. Unsigned programs are not affected by the signature path; for them the LSM admission hook merely moves below fd_array resolution, with minimal bounded work in between. Note, signed loaders generated by older libbpf/ bpftool versions need to be regenerated; some of the recent fixes we've had on the signed loader side require the latter already to close gaps. Finally, some remarks around the security_bpf_prog_load() placement given there was discussion on whether a new hook is needed or the existing security_bpf_prog() hook should be reused [3]: For a new hook it would mean that just for loading a single BPF program it has to pass through four layers of LSM hooks: 1) security_bpf (cmd=PROG_LOAD): for gating various bpf subcmds 2) security_bpf_prog_load: historical admission hook (CAP/token, prog_type, attach point), pre-verification 3) security_bpf_prog_verify_signature: newly asked admission hook, same role as 2), plus the BPF signature verdict 4) security_bpf_prog: gate handing the prog fd back to userspace, verification done & signature verified The use-cases of 2) and 3) conflate, thus BPF community prefers to just keep a total of 3 LSM hooks (as-is today): 3) makes 2) incoherent given they are the /same class/ of hook, that is, access-control admission on the load and split only by _what_ they can see. Worse, with the split, for a signed BPF program security_bpf_prog_load 2) admits a program whose signature has not been checked, so a policy gating at 2) is structurally unable to express "admit only verified" and every such policy is forced onto 3) *anyway*. In other words, one doesn't get two complementary hooks, but rather, one real admission hook aka 3) plus a now-degraded /legacy/ hook 2) that can't answer the question operators actually want to ask. Reusing security_bpf_prog() 4) for admission is no alternative either: it fires only after the entire verifier (and JIT) pipeline ran, so denying a not-yet-verified program at that point burns exactly the work a denial is supposed to avoid, and by then the program has an id assigned and the kallsyms/perf/audit load events fired. Policies are free to also consume the signature verdict at 4), but admission control belongs into security_bpf_prog_load(). Hence the latter remains the only admission hook, merely moved past signature verification; with moving large allocations further down into the BPF verifier, there is now only minimal work between the old and new location: The preparation work in bpf_check() is reordered such that only the minimally necessary setup happens up front: Allocating the env, initializing the verifier log and resolving the fd_array that a signed BPF metadata map needs. The worst case allocation up until security_bpf_prog_load() is ~90K which is the env itself (~54K) plus the continuous fd_array cache (at most 32K). The insn_aux_data array is moved into a later stage in the verification. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/2f71d6c03698eb17d51f7247efde777627ee578a.camel@HansenPartnership.com [0] Link: https://lore.kernel.org/lkml/ecf0521ed302db672672ebfbc670ecfba36a6e00.camel@HansenPartnership.com [1] Link: https://lore.kernel.org/bpf/88703f00d5b7a779728451008626efa45e42db3d.camel@HansenPartnership.com [2] Link: https://lore.kernel.org/bpf/DJOFY21DYUI4.19WKQ3NPZ4H5R@gmail.com [3] Link: https://lore.kernel.org/bpf/20260708075343.358712-3-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 1 + kernel/bpf/syscall.c | 76 +---------- kernel/bpf/verifier.c | 238 +++++++++++++++++++++++++++++++---- 3 files changed, 217 insertions(+), 98 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index bb57773cde37..317e99b9acc0 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -947,6 +947,7 @@ struct bpf_verifier_env { bool bypass_spec_v4; bool seen_direct_write; bool seen_exception; + bool signature; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6db306d23b47..e898fad01aaf 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -40,7 +40,6 @@ #include #include #include -#include #include #include @@ -2886,64 +2885,6 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) } } -static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id) -{ - switch (keyring_id) { - case 0: - return BPF_SIG_KEYRING_BUILTIN; - case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING: - return BPF_SIG_KEYRING_SECONDARY; - case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING: - return BPF_SIG_KEYRING_PLATFORM; - default: - return BPF_SIG_KEYRING_USER; - } -} - -static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr, - bool is_kernel, s32 *keyring_serial) -{ - bpfptr_t usig = make_bpfptr(attr->signature, is_kernel); - struct bpf_dynptr_kern sig_ptr, insns_ptr; - struct bpf_key *key = NULL; - void *sig; - int err = 0; - - /* - * Don't attempt to use kmalloc_large or vmalloc for signatures. - * Practical signature for BPF program should be below this limit. - */ - if (attr->signature_size > KMALLOC_MAX_CACHE_SIZE) - return -EINVAL; - - if (system_keyring_id_check(attr->keyring_id) == 0) - key = bpf_lookup_system_key(attr->keyring_id); - else - key = bpf_lookup_user_key(attr->keyring_id, 0); - - if (!key) - return -EINVAL; - - sig = kvmemdup_bpfptr(usig, attr->signature_size); - if (IS_ERR(sig)) { - bpf_key_put(key); - return PTR_ERR(sig); - } - - bpf_dynptr_init(&sig_ptr, sig, BPF_DYNPTR_TYPE_LOCAL, 0, - attr->signature_size); - bpf_dynptr_init(&insns_ptr, prog->insnsi, BPF_DYNPTR_TYPE_LOCAL, 0, - prog->len * sizeof(struct bpf_insn)); - - err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr, - (struct bpf_dynptr *)&sig_ptr, key); - if (!err) - *keyring_serial = bpf_key_serial(key); - bpf_key_put(key); - kvfree(sig); - return err; -} - static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog) { int err; @@ -3133,17 +3074,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at /* eBPF programs must be GPL compatible to use GPL-ed functions */ prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0; - if (attr->signature) { - err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel, - &prog->aux->sig.keyring_serial); - if (err) - goto free_prog; - prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id); - prog->aux->sig.verdict = BPF_SIG_VERIFIED; - } else { - prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE; - prog->aux->sig.verdict = BPF_SIG_UNSIGNED; - } + prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE; + prog->aux->sig.verdict = BPF_SIG_UNSIGNED; prog->orig_prog = NULL; prog->jited = 0; @@ -3189,10 +3121,6 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at if (err < 0) goto free_prog; - err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel); - if (err) - goto free_prog; - /* run eBPF verifier */ err = bpf_check(&prog, attr, uattr, attr_log); if (err < 0) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e8e21d1a919a..001ac53825da 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -22,6 +22,8 @@ #include #include #include +#include +#include #include #include #include @@ -2554,6 +2556,10 @@ fd_array_get_btf_sparse(struct bpf_verifier_env *env, u32 idx) static struct btf *fd_array_get_btf(struct bpf_verifier_env *env, u32 idx) { + if (env->signature) { + verbose(env, "signed program cannot bind any BTF\n"); + return ERR_PTR(-EACCES); + } if (env->fd_array) return fd_array_get_btf_continuous(env, idx); if (!bpfptr_is_null(env->fd_array_raw)) @@ -17627,6 +17633,11 @@ static int __add_used_btf(struct bpf_verifier_env *env, struct btf *btf) if (env->used_btfs[i].btf == btf) goto ret_put; + if (env->signature) { + verbose(env, "signed program cannot bind any BTF\n"); + ret = -EACCES; + goto ret_put; + } if (env->used_btf_cnt >= MAX_USED_BTFS) { verbose(env, "The total number of btfs per program has reached the limit of %u\n", MAX_USED_BTFS); @@ -17909,6 +17920,12 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map) if (env->used_maps[i] == map) return i; + if (env->signature && + env->prog->aux->sig.verdict == BPF_SIG_VERIFIED) { + verbose(env, "signed program cannot bind map '%s' not covered by the signature\n", + map->name); + return -EACCES; + } if (env->used_map_cnt >= MAX_USED_MAPS) { verbose(env, "The total number of maps per program has reached the limit of %u\n", MAX_USED_MAPS); @@ -17992,6 +18009,10 @@ static int fd_array_get_map_idx(struct bpf_verifier_env *env, u32 idx) { if (env->fd_array) return fd_array_get_map_idx_continuous(env, idx); + if (env->signature) { + verbose(env, "signed program must bind maps via a continuous fd_array (fd_array_cnt)\n"); + return -EACCES; + } if (!bpfptr_is_null(env->fd_array_raw)) return fd_array_get_map_idx_sparse(env, idx); @@ -18270,6 +18291,10 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env) map_idx = fd_array_get_map_idx(env, insn[0].imm); break; default: + if (env->signature) { + verbose(env, "signed program cannot reference a map by fd, only via fd_array index\n"); + return -EINVAL; + } map_idx = add_used_map(env, insn[0].imm); break; } @@ -19851,6 +19876,146 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return 0; } +static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id) +{ + switch (keyring_id) { + case 0: + return BPF_SIG_KEYRING_BUILTIN; + case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING: + return BPF_SIG_KEYRING_SECONDARY; + case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING: + return BPF_SIG_KEYRING_PLATFORM; + default: + return BPF_SIG_KEYRING_USER; + } +} + +/* + * Verify the PKCS#7 signature of a loaded program. Called from bpf_check() + * once the program's metadata maps have been resolved into used_maps, so + * the exact maps folded into the signature are the ones the program binds. + * + * The signature covers the instructions followed by the frozen contents of + * each map, in @maps order: insns || map_0 || map_1 || [...]. On success the + * verdict and keyring info are recorded on prog->aux. + */ +static int bpf_prog_verify_signature(struct bpf_verifier_env *env, + union bpf_attr *attr, bool is_kernel) +{ + bpfptr_t usig = make_bpfptr(attr->signature, is_kernel); + struct bpf_dynptr_kern sig_ptr, data_ptr; + struct bpf_prog *prog = env->prog; + struct bpf_map **maps = env->used_maps; + struct bpf_key *key = NULL; + void *sig, *data = NULL; + u32 map_cnt = env->used_map_cnt; + u32 i, off, insns_sz; + u64 data_sz; + int err = 0; + + /* + * Don't attempt to use kmalloc_large or vmalloc for signatures. + * Practical signature for BPF program should be below this limit. + */ + if (!attr->signature_size || + attr->signature_size > KMALLOC_MAX_CACHE_SIZE) + return -EINVAL; + if (system_keyring_id_check(attr->keyring_id) == 0) + key = bpf_lookup_system_key(attr->keyring_id); + else + key = bpf_lookup_user_key(attr->keyring_id, 0); + if (!key) { + verbose(env, "cannot resolve signing keyring with keyring_id %d\n", + attr->keyring_id); + return -EINVAL; + } + + sig = kvmemdup_bpfptr(usig, attr->signature_size); + if (IS_ERR(sig)) { + bpf_key_put(key); + return PTR_ERR(sig); + } + + insns_sz = prog->len * sizeof(struct bpf_insn); + data_sz = insns_sz; + for (i = 0; i < map_cnt; i++) { + struct bpf_map *map = maps[i]; + + if (map->map_type != BPF_MAP_TYPE_ARRAY || + !map->ops->map_direct_value_addr) { + verbose(env, "signed program metadata map '%s' must be an array\n", + map->name); + err = -EINVAL; + goto out; + } + if (!READ_ONCE(map->frozen)) { + verbose(env, "signed program metadata map '%s' must be frozen\n", + map->name); + err = -EPERM; + goto out; + } + if (bpf_map_write_active(map)) { + verbose(env, "signed program metadata map '%s' has active writers\n", + map->name); + err = -EBUSY; + goto out; + } + if (!map->excl_prog_sha) { + verbose(env, "signed program metadata map '%s' must be exclusive\n", + map->name); + err = -EPERM; + goto out; + } + data_sz += map->value_size; + } + if (bpf_dynptr_check_size(data_sz)) { + verbose(env, "signed payload too large: %llu bytes\n", data_sz); + err = -E2BIG; + goto out; + } + data = kvmalloc(data_sz, GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (!data) { + err = -ENOMEM; + goto out; + } + memcpy(data, prog->insnsi, insns_sz); + off = insns_sz; + for (i = 0; i < map_cnt; i++) { + struct bpf_map *map = maps[i]; + u64 addr; + + err = map->ops->map_direct_value_addr(map, &addr, 0); + if (err) { + verbose(env, "failed to read signed metadata map '%s': %d\n", + map->name, err); + goto out; + } + memcpy(data + off, (void *)(unsigned long)addr, + map->value_size); + off += map->value_size; + } + + bpf_dynptr_init(&data_ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, data_sz); + bpf_dynptr_init(&sig_ptr, sig, BPF_DYNPTR_TYPE_LOCAL, 0, + attr->signature_size); + + err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&data_ptr, + (struct bpf_dynptr *)&sig_ptr, key); + if (err) { + verbose(env, "signature verification failed: %d\n", err); + } else { + verbose(env, "signature verification passed\n"); + prog->aux->sig.keyring_serial = bpf_key_serial(key); + prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id); + prog->aux->sig.verdict = BPF_SIG_VERIFIED; + } +out: + kvfree(data); + bpf_key_put(key); + kvfree(sig); + return err; +} + int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) { @@ -19873,18 +20038,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, return -ENOMEM; env->bt.env = env; - - len = (*prog)->len; - env->insn_aux_data = - vzalloc(array_size(sizeof(struct bpf_insn_aux_data), len)); - ret = -ENOMEM; - if (!env->insn_aux_data) - goto err_free_env; - for (i = 0; i < len; i++) - env->insn_aux_data[i].orig_idx = i; - env->succ = bpf_iarray_realloc(NULL, 2); - if (!env->succ) - goto err_free_env; env->prog = *prog; env->ops = bpf_verifier_ops[env->prog->type]; @@ -19893,6 +20046,34 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, env->bypass_spec_v1 = bpf_bypass_spec_v1(env->prog->aux->token); env->bypass_spec_v4 = bpf_bypass_spec_v4(env->prog->aux->token); env->bpf_capable = is_priv = bpf_token_capable(env->prog->aux->token, CAP_BPF); + env->signature = attr->signature; + + /* user could have requested verbose verifier output + * and supplied buffer to store the verification trace + */ + ret = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size); + if (ret) + goto err_free_env; + if (env->signature) { + ret = bpf_prog_calc_tag(env->prog); + if (ret < 0) + goto err_prep; + } + + ret = process_fd_array(env, attr, uattr); + if (ret) + goto err_prep; + + if (env->signature) { + ret = bpf_prog_verify_signature(env, attr, uattr.is_kernel); + if (ret) + goto err_prep; + } + + ret = security_bpf_prog_load(env->prog, attr, env->prog->aux->token, + uattr.is_kernel); + if (ret) + goto err_prep; bpf_get_btf_vmlinux(); @@ -19900,15 +20081,16 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (!is_priv) mutex_lock(&bpf_verifier_lock); - /* user could have requested verbose verifier output - * and supplied buffer to store the verification trace - */ - ret = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size); - if (ret) - goto err_unlock; - - ret = process_fd_array(env, attr, uattr); - if (ret) + len = env->prog->len; + env->insn_aux_data = + vzalloc(array_size(sizeof(struct bpf_insn_aux_data), len)); + ret = -ENOMEM; + if (!env->insn_aux_data) + goto skip_full_check; + for (i = 0; i < len; i++) + env->insn_aux_data[i].orig_idx = i; + env->succ = bpf_iarray_realloc(NULL, 2); + if (!env->succ) goto skip_full_check; mark_verifier_state_clean(env); @@ -20132,18 +20314,26 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, *prog = env->prog; module_put(env->attach_btf_mod); -err_unlock: if (!is_priv) mutex_unlock(&bpf_verifier_lock); - bpf_clear_insn_aux_data(env, 0, env->prog->len); + goto err_free_env; +err_prep: + err = bpf_log_attr_finalize(attr_log, &env->log); + if (err) + ret = err; + release_insn_arrays(env); + release_maps(env); + release_btfs(env); err_free_env: + if (env->insn_aux_data) + bpf_clear_insn_aux_data(env, 0, env->prog->len); + vfree(env->insn_aux_data); kvfree(env->fd_array); bpf_stack_liveness_free(env); kvfree(env->cfg.insn_postorder); kvfree(env->scc_info); kvfree(env->succ); kvfree(env->gotox_tmp_buf); - vfree(env->insn_aux_data); kvfree(env); return ret; } From a2d784869a0f252e1a277db7dc2c16d55694da72 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:38 +0200 Subject: [PATCH 048/373] libbpf: Drop in-loader metadata check for load-time verification The signed gen_loader used to police its own metadata map from within BPF: emit_signature_match() read the kernel-cached map->sha[] back through hardcoded struct bpf_map offsets and compared it against a hash that compute_sha_update_offsets() baked into the signed instructions, after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[]. The kernel now verifies the metadata at BPF_PROG_LOAD time by folding the frozen contents of the loader's exclusive fd_array maps into the signature, so the loader no longer checks anything itself. Generated loaders thus carry no verification logic of their own anymore: Nothing in the signing chain depends on emitted loader bytecode doing the right thing. On the loading side, skel_internal.h now sets fd_array_cnt for a signed load so the kernel scans fd_array for the exclusive metadata map - still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[] is gone. The struct bpf_map layout BUILD_BUG_ON()s on the kernel side are removed as well: they only pinned the ABI for the in-BPF read of map->sha[] that is no longer needed. Same for the map->excl member. Note: gen_hash is retained; it still marks a loader as signed so an untrusted host cannot re-dimension maps or override initial values now covered by the signature. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 1 - kernel/bpf/syscall.c | 7 --- tools/lib/bpf/bpf_gen_internal.h | 1 - tools/lib/bpf/gen_loader.c | 76 +++----------------------------- tools/lib/bpf/libbpf_internal.h | 1 - tools/lib/bpf/skel_internal.h | 31 +------------ 6 files changed, 9 insertions(+), 108 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index adf53f7edf28..c1a98fa36738 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -299,7 +299,6 @@ struct bpf_map_owner { struct bpf_map { u8 sha[SHA256_DIGEST_SIZE]; - u32 excl; const struct bpf_map_ops *ops; struct bpf_map *inner_map_meta; #ifdef CONFIG_SECURITY diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index e898fad01aaf..358f2b0ce2bd 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1598,13 +1598,6 @@ static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_ver err = -EFAULT; goto free_map; } - - /* See libbpf: emit_signature_match() */ - BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE); - BUILD_BUG_ON(!__same_type(map->excl, u32)); - BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0); - BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE])); - map->excl = 1; } else if (attr->excl_prog_hash_size) { bpf_log(log, "Invalid excl_prog_hash_size.\n"); err = -EINVAL; diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h index 49af4260b8e6..042569187752 100644 --- a/tools/lib/bpf/bpf_gen_internal.h +++ b/tools/lib/bpf/bpf_gen_internal.h @@ -51,7 +51,6 @@ struct bpf_gen { __u32 nr_ksyms; int fd_array; int nr_fd_array; - int hash_insn_offset[SHA256_DWORD_SIZE]; }; void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps); diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index c7f2d2ac7bb3..6e3dd5242761 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -111,7 +111,6 @@ static void emit2(struct bpf_gen *gen, struct bpf_insn insn1, struct bpf_insn in static int add_data(struct bpf_gen *gen, const void *data, __u32 size); static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off); -static void emit_signature_match(struct bpf_gen *gen); void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps) { @@ -154,8 +153,6 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps /* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */ emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_7)); emit(gen, BPF_EXIT_INSN()); - if (OPTS_GET(gen->opts, gen_hash, false)) - emit_signature_match(gen); } static int add_data(struct bpf_gen *gen, const void *data, __u32 size) @@ -377,8 +374,6 @@ static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off) __emit_sys_close(gen); } -static void compute_sha_update_offsets(struct bpf_gen *gen); - int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) { int i; @@ -408,9 +403,6 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) if (!gen->error) { struct gen_loader_opts *opts = gen->opts; - if (OPTS_GET(opts, gen_hash, false)) - compute_sha_update_offsets(gen); - opts->insns = gen->insn_start; opts->insns_sz = gen->insn_cur - gen->insn_start; opts->data = gen->data_start; @@ -460,22 +452,6 @@ void bpf_gen__free(struct bpf_gen *gen) _val; \ }) -static void compute_sha_update_offsets(struct bpf_gen *gen) -{ - __u64 sha[SHA256_DWORD_SIZE]; - __u64 sha_dw; - int i; - - libbpf_sha256(gen->data_start, gen->data_cur - gen->data_start, (__u8 *)sha); - for (i = 0; i < SHA256_DWORD_SIZE; i++) { - struct bpf_insn *insn = - (struct bpf_insn *)(gen->insn_start + gen->hash_insn_offset[i]); - sha_dw = tgt_endian(sha[i]); - insn[0].imm = (__u32)sha_dw; - insn[1].imm = sha_dw >> 32; - } -} - void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data, __u32 btf_raw_size) { @@ -557,8 +533,9 @@ void bpf_gen__map_create(struct bpf_gen *gen, * Conditionally update max_entries from the host-supplied loader * ctx. This sizes the map at runtime, but for a signed loader * (gen_hash) it would let an untrusted host re-dimension the - * program's maps after emit_signature_match(), outside what the - * signature attests to. Keep the signer-provided max_entries + * program's maps, outside what the signature attests to: the + * metadata blob is covered by the program signature and verified + * by the kernel at load time. Keep the signer-provided max_entries * baked into the blob in that case. */ if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false)) @@ -596,45 +573,6 @@ void bpf_gen__map_create(struct bpf_gen *gen, emit_sys_close_stack(gen, stack_off(inner_map_fd)); } -static void emit_signature_match(struct bpf_gen *gen) -{ - __s64 off; - int i; - - /* - * Reject if the metadata map is not exclusive. Without exclusivity - * the cached map->sha[] verified above can be stale: another BPF - * program with map access could have mutated the contents between - * BPF_OBJ_GET_INFO_BY_FD and loader execution. - */ - emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, - 0, 0, 0, 0)); - emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH)); - off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2; - if (is_simm16(off)) { - emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL)); - emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off)); - } else { - gen->error = -ERANGE; - } - - for (i = 0; i < SHA256_DWORD_SIZE; i++) { - emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, - 0, 0, 0, 0)); - emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, i * sizeof(__u64))); - gen->hash_insn_offset[i] = gen->insn_cur - gen->insn_start; - emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_3, 0, 0, 0, 0, 0)); - - off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2; - if (is_simm16(off)) { - emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL)); - emit(gen, BPF_JMP_REG(BPF_JNE, BPF_REG_2, BPF_REG_3, off)); - } else { - gen->error = -ERANGE; - } - } -} - void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *attach_name, enum bpf_attach_type type) { @@ -1211,10 +1149,10 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, * } * * The runtime initial_value comes from the host-supplied loader - * ctx and would overwrite the blob value after emit_signature_match() - * has already validated map->sha[]. For a signed loader (gen_hash) - * the attested blob value must be authoritative, so skip the override - * and leave the hashed value in place. + * ctx and would overwrite the blob value that the program signature + * covers and the kernel verifies at load time. For a signed loader + * (gen_hash) the attested blob value must be authoritative, so skip + * the override and leave the signed value in place. */ if (!OPTS_GET(gen->opts, gen_hash, false)) { emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6, diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 04cd303fb5a8..d5b7db703b3f 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -768,7 +768,6 @@ int elf_resolve_pattern_offsets(const char *binary_path, const char *pattern, int probe_fd(int fd); #define SHA256_DIGEST_LENGTH 32 -#define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64) void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]); int probe_sys_bpf_ext(void); diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index 74503d358bc8..53fee53d36d5 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -18,10 +18,6 @@ #include "bpf.h" #endif -#ifndef SHA256_DIGEST_LENGTH -#define SHA256_DIGEST_LENGTH 32 -#endif - #ifndef __NR_bpf # if defined(__mips__) && defined(_ABIO32) # define __NR_bpf 4355 @@ -320,25 +316,6 @@ static inline int skel_link_create(int prog_fd, int target_fd, return skel_sys_bpf(BPF_LINK_CREATE, &attr, attr_sz); } -static inline int skel_obj_get_info_by_fd(int fd) -{ - const size_t attr_sz = offsetofend(union bpf_attr, info); - __u8 sha[SHA256_DIGEST_LENGTH]; - struct bpf_map_info info; - __u32 info_len = sizeof(info); - union bpf_attr attr; - - memset(&info, 0, sizeof(info)); - info.hash = (long) &sha; - info.hash_size = SHA256_DIGEST_LENGTH; - - memset(&attr, 0, attr_sz); - attr.info.bpf_fd = fd; - attr.info.info = (long) &info; - attr.info.info_len = info_len; - return skel_sys_bpf(BPF_OBJ_GET_INFO_BY_FD, &attr, attr_sz); -} - static inline int skel_map_freeze(int fd) { const size_t attr_sz = offsetofend(union bpf_attr, map_fd); @@ -384,12 +361,6 @@ static inline int bpf_load_and_run(struct bpf_load_and_run_opts *opts) set_err; goto out; } - err = skel_obj_get_info_by_fd(map_fd); - if (err < 0) { - opts->errstr = "failed to fetch obj info"; - set_err; - goto out; - } #endif memset(&attr, 0, prog_load_attr_sz); @@ -400,6 +371,8 @@ static inline int bpf_load_and_run(struct bpf_load_and_run_opts *opts) #ifndef __KERNEL__ attr.signature = (long) opts->signature; attr.signature_size = opts->signature_sz; + if (opts->signature) + attr.fd_array_cnt = 1; #else if (opts->signature || opts->signature_sz) pr_warn("signatures are not supported from bpf_preload\n"); From 576bcaa1f5c208af0f590c9622247da87b49c05f Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:39 +0200 Subject: [PATCH 049/373] bpftool: Check EVP_Digest when computing excl_prog_hash bpftool_prog_sign() ignores the return value of EVP_Digest(). If the digest computation fails (context allocation failure, or a digest fetch failure under OpenSSL), EVP_Digest() returns 0 and leaves the output buffer untouched, but the function still reports success. Fixes: 40863f4d6ef2 ("bpftool: Add support for signing BPF programs") Signed-off-by: Daniel Borkmann Reviewed-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20260708075343.358712-5-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- tools/bpf/bpftool/sign.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tools/bpf/bpftool/sign.c b/tools/bpf/bpftool/sign.c index f9b742f4bb10..1257dba8ef2f 100644 --- a/tools/bpf/bpftool/sign.c +++ b/tools/bpf/bpftool/sign.c @@ -175,8 +175,11 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts) goto cleanup; } - EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash, - &opts->excl_prog_hash_sz, EVP_sha256(), NULL); + if (EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash, + &opts->excl_prog_hash_sz, EVP_sha256(), NULL) != 1) { + err = -EIO; + goto cleanup; + } bd_out = BIO_new(BIO_s_mem()); if (!bd_out) { From 92c7717981bb43ff91c04d8588bd79d570ad12c1 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:40 +0200 Subject: [PATCH 050/373] bpftool: Cover loader metadata with the program signature bpftool_prog_sign() signed only the loader instructions. The metadata blob the loader installs was left to an in-loader hash check, which the kernel now performs at load time over insns || metadata. Sign that same concatenation: pass the metadata blob (gen_loader_opts data) through to bpftool_prog_sign() and feed insns || metadata to CMS_final(). The excl_prog_hash stays a digest of the instructions alone; it binds the metadata map to the loader and is matched against prog->digest by the verifier, independent of what the signature covers. The signed artifact is now plain data: both bytes the signature covers are embedded verbatim in the generated skeleton, so signing and verifying an lskel is an ordinary CMS operation that a signer or auditor can perform (or reproduce) offline, without analyzing loader bytecode to establish what the signature actually attests to. Signed-off-by: Daniel Borkmann Reviewed-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20260708075343.358712-6-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- tools/bpf/bpftool/gen.c | 2 ++ tools/bpf/bpftool/sign.c | 17 +++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index 6ae7262ebe0c..a01d06d22d1a 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -793,6 +793,8 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h if (sign_progs) { sopts.insns = opts.insns; sopts.insns_sz = opts.insns_sz; + sopts.data = opts.data; + sopts.data_sz = opts.data_sz; sopts.excl_prog_hash = prog_sha; sopts.excl_prog_hash_sz = sizeof(prog_sha); sopts.signature = sig_buf; diff --git a/tools/bpf/bpftool/sign.c b/tools/bpf/bpftool/sign.c index 1257dba8ef2f..88726a6db6d0 100644 --- a/tools/bpf/bpftool/sign.c +++ b/tools/bpf/bpftool/sign.c @@ -135,9 +135,21 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts) CMS_ContentInfo *cms = NULL; long actual_sig_len = 0; X509 *x509 = NULL; + void *data = NULL; + size_t data_sz; int err = 0; - bd_in = BIO_new_mem_buf(opts->insns, opts->insns_sz); + data_sz = (size_t)opts->insns_sz + opts->data_sz; + data = malloc(data_sz); + if (!data) { + err = -ENOMEM; + goto cleanup; + } + memcpy(data, opts->insns, opts->insns_sz); + if (opts->data_sz) + memcpy((char *)data + opts->insns_sz, opts->data, opts->data_sz); + + bd_in = BIO_new_mem_buf(data, data_sz); if (!bd_in) { err = -ENOMEM; goto cleanup; @@ -181,7 +193,7 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts) goto cleanup; } - bd_out = BIO_new(BIO_s_mem()); + bd_out = BIO_new(BIO_s_mem()); if (!bd_out) { err = -ENOMEM; goto cleanup; @@ -215,6 +227,7 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts) X509_free(x509); EVP_PKEY_free(private_key); BIO_free(bd_in); + free(data); DISPLAY_OSSL_ERR(err < 0); return err; } From 77e5f3c91453307096d653f4fbf9647fcb5b6e95 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:41 +0200 Subject: [PATCH 051/373] selftests/bpf: Adjust bpf_map layout in verifier_map_ptr With write-only excl member removed from struct bpf_map, ops moves to offset 32 and inner_map_meta to offset 40. Update the expected verifier message for the former and retarget the latter at the sha byte array, so the beyond-member-end rejection path stays covered: # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_map_ptr [...] #619/5 verifier_map_ptr/bpf_map_ptr: read non-existent field rejected:OK #619/6 verifier_map_ptr/bpf_map_ptr: read non-existent field rejected @unpriv:OK #619/7 verifier_map_ptr/bpf_map_ptr: read beyond sha field rejected:OK #619/8 verifier_map_ptr/bpf_map_ptr: read beyond sha field rejected @unpriv:OK #619/9 verifier_map_ptr/bpf_map_ptr: read ops field accepted:OK #619/10 verifier_map_ptr/bpf_map_ptr: read ops field accepted @unpriv:OK [...] #620 verifier_map_ptr_mixing:OK Summary: 2/20 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708075343.358712-7-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_map_ptr.c | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ptr.c b/tools/testing/selftests/bpf/progs/verifier_map_ptr.c index 166193659870..e0a65835c861 100644 --- a/tools/testing/selftests/bpf/progs/verifier_map_ptr.c +++ b/tools/testing/selftests/bpf/progs/verifier_map_ptr.c @@ -72,14 +72,15 @@ __naked void bpf_map_ptr_write_rejected(void) /* * struct bpf_map starts with the SHA256 hash sha[32] at offset 0 (a readable - * byte array), the u32 excl field at offset 32, and the ops pointer at offset - * 40. Reading a u32 at offset 41 reaches into the middle of the ops pointer, - * i.e. a partial pointer access, which is rejected. + * byte array), followed by the ops pointer at offset 32 and the inner_map_meta + * pointer at offset 40. Reading a u32 at offset 41 reaches into the middle of + * the inner_map_meta pointer, i.e. a partial pointer access, which is + * rejected. */ SEC("socket") __description("bpf_map_ptr: read non-existent field rejected") __failure -__msg("cannot access ptr member ops with moff 40 in struct bpf_map with off 41 size 4") +__msg("cannot access ptr member inner_map_meta with moff 40 in struct bpf_map with off 41 size 4") __failure_unpriv __msg_unpriv("access is allowed only to CAP_PERFMON and CAP_SYS_ADMIN") __flag(BPF_F_ANY_ALIGNMENT) @@ -97,23 +98,23 @@ __naked void read_non_existent_field_rejected(void) } /* - * The u32 excl field spans offsets 32..35 (mend 36). Reading a u32 at offset - * 33 starts inside excl but extends past its end, which the verifier rejects + * The sha byte array spans offsets 0..31 (mend 32). Reading a u32 at offset + * 30 starts inside sha but extends past its end, which the verifier rejects * as an out-of-bounds scalar access. */ SEC("socket") -__description("bpf_map_ptr: read beyond excl field rejected") +__description("bpf_map_ptr: read beyond sha field rejected") __failure -__msg("access beyond the end of member excl (mend:36) in struct bpf_map with off 33 size 4") +__msg("access beyond the end of member sha (mend:32) in struct bpf_map with off 30 size 4") __failure_unpriv __msg_unpriv("access is allowed only to CAP_PERFMON and CAP_SYS_ADMIN") __flag(BPF_F_ANY_ALIGNMENT) -__naked void read_beyond_excl_field_rejected(void) +__naked void read_beyond_sha_field_rejected(void) { asm volatile (" \ r6 = 0; \ r1 = %[map_array_48b] ll; \ - r6 = *(u32*)(r1 + 33); \ + r6 = *(u32*)(r1 + 30); \ r0 = 1; \ exit; \ " : @@ -131,7 +132,7 @@ __naked void ptr_read_ops_field_accepted(void) asm volatile (" \ r6 = 0; \ r1 = %[map_array_48b] ll; \ - r6 = *(u64*)(r1 + 40); \ + r6 = *(u64*)(r1 + 32); \ r0 = 1; \ exit; \ " : From 99b321dde704e38d3e6c1f425613d7b138ba5696 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:42 +0200 Subject: [PATCH 052/373] selftests/bpf: Verify load-time signed loader metadata The signed gen_loader no longer checks its metadata map from within BPF; the kernel does it at BPF_PROG_LOAD by folding the loader's frozen exclusive fd_array maps into the signature. Exercise that path end to end. Extend with more test cases (e.g. map-less program, asserting the LSM admission hook observes BPF_SIG_UNSIGNED and BPF_SIG_VERIFIED), and retire the subtests that asserted the old in-loader check, which no longer exists. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t signed_loader [...] #412/1 signed_loader/loadtime_no_map:OK #412/2 signed_loader/loadtime_with_map:OK #412/3 signed_loader/metadata_match:OK #412/4 signed_loader/signature_enforced:OK #412/5 signed_loader/signed_nonexcl_fd_array_rejected:OK #412/6 signed_loader/signed_unfrozen_fd_array_rejected:OK #412/7 signed_loader/signed_nonarray_fd_array_rejected:OK #412/8 signed_loader/signed_btf_fd_array_rejected:OK #412/9 signed_loader/signed_module_kfunc_rejected:OK #412/10 signed_loader/signature_failure_logs:OK #412/11 signed_loader/signature_too_large:OK #412/12 signed_loader/signature_zero_size:OK #412/13 signed_loader/signature_bad_keyring:OK #412/14 signed_loader/metadata_ctx_max_entries_ignored:OK #412/15 signed_loader/metadata_ctx_initial_value_ignored:OK #412/16 signed_loader/signature_authenticates_insns:OK #412/17 signed_loader/signature_authenticates_metadata:OK #412/18 signed_loader/hash_requires_frozen:OK #412/19 signed_loader/no_update_after_freeze:OK #412/20 signed_loader/freeze_writable_mmap:OK #412/21 signed_loader/no_writable_mmap_frozen:OK #412/22 signed_loader/map_hash_matches_libbpf:OK #412/23 signed_loader/map_hash_multi_element:OK #412/24 signed_loader/map_hash_bad_size:OK #412/25 signed_loader/map_hash_unsupported_type:OK #412/26 signed_loader/lsm_signature_verdict:OK #412/27 signed_loader/signed_no_fd_array:OK #412/28 signed_loader/signed_map_by_fd_rejected:OK #412/29 signed_loader/signed_sparse_fd_array_rejected:OK #412 signed_loader:OK Summary: 1/29 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708075343.358712-8-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/signed_loader.c | 1075 ++++++++++++++--- .../selftests/bpf/progs/test_signed_loader.c | 9 +- 2 files changed, 891 insertions(+), 193 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/signed_loader.c b/tools/testing/selftests/bpf/prog_tests/signed_loader.c index 5fc417e31fc6..0019492cf07a 100644 --- a/tools/testing/selftests/bpf/prog_tests/signed_loader.c +++ b/tools/testing/selftests/bpf/prog_tests/signed_loader.c @@ -11,6 +11,8 @@ #include #include +#include + #include "bpf/libbpf_internal.h" /* for libbpf_sha256() */ #include "bpf/skel_internal.h" /* for loader ctx layout (bpf_loader_ctx etc) */ @@ -19,8 +21,6 @@ #include "test_signed_loader_data.skel.h" #include "test_signed_loader_lsm.skel.h" -#define SIG_MATCH_INSNS 33 /* excl (5) + 4 * sha-dword (7) */ - enum { BPF_SIG_UNSIGNED = 0, BPF_SIG_VERIFIED, @@ -35,7 +35,8 @@ enum { }; static int load_loader(const void *insns, __u32 insns_sz, int map_fd, - const void *sig, __u32 sig_sz, __s32 keyring_id) + const void *sig, __u32 sig_sz, __s32 keyring_id, + __u32 fd_array_cnt) { union bpf_attr attr; int fd; @@ -52,6 +53,7 @@ static int load_loader(const void *insns, __u32 insns_sz, int map_fd, attr.signature_size = sig_sz; attr.keyring_id = keyring_id; } + attr.fd_array_cnt = fd_array_cnt; memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog")); fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, offsetofend(union bpf_attr, keyring_id)); @@ -62,14 +64,12 @@ static int run_gen_loader(const void *insns, __u32 insns_sz, const void *data, __u32 data_sz, const void *excl, __u32 excl_sz, const void *sig, __u32 sig_sz, - bool get_hash, void *ctx, __u32 ctx_sz, bool *loader_ran) + void *ctx, __u32 ctx_sz, bool *loader_ran) { LIBBPF_OPTS(bpf_map_create_opts, mopts, .excl_prog_hash = excl, .excl_prog_hash_size = excl_sz); - __u8 hbuf[SHA256_DIGEST_LENGTH]; - struct bpf_map_info info; - __u32 ilen = sizeof(info), key = 0; + __u32 key = 0; union bpf_attr attr; int map_fd, prog_fd, ret; @@ -87,15 +87,6 @@ static int run_gen_loader(const void *insns, __u32 insns_sz, ret = -errno; goto out_map; } - if (get_hash) { - memset(&info, 0, sizeof(info)); - info.hash = ptr_to_u64(hbuf); - info.hash_size = sizeof(hbuf); - if (bpf_map_get_info_by_fd(map_fd, &info, &ilen)) { - ret = -errno; - goto out_map; - } - } memset(&attr, 0, sizeof(attr)); attr.prog_type = BPF_PROG_TYPE_SYSCALL; @@ -108,6 +99,7 @@ static int run_gen_loader(const void *insns, __u32 insns_sz, attr.signature = ptr_to_u64(sig); attr.signature_size = sig_sz; attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + attr.fd_array_cnt = 1; } memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog")); prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, @@ -236,79 +228,6 @@ static int sign_buf(const char *dir, const void *buf, __u32 len, return ret; } -static void check_sig_match_shape(const struct bpf_insn *in, int n) -{ - int a = -1, cleanup = -1, i, base, t, br[5], nb = 0; - - /* BPF_PSEUDO_MAP_IDX (the struct bpf_map * form) is used only here. */ - for (i = 0; i + 1 < n; i++) { - if (in[i].code == (BPF_LD | BPF_IMM | BPF_DW) && - in[i].src_reg == BPF_PSEUDO_MAP_IDX) { - a = i; - break; - } - } - if (!ASSERT_GE(a, 0, "emit_signature_match present")) - return; - if (!ASSERT_LE(a + SIG_MATCH_INSNS, n, "block fits in program")) - return; - - /* excl check: r2 = *(u32 *)(map + 32); if r2 != 1 goto cleanup */ - ASSERT_EQ(in[a + 2].code, (BPF_LDX | BPF_MEM | BPF_W), "excl load width"); - ASSERT_EQ(in[a + 2].off, SHA256_DIGEST_LENGTH, "excl field offset"); - ASSERT_EQ(in[a + 4].code, (BPF_JMP | BPF_JNE | BPF_K), "excl branch op"); - ASSERT_EQ(in[a + 4].imm, 1, "excl compared to 1"); - br[nb++] = a + 4; - - /* 4 sha-dword checks: r2 = *(u64 *)(map + i*8); if r2 != r3 goto cleanup */ - for (i = 0; i < 4; i++) { - base = a + 5 + i * 7; - ASSERT_EQ(in[base + 2].code, (BPF_LDX | BPF_MEM | BPF_DW), "sha load width"); - ASSERT_EQ(in[base + 2].off, i * 8, "sha dword offset"); - ASSERT_EQ(in[base + 3].code, (BPF_LD | BPF_IMM | BPF_DW), "sha imm64 (H_meta)"); - ASSERT_EQ(in[base + 6].code, (BPF_JMP | BPF_JNE | BPF_X), "sha branch op"); - br[nb++] = base + 6; - } - - /* - * Locate the real cleanup label so we can pin the exact jump target, - * not just "some backward label". bpf_gen__init() emits the cleanup - * block as a prog-fd close loop whose first instruction is the label - * every error branch jumps to. - */ - for (i = 0; i + 2 < a; i++) { - if (in[i].code == (BPF_LDX | BPF_MEM | BPF_W) && - in[i].dst_reg == BPF_REG_1 && in[i].src_reg == BPF_REG_10 && - in[i + 1].code == (BPF_JMP | BPF_JSLE | BPF_K) && - in[i + 1].dst_reg == BPF_REG_1 && in[i + 1].imm == 0 && - in[i + 1].off == 1 && - in[i + 2].code == (BPF_JMP | BPF_CALL) && - in[i + 2].imm == BPF_FUNC_sys_close) { - cleanup = i; - break; - } - } - if (!ASSERT_GE(cleanup, 0, "cleanup label located")) - return; - for (i = 0; i < nb; i++) { - t = br[i] + 1 + in[br[i]].off; - ASSERT_EQ(t, cleanup, "sig-match lands on cleanup"); - } - /* - * Same invariant for every other cleanup-bound jump in the program: - * emit_check_err() is the only source of "if (r7 < 0) goto cleanup", - * so each of those must also resolve exactly to cleanup. - */ - for (i = 0, t = 0; i < n; i++) { - if (in[i].code != (BPF_JMP | BPF_JSLT | BPF_K) || - in[i].dst_reg != BPF_REG_7 || in[i].imm != 0 || in[i].off >= 0) - continue; - ASSERT_EQ(i + 1 + in[i].off, cleanup, "err-check lands on cleanup"); - t++; - } - ASSERT_GT(t, 0, "found emit_check_err jumps"); -} - struct gen_loader_fixture { struct test_signed_loader *skel; struct gen_loader_opts gopts; @@ -372,16 +291,6 @@ static void gen_loader_fixture_fini(struct gen_loader_fixture *f) test_signed_loader__destroy(f->skel); } -static void metadata_check_shape(void) -{ - struct gen_loader_fixture f; - - if (gen_loader_fixture_init(&f) == 0) - check_sig_match_shape((const struct bpf_insn *)f.gopts.insns, - f.gopts.insns_sz / sizeof(struct bpf_insn)); - gen_loader_fixture_fini(&f); -} - static void metadata_match(void) { struct gen_loader_fixture f; @@ -391,78 +300,13 @@ static void metadata_match(void) if (gen_loader_fixture_init(&f) == 0) { r = run_gen_loader(f.gopts.insns, f.gopts.insns_sz, f.blob, f.data_sz, f.excl, sizeof(f.excl), NULL, 0, - true, f.ctx, f.ctx_sz, &ran); + f.ctx, f.ctx_sz, &ran); ASSERT_TRUE(ran, "loader ran"); ASSERT_EQ(r, 0, "honest loader retval"); } gen_loader_fixture_fini(&f); } -static void metadata_sha_mismatch(void) -{ - struct gen_loader_fixture f; - bool ran; - int r; - - if (gen_loader_fixture_init(&f) == 0) { - /* - * blob[0] lives in the loader's fd_array scratch (first add_data in - * bpf_gen__init); a 0-map program never reads it, so flipping it - * changes only map->sha. The metadata check is the only thing that - * can notice -> isolates emit_signature_match. - */ - f.blob[0] ^= 0xff; - r = run_gen_loader(f.gopts.insns, f.gopts.insns_sz, f.blob, - f.data_sz, f.excl, sizeof(f.excl), NULL, 0, - true, f.ctx, f.ctx_sz, &ran); - ASSERT_TRUE(ran, "loader ran"); - ASSERT_EQ(r, -EINVAL, "tampered blob rejected by emit_signature_match"); - } - gen_loader_fixture_fini(&f); -} - -static void metadata_not_exclusive(void) -{ - struct gen_loader_fixture f; - bool ran; - int r; - - if (gen_loader_fixture_init(&f) == 0) { - /* - * Correct blob but a non-exclusive metadata map: the verifier does - * not reject (excl_prog_sha unset), so the runtime map->excl == 1 - * check in the loader must. - */ - r = run_gen_loader(f.gopts.insns, f.gopts.insns_sz, f.blob, - f.data_sz, NULL, 0, NULL, 0, true, f.ctx, - f.ctx_sz, &ran); - ASSERT_TRUE(ran, "loader ran"); - ASSERT_EQ(r, -EINVAL, "non-exclusive metadata map rejected"); - } - gen_loader_fixture_fini(&f); -} - -static void metadata_hash_not_computed(void) -{ - struct gen_loader_fixture f; - bool ran; - int r; - - if (gen_loader_fixture_init(&f) == 0) { - /* - * Correct, exclusive, frozen map, but its hash was never computed - * (no OBJ_GET_INFO_BY_FD), so map->sha stays zero. The loader must - * fail closed rather than treat an unset hash as a match. - */ - r = run_gen_loader(f.gopts.insns, f.gopts.insns_sz, f.blob, - f.data_sz, f.excl, sizeof(f.excl), NULL, 0, - false, f.ctx, f.ctx_sz, &ran); - ASSERT_TRUE(ran, "loader ran"); - ASSERT_EQ(r, -EINVAL, "uncomputed metadata hash rejected"); - } - gen_loader_fixture_fini(&f); -} - static void signature_enforced(void) { static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, }; @@ -474,11 +318,245 @@ static void signature_enforced(void) * A present-but-invalid signature (the cert bytes are not a * PKCS#7 signature) must be rejected at load: the signature * path is honored, not ignored. (The valid path is covered by - * the signed lskels.) + * the signed lskels.) Pin -EBADMSG, the PKCS#7 parse failure: + * a looser fd < 0 check could also be satisfied by the sparse + * fd_array rejection (-EACCES) that the loader's map reference + * would trip even if the signature were silently ignored. */ fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, - sizeof(junk), KEY_SPEC_SESSION_KEYRING); + sizeof(junk), KEY_SPEC_SESSION_KEYRING, 0); + ASSERT_EQ(fd, -EBADMSG, "invalid signature rejected at load"); + } + gen_loader_fixture_fini(&f); +} + +static void signed_nonexcl_fd_array_rejected(void) +{ + static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, }; + struct gen_loader_fixture f; + int map_fd, fd; + + if (gen_loader_fixture_init(&f) == 0) { + /* + * A signed program may only bind exclusive maps through fd_array + * (their contents are folded into the signature). Binding a + * non-exclusive map is rejected, before the signature is even + * examined. + */ + map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "nonexcl", 4, + f.data_sz, 1, NULL); + if (ASSERT_OK_FD(map_fd, "nonexcl_map")) { + if (ASSERT_OK(bpf_map_freeze(map_fd), "freeze")) { + fd = load_loader(f.gopts.insns, f.gopts.insns_sz, + map_fd, junk, sizeof(junk), + KEY_SPEC_SESSION_KEYRING, 1); + ASSERT_EQ(fd, -EPERM, + "non-exclusive map in signed fd_array rejected"); + if (fd >= 0) + close(fd); + } + close(map_fd); + } + } + gen_loader_fixture_fini(&f); +} + +static void signed_unfrozen_fd_array_rejected(void) +{ + static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, }; + LIBBPF_OPTS(bpf_map_create_opts, mopts); + struct gen_loader_fixture f; + __u32 key = 0; + int map_fd, fd; + + if (gen_loader_fixture_init(&f) == 0) { + /* + * The metadata map must be frozen before a signed load so the + * folded bytes cannot change afterwards. Bind an exclusive map + * with matching contents but skip the freeze: the load must be + * rejected by the frozen check with -EPERM. The exclusivity + * check right after it would pass, so the errno uniquely pins + * the freeze requirement. + */ + mopts.excl_prog_hash = f.excl; + mopts.excl_prog_hash_size = sizeof(f.excl); + map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "unfrozen", 4, + f.data_sz, 1, &mopts); + if (ASSERT_OK_FD(map_fd, "unfrozen_map")) { + if (ASSERT_OK(bpf_map_update_elem(map_fd, &key, f.blob, 0), + "update")) { + fd = load_loader(f.gopts.insns, f.gopts.insns_sz, + map_fd, junk, sizeof(junk), + KEY_SPEC_SESSION_KEYRING, 1); + ASSERT_EQ(fd, -EPERM, + "unfrozen map in signed fd_array rejected"); + if (fd >= 0) + close(fd); + } + close(map_fd); + } + } + gen_loader_fixture_fini(&f); +} + +static void signed_nonarray_fd_array_rejected(void) +{ + static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, }; + LIBBPF_OPTS(bpf_map_create_opts, mopts); + struct gen_loader_fixture f; + int map_fd, fd; + + if (gen_loader_fixture_init(&f) == 0) { + /* + * Only a plain BPF_MAP_TYPE_ARRAY may be folded into the + * signature. An exclusive map of any other type is rejected + * (-EINVAL) rather than folded - this is the type gate that + * keeps arena maps (map_direct_value_addr() returns a user + * address) and insn-array maps (buffer smaller than value_size) + * out of the hashed region, where the old code would have + * memcpy()'d from them. A hash map stands in here: it is + * exclusive (bound to the loader digest) but not an array. + */ + mopts.excl_prog_hash = f.excl; + mopts.excl_prog_hash_size = sizeof(f.excl); + map_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "excl_hash", 4, 4, 1, + &mopts); + if (ASSERT_OK_FD(map_fd, "excl_hash_map")) { + fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, + junk, sizeof(junk), + KEY_SPEC_SESSION_KEYRING, 1); + ASSERT_EQ(fd, -EINVAL, + "non-array map in signed fd_array rejected"); + if (fd >= 0) + close(fd); + close(map_fd); + } + } + gen_loader_fixture_fini(&f); +} + +static int setup_meta_map(const struct gen_loader_fixture *f); + +static void signed_btf_fd_array_rejected(void) +{ + char dir_tmpl[] = "/tmp/signed_loader_btfXXXXXX", *dir = NULL; + __u32 sig_sz = 8192; + int map_fd = -1, prog_fd = -1; + unsigned char *buf = NULL; + struct gen_loader_fixture f; + bool have_fixture = false; + struct btf *btf = NULL; + union bpf_attr attr; + int fds[2]; + __u8 sig[8192]; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + return; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + return; + } + have_fixture = true; + if (gen_loader_fixture_init(&f) != 0) + goto out; + + /* + * fd_array binds maps and BTFs alike, but only exclusive array maps are + * folded into the signature. Build an otherwise genuinely signed load - + * insns || metadata, exclusive frozen map at fd_array[0] - then smuggle + * an extra BTF into fd_array[1]. A signed program may not bind any BTF, + * so resolving the fd_array entries rejects the BTF with -EACCES (in + * __add_used_btf(), before the signature is even verified). + */ + buf = malloc((size_t)f.gopts.insns_sz + f.data_sz); + if (!ASSERT_OK_PTR(buf, "signbuf")) + goto out; + memcpy(buf, f.gopts.insns, f.gopts.insns_sz); + memcpy(buf + f.gopts.insns_sz, f.blob, f.data_sz); + if (!ASSERT_OK(sign_buf(dir, buf, f.gopts.insns_sz + f.data_sz, sig, + &sig_sz), "sign insns||metadata")) + goto out; + + map_fd = setup_meta_map(&f); + if (!ASSERT_OK_FD(map_fd, "meta_map")) + goto out; + btf = btf__new_empty(); + if (!ASSERT_OK_PTR(btf, "btf_new_empty")) + goto out; + btf__add_int(btf, "int", 4, BTF_INT_SIGNED); + if (!ASSERT_OK(btf__load_into_kernel(btf), "btf_load")) + goto out; + + fds[0] = map_fd; + fds[1] = btf__fd(btf); + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(f.gopts.insns); + attr.insn_cnt = f.gopts.insns_sz / sizeof(struct bpf_insn); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.fd_array = ptr_to_u64(fds); + attr.fd_array_cnt = 2; + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog")); + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + ASSERT_EQ(prog_fd < 0 ? -errno : prog_fd, -EACCES, + "BTF in signed fd_array rejected"); + if (prog_fd >= 0) + close(prog_fd); +out: + if (btf) + btf__free(btf); + if (map_fd >= 0) + close(map_fd); + if (have_fixture) + gen_loader_fixture_fini(&f); + if (dir) + run_setup("cleanup", dir); + free(buf); +} + +static void signature_failure_logs(void) +{ + static const __u8 junk[64] = { 0x30, 0x42, 0x13, 0x37, }; + char log_buf[1024] = {}; + struct gen_loader_fixture f; + union bpf_attr attr; + int fd; + + if (gen_loader_fixture_init(&f) == 0) { + /* + * Signature verification now runs inside bpf_check(), so a + * failure is reported through the verifier log. A present-but- + * invalid signature is rejected and the log says why. + */ + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(f.gopts.insns); + attr.insn_cnt = f.gopts.insns_sz / sizeof(struct bpf_insn); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.signature = ptr_to_u64(junk); + attr.signature_size = sizeof(junk); + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + attr.log_level = 1; + attr.log_buf = ptr_to_u64(log_buf); + attr.log_size = sizeof(log_buf); + memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog")); + + fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); ASSERT_LT(fd, 0, "invalid signature rejected at load"); + if (fd >= 0) + close(fd); + ASSERT_HAS_SUBSTR(log_buf, "signature verification failed", + "verifier logs signature failure"); } gen_loader_fixture_fini(&f); } @@ -495,12 +573,31 @@ static void signature_too_large(void) * is rejected before the buffer is read. */ fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, - 64 << 20, KEY_SPEC_SESSION_KEYRING); + 64 << 20, KEY_SPEC_SESSION_KEYRING, 0); ASSERT_EQ(fd, -EINVAL, "oversized signature rejected"); } gen_loader_fixture_fini(&f); } +static void signature_zero_size(void) +{ + static const __u8 junk[64] = {}; + struct gen_loader_fixture f; + int fd; + + if (gen_loader_fixture_init(&f) == 0) { + /* + * A present signature with signature_size == 0 is rejected + * up front, before the keyring is resolved or the signature + * buffer is read. + */ + fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, + 0, KEY_SPEC_SESSION_KEYRING, 0); + ASSERT_EQ(fd, -EINVAL, "zero-size signature rejected"); + } + gen_loader_fixture_fini(&f); +} + static void signature_bad_keyring(void) { static const __u8 junk[64] = {}; @@ -515,7 +612,7 @@ static void signature_bad_keyring(void) * large positive serial takes the user-keyring path and won't exist. */ fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, - sizeof(junk), INT_MAX); + sizeof(junk), INT_MAX, 0); ASSERT_EQ(fd, -EINVAL, "signature with bad keyring_id rejected"); } gen_loader_fixture_fini(&f); @@ -575,7 +672,7 @@ static void metadata_ctx_max_entries_ignored(void) memcpy(blob, gopts.data, data_sz); r = run_gen_loader(gopts.insns, gopts.insns_sz, blob, data_sz, - excl, sizeof(excl), NULL, 0, true, ctx, ctx_sz, &ran); + excl, sizeof(excl), NULL, 0, ctx, ctx_sz, &ran); if (!ASSERT_TRUE(ran, "loader ran") || !ASSERT_EQ(r, 0, "loader retval")) goto free_blob; @@ -661,7 +758,7 @@ static void metadata_ctx_initial_value_ignored(void) memcpy(blob, gopts.data, data_sz); r = run_gen_loader(gopts.insns, gopts.insns_sz, blob, data_sz, - excl, sizeof(excl), NULL, 0, true, ctx, ctx_sz, &ran); + excl, sizeof(excl), NULL, 0, ctx, ctx_sz, &ran); if (!ASSERT_TRUE(ran, "loader ran") || !ASSERT_EQ(r, 0, "loader retval")) goto free_blob; @@ -714,6 +811,7 @@ static void signature_authenticates_insns(void) __u8 excl[SHA256_DIGEST_LENGTH], sig[8192]; __u32 sig_sz = sizeof(sig), insns_sz, data_sz, ctx_sz; unsigned char *insns = NULL, *tampered = NULL, *blob = NULL; + unsigned char *signbuf = NULL; int nr_maps = 0, nr_progs = 0, r; struct bpf_program *p; struct bpf_map *m; @@ -760,29 +858,141 @@ static void signature_authenticates_insns(void) memcpy(blob, gopts.data, data_sz); libbpf_sha256(insns, insns_sz, excl); - if (!ASSERT_OK(sign_buf(dir, insns, insns_sz, sig, &sig_sz), "sign-file")) + signbuf = malloc((size_t)insns_sz + data_sz); + if (!ASSERT_OK_PTR(signbuf, "signbuf")) + goto cleanup; + memcpy(signbuf, insns, insns_sz); + memcpy(signbuf + insns_sz, blob, data_sz); + if (!ASSERT_OK(sign_buf(dir, signbuf, insns_sz + data_sz, sig, &sig_sz), + "sign-file")) goto cleanup; memset(ctx, 0, ctx_sz); ((struct bpf_loader_ctx *)ctx)->sz = ctx_sz; r = run_gen_loader(insns, insns_sz, blob, data_sz, excl, sizeof(excl), - sig, sig_sz, true, ctx, ctx_sz, &ran); + sig, sig_sz, ctx, ctx_sz, &ran); ASSERT_TRUE(ran, "valid signature: loader loaded and ran"); ASSERT_EQ(r, 0, "valid signature accepted"); close_loader_ctx_fds(ctx, nr_maps, nr_progs); memcpy(tampered, insns, insns_sz); tampered[insns_sz / 2] ^= 0xff; + /* + * Bind the metadata map to the tampered loader's own digest, so the + * verifier's exclusive-map check (excl_prog_sha == prog->digest) passes + * and the signature - verified after the maps are resolved - is what + * rejects the load. This is the attacker's best case: even after + * re-binding the exclusive map to their tampered loader, the signature + * over the original insns || metadata still fails. (Leaving the map + * bound to the original digest would instead trip the excl check first.) + */ + libbpf_sha256(tampered, insns_sz, excl); memset(ctx, 0, ctx_sz); ((struct bpf_loader_ctx *)ctx)->sz = ctx_sz; r = run_gen_loader(tampered, insns_sz, blob, data_sz, excl, sizeof(excl), - sig, sig_sz, true, ctx, ctx_sz, &ran); + sig, sig_sz, ctx, ctx_sz, &ran); ASSERT_FALSE(ran, "tampered loader rejected before run"); ASSERT_EQ(r, -EKEYREJECTED, "signature is bound to the instructions"); cleanup: free(insns); free(tampered); free(blob); + free(signbuf); + free(ctx); + test_signed_loader__destroy(skel); + run_setup("cleanup", dir); +} + +static void signature_authenticates_metadata(void) +{ + LIBBPF_OPTS(gen_loader_opts, gopts, .gen_hash = true); + char dir_tmpl[] = "/tmp/signed_loaderXXXXXX", *dir; + struct test_signed_loader *skel = NULL; + __u8 excl[SHA256_DIGEST_LENGTH], sig[8192]; + __u32 sig_sz = sizeof(sig), insns_sz, data_sz, ctx_sz; + unsigned char *insns = NULL, *blob = NULL; + unsigned char *signbuf = NULL; + int nr_maps = 0, nr_progs = 0, r; + struct bpf_program *p; + struct bpf_map *m; + void *ctx = NULL; + bool ran; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + return; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + return; + } + + skel = test_signed_loader__open(); + if (!ASSERT_OK_PTR(skel, "skel_open")) + goto cleanup; + if (!ASSERT_OK(bpf_object__gen_loader(skel->obj, &gopts), "gen_loader")) + goto cleanup; + if (!ASSERT_OK(bpf_object__load(skel->obj), "gen_load")) + goto cleanup; + + bpf_object__for_each_program(p, skel->obj) + nr_progs++; + bpf_object__for_each_map(m, skel->obj) + nr_maps++; + ctx_sz = sizeof(struct bpf_loader_ctx) + + nr_maps * sizeof(struct bpf_map_desc) + + nr_progs * sizeof(struct bpf_prog_desc); + insns_sz = gopts.insns_sz; + data_sz = gopts.data_sz; + ctx = calloc(1, ctx_sz); + insns = malloc(insns_sz); + blob = malloc(data_sz); + if (!ASSERT_OK_PTR(ctx, "ctx") || + !ASSERT_OK_PTR(insns, "insns") || + !ASSERT_OK_PTR(blob, "blob")) + goto cleanup; + memcpy(insns, gopts.insns, insns_sz); + memcpy(blob, gopts.data, data_sz); + libbpf_sha256(insns, insns_sz, excl); + + signbuf = malloc((size_t)insns_sz + data_sz); + if (!ASSERT_OK_PTR(signbuf, "signbuf")) + goto cleanup; + memcpy(signbuf, insns, insns_sz); + memcpy(signbuf + insns_sz, blob, data_sz); + if (!ASSERT_OK(sign_buf(dir, signbuf, insns_sz + data_sz, sig, &sig_sz), + "sign-file")) + goto cleanup; + + memset(ctx, 0, ctx_sz); + ((struct bpf_loader_ctx *)ctx)->sz = ctx_sz; + r = run_gen_loader(insns, insns_sz, blob, data_sz, excl, sizeof(excl), + sig, sig_sz, ctx, ctx_sz, &ran); + ASSERT_TRUE(ran, "valid signature: loader loaded and ran"); + ASSERT_EQ(r, 0, "valid signature accepted"); + close_loader_ctx_fds(ctx, nr_maps, nr_progs); + + /* + * Tamper the metadata after signing while leaving the instructions + * and thus the exclusive hash binding untouched: the map freezes + * fine and excl_prog_sha still matches the loader's digest, so the + * load reaches signature verification, which folds the live frozen + * map bytes into the checked payload and must reject the modified + * blob. A kernel folding anything but the map contents themselves + * would wrongly accept this load. + */ + blob[data_sz / 2] ^= 0xff; + memset(ctx, 0, ctx_sz); + ((struct bpf_loader_ctx *)ctx)->sz = ctx_sz; + r = run_gen_loader(insns, insns_sz, blob, data_sz, excl, sizeof(excl), + sig, sig_sz, ctx, ctx_sz, &ran); + ASSERT_FALSE(ran, "tampered metadata rejected before run"); + ASSERT_EQ(r, -EKEYREJECTED, "signature is bound to the metadata"); +cleanup: + free(insns); + free(blob); + free(signbuf); free(ctx); test_signed_loader__destroy(skel); run_setup("cleanup", dir); @@ -1007,10 +1217,11 @@ static void lsm_signature_verdict(void) { char dir_tmpl[] = "/tmp/signed_loader_lsmXXXXXX", *dir = NULL; struct test_signed_loader_lsm *lsm = NULL; + __u32 sig_sz = 8192, msig_sz = 8192; int map_fd = -1, prog_fd = -1; bool have_fixture = false; struct gen_loader_fixture f; - __u32 sig_sz = 8192; + unsigned char *buf; __s32 ses_serial; __u8 sig[8192]; @@ -1029,7 +1240,7 @@ static void lsm_signature_verdict(void) if (!ASSERT_OK_FD(map_fd, "meta_map_unsigned")) goto out; lsm->bss->seen = 0; - prog_fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, NULL, 0, 0); + prog_fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, NULL, 0, 0, 0); close(map_fd); map_fd = -1; if (!ASSERT_OK_FD(prog_fd, "unsigned loader load")) @@ -1062,22 +1273,51 @@ static void lsm_signature_verdict(void) goto out; lsm->bss->seen = 0; prog_fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, sig, - sig_sz, KEY_SPEC_SESSION_KEYRING); + sig_sz, KEY_SPEC_SESSION_KEYRING, 0); close(map_fd); map_fd = -1; - if (!ASSERT_OK_FD(prog_fd, "signed loader load")) - goto out; - close(prog_fd); + ASSERT_EQ(prog_fd, -EACCES, "unfolded metadata rejected"); + if (prog_fd >= 0) + close(prog_fd); prog_fd = -1; ses_serial = syscall(__NR_keyctl, KEYCTL_GET_KEYRING_ID, KEY_SPEC_SESSION_KEYRING, 0); ASSERT_EQ(lsm->bss->seen, 1, "signed: one observed load"); - ASSERT_EQ(lsm->bss->sig_verdict, BPF_SIG_VERIFIED, "signed verdict"); + ASSERT_EQ(lsm->bss->sig_verdict, BPF_SIG_VERIFIED, + "admission saw a valid signature"); ASSERT_EQ(lsm->bss->sig_keyring_type, BPF_SIG_KEYRING_USER, "signed keyring type"); ASSERT_GT(ses_serial, 0, "session keyring serial resolved"); ASSERT_EQ(lsm->bss->sig_keyring_serial, ses_serial, "signed: validated against session keyring"); + + buf = malloc((size_t)f.gopts.insns_sz + f.data_sz); + if (!ASSERT_OK_PTR(buf, "meta_signbuf")) + goto out; + memcpy(buf, f.gopts.insns, f.gopts.insns_sz); + memcpy(buf + f.gopts.insns_sz, f.blob, f.data_sz); + if (!ASSERT_OK(sign_buf(dir, buf, f.gopts.insns_sz + f.data_sz, + sig, &msig_sz), "sign insns||metadata")) { + free(buf); + goto out; + } + free(buf); + + map_fd = setup_meta_map(&f); + if (!ASSERT_OK_FD(map_fd, "meta_map_bound")) + goto out; + lsm->bss->seen = 0; + prog_fd = load_loader(f.gopts.insns, f.gopts.insns_sz, map_fd, sig, + msig_sz, KEY_SPEC_SESSION_KEYRING, 1); + close(map_fd); + map_fd = -1; + if (!ASSERT_OK_FD(prog_fd, "metadata-bound loader load")) + goto out; + close(prog_fd); + prog_fd = -1; + ASSERT_EQ(lsm->bss->seen, 1, "metadata: one observed load"); + ASSERT_EQ(lsm->bss->sig_verdict, BPF_SIG_VERIFIED, + "metadata-bound verdict"); out: if (map_fd >= 0) close(map_fd); @@ -1090,22 +1330,471 @@ static void lsm_signature_verdict(void) test_signed_loader_lsm__destroy(lsm); } +/* + * Load-time metadata verification: the kernel folds the frozen metadata map + * into the signature (insns || metadata) and checks it at BPF_PROG_LOAD via + * fd_array_cnt, rather than the loader checking from within BPF. Sign that + * concatenation, hand the kernel the map, and confirm the signed loader loads, + * runs, and installs its target. + */ +static int loadtime_drive(const char *dir, const void *insns, __u32 insns_sz, + const void *data, __u32 data_sz, const __u8 *excl, + void *ctx, __u32 ctx_sz, int *load_ret, bool *ran) +{ + LIBBPF_OPTS(bpf_map_create_opts, mopts, + .excl_prog_hash = excl, + .excl_prog_hash_size = SHA256_DIGEST_LENGTH); + __u32 sig_sz = 8192, key = 0; + unsigned char *buf = NULL; + int map_fd, prog_fd, ret = 0; + union bpf_attr attr; + __u8 sig[8192]; + + *ran = false; + *load_ret = 0; + + /* + * Metadata map, bound to the loader digest and frozen, exactly as + * skel_internal.h's bpf_load_and_run() sets it up. + */ + map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "__loader.map", 4, + data_sz, 1, &mopts); + if (map_fd < 0) { + ret = -errno; + goto out_load; + } + if (bpf_map_update_elem(map_fd, &key, data, 0) || bpf_map_freeze(map_fd)) { + ret = -errno; + goto out_load; + } + + /* Sign insns || metadata, the same bytes the kernel reconstructs. */ + buf = malloc((size_t)insns_sz + data_sz); + if (!buf) { + ret = -ENOMEM; + goto out_load; + } + memcpy(buf, insns, insns_sz); + memcpy(buf + insns_sz, data, data_sz); + ret = sign_buf(dir, buf, insns_sz + data_sz, sig, &sig_sz); + if (ret) + goto out_load; + + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(insns); + attr.insn_cnt = insns_sz / sizeof(struct bpf_insn); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.fd_array = ptr_to_u64(&map_fd); + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + attr.fd_array_cnt = 1; + memcpy(attr.prog_name, "__loader.prog", sizeof("__loader.prog")); + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + if (prog_fd < 0) { + ret = -errno; + goto out_load; + } + + memset(&attr, 0, sizeof(attr)); + attr.test.prog_fd = prog_fd; + attr.test.ctx_in = ptr_to_u64(ctx); + attr.test.ctx_size_in = ctx_sz; + if (syscall(__NR_bpf, BPF_PROG_RUN, &attr, + offsetofend(union bpf_attr, test)) < 0) { + ret = -errno; + goto out_prog; + } + *ran = true; + ret = (int)attr.test.retval; +out_prog: + close(prog_fd); + goto out_map; +out_load: + *load_ret = ret; +out_map: + free(buf); + if (map_fd >= 0) + close(map_fd); + return ret; +} + +static void loadtime_verify(struct bpf_object *obj, int expect_maps) +{ + LIBBPF_OPTS(gen_loader_opts, gopts, .gen_hash = true); + char dir_tmpl[] = "/tmp/signed_loader_ltXXXXXX", *dir = NULL; + int nr_maps = 0, nr_progs = 0, load_ret = 0, r; + __u8 excl[SHA256_DIGEST_LENGTH]; + struct bpf_prog_desc *pd; + struct bpf_map_desc *md; + unsigned char *blob = NULL; + struct bpf_program *p; + struct bpf_map *m; + __u32 ctx_sz, data_sz; + void *ctx = NULL; + bool ran = false; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + return; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + return; + } + + if (!ASSERT_OK(bpf_object__gen_loader(obj, &gopts), "gen_loader")) + goto out; + if (!ASSERT_OK(bpf_object__load(obj), "gen_load")) + goto out; + + bpf_object__for_each_program(p, obj) + nr_progs++; + bpf_object__for_each_map(m, obj) + nr_maps++; + if (!ASSERT_EQ(nr_maps, expect_maps, "fixture map count")) + goto out; + + ctx_sz = sizeof(struct bpf_loader_ctx) + + nr_maps * sizeof(struct bpf_map_desc) + + nr_progs * sizeof(struct bpf_prog_desc); + ctx = calloc(1, ctx_sz); + if (!ASSERT_OK_PTR(ctx, "ctx_alloc")) + goto out; + ((struct bpf_loader_ctx *)ctx)->sz = ctx_sz; + + data_sz = gopts.data_sz; + blob = malloc(data_sz); + if (!ASSERT_OK_PTR(blob, "blob_alloc")) + goto out; + memcpy(blob, gopts.data, data_sz); + + /* excl_prog_hash = SHA256(loader insns) == the loader's prog->digest. */ + libbpf_sha256(gopts.insns, gopts.insns_sz, excl); + + r = loadtime_drive(dir, gopts.insns, gopts.insns_sz, blob, data_sz, + excl, ctx, ctx_sz, &load_ret, &ran); + ASSERT_OK(load_ret, "signed loader loaded (insns || metadata)"); + ASSERT_TRUE(ran, "loader ran"); + ASSERT_EQ(r, 0, "loader installed its target"); + + md = (struct bpf_map_desc *)((char *)ctx + sizeof(struct bpf_loader_ctx)); + pd = (struct bpf_prog_desc *)(md + nr_maps); + ASSERT_GT(pd[0].prog_fd, 0, "target program installed"); + if (nr_maps) + ASSERT_GT(md[0].map_fd, 0, "target map installed"); + + close_loader_ctx_fds(ctx, nr_maps, nr_progs); +out: + free(blob); + free(ctx); + if (dir) + run_setup("cleanup", dir); +} + +static void loadtime_no_map(void) +{ + struct test_signed_loader *skel = test_signed_loader__open(); + + if (!ASSERT_OK_PTR(skel, "skel_open")) + return; + loadtime_verify(skel->obj, 0); + test_signed_loader__destroy(skel); +} + +static void loadtime_with_map(void) +{ + struct test_signed_loader_map *skel = test_signed_loader_map__open(); + + if (!ASSERT_OK_PTR(skel, "skel_open")) + return; + loadtime_verify(skel->obj, 1); + test_signed_loader_map__destroy(skel); +} + +/* + * A signed program need not bind any map. A plain BPF_PROG_TYPE_SYSCALL + * program with no fd_array is signed over its instructions alone: the kernel + * verifies the signature, folds no metadata, and the program loads. Exercise + * the fd_array == NULL / fd_array_cnt == 0 path, and confirm the signature + * still authenticates the instructions (a tampered copy is rejected). + */ +static void signed_no_fd_array(void) +{ + struct bpf_insn insns[] = { + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + char dir_tmpl[] = "/tmp/signed_loaderXXXXXX", *dir; + __u32 sig_sz = 8192; + union bpf_attr attr; + __u8 sig[8192]; + int prog_fd, err; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + return; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + return; + } + + /* No metadata map: the signed payload is the instructions alone. */ + if (!ASSERT_OK(sign_buf(dir, insns, sizeof(insns), sig, &sig_sz), + "sign-file")) + goto cleanup; + + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(insns); + attr.insn_cnt = ARRAY_SIZE(insns); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + /* fd_array and fd_array_cnt deliberately left NULL/0. */ + memcpy(attr.prog_name, "signed_nomap", sizeof("signed_nomap")); + + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + if (!ASSERT_GE(prog_fd, 0, "map-less signed program loaded")) { + if (prog_fd >= 0) + close(prog_fd); + goto cleanup; + } + close(prog_fd); + + /* The signature covers the instructions, so tampering must be rejected. */ + insns[0].imm = 1; + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + err = prog_fd < 0 ? -errno : prog_fd; + ASSERT_EQ(err, -EKEYREJECTED, "tampered map-less program rejected"); + if (prog_fd >= 0) + close(prog_fd); +cleanup: + run_setup("cleanup", dir); +} + +/* + * A signed program may reach maps only through fd_array indices, so the kernel + * folds (and thus attests) them. A direct BPF_PSEUDO_MAP_FD reference - a raw, + * unfolded fd baked into the signed instructions - is rejected by the verifier. + */ +static void signed_map_by_fd_rejected(void) +{ + struct bpf_insn insns[] = { + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + char dir_tmpl[] = "/tmp/signed_loaderXXXXXX", *dir; + __u32 sig_sz = 8192; + union bpf_attr attr; + __u8 sig[8192]; + int map_fd, prog_fd, err; + + map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "sig_mapfd", 4, 4, 1, NULL); + if (!ASSERT_GE(map_fd, 0, "map_create")) + return; + insns[0].imm = map_fd; /* bake the raw map fd into the ld_imm64 */ + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + goto out_map; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + goto out_map; + } + + /* Sign the instructions, raw map fd and all. */ + if (!ASSERT_OK(sign_buf(dir, insns, sizeof(insns), sig, &sig_sz), + "sign-file")) + goto cleanup; + + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(insns); + attr.insn_cnt = ARRAY_SIZE(insns); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + /* No fd_array: the map is reached by a raw fd in the instructions. */ + memcpy(attr.prog_name, "signed_mapfd", sizeof("signed_mapfd")); + + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + err = prog_fd < 0 ? -errno : prog_fd; + ASSERT_EQ(err, -EINVAL, "signed program referencing a map by fd rejected"); + if (prog_fd >= 0) + close(prog_fd); +cleanup: + run_setup("cleanup", dir); +out_map: + close(map_fd); +} + +/* + * A signed program may reach maps only through the continuous fd_array, so the + * kernel folds (and thus attests) them. Referencing a map by fd_array *index* + * while leaving fd_array_cnt at 0 selects the sparse path, which resolves a map + * the signature never covered; the verifier rejects it up front with -EACCES. + */ +static void signed_sparse_fd_array_rejected(void) +{ + struct bpf_insn insns[] = { + BPF_LD_IMM64_RAW(BPF_REG_1, BPF_PSEUDO_MAP_IDX, 0), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + char dir_tmpl[] = "/tmp/signed_loader_spXXXXXX", *dir; + __u32 sig_sz = 8192; + union bpf_attr attr; + __u8 sig[8192]; + int map_fd, prog_fd, err; + + map_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY, "sig_sparse", 4, 4, 1, NULL); + if (!ASSERT_GE(map_fd, 0, "map_create")) + return; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + goto out_map; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + goto out_map; + } + + /* Sign the instructions alone; the sparse map is not folded. */ + if (!ASSERT_OK(sign_buf(dir, insns, sizeof(insns), sig, &sig_sz), + "sign-file")) + goto cleanup; + + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(insns); + attr.insn_cnt = ARRAY_SIZE(insns); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.fd_array = ptr_to_u64(&map_fd); + attr.fd_array_cnt = 0; /* sparse: force lazy map resolution */ + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + memcpy(attr.prog_name, "signed_sparse", sizeof("signed_sparse")); + + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + err = prog_fd < 0 ? -errno : prog_fd; + ASSERT_EQ(err, -EACCES, "signed program binding a sparse fd_array map rejected"); + if (prog_fd >= 0) + close(prog_fd); +cleanup: + run_setup("cleanup", dir); +out_map: + close(map_fd); +} + +static void signed_module_kfunc_rejected(void) +{ + struct bpf_insn insns[] = { + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 1, 1), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + char dir_tmpl[] = "/tmp/signed_loader_kfnXXXXXX", *dir; + int prog_fd, err, fds[2]; + struct btf *btf = NULL; + __u32 sig_sz = 8192; + union bpf_attr attr; + __u8 sig[8192]; + + syscall(__NR_request_key, "keyring", "_uid.0", NULL, + KEY_SPEC_SESSION_KEYRING); + dir = mkdtemp(dir_tmpl); + if (!ASSERT_OK_PTR(dir, "mkdtemp")) + return; + if (!ASSERT_OK(run_setup("setup", dir), "verify_sig_setup")) { + rmdir(dir); + return; + } + if (!ASSERT_OK(sign_buf(dir, insns, sizeof(insns), sig, &sig_sz), + "sign-file")) + goto cleanup; + btf = btf__new_empty(); + if (!ASSERT_OK_PTR(btf, "btf_new_empty")) + goto cleanup; + btf__add_int(btf, "int", 4, BTF_INT_SIGNED); + if (!ASSERT_OK(btf__load_into_kernel(btf), "btf_load")) + goto cleanup; + fds[0] = -1; + fds[1] = btf__fd(btf); + + memset(&attr, 0, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SYSCALL; + attr.insns = ptr_to_u64(insns); + attr.insn_cnt = ARRAY_SIZE(insns); + attr.license = ptr_to_u64("Dual BSD/GPL"); + attr.prog_flags = BPF_F_SLEEPABLE; + attr.fd_array = ptr_to_u64(fds); + attr.fd_array_cnt = 0; /* sparse: force lazy kfunc BTF resolution */ + attr.signature = ptr_to_u64(sig); + attr.signature_size = sig_sz; + attr.keyring_id = KEY_SPEC_SESSION_KEYRING; + memcpy(attr.prog_name, "signed_kfunc", sizeof("signed_kfunc")); + + prog_fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, + offsetofend(union bpf_attr, keyring_id)); + err = prog_fd < 0 ? -errno : prog_fd; + if (prog_fd >= 0) + close(prog_fd); + + ASSERT_EQ(err, -EACCES, "module kfunc BTF in signed program rejected"); +cleanup: + if (btf) + btf__free(btf); + run_setup("cleanup", dir); +} + void test_signed_loader(void) { - if (test__start_subtest("metadata_check_shape")) - metadata_check_shape(); + if (test__start_subtest("loadtime_no_map")) + loadtime_no_map(); + if (test__start_subtest("loadtime_with_map")) + loadtime_with_map(); if (test__start_subtest("metadata_match")) metadata_match(); - if (test__start_subtest("metadata_sha_mismatch")) - metadata_sha_mismatch(); - if (test__start_subtest("metadata_not_exclusive")) - metadata_not_exclusive(); - if (test__start_subtest("metadata_hash_not_computed")) - metadata_hash_not_computed(); if (test__start_subtest("signature_enforced")) signature_enforced(); + if (test__start_subtest("signed_nonexcl_fd_array_rejected")) + signed_nonexcl_fd_array_rejected(); + if (test__start_subtest("signed_unfrozen_fd_array_rejected")) + signed_unfrozen_fd_array_rejected(); + if (test__start_subtest("signed_nonarray_fd_array_rejected")) + signed_nonarray_fd_array_rejected(); + if (test__start_subtest("signed_btf_fd_array_rejected")) + signed_btf_fd_array_rejected(); + if (test__start_subtest("signed_module_kfunc_rejected")) + signed_module_kfunc_rejected(); + if (test__start_subtest("signature_failure_logs")) + signature_failure_logs(); if (test__start_subtest("signature_too_large")) signature_too_large(); + if (test__start_subtest("signature_zero_size")) + signature_zero_size(); if (test__start_subtest("signature_bad_keyring")) signature_bad_keyring(); if (test__start_subtest("metadata_ctx_max_entries_ignored")) @@ -1114,6 +1803,8 @@ void test_signed_loader(void) metadata_ctx_initial_value_ignored(); if (test__start_subtest("signature_authenticates_insns")) signature_authenticates_insns(); + if (test__start_subtest("signature_authenticates_metadata")) + signature_authenticates_metadata(); if (test__start_subtest("hash_requires_frozen")) hash_requires_frozen(); if (test__start_subtest("no_update_after_freeze")) @@ -1132,4 +1823,10 @@ void test_signed_loader(void) map_hash_unsupported_type(); if (test__start_subtest("lsm_signature_verdict")) lsm_signature_verdict(); + if (test__start_subtest("signed_no_fd_array")) + signed_no_fd_array(); + if (test__start_subtest("signed_map_by_fd_rejected")) + signed_map_by_fd_rejected(); + if (test__start_subtest("signed_sparse_fd_array_rejected")) + signed_sparse_fd_array_rejected(); } diff --git a/tools/testing/selftests/bpf/progs/test_signed_loader.c b/tools/testing/selftests/bpf/progs/test_signed_loader.c index d9a4b85f9391..50451a69b99a 100644 --- a/tools/testing/selftests/bpf/progs/test_signed_loader.c +++ b/tools/testing/selftests/bpf/progs/test_signed_loader.c @@ -4,10 +4,11 @@ /* * Minimal, map-less program. Driven through libbpf's gen_loader (gen_hash) - * by prog_tests/signed_loader.c so the generated light-skeleton loader (with - * the emit_signature_match metadata check) can be exercised against good - * and tampered metadata. A socket filter needs no load-time attach resolution, - * and having no maps keeps the generated loader's ctx trivial (0 maps, 1 prog). + * by prog_tests/signed_loader.c so the generated light-skeleton loader can be + * exercised against good and tampered metadata, which the kernel now verifies + * at load time via the insns||metadata signature. A socket filter needs no + * load-time attach resolution, and having no maps keeps the generated loader's + * ctx trivial (0 maps, 1 prog). */ SEC("socket") int probe(void *ctx) From 84c42f515f184d4c9bc05da385f39f7ff406c302 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 09:53:43 +0200 Subject: [PATCH 053/373] Documentation/bpf: Add BPF signing and enforcement doc Describe the BPF signing design end to end: why a trusted loader is needed, the signature(insns || metadata) contract, load-time verification via fd_array (exclusive + frozen maps), the binary BPF_SIG_{UNSIGNED,VERIFIED} verdict, and how [BPF] LSMs can enforce policy on it. This writes down the contract on the discussion points with the LSM / integrity folks [0][1]: by the time security_bpf_prog_load() is called, signature verification has fully completed and covers the instructions plus the frozen contents of every bound exclusive map; there is no intermediate "loader verified, payload pending" state to reason about; and what BPF_SIG_VERIFIED means at each hook is spelled out explicitly. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/bc823ddbaf63e0e177eb46d1cc15076e4e2e689d.camel@HansenPartnership.com [0] Link: https://lore.kernel.org/bpf/CAHC9VhSDkwGgPfrBUh7EgBKEJj_JjnY68c0YAmuuLT_i--GskQ@mail.gmail.com [1] Link: https://lore.kernel.org/bpf/20260708075343.358712-9-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- Documentation/bpf/index.rst | 1 + Documentation/bpf/signing.rst | 497 ++++++++++++++++++++++++++++++++++ 2 files changed, 498 insertions(+) create mode 100644 Documentation/bpf/signing.rst diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst index 0d5c6f659266..638a00d42bc2 100644 --- a/Documentation/bpf/index.rst +++ b/Documentation/bpf/index.rst @@ -28,6 +28,7 @@ that goes into great technical depth about the BPF Architecture. classic_vs_extended.rst bpf_iterators bpf_licensing + signing test_debug clang-notes linux-notes diff --git a/Documentation/bpf/signing.rst b/Documentation/bpf/signing.rst new file mode 100644 index 000000000000..e73eaaebd8b1 --- /dev/null +++ b/Documentation/bpf/signing.rst @@ -0,0 +1,497 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +BPF signing +============ + +This document describes how BPF programs are cryptographically signed, how the +kernel verifies them at load time, and how Linux Security Modules (LSMs) - +including the BPF LSM - use the resulting verdict to enforce policy. It is +written for developers who want to produce signed BPF objects, understand what +the signature actually guarantees, or build a policy on top of it. + +Motivation +========== + +A signed BPF program lets the kernel establish that the bytecode being loaded +originates from a trusted producer and was not modified in transit. On its own +the kernel does not *require* signatures - an unsigned program loads exactly as +before - but it records a verdict (see `The verdict`_) that an LSM can gate on. +This is the building block for policies such as "only run BPF that was signed by +a key in the trusted keyring", as could in the future be enforced by an LSM +such as IPE. + +Signing is orthogonal to the existing permission model: it does not replace the +capability checks or the verifier. A signed load still requires the usual +privileges (``CAP_BPF`` and any program-type-specific capability, subject to +``kernel.unprivileged_bpf_disabled``), and the loader's instructions are still +checked by the verifier like any other program. A valid signature establishes +*origin and integrity*, not safety - it lets a policy trust where the bytecode +came from, it does not let a load skip any check it would otherwise face. + +The hard part is *what* gets signed. A naive scheme would sign a program's +instruction buffer at build time and verify that signature at +``BPF_PROG_LOAD``. That does not survive contact with real BPF objects, because +the bytes the kernel finally loads are not the bytes the developer built and +signed. Between the two, libbpf and the kernel rewrite the program: + +- **map file descriptors** are patched into ``ld_imm64`` instructions + (``BPF_PSEUDO_MAP_FD``), and a map's fd is assigned at load time, so it + differs on every run; +- **CO-RE relocations** rewrite field offsets, sizes and existence flags against + the *running* kernel's BTF, so the result differs from one kernel to the next; +- **kfunc and ksym references** are resolved to ids/addresses in the running + kernel; +- **global data** (``.rodata``/``.data``/``.bss``) is created and seeded as maps + at load. + +So a signature over the original instructions cannot match the relocated +instructions the verifier ends up checking, and the relocated form cannot be +produced ahead of time because it depends on the target kernel. There is no +fixed byte string that is both signable at build time and what the kernel +actually loads - which is why a program cannot simply be signed and loaded +directly. + +The trusted loader +================== + +The solution is to move that setup work *into* a small BPF program - the +**loader** - and sign the loader instead of the individual programs. libbpf's +``gen_loader`` machinery (``bpftool gen skeleton -L``, the "light skeleton") +emits a ``BPF_PROG_TYPE_SYSCALL`` program whose body performs the bpf() syscalls +that create maps, apply relocations, and load the real programs. The payload it +installs - the serialized programs, map descriptions, relocation data and +initial values - lives in a separate array map, the **metadata map** +(``__loader.map``). + +So the unit of trust is the loader, and the signing contract is:: + + Sig(I_loader || D_meta) + +where ``I_loader`` is the loader's instruction stream and ``D_meta`` is the +content of the metadata map. Verifying the loader's signature establishes that +both the loader *and* the payload it is about to install are authentic. The +loader is reproducible: ``gen_loader`` builds it from primitives so the same +object yields the same bytes on any build host. + +Why the loader is signable when the program is not +-------------------------------------------------- + +The loader sidesteps every rewrite listed above, because the bytes that are +signed are *relocation-invariant*: + +- The loader's own instructions are a fixed sequence of bpf() syscalls emitted + by ``gen_loader``; they carry no CO-RE relocations and resolve no ksyms, so + they are identical on every kernel. The metadata map is referenced by *index* + into ``fd_array`` (``BPF_PSEUDO_MAP_IDX_VALUE``), not by a baked-in file + descriptor, so even that reference does not change between build and load. + The loader instruction bytes the kernel verifies are exactly the bytes that + were signed. +- The metadata map is opaque, frozen data - the serialized target programs, + their relocation records, map descriptions and initial values. Its bytes are + identical at build time and at load time, so they are simply appended to the + instructions and covered by the same signature (there is no separate metadata + hash to compute or compare). + +All the host-specific rewriting - creating maps, patching their fds into the +target programs, applying CO-RE, resolving ksyms, seeding global data - still +happens, but it happens *inside the loader at runtime*, on the verified +metadata, **after** the kernel has verified the ``insns || metadata`` signature. +The kernel never has to verify the relocated target programs: it verifies the +loader and its inputs once, and trust transfers to whatever that now-trusted, +deterministic loader installs. The relocation step is moved from "before the +signature can be checked" to "after a trusted program runs" - which is exactly +what makes it signable. + +Because the metadata map is the loader's only untrusted input, two existing map +properties are reused to keep it trustworthy across the load: + +Exclusive maps + A map created with ``excl_prog_hash`` (see ``BPF_MAP_CREATE``) may only be + accessed by a program whose digest matches that hash. The verifier enforces + ``map->excl_prog_sha == prog->digest`` for every map a program uses, so the + metadata map is bound to exactly the signed loader and cannot be shared with + or mutated by another program. + +Frozen maps + The metadata map is frozen (``BPF_MAP_FREEZE``) before the loader is loaded. + Freezing blocks further userspace writes, so the bytes folded into the + signature cannot change before the loader runs. (Freezing does not make the + map read-only to the loader program itself, which still writes created file + descriptors back into the blob's scratch area.) + +Load-time verification +======================= + +Rather than have the loader check its own metadata from within BPF, the kernel +verifies it directly at ``BPF_PROG_LOAD``, with no new UAPI. The mechanism +reuses the existing ``fd_array``: + +#. Userspace creates the metadata map with ``excl_prog_hash`` set to the + loader's digest, populates it, and freezes it. +#. The loader is loaded with ``signature``/``signature_size``/``keyring_id`` + set, the metadata map referenced through ``fd_array``, and ``fd_array_cnt`` + set so the kernel knows the array's length. +#. Signature verification runs inside the verifier (``bpf_check()``), once it + has resolved the ``fd_array`` entries into the program's ``used_maps``. The + maps folded into the signature are therefore the very objects the program + binds - a single resolution of ``fd_array``, not a separate read, so the + verified bytes cannot be swapped for a different map after the check (no + time-of-check/time-of-use window). Each folded map must be exclusive (carry + ``excl_prog_sha``) and a plain array map (``BPF_MAP_TYPE_ARRAY``); only an + array map exposes its value buffer through ``map_direct_value_addr()`` as a + kernel address spanning ``value_size`` bytes. A map that is not exclusive, not + frozen, or not a plain array is rejected, with a verifier log message naming + the offending map. The kernel appends each map's frozen + contents to the instruction buffer and verifies the PKCS#7 signature over the + concatenation ``insns || metadata_0 || metadata_1 || ...`` in ``used_maps`` + order, before it rewrites the (signed) instructions. + +A signed program therefore takes one of exactly two shapes, both fully +supported: + +- **No bound maps** (``fd_array_cnt == 0``): there is nothing to append, so the + kernel verifies the signature over the instructions alone. A valid signature + yields ``BPF_SIG_VERIFIED`` and the program loads. This is the ordinary case + for a directly-loaded signed program with no separate payload; it is *not* + rejected for "missing" metadata, because it has none to cover. +- **Exclusive bound maps** (``fd_array_cnt > 0``): every entry is exclusive and + folded, so the signature covers ``insns || metadata``. + +There is no third shape: a non-exclusive map in a signed program's ``fd_array`` +is rejected rather than silently left out of the signature, so a signed loader +never binds a map its signature does not cover. + +The digest binding (``excl_prog_sha == prog->digest``) is enforced by the +verifier as usual; because that check runs while ``fd_array`` is resolved - +before the verifier would otherwise compute the tag - ``prog->digest`` is +computed up front in the verifier, over the unmodified (signature-covered) +instructions, for any signed load. + +Coverage is then enforced as the verifier resolves instructions, at the point +each object is bound rather than by a count taken afterwards. Once the signature +has been verified, binding any further map is refused: a map reached by a +directly-referenced fd, or a map swapped into an ``fd_array`` slot the loader +reads, is not among those already folded, so it is rejected the moment the +verifier tries to bind it. A BTF is refused outright for a signed program - a +ksym or a BTF fd in ``fd_array``, whether resolved up front or lazily for a +module kfunc, is rejected when it would be bound. Together with the fold rule +above this keeps the verdict binary: a signed program cannot use a map its +signature does not cover, and a different but equally digest-bound map cannot be +substituted at an ``fd_array`` slot. Non-exclusive maps are never folded, so a +signed program cannot use one at all. + +The verdict +=========== + +A program is either unsigned or fully verified - there is no intermediate +state. The outcome is recorded in ``prog->aux->sig.verdict``: + +.. code-block:: c + + enum bpf_sig_verdict { + BPF_SIG_UNSIGNED = 0, + BPF_SIG_VERIFIED, + }; + +``BPF_SIG_VERIFIED`` means the signature is valid and covers the instructions +*and* the frozen contents of every exclusive map the program uses: + +- For an ordinary, directly-loaded signed program the instructions are the whole + artifact and it uses no exclusive maps, so a valid instruction signature is + the complete verification. +- For a signed loader the metadata map is exclusive, so its contents are folded + in and the signature covers ``insns || metadata``. + +There is deliberately no "instructions verified but metadata not" verdict: a +signed loader that fails to cover its metadata is *rejected* (see above), not +recorded with a weaker verdict. ``BPF_SIG_VERIFIED`` therefore always means the +program and everything the signature is responsible for are authentic, which is +what a policy can rely on. + +Alongside the verdict the kernel records which keyring validated the signature; +see `Keyrings`_. + +Enforcement via LSMs +==================== + +Signing only *records* a verdict; an LSM turns it into policy. The verdict and +keyring fields live in ``struct bpf_prog_aux``, so a BPF LSM program can read +them directly (see Documentation/bpf/prog_lsm.rst for writing and attaching BPF +LSM programs); the same fields are equally available to in-tree LSMs. Two hooks +are useful at different points of the load: the dedicated +``security_bpf_prog_load()`` gates admission before the main verification work, +and the existing ``security_bpf_prog()`` observes a program that has fully +loaded. + +Admission: ``security_bpf_prog_load()`` +--------------------------------------- + +This hook gates admission **for every load**, from a single call site inside the +verifier (``bpf_check()``), before the main verification work. It runs after the +optional signature verification, so the verdict and keyring fields are final - the +hook can see whether, and how strongly, the program was signed, which keyring +validated it, the load ``attr``, the BPF token and whether the load came from the +kernel. For a signed load the verdict is ``BPF_SIG_VERIFIED`` here (the signature +has just been checked); for an unsigned load it is ``BPF_SIG_UNSIGNED``. + +This is the place for *coarse admission* that must also see unsigned and +not-yet-verified loads: require a signature at all, restrict the acceptable +keyring, restrict which token/credentials may load BPF, apply per-program-type +rules, or audit every load attempt that makes it past signature verification - +attempts failing the signature or the metadata binding abort before this hook +fires. It is the primary deny point. + +One subtlety: this hook runs *before* the verifier finishes its work, so +``BPF_SIG_VERIFIED`` *here* means only "validly signed" - not "loaded". Allowing +a load at this point lets it *proceed*; it does not guarantee the program will +load. A validly signed program can still be rejected afterwards on two +independent grounds: the verifier may reject it like any other program (unsafe +memory access, bad control flow, resource limits, ...), and the kernel separately +refuses - as the verifier resolves instructions and binds each object - any map +the signature does not cover or any BTF at all, regardless of what this hook +returned. Only after the program has fully loaded, at the next hook +(``security_bpf_prog()``), does ``BPF_SIG_VERIFIED`` carry its full meaning: +validly signed *and* fully verified. + +A more realistic admission policy than "is it signed at all": accept programs +signed by a system keyring, accept a user-keyring signature only if the +key/keyring it was verified against is on an explicit allowlist, and emit a +tamper-evident record of every decision so that even denied attempts are +auditable. (Illustrative - error checking elided.) + +.. code-block:: c + + /* Serials of user keys/keyrings we additionally trust. */ + struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __s32); /* keyring_serial */ + __type(value, __u8); + __uint(max_entries, 64); + } trusted_user_keys SEC(".maps"); + + /* Audit stream consumed by a userspace logger. */ + struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 1 << 16); + } audit SEC(".maps"); + + struct decision { __u32 prog_type, verdict, ktype; __s32 serial, ret; }; + + SEC("lsm/bpf_prog_load") + int BPF_PROG(admit, struct bpf_prog *prog, union bpf_attr *attr, + struct bpf_token *token, bool kernel) + { + __u32 verdict = prog->aux->sig.verdict; + __u32 ktype = prog->aux->sig.keyring_type; + __s32 serial = prog->aux->sig.keyring_serial; + struct decision *d; + int ret = 0; + + if (kernel) + return 0; /* trust in-kernel loads */ + + if (verdict != BPF_SIG_VERIFIED) + ret = -EPERM; /* must be validly signed */ + else if (ktype == BPF_SIG_KEYRING_USER && + !bpf_map_lookup_elem(&trusted_user_keys, &serial)) + ret = -EPERM; /* key/keyring not allowlisted */ + + d = bpf_ringbuf_reserve(&audit, sizeof(*d), 0); + if (d) { + d->prog_type = attr->prog_type; + d->verdict = verdict; + d->ktype = ktype; + d->serial = serial; + d->ret = ret; + bpf_ringbuf_submit(d, 0); /* record allow *and* deny */ + } + return ret; + } + +Observing a verified load: ``security_bpf_prog()`` +-------------------------------------------------- + +There is deliberately no separate "metadata attested" hook. The coverage check +above is enforced by the kernel unconditionally, so a signed loader that fails +to cover its metadata never loads and an LSM never has to re-establish that +fact. To *act on* a program that has successfully and fully loaded, use the +existing ``security_bpf_prog()`` hook (``lsm/bpf_prog``), which fires from +``bpf_prog_new_fd()`` - after the verifier, after the coverage check, and after +``bpf_prog_alloc_id()``. Relative to the admission hook this point is strictly +later and stronger: + +- the program has an id (``prog->aux->id``), so it can be recorded or correlated + with later events; +- ``verdict == BPF_SIG_VERIFIED`` *here* means **fully** verified - a program + that used a map the signature does not cover was already rejected, so it cannot + reach this point; +- it observes only programs that actually loaded; a failed load never mints an + fd, so it never reaches this hook. + +It takes only the ``prog`` and a non-zero return still aborts (the fd is not +handed out), so it can veto as well as observe. One wrinkle: it also fires on +other paths that mint a new program fd - notably ``bpf_prog_get_fd_by_id()`` - +not just on a fresh load. Because the program already has its id here, an LSM +can tell the two apart with a small hash map: the *first* time an id is seen is +the load; a later sighting of the same id is just another fd to a program that +already exists. + +To bound the map and let a reused id read as a fresh load, this can be paired +with ``security_bpf_prog_free()`` (``lsm/bpf_prog_free``), which deletes the +entry on teardown - keyed by the same ``prog`` pointer, since +``bpf_prog_free_id()`` has already cleared ``prog->aux->id`` to ``0`` by the time +that hook runs. (Illustrative - privileged LSM, error checking elided.) + +.. code-block:: c + + struct rec { __u32 id, ktype; __s32 serial; }; + + struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, __u64); /* struct bpf_prog * -- stable id */ + __type(value, struct rec); + __uint(max_entries, 4096); + } live SEC(".maps"); + + SEC("lsm/bpf_prog") /* fires after load and on every later fd */ + int BPF_PROG(observe, struct bpf_prog *prog) + { + __u64 key = (__u64)(unsigned long)prog; + struct rec r; + + if (prog->aux->sig.verdict != BPF_SIG_VERIFIED) + return 0; + if (bpf_map_lookup_elem(&live, &key)) + return 0; /* seen before: a later fd, not a load */ + + /* First sighting == this program just loaded; id is valid here. */ + r.id = prog->aux->id; + r.ktype = prog->aux->sig.keyring_type; + r.serial = prog->aux->sig.keyring_serial; + bpf_map_update_elem(&live, &key, &r, BPF_NOEXIST); + /* ... newly-loaded verified-program action, e.g. record r.id ... */ + return 0; + } + +Putting them together: to *require* verified BPF, deny at the admission hook +unless the verdict is ``BPF_SIG_VERIFIED`` (and, if desired, restrict the +keyring). The kernel then guarantees that any program which actually loads with +that verdict covered all of its exclusive maps, rejecting any that did not - so +a deny-by-default admission policy needs no second enforcement point. Use +``security_bpf_prog()`` to record or finally gate the verified programs once +they carry an id. The ``verdict``, ``keyring_type`` and ``keyring_serial`` fields +let a policy distinguish, for example, "verified and signed by a builtin key" +from "verified by a user key". A policy LSM such as IPE could consume the same +hooks to enforce system policy without writing any BPF, though none implements +this today. + +Keyrings +======== + +``keyring_id`` selects the trusted keyring the PKCS#7 signature is verified +against. The well-known ids ``0`` (builtin), ``VERIFY_USE_SECONDARY_KEYRING`` +and ``VERIFY_USE_PLATFORM_KEYRING`` select the corresponding system keyrings; +any other value is treated as the serial of a user/session key or keyring. +The keyring is looked up first, before the signature bytes are examined, so a +signature naming a non-existent keyring is rejected up front, and a failed +verification aborts the load - so a program that loads successfully with a +signature always has consistent keyring fields recorded. + +Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect: + +``keyring_type`` (``enum bpf_sig_keyring``) + Classified purely from ``keyring_id`` whenever the program is signed: + ``BPF_SIG_KEYRING_BUILTIN``, ``_SECONDARY``, ``_PLATFORM`` for the system + keyrings, or ``_USER`` for a user/session keyring. It is + ``BPF_SIG_KEYRING_NONE`` for an unsigned program. + +``keyring_serial`` (``s32``) + Set **only** on a successful verification, to the serial of the + **user/session key or keyring** that ``keyring_id`` resolved to - the + object the signature was verified against, not the individual asymmetric + key inside it that matched the signer. Passing + ``KEY_SPEC_SESSION_KEYRING``, for example, records the session keyring's + serial. The system keyrings are trusted as a whole and expose no serial + here, so the serial is ``0`` for builtin, secondary and platform + signatures, and ``0`` for unsigned programs. In other words, a non-zero + ``keyring_serial`` is exactly "verified against the user key/keyring with + this serial". + +.. list-table:: + :header-rows: 1 + + * - ``keyring_id`` + - ``keyring_type`` + - ``keyring_serial`` + * - (no signature) + - ``BPF_SIG_KEYRING_NONE`` + - ``0`` + * - ``0`` + - ``BPF_SIG_KEYRING_BUILTIN`` + - ``0`` + * - ``VERIFY_USE_SECONDARY_KEYRING`` + - ``BPF_SIG_KEYRING_SECONDARY`` + - ``0`` + * - ``VERIFY_USE_PLATFORM_KEYRING`` + - ``BPF_SIG_KEYRING_PLATFORM`` + - ``0`` + * - other (a user/session key serial) + - ``BPF_SIG_KEYRING_USER`` + - serial of the resolved key/keyring + +Producing a signed object +========================== + +``bpftool`` generates and signs a light skeleton in one step:: + + bpftool gen skeleton -L -S -k -i \ + obj.bpf.o > obj.lskel.h + +``-L`` selects the light-skeleton (``gen_loader``) backend and ``-S`` enables +signing; ``-k`` and ``-i`` supply the signing key and its X.509 certificate. +``bpftool`` signs ``insns || metadata`` - the exact bytes the kernel +reconstructs - and also computes ``excl_prog_hash`` as the digest of the loader +instructions so the metadata map can be bound to the loader. The signature and +hash are embedded in the generated header; the certificate is used only for +signing and is not included. Loading the skeleton performs the +create/populate/freeze/load sequence described above. + +At runtime the trusted public key must be present in the chosen keyring (for +example added to the session keyring, or built into the kernel's builtin trusted +keyring) for verification to succeed. + +UAPI reference +============== + +``BPF_PROG_LOAD`` (``union bpf_attr``): + +``signature``, ``signature_size`` + Pointer to and length of the PKCS#7 signature blob. + +``keyring_id`` + Trusted keyring selector (see `Keyrings`_). + +``fd_array``, ``fd_array_cnt`` + Array of map (and module BTF) file descriptors bound to the program. + ``fd_array_cnt`` must be set for the kernel to scan the array. When a + signature is present, a BTF entry is rejected outright, and every map must + be exclusive; its frozen contents are folded into the verified buffer, and + a non-exclusive entry is rejected. + +``BPF_MAP_CREATE`` (``union bpf_attr``): + +``excl_prog_hash``, ``excl_prog_hash_size`` + SHA-256 digest of the program permitted to access this (exclusive) map. This + binds the metadata map to the loader; it is not a hash of the map *content*. + The map content is not hashed separately at all - it is covered, as bytes, + by the program signature. + +Notes and limitations +====================== + +- The instructions plus folded metadata are verified as one ``bpf_dynptr``, + which bounds the combined size (currently ~16 MiB); very large objects can + exceed it. +- The metadata container is a single-element array map, accessed through + ``map_direct_value_addr``. From 43f129d2148983f69f8f9c34f2a64d27f888bef1 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 20:41:07 +0200 Subject: [PATCH 054/373] selftests/bpf: Close fd on unexpected success in signed loader The signature_enforced, signature_too_large, signature_zero_size and signature_bad_keyring subtests load a program that must be rejected, but leave the fd open if the kernel unexpectedly accepts the load: test_progs asserts record the failure and continue, so the fd would linger for the rest of the run. Just close it. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708184107.369182-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/signed_loader.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/signed_loader.c b/tools/testing/selftests/bpf/prog_tests/signed_loader.c index 0019492cf07a..77381d345435 100644 --- a/tools/testing/selftests/bpf/prog_tests/signed_loader.c +++ b/tools/testing/selftests/bpf/prog_tests/signed_loader.c @@ -326,6 +326,8 @@ static void signature_enforced(void) fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, sizeof(junk), KEY_SPEC_SESSION_KEYRING, 0); ASSERT_EQ(fd, -EBADMSG, "invalid signature rejected at load"); + if (fd >= 0) + close(fd); } gen_loader_fixture_fini(&f); } @@ -575,6 +577,8 @@ static void signature_too_large(void) fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, 64 << 20, KEY_SPEC_SESSION_KEYRING, 0); ASSERT_EQ(fd, -EINVAL, "oversized signature rejected"); + if (fd >= 0) + close(fd); } gen_loader_fixture_fini(&f); } @@ -594,6 +598,8 @@ static void signature_zero_size(void) fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, 0, KEY_SPEC_SESSION_KEYRING, 0); ASSERT_EQ(fd, -EINVAL, "zero-size signature rejected"); + if (fd >= 0) + close(fd); } gen_loader_fixture_fini(&f); } @@ -614,6 +620,8 @@ static void signature_bad_keyring(void) fd = load_loader(f.gopts.insns, f.gopts.insns_sz, -1, junk, sizeof(junk), INT_MAX, 0); ASSERT_EQ(fd, -EINVAL, "signature with bad keyring_id rejected"); + if (fd >= 0) + close(fd); } gen_loader_fixture_fini(&f); } From 41ec7e4a17792599af6c208dcc32c038a9f78da8 Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Wed, 8 Jul 2026 12:47:17 +0200 Subject: [PATCH 055/373] selftests/bpf: Skip res_spin_lock_stress if no perf support Probe PMU support before loading bpf_test_rqspinlock.ko, otherwise the test fails with not obvious error without proper perf event support: Failed to load bpf_test_rqspinlock.ko into the kernel: -2 serial_test_res_spin_lock_stress:FAIL:load module AA unexpected error: -22 (errno 2) Reported-by: Ilya Leoshkevich Signed-off-by: Maxim Khmelevskii Reviewed-by: Ilya Leoshkevich Link: https://lore.kernel.org/bpf/20260708104729.1248234-2-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/res_spin_lock.c | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/res_spin_lock.c b/tools/testing/selftests/bpf/prog_tests/res_spin_lock.c index f0a8c828f8f1..7541f4966abc 100644 --- a/tools/testing/selftests/bpf/prog_tests/res_spin_lock.c +++ b/tools/testing/selftests/bpf/prog_tests/res_spin_lock.c @@ -3,6 +3,8 @@ #include #include #include +#include +#include #include "res_spin_lock.skel.h" #include "res_spin_lock_fail.skel.h" @@ -102,11 +104,29 @@ void test_res_spin_lock_success(void) void serial_test_res_spin_lock_stress(void) { + struct perf_event_attr attr = { + .size = sizeof(attr), + .type = PERF_TYPE_HARDWARE, + .config = PERF_COUNT_HW_CPU_CYCLES, + }; + int pmu_fd; + if (libbpf_num_possible_cpus() < 3) { test__skip(); return; } + pmu_fd = syscall(__NR_perf_event_open, &attr, 0, -1, -1, 0); + if (pmu_fd < 0) { + if (errno == ENOENT || errno == EOPNOTSUPP) { + test__skip(); + return; + } + ASSERT_OK(-errno, "perf_event_open pmu probe"); + return; + } + close(pmu_fd); + ASSERT_OK(load_module("bpf_test_rqspinlock.ko", false), "load module AA"); sleep(5); unload_module("bpf_test_rqspinlock", false); From 92863e678070f57c17c868e4bfa2441a5c61ad2b Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 23:15:34 +0200 Subject: [PATCH 056/373] bpf: Fix vmlinux BTF prep race in bpf_get_btf_vmlinux bpf_get_btf_vmlinux() lazily parses the vmlinux BTF under the bpf_verifier_lock, but publishes the result through a plain store and re-checks it through a plain lockless load. Nothing orders the stores initializing the struct btf inside btf_parse_vmlinux() against the store publishing the pointer: On a weakly ordered arch, a concurrent first-time caller taking the lockless fast path could in principle observe the pointer before the parsed contents are visible. The mutex_unlock() does not help such a reader given it only synchronizes with a later acquisition of the same lock. Thus, publish the pointer with smp_store_release() and read it on the fast path with smp_load_acquire(). Acquire semantics are needed rather than a dependency-ordered READ_ONCE(): btf_parse_vmlinux() also populates globals outside the returned object (e.g. bpf_ctx_convert.t). An address dependency would only order accesses performed through the pointer and not cover other globals. Fixes: 8580ac9404f6 ("bpf: Process in-kernel BTF") Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708211537.371874-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 001ac53825da..9217e0f87cb5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19559,13 +19559,25 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt struct btf *bpf_get_btf_vmlinux(void) { - if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { + /* Pairs with the smp_store_release() on the parse path below. */ + struct btf *btf = smp_load_acquire(&btf_vmlinux); + + if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { mutex_lock(&bpf_verifier_lock); - if (!btf_vmlinux) - btf_vmlinux = btf_parse_vmlinux(); + btf = btf_vmlinux; + if (!btf) { + btf = btf_parse_vmlinux(); + /* + * Order the parsed BTF contents and the globals the + * parse populated (e.g. bpf_ctx_convert.t) before + * the pointer publication. Pairs with the acquire + * on the lockless fast path above. + */ + smp_store_release(&btf_vmlinux, btf); + } mutex_unlock(&bpf_verifier_lock); } - return btf_vmlinux; + return btf; } /* From 5e5e94d87dea92cc2e2fadaf3be84771509a86ca Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 23:15:35 +0200 Subject: [PATCH 057/373] bpf: Give vmlinux BTF init its own mutex bpf_get_btf_vmlinux() serializes the lazy vmlinux BTF parse with bpf_verifier_lock, the same mutex bpf_check() holds across the whole verification of an unprivileged program (if enabled; it's disabled by default). The latter can potentially stall the mutex holder for a long time (e.g. via userfaultfd), and therefore block first-time bpf_get_btf_vmlinux() caller from any context, including privileged program loads. Give the vmlinux BTF initialization a dedicated btf_vmlinux_lock so it is independent of the unprivileged verification mutex. The parse only needs mutual exclusion against itself. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708211537.371874-3-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/btf.c | 2 +- kernel/bpf/verifier.c | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index dff5c0d91641..8c04c340f499 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6451,7 +6451,7 @@ struct btf *btf_parse_vmlinux(void) if (IS_ERR(btf)) goto err_out; - /* btf_parse_vmlinux() runs under bpf_verifier_lock */ + /* btf_parse_vmlinux() runs under btf_vmlinux_lock */ bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]); err = btf_alloc_id(btf); if (err) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9217e0f87cb5..40e20dfa3212 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -324,6 +324,7 @@ static const char *btf_type_name(const struct btf *btf, u32 id) } static DEFINE_MUTEX(bpf_verifier_lock); +static DEFINE_MUTEX(btf_vmlinux_lock); static DEFINE_MUTEX(bpf_percpu_ma_lock); __printf(2, 3) static void verbose(void *private_data, const char *fmt, ...) @@ -19563,7 +19564,7 @@ struct btf *bpf_get_btf_vmlinux(void) struct btf *btf = smp_load_acquire(&btf_vmlinux); if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { - mutex_lock(&bpf_verifier_lock); + mutex_lock(&btf_vmlinux_lock); btf = btf_vmlinux; if (!btf) { btf = btf_parse_vmlinux(); @@ -19575,7 +19576,7 @@ struct btf *bpf_get_btf_vmlinux(void) */ smp_store_release(&btf_vmlinux, btf); } - mutex_unlock(&bpf_verifier_lock); + mutex_unlock(&btf_vmlinux_lock); } return btf; } @@ -20089,7 +20090,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, bpf_get_btf_vmlinux(); - /* grab the mutex to protect few globals used by verifier */ + /* Serialize verification of unprivileged programs. */ if (!is_priv) mutex_lock(&bpf_verifier_lock); From 42560699a83db261d1a671a5eadade460d0f9eee Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 23:15:36 +0200 Subject: [PATCH 058/373] bpf: Account insn_aux_data allocation in bpf_check The insn_aux_data array is allocated with a plain vzalloc(), while every other allocation scoped to the verification - verifier states, explored states, the cfg/scc arrays, liveness masks, jump history - is charged to the loader's memcg via GFP_KERNEL_ACCOUNT. At 136 bytes per instruction it is one of the largest verification-time buffers, in the range of ~130MB for a program at the 1M instruction limit (worst case), and it lives across the whole verification. The buffer is also inconsistent with itself: when instruction patching grows it, the vrealloc() in bpf_patch_insn_data() already passes GFP_KERNEL_ACCOUNT. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708211537.371874-4-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 40e20dfa3212..ad8ff228c963 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20096,7 +20096,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, len = env->prog->len; env->insn_aux_data = - vzalloc(array_size(sizeof(struct bpf_insn_aux_data), len)); + __vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); ret = -ENOMEM; if (!env->insn_aux_data) goto skip_full_check; From ff755b6007908730946c155bb0d90ebc55926da7 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Wed, 8 Jul 2026 23:15:37 +0200 Subject: [PATCH 059/373] bpf: Account scratch buffer in bpf_prog_calc_tag bpf_prog_calc_tag() copies the instructions into a plain vmalloc() scratch buffer to blind the map fds before hashing. The buffer scales with the program, up to ~8MB at the 1M instruction limit, and is allocated on every program load, but unlike the rest of the load-time scratch memory it is not charged to the loader's memcg. Use GFP_KERNEL_ACCOUNT to account it like the other allocations scoped to the verification/load. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260708211537.371874-5-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6e19a030da6f..f2b6e4c888af 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -305,7 +305,7 @@ int bpf_prog_calc_tag(struct bpf_prog *fp) bool was_ld_map; u32 i; - dst = vmalloc(size); + dst = __vmalloc(size, GFP_KERNEL_ACCOUNT); if (!dst) return -ENOMEM; From 47b079e2117a2ee52e21f8b72935900c702fc0b5 Mon Sep 17 00:00:00 2001 From: Sanghyun Park Date: Wed, 8 Jul 2026 16:21:04 +0900 Subject: [PATCH 060/373] bpf: Fix use-after-free on mm_struct in bpf_find_vma() bpf_find_vma() reads task->mm and calls mmap_read_trylock(mm) without holding a reference on the mm. On a foreign task, a concurrent exit_mm() can free the mm_struct between the lockless read and the trylock, resulting in a use-after-free. mm_struct is not SLAB_TYPESAFE_BY_RCU. For the current task, task->mm is stable. For a foreign task, pin the mm under task->alloc_lock and release it with mmput_async(), mirroring commit d8e27d2d22b6 ("bpf: fix mm lifecycle in open-coded task_vma iterator"). Use spin_trylock() instead of get_task_mm() so BPF context does not block on alloc_lock. Reject irqs-disabled contexts and !CONFIG_MMU on the foreign-task path because dropping the mm reference is not safe there. Race: CPU0 (BPF program) CPU1 (exiting task) ============================ ========================== bpf_find_vma(foreign_task): mm = task->mm exit_mm(): task->mm = NULL mmput(mm) -> frees mm_struct mmap_read_trylock(mm) // UAF on mm Fixes: 7c7e3d31e785 ("bpf: Introduce helper bpf_find_vma") Signed-off-by: Sanghyun Park Reviewed-by: Puranjay Mohan Acked-by: Yonghong Song Link: https://lore.kernel.org/bpf/20260708072106.199637-2-sanghyun.park.cnu@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/task_iter.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index e791ae065c39..b256fb9c1214 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -756,6 +756,7 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, struct mmap_unlock_irq_work *work = NULL; struct vm_area_struct *vma; bool irq_work_busy = false; + bool __maybe_unused mmput_needed = false; struct mm_struct *mm; int ret = -ENOENT; @@ -765,14 +766,38 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, if (!task) return -ENOENT; - mm = task->mm; + if (task == current) { + mm = task->mm; + } else { + /* + * Foreign task: pin task->mm against a concurrent exit_mm(). + * Use trylock on alloc_lock instead of get_task_mm()'s + * blocking task_lock() to avoid deadlocking the target task. + */ + if (!IS_ENABLED(CONFIG_MMU)) + return -EOPNOTSUPP; + if (irqs_disabled()) + return -EBUSY; + if (!spin_trylock(&task->alloc_lock)) + return -EBUSY; + mm = task->mm; + if (mm && !(task->flags & PF_KTHREAD)) { + mmget(mm); + mmput_needed = true; + } else { + mm = NULL; + } + spin_unlock(&task->alloc_lock); + } if (!mm) return -ENOENT; irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); - if (irq_work_busy || !mmap_read_trylock(mm)) - return -EBUSY; + if (irq_work_busy || !mmap_read_trylock(mm)) { + ret = -EBUSY; + goto out; + } vma = find_vma(mm, start); @@ -782,6 +807,11 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, ret = 0; } bpf_mmap_unlock_mm(work, mm); +out: +#ifdef CONFIG_MMU + if (mmput_needed) + mmput_async(mm); +#endif return ret; } From 9a6df65d5c6a9947ddab4e563e329720f44b8747 Mon Sep 17 00:00:00 2001 From: Tiezhu Yang Date: Wed, 8 Jul 2026 18:18:05 +0800 Subject: [PATCH 061/373] bpf: Introduce jit_required flag and remove bpf_prog_has_kfunc_call() Introduce a 'jit_required' bitfield flag in struct bpf_prog to track whether a BPF program strictly requires the JIT compiler to run. This prevents a dangerous runtime fallback to the interpreter for features that are only implemented in the JIT compiler. Currently, bpf_prog_has_kfunc_call() is used only for kernel function calls, replace the kfunc-specific helper with the new 'jit_required' flag. This makes it easy to support other JIT-only BPF features, such as inlined helpers. Suggested-by: Alexei Starovoitov Suggested-by: KaFai Wan Suggested-by: Leon Hwang Acked-by: Leon Hwang Signed-off-by: Tiezhu Yang Signed-off-by: Eduard Zingerman --- include/linux/bpf.h | 9 ++------- kernel/bpf/core.c | 7 ++----- kernel/bpf/fixups.c | 5 ++--- kernel/bpf/verifier.c | 7 ++----- 4 files changed, 8 insertions(+), 20 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c1a98fa36738..31181e0c2b80 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1864,8 +1864,9 @@ struct bpf_prog_aux { struct bpf_prog { u16 pages; /* Number of allocated pages */ - u16 jited:1, /* Is our filter JIT'ed? */ + u32 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ + jit_required:1, /* program strictly requires JIT compiler */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -3169,7 +3170,6 @@ const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog); void bpf_task_storage_free(struct task_struct *task); void bpf_cgrp_storage_free(struct cgroup *cgroup); -bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog); const struct btf_func_model * bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn); @@ -3508,11 +3508,6 @@ static inline void bpf_task_storage_free(struct task_struct *task) { } -static inline bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog) -{ - return false; -} - static inline const struct btf_func_model * bpf_jit_find_kfunc_model(const struct bpf_prog *prog, const struct bpf_insn *insn) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index f2b6e4c888af..47fe047ad30b 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -126,6 +126,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag fp->aux->main_prog_aux = aux; fp->aux->prog = fp; fp->jit_requested = ebpf_jit_enabled(); + fp->jit_required = IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON); fp->blinding_requested = bpf_jit_blinding_enabled(fp); #ifdef CONFIG_CGROUP_BPF aux->cgroup_atype = CGROUP_BPF_ATTACH_TYPE_INVALID; @@ -2670,15 +2671,11 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct /* In case of BPF to BPF calls, verifier did all the prep * work with regards to JITing, etc. */ - bool jit_needed = false; + bool jit_needed = fp->jit_required; if (fp->bpf_func) goto finalize; - if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) || - bpf_prog_has_kfunc_call(fp)) - jit_needed = true; - if (!bpf_prog_select_interpreter(fp)) jit_needed = true; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 12a8a4eb757f..02246df2f6c3 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1378,7 +1378,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) #ifndef CONFIG_BPF_JIT_ALWAYS_ON struct bpf_prog *prog = env->prog; struct bpf_insn *insn = prog->insnsi; - bool has_kfunc_call = bpf_prog_has_kfunc_call(prog); int depth; #endif int i, err = 0; @@ -1404,8 +1403,8 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) return err; } #ifndef CONFIG_BPF_JIT_ALWAYS_ON - if (has_kfunc_call) { - verbose(env, "calling kernel functions are not allowed in non-JITed programs\n"); + if (prog->jit_required) { + verbose(env, "program requires BPF JIT compiler but it is not available\n"); return -EINVAL; } for (i = 0; i < env->subprog_cnt; i++) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ad8ff228c963..233472a871be 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2780,6 +2780,8 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) prog_aux->kfunc_tab = tab; } + env->prog->jit_required = 1; + /* func_id == 0 is always invalid, but instead of returning an error, be * conservative and wait until the code elimination pass before returning * error, so that invalid calls that get pruned out can be in BPF programs @@ -2834,11 +2836,6 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) return 0; } -bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog) -{ - return !!prog->aux->kfunc_tab; -} - static int add_subprog_and_kfunc(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; From f1c27922576edccb99d0257827d09bd05c0304a6 Mon Sep 17 00:00:00 2001 From: Tiezhu Yang Date: Wed, 8 Jul 2026 18:18:06 +0800 Subject: [PATCH 062/373] bpf: Reject programs with inlined helpers if JIT is not available When an architecture (such as LoongArch, ARM64, and RISC-V) implements bpf_jit_inlines_helper_call(), the verifier skips rewriting the helper call offset (insn->imm) in bpf_do_misc_fixups(). This is because the helper is expected to be inlined by the JIT compiler later. Therefore, insn->imm remains as the raw helper enum ID. However, if JIT is disabled at runtime (net.core.bpf_jit_enable=0) or if JIT compilation fails dynamically (e.g., due to OOM), the program falls back to the BPF interpreter. When the interpreter executes (__bpf_call_base + insn->imm) with the unpatched raw ID, it jumps into an invalid address space, triggering an instruction alignment fault or a kernel panic. Although these helpers have valid C implementations in the kernel, the omission of offset rewriting makes runtime interpreter fallback fatal. Fix this by setting 'prog->jit_required = 1' when helper call rewriting is skipped for JIT inlining. This ensures that such programs are safely rejected if JIT is not available, preventing the runtime kernel panic. Fixes: 2ddec2c80b44 ("riscv, bpf: inline bpf_get_smp_processor_id()") Suggested-by: Alexei Starovoitov Suggested-by: KaFai Wan Acked-by: Leon Hwang Signed-off-by: Tiezhu Yang Signed-off-by: Eduard Zingerman --- kernel/bpf/fixups.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 02246df2f6c3..d3be972714b2 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1840,8 +1840,10 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) } /* Skip inlining the helper call if the JIT does it. */ - if (bpf_jit_inlines_helper_call(insn->imm)) + if (bpf_jit_inlines_helper_call(insn->imm)) { + prog->jit_required = 1; goto next_insn; + } if (insn->imm == BPF_FUNC_get_route_realm) prog->dst_needed = 1; From 36ffa86c42f91c8a57071e024afc4ffb51a8958f Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 9 Jul 2026 09:34:22 +0200 Subject: [PATCH 063/373] bpf: Fix security_bpf_map_create error handling Commit 5816bf4273ed ("lsm,selinux: Add LSM blob support for BPF objects") made the LSM hook wrappers for BPF object creation clean up the LSM state internally upon denial, e.g. security_bpf_map_create() internally calls security_bpf_map_free() when the bpf_map_create hook returns an error. map_create() however still routes a denial to its free_map_sec label, which invokes security_bpf_map_free() a second time, so the bpf_map_free hook fires twice for a single denied map. In-tree LSMs are unaffected in practice since the blob kfree() inside security_bpf_map_free() is NULL-safe and idempotent and none of them implement bpf_map_free, but a BPF LSM program attached to that hook observes double invocations. Route the denial to free_map instead. Fixes: 5816bf4273ed ("lsm,selinux: Add LSM blob support for BPF objects") Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260709073422.379247-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/syscall.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 358f2b0ce2bd..0ff9e3aa293d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1649,7 +1649,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_at err = security_bpf_map_create(map, attr, token, uattr.is_kernel); if (err) - goto free_map_sec; + goto free_map; err = bpf_map_alloc_id(map); if (err) From 2cb5f4ca695ebe552647e5ba4aad6934d6a43bae Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 9 Jul 2026 17:31:30 +0200 Subject: [PATCH 064/373] bpf: Drop scalar id on sign-extending narrowing stack fills When a spilled scalar is filled back with a sign-extending narrowing load (BPF_MEMSX), check_stack_read_fixed_off() copies the spilled register including its scalar id, but coerce_reg_to_size_sx() then sign-extends the filled register's value. If the same slot is also filled with a plain zero-extending load (BPF_MEM), both destination registers share the id yet hold different values. A later 'if == const' then refines the sign-extended register through sync_linked_regs() to a value it does not have at runtime (e.g. the verifier believes 0x80000000 while the register is 0xffffffff80000000), which can be turned into an out-of-bounds access. Drop the shared scalar id at the sign-extension site in check_mem_access() when sign extension actually changes the value, mirroring the BPF_MOVSX handling in check_alu_op() (no_sext = reg_umax < 2^(size*8-1)). Fixes: 3cd5c890652b ("bpf: Let the verifier assign ids on stack fills") Reported-by: STAR Labs SG Signed-off-by: Daniel Borkmann Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 233472a871be..a0830ad6bebb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6394,11 +6394,23 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b if (!err && size < BPF_REG_SIZE && value_regno >= 0 && t == BPF_READ && regs[value_regno].type == SCALAR_VALUE) { - if (!is_ldsx) + if (!is_ldsx) { /* b/h/w load zero-extends, mark upper bits as known 0 */ coerce_reg_to_size(®s[value_regno], size); - else + } else { + /* + * Sign-extension can change the register value relative + * to a scalar it is linked with by id (e.g. a zero- + * extending fill of the same spilled stack slot), thus + * drop the shared id in that case. + */ + bool no_sext = reg_umax(®s[value_regno]) < + (1ULL << (size * BITS_PER_BYTE - 1)); + coerce_reg_to_size_sx(®s[value_regno], size); + if (!no_sext) + clear_scalar_id(®s[value_regno]); + } } return err; } From c3d5ef291a2a335d2e33fe75b3b3806fdbd86ad1 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 9 Jul 2026 17:31:31 +0200 Subject: [PATCH 065/373] selftests/bpf: Add test for scalar id on sign-extending stack fill Add a verifier test where a spilled scalar is filled once via a sign- extending load (BPF_MEMSX) and once via a zero-extending load (BPF_MEM). The two destination registers must not share a scalar id. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_scalar_ids [...] #643/1 verifier_scalar_ids/linked_regs_bpf_k:OK #643/2 verifier_scalar_ids/linked_regs_bpf_x_src:OK #643/3 verifier_scalar_ids/linked_regs_bpf_x_dst:OK #643/4 verifier_scalar_ids/linked_regs_broken_link:OK #643/5 verifier_scalar_ids/precision_many_frames:OK #643/6 verifier_scalar_ids/precision_stack:OK #643/7 verifier_scalar_ids/precision_two_ids:OK #643/8 verifier_scalar_ids/linked_regs_too_many_regs:OK #643/9 verifier_scalar_ids/linked_regs_broken_link_2:OK #643/10 verifier_scalar_ids/cjmp_no_linked_regs_trigger:OK #643/11 verifier_scalar_ids/check_ids_in_regsafe:OK #643/12 verifier_scalar_ids/check_ids_in_regsafe_2:OK #643/13 verifier_scalar_ids/no_scalar_id_for_const:OK #643/14 verifier_scalar_ids/no_scalar_id_for_const32:OK #643/15 verifier_scalar_ids/ignore_unique_scalar_ids_cur:OK #643/16 verifier_scalar_ids/ignore_unique_scalar_ids_old:OK #643/17 verifier_scalar_ids/two_nil_old_ids_one_cur_id:OK #643/18 verifier_scalar_ids/two_old_ids_one_cur_id:OK #643/19 verifier_scalar_ids/linked_regs_and_subreg_def:OK #643/20 verifier_scalar_ids/ldsx_fill_scalar_id_not_shared:OK #643 verifier_scalar_ids:OK Summary: 1/20 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Signed-off-by: Eduard Zingerman --- .../selftests/bpf/progs/verifier_scalar_ids.c | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c index e38f102da45f..663d15fc5fd2 100644 --- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c @@ -4,6 +4,13 @@ #include #include "bpf_misc.h" +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + /* Check that precision marks propagate through scalar IDs. * Registers r{0,1,2} have the same scalar ID. * Range information is propagated for scalars sharing same ID. @@ -915,4 +922,53 @@ __naked void linked_regs_and_subreg_def(void) : __clobber_all); } +/* + * A scalar is spilled to the stack and then filled twice: once via a + * sign-extending load (BPF_MEMSX) into r4 and once via a zero-extending + * load (BPF_MEM) into r5. coerce_reg_to_size_sx() gives r4 a different + * value than the spilled/zero-extended siblings, so r4 must not keep the + * shared scalar id. Otherwise the later 'if r5 == 0x80000000' refines r4 + * through sync_linked_regs() to a known 0x80000000, while at runtime r4 + * is the sign-extended 0xffffffff80000000. The test turns that discrepancy + * into an out-of-bounds map value access (r4 >> 63 is believed 0 but is 1 + * at runtime), which must be rejected. + */ +SEC("socket") +__failure __msg("R0 max value is outside of the allowed memory range") +__naked void ldsx_fill_scalar_id_not_shared(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + /* r7 = unknown u32, keep only bit 31 */ \ + r7 = *(u32*)(r0 + 0); \ + r2 = 0x80000000 ll; \ + r7 &= r2; \ + /* link r6 and r7 via a fresh scalar id */ \ + r6 = r7; \ + /* spill r7 (u32) to the stack */ \ + *(u32*)(r10 - 8) = r7; \ + /* sign-extending fill: must drop the id */ \ + r4 = *(s32*)(r10 - 8); \ + /* zero-extending fill: keeps the id */ \ + r5 = *(u32*)(r10 - 8); \ + /* r5 becomes known 0x80000000 on fall-through */\ + if r5 != r2 goto l0_%=; \ + /* verifier believes r4 == 0 here, runtime is 1 */\ + r4 >>= 63; \ + r0 += r4; \ + r0 = *(u8*)(r0 + 7); \ +l0_%=: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; From 2aaf67f0516fde29620d0edfc29c01b9ea7ad430 Mon Sep 17 00:00:00 2001 From: Nicholas Dudar Date: Thu, 9 Jul 2026 11:58:36 -0400 Subject: [PATCH 066/373] bpf: Reject rdonly/rdwr_buf_size kfunc arguments that exceed u32 max check_kfunc_args() detects a kfunc argument named rdonly_buf_size or rdwr_buf_size and stores reg->var_off.value into meta->r0_size, a u64, and does not bound it. check_kfunc_call() later copies that value into the returned register's mem_size field: meta->r0_size = reg->var_off.value; ... regs[BPF_REG_0].mem_size = meta.r0_size; regs[BPF_REG_0].mem_size is u32. A constant whose upper 32 bits are set gets truncated instead of causing a load-time rejection, so the verifier records a PTR_TO_MEM register with an approximately 4 GiB mem_size for whatever allocation the kfunc returned. A later access check against that register uses the truncated, wrong bound. Reject rdonly_buf_size/rdwr_buf_size values that exceed U32_MAX at the point meta->r0_size is set. Fixes: eb1f7f71c126 ("bpf/verifier: allow kfunc to return an allocated mem") Signed-off-by: Nicholas Dudar Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260709155837.1879230-2-main.kalliope@gmail.com Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a0830ad6bebb..03e2202cca13 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12109,6 +12109,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; + if (meta->r0_size > U32_MAX) { + verbose(env, "%s rdonly/rdwr_buf_size exceeds u32 max\n", + reg_arg_name(env, argno)); + return -EINVAL; + } if (regno >= 0) ret = mark_chain_precision(env, regno); else From 12556c3198328df38b3444978391141ad6f7092b Mon Sep 17 00:00:00 2001 From: Nicholas Dudar Date: Thu, 9 Jul 2026 11:58:37 -0400 Subject: [PATCH 067/373] selftests/bpf: Add test for oversized rdonly/rdwr_buf_size kfunc argument Add a load-failure test to the kfunc_call suite using the existing bpf_kfunc_call_test_get_rdwr_mem() test kfunc. Its rdwr_buf_size argument is a const int, so the test uses a 64-bit immediate load in inline asm to place 2^64 - 192 (0xffffffffffffff40) in the argument register. The verifier records r0_size from the full 64-bit register value, and the test asserts that BPF_PROG_LOAD rejects it with "rdonly/rdwr_buf_size exceeds u32 max". Signed-off-by: Nicholas Dudar Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260709155837.1879230-3-main.kalliope@gmail.com Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/kfunc_call.c | 1 + .../selftests/bpf/progs/kfunc_call_fail.c | 33 +++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c index 3df07680f9e0..67a30bf69509 100644 --- a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c +++ b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c @@ -66,6 +66,7 @@ static struct kfunc_test_params kfunc_tests[] = { TC_FAIL(kfunc_call_test_get_mem_fail_rdonly, 0, "R0 cannot write into rdonly_mem"), TC_FAIL(kfunc_call_test_get_mem_fail_use_after_free, 0, "invalid mem access 'scalar'"), TC_FAIL(kfunc_call_test_get_mem_fail_oob, 0, "min value is outside of the allowed memory range"), + TC_FAIL(kfunc_call_test_get_mem_fail_oversized, 0, "rdonly/rdwr_buf_size exceeds u32 max"), TC_FAIL(kfunc_call_test_get_mem_fail_not_const, 0, "is not a const"), TC_FAIL(kfunc_call_test_mem_acquire_fail, 0, "acquire kernel function does not return PTR_TO_BTF_ID"), TC_FAIL(kfunc_call_test_pointer_arg_type_mismatch, 0, "R1 expected pointer to ctx, but got scalar"), diff --git a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c index a1963497f0bf..6144ce3ff0b2 100644 --- a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c +++ b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c @@ -103,6 +103,39 @@ int kfunc_call_test_get_mem_fail_oob(struct __sk_buff *skb) return ret; } +SEC("?tc") +int kfunc_call_test_get_mem_fail_oversized(struct __sk_buff *skb) +{ + struct prog_test_ref_kfunc *pt; + unsigned long s = 0; + int *p = NULL; + int ret = 0; + + pt = bpf_kfunc_call_test_acquire(&s); + if (pt) { + /* + * rdwr_buf_size is a const int, so a C literal is narrowed to + * 32 bits before the call. Force the full 64-bit value 2^64 - 192 + * (0xffffffffffffff40, > U32_MAX) into the argument register with + * a 64-bit immediate load. The verifier records r0_size from the + * full register value and must reject it before that value is + * truncated into R0's u32 mem_size. + */ + asm volatile ( + "r1 = %[pt];" + "r2 = %[oversized] ll;" + "call %[get_rdwr_mem];" + "%[p] = r0;" + : [p] "=r"(p) + : [pt] "r"(pt), + [oversized] "i"(0xffffffffffffff40LL), + [get_rdwr_mem] "i"(bpf_kfunc_call_test_get_rdwr_mem) + : "r0", "r1", "r2", "r3", "r4", "r5"); + bpf_kfunc_call_test_release(pt); + } + return ret; +} + int not_const_size = 2 * sizeof(int); SEC("?tc") From f3603df9aebb2a2fe2f745bd71ca38aeca60e6e7 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 9 Jul 2026 14:48:56 +0000 Subject: [PATCH 068/373] bpf: Add bpf_icmp_send kfunc This is needed in the context of Tetragon to provide improved feedback (in contrast to just dropping packets) to east-west traffic when blocked by policies using cgroup_skb programs. This reuses concepts from netfilter reject target codepath with the differences that: * Packets are cloned since the BPF user can still let the packet pass (SK_PASS from the cgroup_skb progs for example) and the current skb need to stay untouched (cgroup_skb hooks only allow read-only skb payload). * We protect against recursion since the kfunc, by generating an ICMP error message, could retrigger the BPF prog that invoked it. Only ICMP_DEST_UNREACH and ICMPV6_DEST_UNREACH are currently supported. The interface accepts a type parameter to facilitate future extension to other ICMP control message types. For normal cgroup_skb paths, the skb dst route should already be set. However, bpf_prog_test_run_skb can create synthetic IPv4/IPv6 skbs without an attached route. In that case, icmp_send returns early, and the kfunc would otherwise report success despite no ICMP reply being sent. This check also reject metadata dsts, which are not valid struct rtable instances. While IPv6 would stricly require only rejecting metadata dsts, same check is applied for API consistency. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Reviewed-by: Jordan Rife Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260709144900.245904-2-mahe.tardy@gmail.com --- net/core/filter.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/net/core/filter.c b/net/core/filter.c index 4f5cbcac3e78..e4697036c67b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -84,6 +84,9 @@ #include #include #include +#include +#include +#include #include "dev.h" @@ -12548,6 +12551,83 @@ __bpf_kfunc int bpf_xdp_pull_data(struct xdp_md *x, u32 len) return 0; } +/** + * bpf_icmp_send - Send an ICMP control message + * @skb_ctx: Packet that triggered the control message + * @type: ICMP type (only ICMP_DEST_UNREACH/ICMPV6_DEST_UNREACH supported) + * @code: ICMP code (0-15 except ICMP_FRAG_NEEDED for IPv4, 0-6 for IPv6) + * + * Sends an ICMP control message in response to the packet. The original packet + * is cloned before sending the ICMP message, so the BPF program can still let + * the packet pass if desired. + * + * Currently only ICMP_DEST_UNREACH (IPv4) and ICMPV6_DEST_UNREACH (IPv6) are + * supported. + * + * Return: 0 on success (send attempt), negative error code on failure: + * -EBUSY: Recursion detected + * -EPROTONOSUPPORT: Non-IP protocol + * -EOPNOTSUPP: Unsupported ICMP type + * -EINVAL: Invalid code parameter + * -ENETUNREACH: No usable route/dst for the ICMP reply + * -ENOMEM: Memory allocation failed + */ +__bpf_kfunc int bpf_icmp_send(struct __sk_buff *skb_ctx, int type, int code) +{ + struct sk_buff *skb = (struct sk_buff *)skb_ctx; + struct sk_buff *nskb; + struct sock *sk; + + sk = skb_to_full_sk(skb); + if (sk && sk->sk_kern_sock && + (sk->sk_protocol == IPPROTO_ICMP || sk->sk_protocol == IPPROTO_ICMPV6)) + return -EBUSY; + + if (!skb_valid_dst(skb)) + return -ENETUNREACH; + + switch (skb->protocol) { +#if IS_ENABLED(CONFIG_INET) + case htons(ETH_P_IP): { + if (type != ICMP_DEST_UNREACH) + return -EOPNOTSUPP; + if (code < 0 || code > NR_ICMP_UNREACH || + code == ICMP_FRAG_NEEDED) /* needs a valid next-hop MTU */ + return -EINVAL; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + memset(IPCB(nskb), 0, sizeof(*IPCB(nskb))); + icmp_send(nskb, type, code, 0); + consume_skb(nskb); + break; + } +#endif +#if IS_ENABLED(CONFIG_IPV6) + case htons(ETH_P_IPV6): + if (type != ICMPV6_DEST_UNREACH) + return -EOPNOTSUPP; + if (code < 0 || code > ICMPV6_REJECT_ROUTE) + return -EINVAL; + + nskb = skb_clone(skb, GFP_ATOMIC); + if (!nskb) + return -ENOMEM; + + memset(IP6CB(nskb), 0, sizeof(*IP6CB(nskb))); + icmpv6_send(nskb, type, code, 0); + consume_skb(nskb); + break; +#endif + default: + return -EPROTONOSUPPORT; + } + + return 0; +} + __bpf_kfunc_end_defs(); int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags, @@ -12590,6 +12670,10 @@ BTF_KFUNCS_START(bpf_kfunc_check_set_sock_ops) BTF_ID_FLAGS(func, bpf_sock_ops_enable_tx_tstamp) BTF_KFUNCS_END(bpf_kfunc_check_set_sock_ops) +BTF_KFUNCS_START(bpf_kfunc_check_set_icmp_send) +BTF_ID_FLAGS(func, bpf_icmp_send) +BTF_KFUNCS_END(bpf_kfunc_check_set_icmp_send) + static const struct btf_kfunc_id_set bpf_kfunc_set_skb = { .owner = THIS_MODULE, .set = &bpf_kfunc_check_set_skb, @@ -12620,6 +12704,11 @@ static const struct btf_kfunc_id_set bpf_kfunc_set_sock_ops = { .set = &bpf_kfunc_check_set_sock_ops, }; +static const struct btf_kfunc_id_set bpf_kfunc_set_icmp_send = { + .owner = THIS_MODULE, + .set = &bpf_kfunc_check_set_icmp_send, +}; + static int __init bpf_kfunc_init(void) { int ret; @@ -12641,6 +12730,7 @@ static int __init bpf_kfunc_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SOCK_ADDR, &bpf_kfunc_set_sock_addr); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_kfunc_set_tcp_reqsk); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &bpf_kfunc_set_icmp_send); return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SOCK_OPS, &bpf_kfunc_set_sock_ops); } late_initcall(bpf_kfunc_init); From 39b337a3d995abba4aa58d402c968bad8c7ca4c0 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 9 Jul 2026 14:48:57 +0000 Subject: [PATCH 069/373] selftests/bpf: Add bpf_icmp_send kfunc cgroup_skb tests This test opens a server and client, enters a new cgroup, attach a cgroup_skb program on egress and calls the bpf_icmp_send function from the client egress so that an ICMP unreach control message is sent back to the client. It then fetches the message from the error queue to confirm the correct ICMP unreach code has been sent. Note that, for the client, we have to connect in non-blocking mode to let the test execute faster. Otherwise, we need to wait for the TCP three-way handshake to timeout in the kernel before reading the errno. Also note that we don't set IP_RECVERR on the socket in connect_to_fd_nonblock since the error will be transferred anyway in our test because the connection is rejected at the beginning of the TCP handshake. See in net/ipv4/tcp_ipv4.c:tcp_v4_err for more details. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Jordan Rife Reviewed-by: Emil Tsalapatis Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260709144900.245904-3-mahe.tardy@gmail.com --- .../bpf/prog_tests/icmp_send_kfunc.c | 164 ++++++++++++++++++ tools/testing/selftests/bpf/progs/icmp_send.c | 38 ++++ 2 files changed, 202 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/icmp_send.c diff --git a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c new file mode 100644 index 000000000000..b8a98c90053e --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include "icmp_send.skel.h" + +#define TIMEOUT_MS 1000 + +#define ICMP_DEST_UNREACH 3 + +#define ICMP_FRAG_NEEDED 4 +#define NR_ICMP_UNREACH 15 + +#define KFUNC_RET_UNSET -1 + +static int connect_to_fd_nonblock(int server_fd) +{ + struct sockaddr_storage addr; + socklen_t len = sizeof(addr); + int fd, err; + + if (getsockname(server_fd, (struct sockaddr *)&addr, &len)) + return -1; + + fd = socket(addr.ss_family, SOCK_STREAM | SOCK_NONBLOCK, 0); + if (fd < 0) + return -1; + + err = connect(fd, (struct sockaddr *)&addr, len); + if (err < 0 && errno != EINPROGRESS) { + close(fd); + return -1; + } + + return fd; +} + +static void read_icmp_errqueue(int sockfd, int expected_code) +{ + struct sock_extended_err *sock_err; + char ctrl_buf[512]; + struct msghdr msg = { + .msg_control = ctrl_buf, + .msg_controllen = sizeof(ctrl_buf), + }; + struct pollfd pfd = { + .fd = sockfd, + .events = POLLERR, + }; + struct cmsghdr *cm; + ssize_t n; + + if (!ASSERT_GE(poll(&pfd, 1, TIMEOUT_MS), 1, "poll_errqueue")) + return; + + n = recvmsg(sockfd, &msg, MSG_ERRQUEUE); + if (!ASSERT_GE(n, 0, "recvmsg_errqueue")) + return; + + cm = CMSG_FIRSTHDR(&msg); + if (!ASSERT_NEQ(cm, NULL, "cm_firsthdr_null")) + return; + + for (; cm; cm = CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level != IPPROTO_IP || cm->cmsg_type != IP_RECVERR) + continue; + + sock_err = (struct sock_extended_err *)CMSG_DATA(cm); + + if (!ASSERT_EQ(sock_err->ee_origin, SO_EE_ORIGIN_ICMP, + "sock_err_origin_icmp")) + return; + if (!ASSERT_EQ(sock_err->ee_type, ICMP_DEST_UNREACH, + "sock_err_type_dest_unreach")) + return; + ASSERT_EQ(sock_err->ee_code, expected_code, "sock_err_code"); + return; + } + + ASSERT_FAIL("no IP_RECVERR control message found"); +} + +static bool valid_unreach_code(int code) +{ + if (code < 0) + return false; + + return code <= NR_ICMP_UNREACH && code != ICMP_FRAG_NEEDED; +} + +static void trigger_prog_read_icmp_errqueue(struct icmp_send *skel, int code) +{ + int srv_fd = -1, client_fd = -1; + int port; + + srv_fd = start_server(AF_INET, SOCK_STREAM, "127.0.0.1", 0, TIMEOUT_MS); + if (!ASSERT_OK_FD(srv_fd, "start_server")) + return; + + port = get_socket_local_port(srv_fd); + if (!ASSERT_GE(port, 0, "get_socket_local_port")) { + close(srv_fd); + return; + } + + skel->bss->server_port = ntohs(port); + skel->bss->unreach_code = code; + skel->data->kfunc_ret = KFUNC_RET_UNSET; + + client_fd = connect_to_fd_nonblock(srv_fd); + if (!ASSERT_OK_FD(client_fd, "client_connect_nonblock")) { + close(srv_fd); + return; + } + + if (valid_unreach_code(code)) + read_icmp_errqueue(client_fd, code); + + close(client_fd); + close(srv_fd); +} + +void test_icmp_send_unreach_cgroup(void) +{ + struct icmp_send *skel; + int cgroup_fd = -1; + + skel = icmp_send__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open")) + goto cleanup; + + cgroup_fd = test__join_cgroup("/icmp_send_unreach_cgroup"); + if (!ASSERT_OK_FD(cgroup_fd, "join_cgroup")) + goto cleanup; + + skel->links.egress = + bpf_program__attach_cgroup(skel->progs.egress, cgroup_fd); + if (!ASSERT_OK_PTR(skel->links.egress, "prog_attach_cgroup")) + goto cleanup; + + for (int code = 0; code <= NR_ICMP_UNREACH; code++) { + if (code == ICMP_FRAG_NEEDED) + continue; + + trigger_prog_read_icmp_errqueue(skel, code); + ASSERT_EQ(skel->data->kfunc_ret, 0, "kfunc_ret"); + } + + /* Test invalid codes */ + trigger_prog_read_icmp_errqueue(skel, -1); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + + trigger_prog_read_icmp_errqueue(skel, NR_ICMP_UNREACH + 1); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + + trigger_prog_read_icmp_errqueue(skel, ICMP_FRAG_NEEDED); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + +cleanup: + icmp_send__destroy(skel); + if (cgroup_fd >= 0) + close(cgroup_fd); +} diff --git a/tools/testing/selftests/bpf/progs/icmp_send.c b/tools/testing/selftests/bpf/progs/icmp_send.c new file mode 100644 index 000000000000..6d0be0a9afe1 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/icmp_send.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "vmlinux.h" +#include +#include + +/* 127.0.0.1 in host byte order */ +#define SERVER_IP 0x7F000001 + +#define ICMP_DEST_UNREACH 3 + +__u16 server_port = 0; +int unreach_code = 0; +int kfunc_ret = -1; + +SEC("cgroup_skb/egress") +int egress(struct __sk_buff *skb) +{ + void *data = (void *)(long)skb->data; + void *data_end = (void *)(long)skb->data_end; + struct iphdr *iph; + struct tcphdr *tcph; + + iph = data; + if ((void *)(iph + 1) > data_end || iph->version != 4 || + iph->protocol != IPPROTO_TCP || iph->daddr != bpf_htonl(SERVER_IP)) + return SK_PASS; + + tcph = (void *)iph + iph->ihl * 4; + if ((void *)(tcph + 1) > data_end || + tcph->dest != bpf_htons(server_port)) + return SK_PASS; + + kfunc_ret = bpf_icmp_send(skb, ICMP_DEST_UNREACH, unreach_code); + + return SK_DROP; +} + +char LICENSE[] SEC("license") = "Dual BSD/GPL"; From 340a40df94e2566b83f5e8ad529c7fe8c7c96d80 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 9 Jul 2026 14:48:58 +0000 Subject: [PATCH 070/373] selftests/bpf: Add bpf_icmp_send kfunc cgroup_skb IPv6 tests This test extends the existing cgroup_skb tests with IPv6 support. Note that we need to set IPV6_RECVERR on the socket for IPv6 in connect_to_fd_nonblock otherwise the error will be ignored even if we are in the middle of the TCP handshake. See in net/ipv6/datagram.c:ipv6_icmp_error for more details. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Jordan Rife Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260709144900.245904-4-mahe.tardy@gmail.com --- .../bpf/prog_tests/icmp_send_kfunc.c | 91 +++++++++++++------ tools/testing/selftests/bpf/progs/icmp_send.c | 50 ++++++++-- 2 files changed, 102 insertions(+), 39 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c index b8a98c90053e..bbb3c3d4509c 100644 --- a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c @@ -8,9 +8,11 @@ #define TIMEOUT_MS 1000 #define ICMP_DEST_UNREACH 3 +#define ICMPV6_DEST_UNREACH 1 #define ICMP_FRAG_NEEDED 4 #define NR_ICMP_UNREACH 15 +#define ICMPV6_REJECT_ROUTE 6 #define KFUNC_RET_UNSET -1 @@ -18,7 +20,7 @@ static int connect_to_fd_nonblock(int server_fd) { struct sockaddr_storage addr; socklen_t len = sizeof(addr); - int fd, err; + int fd, err, on = 1; if (getsockname(server_fd, (struct sockaddr *)&addr, &len)) return -1; @@ -27,6 +29,12 @@ static int connect_to_fd_nonblock(int server_fd) if (fd < 0) return -1; + if (addr.ss_family == AF_INET6 && + setsockopt(fd, IPPROTO_IPV6, IPV6_RECVERR, &on, sizeof(on)) < 0) { + close(fd); + return -1; + } + err = connect(fd, (struct sockaddr *)&addr, len); if (err < 0 && errno != EINPROGRESS) { close(fd); @@ -36,8 +44,14 @@ static int connect_to_fd_nonblock(int server_fd) return fd; } -static void read_icmp_errqueue(int sockfd, int expected_code) +static void read_icmp_errqueue(int sockfd, int expected_code, int af) { + int expected_ee_type = (af == AF_INET) ? ICMP_DEST_UNREACH : + ICMPV6_DEST_UNREACH; + int expected_origin = (af == AF_INET) ? SO_EE_ORIGIN_ICMP : + SO_EE_ORIGIN_ICMP6; + int expected_level = (af == AF_INET) ? IPPROTO_IP : IPPROTO_IPV6; + int expected_type = (af == AF_INET) ? IP_RECVERR : IPV6_RECVERR; struct sock_extended_err *sock_err; char ctrl_buf[512]; struct msghdr msg = { @@ -63,38 +77,43 @@ static void read_icmp_errqueue(int sockfd, int expected_code) return; for (; cm; cm = CMSG_NXTHDR(&msg, cm)) { - if (cm->cmsg_level != IPPROTO_IP || cm->cmsg_type != IP_RECVERR) + if (cm->cmsg_level != expected_level || + cm->cmsg_type != expected_type) continue; sock_err = (struct sock_extended_err *)CMSG_DATA(cm); - if (!ASSERT_EQ(sock_err->ee_origin, SO_EE_ORIGIN_ICMP, - "sock_err_origin_icmp")) + if (!ASSERT_EQ(sock_err->ee_origin, expected_origin, + "sock_err_origin")) return; - if (!ASSERT_EQ(sock_err->ee_type, ICMP_DEST_UNREACH, + if (!ASSERT_EQ(sock_err->ee_type, expected_ee_type, "sock_err_type_dest_unreach")) return; ASSERT_EQ(sock_err->ee_code, expected_code, "sock_err_code"); return; } - ASSERT_FAIL("no IP_RECVERR control message found"); + ASSERT_FAIL("no IP_RECVERR/IPV6_RECVERR control message found"); } -static bool valid_unreach_code(int code) +static bool valid_unreach_code(int code, int af) { if (code < 0) return false; - return code <= NR_ICMP_UNREACH && code != ICMP_FRAG_NEEDED; + if (af == AF_INET) + return code <= NR_ICMP_UNREACH && code != ICMP_FRAG_NEEDED; + + return code <= ICMPV6_REJECT_ROUTE; } -static void trigger_prog_read_icmp_errqueue(struct icmp_send *skel, int code) +static void trigger_prog_read_icmp_errqueue(struct icmp_send *skel, int code, + int af, const char *ip) { int srv_fd = -1, client_fd = -1; int port; - srv_fd = start_server(AF_INET, SOCK_STREAM, "127.0.0.1", 0, TIMEOUT_MS); + srv_fd = start_server(af, SOCK_STREAM, ip, 0, TIMEOUT_MS); if (!ASSERT_OK_FD(srv_fd, "start_server")) return; @@ -105,6 +124,8 @@ static void trigger_prog_read_icmp_errqueue(struct icmp_send *skel, int code) } skel->bss->server_port = ntohs(port); + skel->bss->unreach_type = (af == AF_INET) ? ICMP_DEST_UNREACH : + ICMPV6_DEST_UNREACH; skel->bss->unreach_code = code; skel->data->kfunc_ret = KFUNC_RET_UNSET; @@ -114,13 +135,37 @@ static void trigger_prog_read_icmp_errqueue(struct icmp_send *skel, int code) return; } - if (valid_unreach_code(code)) - read_icmp_errqueue(client_fd, code); + if (valid_unreach_code(code, af)) + read_icmp_errqueue(client_fd, code, af); close(client_fd); close(srv_fd); } +static void run_icmp_test(struct icmp_send *skel, int af, const char *ip, + int max_code) +{ + for (int code = 0; code <= max_code; code++) { + if (af == AF_INET && code == ICMP_FRAG_NEEDED) + continue; + + trigger_prog_read_icmp_errqueue(skel, code, af, ip); + ASSERT_EQ(skel->data->kfunc_ret, 0, "kfunc_ret"); + } + + /* Test invalid codes */ + trigger_prog_read_icmp_errqueue(skel, -1, af, ip); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + + trigger_prog_read_icmp_errqueue(skel, max_code + 1, af, ip); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + + if (af == AF_INET) { + trigger_prog_read_icmp_errqueue(skel, ICMP_FRAG_NEEDED, af, ip); + ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + } +} + void test_icmp_send_unreach_cgroup(void) { struct icmp_send *skel; @@ -139,23 +184,11 @@ void test_icmp_send_unreach_cgroup(void) if (!ASSERT_OK_PTR(skel->links.egress, "prog_attach_cgroup")) goto cleanup; - for (int code = 0; code <= NR_ICMP_UNREACH; code++) { - if (code == ICMP_FRAG_NEEDED) - continue; + if (test__start_subtest("ipv4")) + run_icmp_test(skel, AF_INET, "127.0.0.1", NR_ICMP_UNREACH); - trigger_prog_read_icmp_errqueue(skel, code); - ASSERT_EQ(skel->data->kfunc_ret, 0, "kfunc_ret"); - } - - /* Test invalid codes */ - trigger_prog_read_icmp_errqueue(skel, -1); - ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); - - trigger_prog_read_icmp_errqueue(skel, NR_ICMP_UNREACH + 1); - ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); - - trigger_prog_read_icmp_errqueue(skel, ICMP_FRAG_NEEDED); - ASSERT_EQ(skel->data->kfunc_ret, -EINVAL, "kfunc_ret"); + if (test__start_subtest("ipv6")) + run_icmp_test(skel, AF_INET6, "::1", ICMPV6_REJECT_ROUTE); cleanup: icmp_send__destroy(skel); diff --git a/tools/testing/selftests/bpf/progs/icmp_send.c b/tools/testing/selftests/bpf/progs/icmp_send.c index 6d0be0a9afe1..6e1ba539eeb0 100644 --- a/tools/testing/selftests/bpf/progs/icmp_send.c +++ b/tools/testing/selftests/bpf/progs/icmp_send.c @@ -5,10 +5,11 @@ /* 127.0.0.1 in host byte order */ #define SERVER_IP 0x7F000001 - -#define ICMP_DEST_UNREACH 3 +/* ::1 in host byte order (last 32-bit word) */ +#define SERVER_IP6_LO 0x00000001 __u16 server_port = 0; +int unreach_type = 0; int unreach_code = 0; int kfunc_ret = -1; @@ -18,19 +19,48 @@ int egress(struct __sk_buff *skb) void *data = (void *)(long)skb->data; void *data_end = (void *)(long)skb->data_end; struct iphdr *iph; + struct ipv6hdr *ip6h; struct tcphdr *tcph; + __u8 version; - iph = data; - if ((void *)(iph + 1) > data_end || iph->version != 4 || - iph->protocol != IPPROTO_TCP || iph->daddr != bpf_htonl(SERVER_IP)) + if (data + 1 > data_end) return SK_PASS; - tcph = (void *)iph + iph->ihl * 4; - if ((void *)(tcph + 1) > data_end || - tcph->dest != bpf_htons(server_port)) - return SK_PASS; + version = (*((__u8 *)data)) >> 4; - kfunc_ret = bpf_icmp_send(skb, ICMP_DEST_UNREACH, unreach_code); + if (version == 4) { + iph = data; + if ((void *)(iph + 1) > data_end || + iph->protocol != IPPROTO_TCP || + iph->daddr != bpf_htonl(SERVER_IP)) + return SK_PASS; + + tcph = (void *)iph + iph->ihl * 4; + if ((void *)(tcph + 1) > data_end || + tcph->dest != bpf_htons(server_port)) + return SK_PASS; + + } else if (version == 6) { + ip6h = data; + if ((void *)(ip6h + 1) > data_end || + ip6h->nexthdr != IPPROTO_TCP) + return SK_PASS; + + if (ip6h->daddr.in6_u.u6_addr32[0] != 0 || + ip6h->daddr.in6_u.u6_addr32[1] != 0 || + ip6h->daddr.in6_u.u6_addr32[2] != 0 || + ip6h->daddr.in6_u.u6_addr32[3] != bpf_htonl(SERVER_IP6_LO)) + return SK_PASS; + + tcph = (void *)(ip6h + 1); + if ((void *)(tcph + 1) > data_end || + tcph->dest != bpf_htons(server_port)) + return SK_PASS; + } else { + return SK_PASS; + } + + kfunc_ret = bpf_icmp_send(skb, unreach_type, unreach_code); return SK_DROP; } From 49d07ba673bbbbde876f6854282961ccf67acd93 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 9 Jul 2026 14:48:59 +0000 Subject: [PATCH 071/373] selftests/bpf: Add bpf_icmp_send recursion test This test is similar to test_icmp_send_unreach_cgroup but checks that, in case of recursion, meaning that the BPF program calling the kfunc was re-triggered by the icmp_send done by the kfunc, the kfunc will stop early and return -EBUSY. The test attaches to the root cgroup to ensure the ICMP packet generated by the kfunc re-triggers the BPF program. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Reviewed-by: Jordan Rife Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260709144900.245904-5-mahe.tardy@gmail.com --- .../bpf/prog_tests/icmp_send_kfunc.c | 46 ++++++++++++++++ tools/testing/selftests/bpf/progs/icmp_send.c | 55 +++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c index bbb3c3d4509c..bb532aa0d158 100644 --- a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c @@ -1,8 +1,10 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include #include +#include #include "icmp_send.skel.h" #define TIMEOUT_MS 1000 @@ -10,6 +12,7 @@ #define ICMP_DEST_UNREACH 3 #define ICMPV6_DEST_UNREACH 1 +#define ICMP_HOST_UNREACH 1 #define ICMP_FRAG_NEEDED 4 #define NR_ICMP_UNREACH 15 #define ICMPV6_REJECT_ROUTE 6 @@ -195,3 +198,46 @@ void test_icmp_send_unreach_cgroup(void) if (cgroup_fd >= 0) close(cgroup_fd); } + +void test_icmp_send_unreach_recursion(void) +{ + struct icmp_send *skel; + int cgroup_fd = -1; + int err; + + err = setup_cgroup_environment(); + if (!ASSERT_OK(err, "setup_cgroup_environment")) + return; + + skel = icmp_send__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open")) + goto cleanup; + + cgroup_fd = get_root_cgroup(); + if (!ASSERT_OK_FD(cgroup_fd, "get_root_cgroup")) + goto cleanup; + + skel->data->target_pid = getpid(); + skel->links.recursion = + bpf_program__attach_cgroup(skel->progs.recursion, cgroup_fd); + if (!ASSERT_OK_PTR(skel->links.recursion, "prog_attach_cgroup")) + goto cleanup; + + trigger_prog_read_icmp_errqueue(skel, ICMP_HOST_UNREACH, AF_INET, + "127.0.0.1"); + + /* + * Because there's recursion involved, the first call will return at + * index 1 since it will return the second, and the second call will + * return at index 0 since it will return the first. + */ + ASSERT_EQ(skel->bss->rec_count, 2, "rec_count"); + ASSERT_EQ(skel->data->rec_kfunc_rets[0], -EBUSY, "kfunc_rets[0]"); + ASSERT_EQ(skel->data->rec_kfunc_rets[1], 0, "kfunc_rets[1]"); + +cleanup: + icmp_send__destroy(skel); + if (cgroup_fd >= 0) + close(cgroup_fd); + cleanup_cgroup_environment(); +} diff --git a/tools/testing/selftests/bpf/progs/icmp_send.c b/tools/testing/selftests/bpf/progs/icmp_send.c index 6e1ba539eeb0..c642ccdf9fd5 100644 --- a/tools/testing/selftests/bpf/progs/icmp_send.c +++ b/tools/testing/selftests/bpf/progs/icmp_send.c @@ -12,6 +12,10 @@ __u16 server_port = 0; int unreach_type = 0; int unreach_code = 0; int kfunc_ret = -1; +int target_pid = -1; + +unsigned int rec_count = 0; +int rec_kfunc_rets[] = { -1, -1 }; SEC("cgroup_skb/egress") int egress(struct __sk_buff *skb) @@ -65,4 +69,55 @@ int egress(struct __sk_buff *skb) return SK_DROP; } +SEC("cgroup_skb/egress") +int recursion(struct __sk_buff *skb) +{ + void *data = (void *)(long)skb->data; + void *data_end = (void *)(long)skb->data_end; + struct icmphdr *icmph; + struct tcphdr *tcph; + struct iphdr *iph; + int ret; + + if ((bpf_get_current_pid_tgid() >> 32) != target_pid) + return SK_PASS; + + iph = data; + if ((void *)(iph + 1) > data_end || iph->version != 4) + return SK_PASS; + + if (iph->daddr != bpf_htonl(SERVER_IP)) + return SK_PASS; + + if (iph->protocol == IPPROTO_TCP) { + tcph = (void *)iph + iph->ihl * 4; + if ((void *)(tcph + 1) > data_end || + tcph->dest != bpf_htons(server_port)) + return SK_PASS; + } else if (iph->protocol == IPPROTO_ICMP) { + icmph = (void *)iph + iph->ihl * 4; + if ((void *)(icmph + 1) > data_end || + icmph->type != unreach_type || icmph->code != unreach_code) + return SK_PASS; + } else { + return SK_PASS; + } + + /* + * This call will provoke a recursion: the ICMP packet generated by the + * kfunc will re-trigger this program since we are in the root cgroup in + * which the kernel ICMP socket belongs. However when re-entering the + * kfunc, it should return EBUSY. + */ + ret = bpf_icmp_send(skb, unreach_type, unreach_code); + rec_kfunc_rets[rec_count & 1] = ret; + __sync_fetch_and_add(&rec_count, 1); + + /* Let the first ICMP error message pass */ + if (iph->protocol == IPPROTO_ICMP) + return SK_PASS; + + return SK_DROP; +} + char LICENSE[] SEC("license") = "Dual BSD/GPL"; From b1d4514ff1f96ee83192e17d3db766d2e4ec77f7 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 9 Jul 2026 14:49:00 +0000 Subject: [PATCH 072/373] selftests/bpf: Add bpf_icmp_send no route test For normal live cgroup_skb paths, the skb should already be routed. The exception is for test run via BPF_PROG_TEST_RUN with packets created via bpf_prog_test_run_skb. Those lack dst route and thus the icmp_send would quietly fail by returning early. This test exercises this and makes sure the kfunc returns -ENETUNREACH. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Reviewed-by: Jordan Rife Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260709144900.245904-6-mahe.tardy@gmail.com --- .../bpf/prog_tests/icmp_send_kfunc.c | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c index bb532aa0d158..9318d4bc7ce8 100644 --- a/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/icmp_send_kfunc.c @@ -169,6 +169,49 @@ static void run_icmp_test(struct icmp_send *skel, int af, const char *ip, } } +static void run_icmp_no_route_test(struct icmp_send *skel, int af) +{ + union { + struct ipv4_packet v4; + struct ipv6_packet v6; + } pkt; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts, + .data_in = &pkt, + ); + int err; + + switch (af) { + case AF_INET: + pkt.v4 = pkt_v4; + pkt.v4.iph.version = 4; + pkt.v4.iph.daddr = htonl(INADDR_LOOPBACK); + pkt.v4.tcp.dest = htons(80); + opts.data_size_in = sizeof(pkt.v4); + skel->bss->unreach_type = ICMP_DEST_UNREACH; + break; + case AF_INET6: + pkt.v6 = pkt_v6; + pkt.v6.iph.version = 6; + pkt.v6.iph.daddr = in6addr_loopback; + pkt.v6.tcp.dest = htons(80); + opts.data_size_in = sizeof(pkt.v6); + skel->bss->unreach_type = ICMPV6_DEST_UNREACH; + break; + default: + ASSERT_FAIL("af_not_supported"); + return; + } + + skel->bss->server_port = 80; + skel->data->kfunc_ret = KFUNC_RET_UNSET; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.egress), &opts); + if (!ASSERT_OK(err, "test_run")) + return; + + ASSERT_EQ(skel->data->kfunc_ret, -ENETUNREACH, "kfunc_ret_no_route"); +} + void test_icmp_send_unreach_cgroup(void) { struct icmp_send *skel; @@ -193,6 +236,12 @@ void test_icmp_send_unreach_cgroup(void) if (test__start_subtest("ipv6")) run_icmp_test(skel, AF_INET6, "::1", ICMPV6_REJECT_ROUTE); + if (test__start_subtest("no_route_ipv4")) + run_icmp_no_route_test(skel, AF_INET); + + if (test__start_subtest("no_route_ipv6")) + run_icmp_no_route_test(skel, AF_INET6); + cleanup: icmp_send__destroy(skel); if (cgroup_fd >= 0) From 30f77a0419382ce061b2418de81526e93be4ecf9 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Fri, 10 Jul 2026 18:09:15 +0000 Subject: [PATCH 073/373] bpf: Fix unused nskb warning in bpf_icmp_send Declare nskb inside the IPv4 and IPv6 case blocks so it is only present when the corresponding code is built. The case braces are intentional to scope the local declarations under the switch labels. Fixes: f3603df9aebb ("bpf: Add bpf_icmp_send kfunc") Reported-by: kernel test robot Signed-off-by: Mahe Tardy Link: https://lore.kernel.org/bpf/20260710180915.7105-1-mahe.tardy@gmail.com Closes: https://lore.kernel.org/oe-kbuild-all/202607110140.JeJZ6GIa-lkp@intel.com/ Signed-off-by: Kumar Kartikeya Dwivedi --- net/core/filter.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/net/core/filter.c b/net/core/filter.c index e4697036c67b..056deb9b3fc3 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -12575,7 +12575,6 @@ __bpf_kfunc int bpf_xdp_pull_data(struct xdp_md *x, u32 len) __bpf_kfunc int bpf_icmp_send(struct __sk_buff *skb_ctx, int type, int code) { struct sk_buff *skb = (struct sk_buff *)skb_ctx; - struct sk_buff *nskb; struct sock *sk; sk = skb_to_full_sk(skb); @@ -12589,6 +12588,8 @@ __bpf_kfunc int bpf_icmp_send(struct __sk_buff *skb_ctx, int type, int code) switch (skb->protocol) { #if IS_ENABLED(CONFIG_INET) case htons(ETH_P_IP): { + struct sk_buff *nskb; + if (type != ICMP_DEST_UNREACH) return -EOPNOTSUPP; if (code < 0 || code > NR_ICMP_UNREACH || @@ -12606,7 +12607,9 @@ __bpf_kfunc int bpf_icmp_send(struct __sk_buff *skb_ctx, int type, int code) } #endif #if IS_ENABLED(CONFIG_IPV6) - case htons(ETH_P_IPV6): + case htons(ETH_P_IPV6): { + struct sk_buff *nskb; + if (type != ICMPV6_DEST_UNREACH) return -EOPNOTSUPP; if (code < 0 || code > ICMPV6_REJECT_ROUTE) @@ -12620,6 +12623,7 @@ __bpf_kfunc int bpf_icmp_send(struct __sk_buff *skb_ctx, int type, int code) icmpv6_send(nskb, type, code, 0); consume_skb(nskb); break; + } #endif default: return -EPROTONOSUPPORT; From e821c223875803760d492f88e357380415f5f438 Mon Sep 17 00:00:00 2001 From: Emil Tsalapatis Date: Wed, 8 Jul 2026 15:32:39 -0400 Subject: [PATCH 074/373] selftests/bpf: veristat: Minimize map size during verification The veristat tool verifies that BPF objects along with their maps pass verification for a given kernel version. To do so, veristat loads the objects and maps into the kernel in order to pass them through the verifier. Currently, veristat sizes the maps according to the max_entries field provided by the program author. Depending on the map type this field may be irrelevant to the verification process. However, loading a large map can fail because of -ENOMEM errors. This is a problem when the map is supposed to run on large machines, but veristat tests it machines with significantly less RAM (e.g., CI). In that case veristat fails even if the program verifies. Expand veristat to resize maps whose max_entries are not relevant to verification. Set the max_entries value as low as possible to avoid -ENOMEM errors. Suggested-by: Andrii Nakryiko Signed-off-by: Emil Tsalapatis Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260708193239.5063-1-emil@etsalapatis.com --- tools/testing/selftests/bpf/veristat.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c index a7db6f04f7e1..c9c257784ee3 100644 --- a/tools/testing/selftests/bpf/veristat.c +++ b/tools/testing/selftests/bpf/veristat.c @@ -1248,6 +1248,29 @@ static void fixup_obj_maps(struct bpf_object *obj) /* fix up map size, if necessary */ switch (bpf_map__type(map)) { + /* + * if the verifier doesn't use max_entries + * then set to 1 to avoid -ENOMEM + */ + case BPF_MAP_TYPE_HASH: + case BPF_MAP_TYPE_PERCPU_HASH: + case BPF_MAP_TYPE_LRU_HASH: + case BPF_MAP_TYPE_LRU_PERCPU_HASH: + case BPF_MAP_TYPE_SOCKHASH: + case BPF_MAP_TYPE_DEVMAP_HASH: + case BPF_MAP_TYPE_QUEUE: + case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_BLOOM_FILTER: + case BPF_MAP_TYPE_STACK_TRACE: + bpf_map__set_max_entries(map, 1); + break; + + /* ringbufs must be page-aligned */ + case BPF_MAP_TYPE_RINGBUF: + case BPF_MAP_TYPE_USER_RINGBUF: + bpf_map__set_max_entries(map, sysconf(_SC_PAGESIZE)); + break; + case BPF_MAP_TYPE_SK_STORAGE: case BPF_MAP_TYPE_TASK_STORAGE: case BPF_MAP_TYPE_INODE_STORAGE: From 77f02c9926e1d58f418a705ee4ecc6975721e117 Mon Sep 17 00:00:00 2001 From: Naveed Khan Date: Wed, 8 Jul 2026 01:48:11 +0530 Subject: [PATCH 075/373] libbpf: Fix double-free of distilled base BTF on .BTF.ext parse error When btf_parse_elf() is called without a caller-supplied base_btf (i.e. via the public btf__parse_elf()) and the object file carries a .BTF.base (distilled base) section, a dist_base_btf object is created and used as the base of the split BTF built from the .BTF section. Because base_btf is NULL, the relocation block that would otherwise free and clear dist_base_btf is skipped, and ownership of dist_base_btf is instead transferred to the split btf by setting btf->owns_base = true. That ownership transfer was performed before the fallible btf_ext__new() call that parses the .BTF.ext section. If .BTF.ext is malformed, btf_ext__new() fails and the function jumps to the error path, which frees dist_base_btf directly and then frees btf. Since owns_base is already set, btf__free(btf) also frees btf->base_btf, which is the same dist_base_btf object. The result is a use-after-free read followed by a double free of the base BTF, driven entirely by a crafted object file (a .BTF + .BTF.base + malformed .BTF.ext combination) passed to btf__parse_elf(), as used by bpftool, pahole and similar tools. Transfer ownership only after .BTF.ext has been parsed successfully, so that any earlier failure leaves dist_base_btf owned solely by the local cleanup path and it is freed exactly once. Signed-off-by: Naveed Khan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/178345549172.94179.7948304165383170781@digiscrypt.com --- tools/lib/bpf/btf.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index bf6a68118405..8417de92d028 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -1506,9 +1506,6 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf, dist_base_btf = NULL; } - if (dist_base_btf) - btf->owns_base = true; - switch (gelf_getclass(elf)) { case ELFCLASS32: btf__set_pointer_size(btf, 4); @@ -1523,13 +1520,16 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf, if (btf_ext && secs.btf_ext_data) { *btf_ext = btf_ext__new(secs.btf_ext_data->d_buf, secs.btf_ext_data->d_size); - if (IS_ERR(*btf_ext)) { - err = PTR_ERR(*btf_ext); + if (!*btf_ext) { + err = -errno; goto done; } } else if (btf_ext) { *btf_ext = NULL; } + + if (dist_base_btf) + btf->owns_base = true; done: if (elf) elf_end(elf); From 8740156ad33be5071b588b594c55f279457f667c Mon Sep 17 00:00:00 2001 From: Nicholas Dudar Date: Thu, 9 Jul 2026 14:27:59 -0400 Subject: [PATCH 076/373] bpf: Require a BPF cpumask for bpf_cpumask_populate() bpf_cpumask_populate() writes to its destination with bitmap_copy(), but the destination is typed as struct cpumask *. That allows the verifier to accept borrowed cpumask pointers returned by read-only kfuncs, such as scx_bpf_get_online_cpumask(), as a writable destination. Make the destination a struct bpf_cpumask * so populate follows the same ownership rule as the other mutating cpumask kfuncs. Query kfuncs continue to accept const struct cpumask * inputs. Fixes: 950ad93df2fc ("bpf: add kfunc for populating cpumask bits") Signed-off-by: Nicholas Dudar Acked-by: Tejun Heo Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260709182800.2037938-2-main.kalliope@gmail.com Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/cpumask.c | 6 +++--- tools/sched_ext/include/scx/compat.bpf.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/cpumask.c b/kernel/bpf/cpumask.c index b8c805b4b06a..1336a4efa755 100644 --- a/kernel/bpf/cpumask.c +++ b/kernel/bpf/cpumask.c @@ -449,12 +449,12 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) * @src__sz: Length of the BPF memory region in bytes. * * Return: - * * 0 if the struct cpumask * instance was populated successfully. + * * 0 if the struct bpf_cpumask * instance was populated successfully. * * -EACCES if the memory region is too small to populate the cpumask. * * -EINVAL if the memory region is not aligned to the size of a long * and the architecture does not support efficient unaligned accesses. */ -__bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t src__sz) +__bpf_kfunc int bpf_cpumask_populate(struct bpf_cpumask *cpumask, void *src, size_t src__sz) { unsigned long source = (unsigned long)src; @@ -467,7 +467,7 @@ __bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t !IS_ALIGNED(source, sizeof(long))) return -EINVAL; - bitmap_copy(cpumask_bits(cpumask), src, nr_cpu_ids); + bitmap_copy(cpumask_bits(&cpumask->cpumask), src, nr_cpu_ids); return 0; } diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index 87f15f296234..3f74d522f7e7 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -84,7 +84,7 @@ bool scx_bpf_dispatch_vtime_from_dsq___old(struct bpf_iter_scx_dsq *it__iter, st * * Compat macro will be dropped on v6.19 release. */ -int bpf_cpumask_populate(struct cpumask *dst, void *src, size_t src__sz) __ksym __weak; +int bpf_cpumask_populate(struct bpf_cpumask *dst, void *src, size_t src__sz) __ksym __weak; #define __COMPAT_bpf_cpumask_populate(cpumask, src, size__sz) \ (bpf_ksym_exists(bpf_cpumask_populate) ? \ From 6267b835286eb552298f22d9e4045b55c3272985 Mon Sep 17 00:00:00 2001 From: Nicholas Dudar Date: Thu, 9 Jul 2026 14:28:00 -0400 Subject: [PATCH 077/373] selftests/bpf: Test bpf_cpumask_populate() rejects a borrowed cpumask bpf_cpumask_populate() now takes a struct bpf_cpumask *, so update the kfunc declaration and drop the struct cpumask * casts in the existing populate tests. Add test_populate_borrowed_destination, which passes a borrowed task->cpus_ptr and asserts the verifier rejects it as a writable destination. Signed-off-by: Nicholas Dudar Acked-by: Tejun Heo Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260709182800.2037938-3-main.kalliope@gmail.com Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/cpumask_common.h | 2 +- .../selftests/bpf/progs/cpumask_failure.c | 23 +++++++++++++++++-- .../selftests/bpf/progs/cpumask_success.c | 6 ++--- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/cpumask_common.h b/tools/testing/selftests/bpf/progs/cpumask_common.h index 86085b79f5ca..8fe01308d210 100644 --- a/tools/testing/selftests/bpf/progs/cpumask_common.h +++ b/tools/testing/selftests/bpf/progs/cpumask_common.h @@ -61,7 +61,7 @@ u32 bpf_cpumask_any_distribute(const struct cpumask *src) __ksym __weak; u32 bpf_cpumask_any_and_distribute(const struct cpumask *src1, const struct cpumask *src2) __ksym __weak; u32 bpf_cpumask_weight(const struct cpumask *cpumask) __ksym __weak; -int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t src__sz) __ksym __weak; +int bpf_cpumask_populate(struct bpf_cpumask *cpumask, void *src, size_t src__sz) __ksym __weak; void bpf_rcu_read_lock(void) __ksym __weak; void bpf_rcu_read_unlock(void) __ksym __weak; diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c index 4c45346fe6f7..74b4cd4bcdbb 100644 --- a/tools/testing/selftests/bpf/progs/cpumask_failure.c +++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c @@ -231,7 +231,7 @@ int BPF_PROG(test_populate_invalid_destination, struct task_struct *task, u64 cl u64 bits; int ret; - ret = bpf_cpumask_populate((struct cpumask *)invalid, &bits, sizeof(bits)); + ret = bpf_cpumask_populate(invalid, &bits, sizeof(bits)); if (!ret) err = 2; @@ -252,7 +252,7 @@ int BPF_PROG(test_populate_invalid_source, struct task_struct *task, u64 clone_f return 0; } - ret = bpf_cpumask_populate((struct cpumask *)local, garbage, 8); + ret = bpf_cpumask_populate(local, garbage, 8); if (!ret) err = 2; @@ -260,3 +260,22 @@ int BPF_PROG(test_populate_invalid_source, struct task_struct *task, u64 clone_f return 0; } + +SEC("tp_btf/task_newtask") +__failure __msg("expected pointer to STRUCT bpf_cpumask but R1 has a pointer to STRUCT cpumask") +int BPF_PROG(test_populate_borrowed_destination, struct task_struct *task, u64 clone_flags) +{ + u64 bits; + int ret; + + /* + * task->cpus_ptr is a borrowed, read-only struct cpumask *, not an + * owned struct bpf_cpumask *. The verifier must reject it as a + * writable destination for bpf_cpumask_populate(). + */ + ret = bpf_cpumask_populate((struct bpf_cpumask *)task->cpus_ptr, &bits, sizeof(bits)); + if (!ret) + err = 2; + + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/cpumask_success.c b/tools/testing/selftests/bpf/progs/cpumask_success.c index 774706e7b058..36f77b9732d4 100644 --- a/tools/testing/selftests/bpf/progs/cpumask_success.c +++ b/tools/testing/selftests/bpf/progs/cpumask_success.c @@ -785,7 +785,7 @@ int BPF_PROG(test_populate_reject_small_mask, struct task_struct *task, u64 clon return 0; /* The kfunc should prevent this operation */ - ret = bpf_cpumask_populate((struct cpumask *)local, &toofewbits, sizeof(toofewbits)); + ret = bpf_cpumask_populate(local, &toofewbits, sizeof(toofewbits)); if (ret != -EACCES) err = 2; @@ -824,7 +824,7 @@ int BPF_PROG(test_populate_reject_unaligned, struct task_struct *task, u64 clone /* Misalign the source array by a byte. */ src = &((char *)bits)[1]; - ret = bpf_cpumask_populate((struct cpumask *)mask, src, CPUMASK_TEST_MASKLEN); + ret = bpf_cpumask_populate(mask, src, CPUMASK_TEST_MASKLEN); if (ret != -EINVAL) err = 2; @@ -855,7 +855,7 @@ int BPF_PROG(test_populate, struct task_struct *task, u64 clone_flags) } /* Pass the entire bits array, the kfunc will only copy the valid bits. */ - ret = bpf_cpumask_populate((struct cpumask *)mask, bits, CPUMASK_TEST_MASKLEN); + ret = bpf_cpumask_populate(mask, bits, CPUMASK_TEST_MASKLEN); if (ret) { err = 2; goto out; From 30bdd6d1384d894931f113eb595636092d8e650c Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Sat, 11 Jul 2026 20:48:21 +0800 Subject: [PATCH 078/373] bpf: Mark tracing_multi trampolines as ftrace managed Since tracing_multi link does not set ftrace_managed, it would fail to release the tracing_multi link when attaching tracing_multi link and then attaching fentry link. [ 3.714215] WARNING: kernel/bpf/trampoline.c:1727 at bpf_trampoline_multi_detach+0x20b/0x240, CPU#1: test_progs/97 ... [ 3.733170] bpf_tracing_multi_link_release+0x14/0x30 [ 3.733890] bpf_link_free+0x58/0x130 [ 3.734414] bpf_link_release+0x23/0x30 Fix it by setting 'ftrace_managed = true' in register_fentry_multi(). Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions") Signed-off-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260711124822.29406-2-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/trampoline.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 1a721fc4bef5..6eadf64f7ec9 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -1536,6 +1536,7 @@ static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_ima if (bpf_trampoline_use_jmp(tr->flags)) addr = ftrace_jmp_set(addr); + tr->func.ftrace_managed = true; ftrace_hash_add(data->reg, data->entry, ip, addr); tr->cur_image = im; return 0; From 539d3edf8e53f10b2d63e2d866e23c414c99e921 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Sat, 11 Jul 2026 20:48:22 +0800 Subject: [PATCH 079/373] selftests/bpf: Test fentry link after tracing_multi link Verify that there's no any issue to attach tracing_multi link, then attach fentry link. Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260711124822.29406-3-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/tracing_multi.c | 69 +++++++++++++++++++ .../progs/tracing_multi_intersect_attach.c | 8 +++ 2 files changed, 77 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c index f02ffc7f41d7..0aa9532a05cf 100644 --- a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c +++ b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c @@ -460,6 +460,73 @@ static void test_intersect(void) tracing_multi_intersect__destroy(skel); } +static void test_fentry_after_multi(void) +{ + static const char * const funcs[] = { + "bpf_fentry_test1", + }; + struct bpf_link *fentry_link = NULL, *multi_link = NULL; + struct tracing_multi_intersect *skel = NULL; + LIBBPF_OPTS(bpf_tracing_multi_opts, opts); + LIBBPF_OPTS(bpf_test_run_opts, topts); + __u32 *ids = NULL; + int err; + + skel = tracing_multi_intersect__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_multi_intersect__open_and_load")) + return; + + skel->bss->pid = getpid(); + + ids = get_ids(funcs, ARRAY_SIZE(funcs), NULL); + if (!ASSERT_OK_PTR(ids, "get_ids")) + goto cleanup; + + opts.ids = ids; + opts.cnt = ARRAY_SIZE(funcs); + multi_link = bpf_program__attach_tracing_multi(skel->progs.fentry_1, NULL, &opts); + if (!ASSERT_OK_PTR(multi_link, "attach_multi")) + goto cleanup; + + fentry_link = bpf_program__attach(skel->progs.fentry); + if (!ASSERT_OK_PTR(fentry_link, "attach_fentry")) + goto cleanup; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.fentry_1), &topts); + if (!ASSERT_OK(err, "test_run")) + goto cleanup; + ASSERT_EQ(skel->bss->test_result_fentry_1, 1, "multi_fentry"); + ASSERT_EQ(skel->bss->test_result_fentry, 1, "fentry"); + + err = bpf_link__destroy(fentry_link); + fentry_link = NULL; + if (!ASSERT_OK(err, "destroy_fentry")) + goto cleanup; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.fentry_1), &topts); + if (!ASSERT_OK(err, "test_run_multi")) + goto cleanup; + ASSERT_EQ(skel->bss->test_result_fentry_1, 2, "multi_fentry_only"); + ASSERT_EQ(skel->bss->test_result_fentry, 1, "fentry_detached"); + + err = bpf_link__destroy(multi_link); + multi_link = NULL; + if (!ASSERT_OK(err, "destroy_multi")) + goto cleanup; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.fentry_1), &topts); + if (!ASSERT_OK(err, "test_run_detached")) + goto cleanup; + ASSERT_EQ(skel->bss->test_result_fentry_1, 2, "multi_fentry_detached"); + ASSERT_EQ(skel->bss->test_result_fentry, 1, "fentry_still_detached"); + +cleanup: + bpf_link__destroy(fentry_link); + bpf_link__destroy(multi_link); + free(ids); + tracing_multi_intersect__destroy(skel); +} + static void test_session(void) { LIBBPF_OPTS(bpf_test_run_opts, topts); @@ -957,4 +1024,6 @@ void test_tracing_multi_test(void) if (test__start_subtest("attach_api_fails")) test_attach_api_fails(); RUN_TESTS(tracing_multi_verifier); + if (test__start_subtest("fentry_after_multi")) + test_fentry_after_multi(); } diff --git a/tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c b/tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c index cd5be0bb6ffd..5b0af8f4c62f 100644 --- a/tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c +++ b/tools/testing/selftests/bpf/progs/tracing_multi_intersect_attach.c @@ -11,6 +11,14 @@ __u64 test_result_fentry_1 = 0; __u64 test_result_fentry_2 = 0; __u64 test_result_fexit_1 = 0; __u64 test_result_fexit_2 = 0; +__u64 test_result_fentry = 0; + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(fentry) +{ + tracing_multi_arg_check(ctx, &test_result_fentry, false); + return 0; +} SEC("fentry.multi") int BPF_PROG(fentry_1) From 35dac1daeb3c8208515047d32f22c4a162e8de5f Mon Sep 17 00:00:00 2001 From: Avinash Duduskar Date: Mon, 13 Jul 2026 21:53:03 +0530 Subject: [PATCH 080/373] bpf: Add BPF_FIB_LOOKUP_VLAN flag to bpf_fib_lookup() helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bpf_fib_lookup() returns the FIB-resolved egress ifindex straight from the fib result. When the egress is a VLAN device, the returned ifindex is the VLAN netdev's, which has no XDP xmit handler; XDP programs that want to forward the frame (e.g. xdp-forward) must instead target the underlying physical device and push the VLAN tag themselves. Today the program has no way to learn either the underlying ifindex or the VLAN tag without maintaining its own VLAN-to-ifindex map in userspace and refreshing it on netlink events. Add BPF_FIB_LOOKUP_VLAN. When the caller sets this flag and the fib result is a VLAN device whose immediate parent is a real (non-VLAN) device in the same network namespace, populate the existing output fields params->h_vlan_proto and params->h_vlan_TCI from the VLAN device and replace params->ifindex with the parent's ifindex. params->h_vlan_TCI carries the VID only, with PCP and DEI bits zero; a consumer wanting to set egress priority writes PCP itself. params->smac is the VLAN device's own address, which can differ from the parent's. Only the immediate parent is resolved, via vlan_dev_priv(dev)->real_dev and not vlan_dev_real_dev(), which walks to the bottom of a stack. When the immediate parent is not a real device in the same namespace, the lookup returns BPF_FIB_LKUP_RET_VLAN_FAILURE and leaves params->ifindex at the input. This covers a stacked VLAN (QinQ), where the immediate parent is itself a VLAN device and one h_vlan_proto/h_vlan_TCI pair cannot describe two tags, and a parent in another network namespace (a VLAN device can be moved while its parent stays), whose ifindex would be meaningless in the caller's namespace. A program that wants the VLAN device's own ifindex re-issues the lookup, with a re-initialized params, without BPF_FIB_LOOKUP_VLAN, so the unreducible case stays distinct from a physical egress. That distinction matters for XDP: a program cannot xmit on a VLAN device, so a success carrying the VLAN ifindex would make it redirect to a device with no ndo_xdp_xmit and drop the frame at xdp_do_flush(). The swap and the vlan fields are written only on the reduce path; other output fields keep their existing behaviour, so a frag-needed result still reports the route mtu in params->mtu_result. BPF_FIB_LOOKUP_VLAN is only useful to XDP, which cannot redirect to a VLAN device. A tc program can redirect to the VLAN device directly, so bpf_skb_fib_lookup() rejects the flag with -EINVAL; bpf_xdp_fib_lookup() accepts it. When the flag is not set, behaviour is unchanged: h_vlan_proto and h_vlan_TCI are zeroed and ifindex is left at the FIB result. The new block is compiled only under CONFIG_VLAN_8021Q since vlan_dev_priv() is not defined otherwise; without that config is_vlan_dev() is constant false and the flag is accepted but never acts. That is safe because no VLAN device can exist there, so every egress is already physical. This lets an XDP redirect target the physical device and learn the tag to push in a single lookup, which xdp-forward's optional VLAN mode (xdp-project/xdp-tools#504) wants from the kernel side. The helper's input semantics are unchanged; the reverse direction (supplying a tag as lookup input) is added in the following patch. Suggested-by: Toke Høiland-Jørgensen Signed-off-by: Avinash Duduskar Reviewed-by: Toke Høiland-Jørgensen Reviewed-by: Emil Tsalapatis Acked-by: David Ahern Link: https://lore.kernel.org/bpf/20260713162305.1237211-2-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/uapi/linux/bpf.h | 33 ++++++++++++++++++++++++++++++++- net/core/filter.c | 33 +++++++++++++++++++++++++++++---- tools/include/uapi/linux/bpf.h | 33 ++++++++++++++++++++++++++++++++- 3 files changed, 93 insertions(+), 6 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 2f1d24fef857..8b41c365ddf9 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3532,6 +3532,31 @@ union bpf_attr { * Use the mark present in *params*->mark for the fib lookup. * This option should not be used with BPF_FIB_LOOKUP_DIRECT, * as it only has meaning for full lookups. + * **BPF_FIB_LOOKUP_VLAN** + * If the fib lookup resolves to a VLAN device whose + * parent is a real (non-VLAN) device, set + * *params*->h_vlan_proto and *params*->h_vlan_TCI from + * the VLAN device and replace *params*->ifindex with the + * parent's ifindex. *params*->h_vlan_TCI carries the VID + * only, with PCP and DEI bits zero; a consumer wanting to + * set egress priority writes PCP itself. *params*->smac is + * the VLAN device's own address, which can differ from the + * parent's. Only the immediate parent is resolved; if it + * is itself a VLAN device (QinQ) or in another namespace, + * the egress cannot be reduced to a physical device plus + * one tag and the lookup returns + * **BPF_FIB_LKUP_RET_VLAN_FAILURE** with *params*->ifindex + * left at the input. To obtain the VLAN device's own + * ifindex, repeat the lookup without + * **BPF_FIB_LOOKUP_VLAN**, re-initializing *params* + * first: output fields overwrite the inputs they share + * storage with. The swap and the vlan fields + * are written only on success; other output fields keep + * the helper's existing behaviour, so a frag-needed result + * still reports the route mtu in *params*->mtu_result. + * This flag is only valid for XDP programs; tc programs + * receive -EINVAL since they can redirect to the VLAN + * device directly. * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -7339,6 +7364,7 @@ enum { BPF_FIB_LOOKUP_TBID = (1U << 3), BPF_FIB_LOOKUP_SRC = (1U << 4), BPF_FIB_LOOKUP_MARK = (1U << 5), + BPF_FIB_LOOKUP_VLAN = (1U << 6), }; enum { @@ -7352,6 +7378,7 @@ enum { BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */ BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */ BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */ + BPF_FIB_LKUP_RET_VLAN_FAILURE, /* VLAN egress, parent unresolvable */ }; struct bpf_fib_lookup { @@ -7405,7 +7432,11 @@ struct bpf_fib_lookup { union { struct { - /* output */ + /* + * output with BPF_FIB_LOOKUP_VLAN: set from the + * resolved egress VLAN device (see the flag); zeroed + * on other successful lookups. + */ __be16 h_vlan_proto; __be16 h_vlan_TCI; }; diff --git a/net/core/filter.c b/net/core/filter.c index 056deb9b3fc3..b8f2595ff582 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -6206,10 +6206,29 @@ static const struct bpf_func_proto bpf_skb_get_xfrm_state_proto = { #endif #if IS_ENABLED(CONFIG_INET) || IS_ENABLED(CONFIG_IPV6) -static int bpf_fib_set_fwd_params(struct bpf_fib_lookup *params, u32 mtu) +static int bpf_fib_set_fwd_params(struct net_device *dev, + struct bpf_fib_lookup *params, + u32 flags, u32 mtu, u32 in_ifindex) { params->h_vlan_TCI = 0; params->h_vlan_proto = 0; + +#if IS_ENABLED(CONFIG_VLAN_8021Q) + if ((flags & BPF_FIB_LOOKUP_VLAN) && is_vlan_dev(dev)) { + struct net_device *real_dev = vlan_dev_priv(dev)->real_dev; + + if (!is_vlan_dev(real_dev) && + net_eq(dev_net(real_dev), dev_net(dev))) { + params->h_vlan_proto = vlan_dev_vlan_proto(dev); + params->h_vlan_TCI = htons(vlan_dev_vlan_id(dev)); + params->ifindex = real_dev->ifindex; + } else { + params->ifindex = in_ifindex; + return BPF_FIB_LKUP_RET_VLAN_FAILURE; + } + } +#endif + if (mtu) params->mtu_result = mtu; /* union with tot_len */ @@ -6221,6 +6240,7 @@ static int bpf_fib_set_fwd_params(struct bpf_fib_lookup *params, u32 mtu) static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, u32 flags, bool check_mtu) { + u32 in_ifindex = params->ifindex; struct neighbour *neigh = NULL; struct fib_nh_common *nhc; struct in_device *in_dev; @@ -6352,7 +6372,7 @@ static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, memcpy(params->smac, dev->dev_addr, ETH_ALEN); set_fwd_params: - return bpf_fib_set_fwd_params(params, mtu); + return bpf_fib_set_fwd_params(dev, params, flags, mtu, in_ifindex); } #endif @@ -6362,6 +6382,7 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, { struct in6_addr *src = (struct in6_addr *) params->ipv6_src; struct in6_addr *dst = (struct in6_addr *) params->ipv6_dst; + u32 in_ifindex = params->ifindex; struct fib6_result res = {}; struct neighbour *neigh; struct net_device *dev; @@ -6491,13 +6512,14 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, memcpy(params->smac, dev->dev_addr, ETH_ALEN); set_fwd_params: - return bpf_fib_set_fwd_params(params, mtu); + return bpf_fib_set_fwd_params(dev, params, flags, mtu, in_ifindex); } #endif #define BPF_FIB_LOOKUP_MASK (BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT | \ BPF_FIB_LOOKUP_SKIP_NEIGH | BPF_FIB_LOOKUP_TBID | \ - BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_MARK) + BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_MARK | \ + BPF_FIB_LOOKUP_VLAN) BPF_CALL_4(bpf_xdp_fib_lookup, struct xdp_buff *, ctx, struct bpf_fib_lookup *, params, int, plen, u32, flags) @@ -6546,6 +6568,9 @@ BPF_CALL_4(bpf_skb_fib_lookup, struct sk_buff *, skb, if (flags & ~BPF_FIB_LOOKUP_MASK) return -EINVAL; + if (flags & BPF_FIB_LOOKUP_VLAN) + return -EINVAL; + if (params->tot_len) check_mtu = true; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 2f1d24fef857..8b41c365ddf9 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3532,6 +3532,31 @@ union bpf_attr { * Use the mark present in *params*->mark for the fib lookup. * This option should not be used with BPF_FIB_LOOKUP_DIRECT, * as it only has meaning for full lookups. + * **BPF_FIB_LOOKUP_VLAN** + * If the fib lookup resolves to a VLAN device whose + * parent is a real (non-VLAN) device, set + * *params*->h_vlan_proto and *params*->h_vlan_TCI from + * the VLAN device and replace *params*->ifindex with the + * parent's ifindex. *params*->h_vlan_TCI carries the VID + * only, with PCP and DEI bits zero; a consumer wanting to + * set egress priority writes PCP itself. *params*->smac is + * the VLAN device's own address, which can differ from the + * parent's. Only the immediate parent is resolved; if it + * is itself a VLAN device (QinQ) or in another namespace, + * the egress cannot be reduced to a physical device plus + * one tag and the lookup returns + * **BPF_FIB_LKUP_RET_VLAN_FAILURE** with *params*->ifindex + * left at the input. To obtain the VLAN device's own + * ifindex, repeat the lookup without + * **BPF_FIB_LOOKUP_VLAN**, re-initializing *params* + * first: output fields overwrite the inputs they share + * storage with. The swap and the vlan fields + * are written only on success; other output fields keep + * the helper's existing behaviour, so a frag-needed result + * still reports the route mtu in *params*->mtu_result. + * This flag is only valid for XDP programs; tc programs + * receive -EINVAL since they can redirect to the VLAN + * device directly. * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -7339,6 +7364,7 @@ enum { BPF_FIB_LOOKUP_TBID = (1U << 3), BPF_FIB_LOOKUP_SRC = (1U << 4), BPF_FIB_LOOKUP_MARK = (1U << 5), + BPF_FIB_LOOKUP_VLAN = (1U << 6), }; enum { @@ -7352,6 +7378,7 @@ enum { BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */ BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */ BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */ + BPF_FIB_LKUP_RET_VLAN_FAILURE, /* VLAN egress, parent unresolvable */ }; struct bpf_fib_lookup { @@ -7405,7 +7432,11 @@ struct bpf_fib_lookup { union { struct { - /* output */ + /* + * output with BPF_FIB_LOOKUP_VLAN: set from the + * resolved egress VLAN device (see the flag); zeroed + * on other successful lookups. + */ __be16 h_vlan_proto; __be16 h_vlan_TCI; }; From 217828aad80d091fa1d840587a3d9b6187ee170f Mon Sep 17 00:00:00 2001 From: Avinash Duduskar Date: Mon, 13 Jul 2026 21:53:04 +0530 Subject: [PATCH 081/373] bpf: Add BPF_FIB_LOOKUP_VLAN_INPUT flag to bpf_fib_lookup() helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BPF_FIB_LOOKUP_VLAN resolves a VLAN egress. The reverse is also useful: an XDP program receiving a VLAN-tagged frame on a physical device wants the lookup to behave as if the packet had arrived on the corresponding VLAN subinterface, so iif-based policy routing and VRF table selection use the right ingress. Add BPF_FIB_LOOKUP_VLAN_INPUT. When set, params->h_vlan_proto and params->h_vlan_TCI are read as an input VLAN tag and the matching VLAN device of params->ifindex is resolved with __vlan_find_dev_deep_rcu(). The device must be up and in the same network namespace as params->ifindex (a VLAN device can be moved to another netns while registered on its parent; receive would deliver into that other namespace, which a lookup here cannot represent). If params->ifindex is itself a VLAN device, its inner (QinQ) subinterface is matched. For a bond or team, a tag on a port matches no device and returns NOT_FWDED; pass the master's ifindex. The lookup then runs with the resolved device as the ingress; params->ifindex itself is not modified on the input side. When the resolved device is enslaved to a VRF, both the full lookup (via the l3mdev rule) and BPF_FIB_LOOKUP_DIRECT (via l3mdev_fib_table_rcu()) select the VRF's table from the resolved ingress. That follows from feeding the resolved device to the flow as the ingress (fl4.flowi4_iif = dev->ifindex), which is what makes l3mdev resolve the VRF master from the subinterface rather than from params->ifindex. The two failure classes get different treatment on purpose. A h_vlan_proto other than 802.1Q/802.1ad is API misuse and returns -EINVAL, since it would otherwise reach the WARN in vlan_proto_idx() with a program-controlled value. An unmatched VID, a device that is down, or one in another namespace is a data outcome and returns BPF_FIB_LKUP_RET_NOT_FWDED, matching the DIRECT path when fib_get_table() finds no table and mirroring real ingress, where the receive path drops such frames. A VID of 0 (a priority tag) is looked up literally and normally fails the same way; receive instead processes such frames untagged, so callers should not set the flag for priority tags. Proceeding on the physical device for any of these would be fail-open for the policy-routing cases above. The h_vlan fields share a union with tbid, so the flag cannot be combined with BPF_FIB_LOOKUP_TBID. It describes ingress, so it also cannot be combined with BPF_FIB_LOOKUP_OUTPUT. Both combinations return -EINVAL; restricting now keeps a later relaxation backward compatible. Combining with BPF_FIB_LOOKUP_VLAN is allowed: the tag is consumed on the ingress side and the egress tag is written on success. Under !CONFIG_VLAN_8021Q the __vlan_find_dev_deep_rcu() stub returns NULL, so every lookup with a valid proto returns NOT_FWDED, which is correct since no VLAN device can exist. Suggested-by: Toke Høiland-Jørgensen Signed-off-by: Avinash Duduskar Reviewed-by: Toke Høiland-Jørgensen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260713162305.1237211-3-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/uapi/linux/bpf.h | 21 ++++++++++- net/core/filter.c | 66 +++++++++++++++++++++++++++++++--- tools/include/uapi/linux/bpf.h | 21 ++++++++++- 3 files changed, 101 insertions(+), 7 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 8b41c365ddf9..005038fbeea4 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3557,6 +3557,22 @@ union bpf_attr { * This flag is only valid for XDP programs; tc programs * receive -EINVAL since they can redirect to the VLAN * device directly. + * **BPF_FIB_LOOKUP_VLAN_INPUT** + * Treat *params*->h_vlan_proto and *params*->h_vlan_TCI + * as an input VLAN tag and run the lookup as if ingress + * had happened on the VLAN subinterface carrying that tag + * on *params*->ifindex. The VID is the low 12 bits of + * *params*->h_vlan_TCI; *params*->h_vlan_proto must be + * ETH_P_8021Q or ETH_P_8021AD in network byte order, else + * **-EINVAL**. If *params*->ifindex is itself a VLAN + * device, its inner (QinQ) subinterface is matched; for a + * bond or team, pass the master's ifindex. An unmatched + * tag, a down device, or one in another namespace returns + * **BPF_FIB_LKUP_RET_NOT_FWDED**, mirroring real ingress. + * A VID of 0 is looked up literally, so do not set this + * flag for priority-tagged frames. Cannot be combined with + * **BPF_FIB_LOOKUP_TBID** or **BPF_FIB_LOOKUP_OUTPUT** + * (returns **-EINVAL**). * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -7365,6 +7381,7 @@ enum { BPF_FIB_LOOKUP_SRC = (1U << 4), BPF_FIB_LOOKUP_MARK = (1U << 5), BPF_FIB_LOOKUP_VLAN = (1U << 6), + BPF_FIB_LOOKUP_VLAN_INPUT = (1U << 7), }; enum { @@ -7435,7 +7452,9 @@ struct bpf_fib_lookup { /* * output with BPF_FIB_LOOKUP_VLAN: set from the * resolved egress VLAN device (see the flag); zeroed - * on other successful lookups. + * on other successful lookups. input with + * BPF_FIB_LOOKUP_VLAN_INPUT: the VLAN tag to scope + * the lookup by. */ __be16 h_vlan_proto; __be16 h_vlan_TCI; diff --git a/net/core/filter.c b/net/core/filter.c index b8f2595ff582..0b7afdd0ae47 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -6234,6 +6234,25 @@ static int bpf_fib_set_fwd_params(struct net_device *dev, return 0; } + +static struct net_device *bpf_fib_vlan_input_dev(struct net_device *dev, + const struct bpf_fib_lookup *params) +{ + __be16 proto = params->h_vlan_proto; + struct net_device *vlan_dev; + u16 vid; + + if (proto != htons(ETH_P_8021Q) && proto != htons(ETH_P_8021AD)) + return ERR_PTR(-EINVAL); + + vid = ntohs(params->h_vlan_TCI) & VLAN_VID_MASK; + vlan_dev = __vlan_find_dev_deep_rcu(dev, proto, vid); + if (!vlan_dev || !(vlan_dev->flags & IFF_UP) || + !net_eq(dev_net(vlan_dev), dev_net(dev))) + return NULL; + + return vlan_dev; +} #endif #if IS_ENABLED(CONFIG_INET) @@ -6254,6 +6273,14 @@ static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, if (unlikely(!dev)) return -ENODEV; + if (flags & BPF_FIB_LOOKUP_VLAN_INPUT) { + dev = bpf_fib_vlan_input_dev(dev, params); + if (IS_ERR(dev)) + return PTR_ERR(dev); + if (!dev) + return BPF_FIB_LKUP_RET_NOT_FWDED; + } + /* verify forwarding is enabled on this interface */ in_dev = __in_dev_get_rcu(dev); if (unlikely(!in_dev || !IN_DEV_FORWARD(in_dev))) @@ -6263,7 +6290,11 @@ static int bpf_ipv4_fib_lookup(struct net *net, struct bpf_fib_lookup *params, fl4.flowi4_iif = 1; fl4.flowi4_oif = params->ifindex; } else { - fl4.flowi4_iif = params->ifindex; + /* + * dev->ifindex, not params->ifindex: VLAN_INPUT may have + * resolved dev to a subinterface above. + */ + fl4.flowi4_iif = dev->ifindex; fl4.flowi4_oif = 0; } fl4.flowi4_dscp = inet_dsfield_to_dscp(params->tos); @@ -6400,6 +6431,14 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, if (unlikely(!dev)) return -ENODEV; + if (flags & BPF_FIB_LOOKUP_VLAN_INPUT) { + dev = bpf_fib_vlan_input_dev(dev, params); + if (IS_ERR(dev)) + return PTR_ERR(dev); + if (!dev) + return BPF_FIB_LKUP_RET_NOT_FWDED; + } + idev = __in6_dev_get_safely(dev); if (unlikely(!idev || !READ_ONCE(idev->cnf.forwarding))) return BPF_FIB_LKUP_RET_FWD_DISABLED; @@ -6408,7 +6447,12 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, fl6.flowi6_iif = 1; oif = fl6.flowi6_oif = params->ifindex; } else { - oif = fl6.flowi6_iif = params->ifindex; + /* + * dev->ifindex, not params->ifindex: VLAN_INPUT may have + * resolved dev to a subinterface above. + */ + oif = dev->ifindex; + fl6.flowi6_iif = oif; fl6.flowi6_oif = 0; strict = RT6_LOOKUP_F_HAS_SADDR; } @@ -6519,7 +6563,19 @@ static int bpf_ipv6_fib_lookup(struct net *net, struct bpf_fib_lookup *params, #define BPF_FIB_LOOKUP_MASK (BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT | \ BPF_FIB_LOOKUP_SKIP_NEIGH | BPF_FIB_LOOKUP_TBID | \ BPF_FIB_LOOKUP_SRC | BPF_FIB_LOOKUP_MARK | \ - BPF_FIB_LOOKUP_VLAN) + BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_VLAN_INPUT) + +static bool bpf_fib_lookup_flags_ok(u32 flags) +{ + if (flags & ~BPF_FIB_LOOKUP_MASK) + return false; + + if ((flags & BPF_FIB_LOOKUP_VLAN_INPUT) && + (flags & (BPF_FIB_LOOKUP_TBID | BPF_FIB_LOOKUP_OUTPUT))) + return false; + + return true; +} BPF_CALL_4(bpf_xdp_fib_lookup, struct xdp_buff *, ctx, struct bpf_fib_lookup *, params, int, plen, u32, flags) @@ -6527,7 +6583,7 @@ BPF_CALL_4(bpf_xdp_fib_lookup, struct xdp_buff *, ctx, if (plen < sizeof(*params)) return -EINVAL; - if (flags & ~BPF_FIB_LOOKUP_MASK) + if (!bpf_fib_lookup_flags_ok(flags)) return -EINVAL; switch (params->family) { @@ -6565,7 +6621,7 @@ BPF_CALL_4(bpf_skb_fib_lookup, struct sk_buff *, skb, if (plen < sizeof(*params)) return -EINVAL; - if (flags & ~BPF_FIB_LOOKUP_MASK) + if (!bpf_fib_lookup_flags_ok(flags)) return -EINVAL; if (flags & BPF_FIB_LOOKUP_VLAN) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 8b41c365ddf9..005038fbeea4 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3557,6 +3557,22 @@ union bpf_attr { * This flag is only valid for XDP programs; tc programs * receive -EINVAL since they can redirect to the VLAN * device directly. + * **BPF_FIB_LOOKUP_VLAN_INPUT** + * Treat *params*->h_vlan_proto and *params*->h_vlan_TCI + * as an input VLAN tag and run the lookup as if ingress + * had happened on the VLAN subinterface carrying that tag + * on *params*->ifindex. The VID is the low 12 bits of + * *params*->h_vlan_TCI; *params*->h_vlan_proto must be + * ETH_P_8021Q or ETH_P_8021AD in network byte order, else + * **-EINVAL**. If *params*->ifindex is itself a VLAN + * device, its inner (QinQ) subinterface is matched; for a + * bond or team, pass the master's ifindex. An unmatched + * tag, a down device, or one in another namespace returns + * **BPF_FIB_LKUP_RET_NOT_FWDED**, mirroring real ingress. + * A VID of 0 is looked up literally, so do not set this + * flag for priority-tagged frames. Cannot be combined with + * **BPF_FIB_LOOKUP_TBID** or **BPF_FIB_LOOKUP_OUTPUT** + * (returns **-EINVAL**). * * *ctx* is either **struct xdp_md** for XDP programs or * **struct sk_buff** tc cls_act programs. @@ -7365,6 +7381,7 @@ enum { BPF_FIB_LOOKUP_SRC = (1U << 4), BPF_FIB_LOOKUP_MARK = (1U << 5), BPF_FIB_LOOKUP_VLAN = (1U << 6), + BPF_FIB_LOOKUP_VLAN_INPUT = (1U << 7), }; enum { @@ -7435,7 +7452,9 @@ struct bpf_fib_lookup { /* * output with BPF_FIB_LOOKUP_VLAN: set from the * resolved egress VLAN device (see the flag); zeroed - * on other successful lookups. + * on other successful lookups. input with + * BPF_FIB_LOOKUP_VLAN_INPUT: the VLAN tag to scope + * the lookup by. */ __be16 h_vlan_proto; __be16 h_vlan_TCI; From e54a87872e34d97333dcdfb8c8e0327f5bd8bb43 Mon Sep 17 00:00:00 2001 From: Avinash Duduskar Date: Mon, 13 Jul 2026 21:53:05 +0530 Subject: [PATCH 082/373] selftests/bpf: Add bpf_fib_lookup() VLAN flag tests Cover both new VLAN flags in the fib_lookup test. BPF_FIB_LOOKUP_VLAN reduces a VLAN egress to its physical parent plus the tag, and BPF_FIB_LOOKUP_VLAN_INPUT scopes the lookup to a VLAN subinterface. BPF_FIB_LOOKUP_VLAN is XDP-only, since VLAN devices have no XDP xmit; the tc helper rejects it with -EINVAL, which the table runner asserts for every flag arm, and the egress result is checked through bpf_xdp_fib_lookup(). Non-VLAN cases run through both helpers and assert the path-independent results match; the XDP loop also checks dmac and, for the tot_len cases, the route mtu_result, so the VLAN-egress dmac and frag-needed coverage stays even though the tc path no longer reaches it. The egress arms pin the reduction (parent ifindex plus tag, including via a neighbour on the VLAN device, in OUTPUT mode, over a bond, and through a DIRECT|TBID table) and the failure contract: a stacked-VLAN (QinQ) egress returns BPF_FIB_LKUP_RET_VLAN_FAILURE with params->ifindex left at the input. That is distinct from a no-neighbour return, which reports the egress ifindex; only VLAN_FAILURE rewinds params->ifindex, and a guard arm whose input and egress devices differ pins the distinction. The VLAN_FAILURE arms are IPv4; the IPv6 path reaches it through the same shared code, so an IPv6 arm would only re-test that. The input arms use an iif rule that routes one destination to two gateways, so the asserted gateway reveals which device the lookup used as ingress, including VRF table selection through the l3mdev rule and l3mdev_fib_table_rcu(). The VRF arms are IPv4-only: the l3mdev match and table resolution are family-independent core shared by both rule paths, and the IPv6 iif feed is pinned by the IPv6 VLAN input arm. A cross-netns subtest moves a VLAN device into a second netns while it stays registered on its parent and checks both directions fail closed at the boundary. A live-frames subtest (test_fib_lookup_vlan_redirect, with BPF_F_TEST_XDP_LIVE_FRAMES) drives real frames through the native xdp_do_redirect() / xdp_do_flush() path: a reducible egress is redirected to the parent and delivered to its peer, while a QinQ egress is passed to the stack, since redirecting to the VLAN device would drop the frame at flush (no ndo_xdp_xmit). The remaining per-case assertions are in the test table: resolution semantics, the -EINVAL and NOT_FWDED error arms, and the SRC/SKIP_NEIGH combinations. Signed-off-by: Avinash Duduskar Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260713162305.1237211-4-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/fib_lookup.c | 720 +++++++++++++++++- .../testing/selftests/bpf/progs/fib_lookup.c | 57 ++ 2 files changed, 773 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/fib_lookup.c b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c index bd7658958004..f7361f9a3459 100644 --- a/tools/testing/selftests/bpf/prog_tests/fib_lookup.c +++ b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c @@ -2,6 +2,7 @@ /* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ #include +#include #include #include @@ -23,6 +24,7 @@ #define IPV4_TBID_ADDR "172.0.0.254" #define IPV4_TBID_NET "172.0.0.0" #define IPV4_TBID_DST "172.0.0.2" +#define IPV4_TBID_NONEIGH_DST "172.0.0.5" #define IPV6_TBID_ADDR "fd00::FFFF" #define IPV6_TBID_NET "fd00::" #define IPV6_TBID_DST "fd00::2" @@ -37,6 +39,41 @@ #define IPV6_LOCAL "fd01::3" #define IPV6_GW1 "fd01::1" #define IPV6_GW2 "fd01::2" +#define VLAN_ID 100 +#define VLAN_IFACE "veth1.100" +#define VLAN_ID_DOWN 102 +#define VLAN_IFACE_DOWN "veth1.102" +#define QINQ_OUTER_IFACE "veth1.200" +#define QINQ_INNER_IFACE "veth1.200.300" +#define VLAN_TABLE "300" +#define IPV4_VLAN_IFACE_ADDR "10.5.0.254" +#define IPV4_VLAN_EGRESS_DST "10.5.0.2" +#define IPV4_QINQ_DST "10.7.0.2" +#define IPV4_VLAN_DST "10.6.0.2" +#define IPV4_VLAN_GW "10.5.0.1" +#define IPV6_VLAN_IFACE_ADDR "fd02::254" +#define IPV6_VLAN_EGRESS_DST "fd02::2" +#define IPV6_VLAN_DST "fd03::2" +#define IPV6_VLAN_GW "fd02::1" +#define VLAN_VID_UNUSED 999 +#define VRF_IFACE "vrf-blue" +#define VRF_TABLE "1000" +#define VRF_VLAN_ID 101 +#define VRF_VLAN_IFACE "veth1.101" +#define IPV4_VRF_IFACE_ADDR "10.8.0.254" +#define IPV4_VRF_GW "10.8.0.1" +#define IPV4_VRF_DST "10.9.0.2" +#define TBID_VLAN_ID 50 +#define TBID_VLAN_IFACE "veth2.50" +#define IPV4_TBID_VLAN_DST "172.2.0.2" +#define IPV4_BOND_VLAN_DST "10.11.0.2" +#define IPV4_VLAN_MTU_DST "10.5.9.2" +#define QINQ_AD_VLAN_ID 200 +#define QINQ_INNER_VLAN_ID 300 +#define BOND_IFACE "bond99" +#define BOND_PORT "veth3" +#define BOND_PORT_PEER "veth4" +#define BOND_VLAN_ID 500 #define DMAC "11:11:11:11:11:11" #define DMAC_INIT { 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, } #define DMAC2 "01:01:01:01:01:01" @@ -52,6 +89,17 @@ struct fib_lookup_test { __u32 tbid; __u8 dmac[6]; __u32 mark; + /* + * input tag with BPF_FIB_LOOKUP_VLAN_INPUT; expected output tag + * with BPF_FIB_LOOKUP_VLAN (checked when check_vlan is set) + */ + __u16 vlan_proto; + __u16 vlan_id; + bool check_vlan; + const char *expected_dev; /* expected params->ifindex after lookup */ + const char *iif; /* override the default veth1 input device */ + __u16 tot_len; /* triggers the in-lookup mtu check when set */ + __u16 expected_mtu; /* expected mtu_result (union with tot_len) */ }; static const struct fib_lookup_test tests[] = { @@ -79,6 +127,17 @@ static const struct fib_lookup_test tests[] = { .daddr = IPV4_TBID_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, .lookup_flags = BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID, .tbid = 100, .dmac = DMAC_INIT2, }, + /* + * An error that returns after the egress device is resolved must + * report the egress ifindex, not the input. This routes from input + * veth1 via veth2 (table 100) to a dst with no neighbour, so + * input != egress, pinning NO_NEIGH to the egress device. + */ + { .desc = "IPv4 NO_NEIGH reports the egress ifindex, not the input", + .daddr = IPV4_TBID_NONEIGH_DST, + .expected_ret = BPF_FIB_LKUP_RET_NO_NEIGH, + .lookup_flags = BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID, .tbid = 100, + .expected_dev = "veth2", }, { .desc = "IPv6 TBID lookup failure", .daddr = IPV6_TBID_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, .lookup_flags = BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID, @@ -142,6 +201,218 @@ static const struct fib_lookup_test tests[] = { .expected_dst = IPV6_GW1, .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, .mark = MARK, }, + /* vlan egress resolution */ + /* + * Invariant the VLAN-egress arms jointly enforce: a + * BPF_FIB_LOOKUP_VLAN SUCCESS always carries a physical, + * xmit-capable ifindex; no SUCCESS ever returns a VLAN-device + * ifindex. Reducible arms pin ifindex == the physical parent; the + * QinQ and foreign-netns arms pin VLAN_FAILURE with params->ifindex + * left at the input, so a regression to best-effort (SUCCESS + the + * VLAN ifindex) fails one. + */ + { .desc = "IPv4 VLAN egress, no flag", + .daddr = IPV4_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = VLAN_IFACE, .check_vlan = true, }, + { .desc = "IPv4 VLAN egress, single VLAN", + .daddr = IPV4_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + /* + * skb path without tot_len: mtu_result is the VLAN device's mtu + * (1400), not the parent's (1500) + */ + { .desc = "IPv4 VLAN egress, skb-path mtu is the VLAN device's without the flag", + .daddr = IPV4_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = VLAN_IFACE, .check_vlan = true, .expected_mtu = 1400, }, + { .desc = "IPv4 VLAN egress, flag set but egress is not a VLAN", + .daddr = IPV4_NUD_FAILED_ADDR, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, }, + { .desc = "IPv4 VLAN egress, QinQ not reducible (VLAN_FAILURE)", + .daddr = IPV4_QINQ_DST, + .expected_ret = BPF_FIB_LKUP_RET_VLAN_FAILURE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, }, + { .desc = "IPv4 QinQ egress without the flag (escape hatch)", + .daddr = IPV4_QINQ_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = QINQ_INNER_IFACE, }, + { .desc = "IPv6 VLAN egress, single VLAN", + .daddr = IPV6_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN egress, neighbour on the VLAN device", + .daddr = IPV4_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_VLAN, + .expected_dev = "veth1", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, .dmac = DMAC_INIT, }, + { .desc = "IPv4 VLAN egress in OUTPUT mode", + .daddr = IPV4_VLAN_EGRESS_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .iif = VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_OUTPUT | BPF_FIB_LOOKUP_VLAN | + BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN egress over a bond", + .daddr = IPV4_BOND_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = BOND_IFACE, .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = BOND_VLAN_ID, }, + { .desc = "IPv4 VLAN egress via TBID table", + .daddr = IPV4_TBID_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .lookup_flags = BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID | + BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .tbid = 100, + .expected_dev = "veth2", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = TBID_VLAN_ID, }, + { .desc = "IPv4 VLAN egress, success writes mtu_result with the swap", + .daddr = IPV4_VLAN_MTU_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .tot_len = 500, .expected_mtu = 1000, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN egress, FRAG_NEEDED reports mtu, swap unwritten", + .daddr = IPV4_VLAN_MTU_DST, .expected_ret = BPF_FIB_LKUP_RET_FRAG_NEEDED, + .tot_len = 1400, .expected_mtu = 1000, + .lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH, + .expected_dev = "veth1", .check_vlan = true, }, + /* vlan tag as lookup input */ + { .desc = "IPv4 VLAN input, no flag", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_GW1, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, }, + { .desc = "IPv4 VLAN input, tag selects subinterface route", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VLAN_GW, .expected_dev = VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv6 VLAN input, tag selects subinterface route", + .daddr = IPV6_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV6_VLAN_GW, .expected_dev = VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN input and egress combined", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VLAN_GW, .expected_dev = "veth1", + .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_VLAN | + BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN input, neighbour resolved on the route", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VLAN_GW, .expected_dev = VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, .dmac = DMAC_INIT2, }, + { .desc = "IPv4 VLAN input, source address from the subinterface", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_src = IPV4_VLAN_IFACE_ADDR, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SRC | + BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + /* + * VRF: the resolved subinterface is enslaved, so the l3mdev rule + * (full lookup) and l3mdev_fib_table_rcu() (DIRECT) must select + * the VRF table from the resolved ingress + */ + { .desc = "IPv4 VLAN input, VRF subinterface, no flag", + .daddr = IPV4_VRF_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_GW1, + .lookup_flags = BPF_FIB_LOOKUP_SKIP_NEIGH, }, + { .desc = "IPv4 VLAN input, tag selects VRF table", + .daddr = IPV4_VRF_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VRF_GW, .expected_dev = VRF_VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VRF_VLAN_ID, }, + { .desc = "IPv4 VLAN input, DIRECT uses VRF table from resolved ingress", + .daddr = IPV4_VRF_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VRF_GW, .expected_dev = VRF_VLAN_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_DIRECT | + BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VRF_VLAN_ID, }, + /* + * failure arms also assert params is left untouched: ifindex still + * names the physical device and the input tag bytes survive + */ + { .desc = "IPv4 VLAN input, invalid proto", + .daddr = IPV4_VLAN_DST, .expected_ret = -EINVAL, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = 0x1234, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN input, unmatched VID", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_VID_UNUSED, }, + { .desc = "IPv4 VLAN input, subinterface down", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID_DOWN, }, + /* + * the resolver runs before the forwarding check, so on devices + * with forwarding off FWD_DISABLED (not NOT_FWDED) proves the tag + * resolved to that device and the lookup used it as ingress + */ + { .desc = "IPv4 VLAN input, 802.1ad tag", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_FWD_DISABLED, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021AD, .vlan_id = QINQ_AD_VLAN_ID, }, + { .desc = "IPv4 VLAN input, PCP and DEI bits ignored in TCI", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_SUCCESS, + .expected_dst = IPV4_VLAN_GW, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = 0xe000 | VLAN_ID, }, + { .desc = "IPv4 VLAN input, inner QinQ device from VLAN ifindex", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_FWD_DISABLED, + .iif = QINQ_OUTER_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = QINQ_INNER_VLAN_ID, }, + /* + * bonding: the VLANs live on the master, as on receive, where the + * frame is steered to the master before VLAN processing; a port + * ifindex does not match (ports carry vid state but no VLAN devs) + */ + { .desc = "IPv4 VLAN input, tag on bond master resolves", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_FWD_DISABLED, + .iif = BOND_IFACE, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = BOND_VLAN_ID, }, + { .desc = "IPv4 VLAN input, tag on bond port does not match", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, + .iif = BOND_PORT, .expected_dev = BOND_PORT, .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = BOND_VLAN_ID, }, + { .desc = "IPv6 VLAN input, invalid proto", + .daddr = IPV6_VLAN_DST, .expected_ret = -EINVAL, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = 0x1234, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN input, VID 0 priority tag fails closed", + .daddr = IPV4_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = 0, }, + { .desc = "IPv6 VLAN input, unmatched VID", + .daddr = IPV6_VLAN_DST, .expected_ret = BPF_FIB_LKUP_RET_NOT_FWDED, + .expected_dev = "veth1", .check_vlan = true, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_SKIP_NEIGH, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_VID_UNUSED, }, + { .desc = "unknown flag bit rejected", + .daddr = IPV4_VLAN_DST, .expected_ret = -EINVAL, + .lookup_flags = (1 << 14) | BPF_FIB_LOOKUP_SKIP_NEIGH, }, + { .desc = "IPv4 VLAN input rejected with TBID", + .daddr = IPV4_VLAN_DST, .expected_ret = -EINVAL, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_TBID, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, + { .desc = "IPv4 VLAN input rejected with OUTPUT", + .daddr = IPV4_VLAN_DST, .expected_ret = -EINVAL, + .lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | BPF_FIB_LOOKUP_OUTPUT, + .vlan_proto = ETH_P_8021Q, .vlan_id = VLAN_ID, }, }; static int setup_netns(void) @@ -204,6 +475,105 @@ static int setup_netns(void) SYS(fail, "ip rule add prio 2 fwmark %d lookup %s", MARK, MARK_TABLE); SYS(fail, "ip -6 rule add prio 2 fwmark %d lookup %s", MARK, MARK_TABLE); + /* + * Setup for vlan tests: a subinterface for egress resolution and + * tag-as-input, a QinQ stack, and an iif rule so the input tests + * observe which device the lookup used as ingress. + */ + SYS(fail, "ip link add link veth1 name %s type vlan id %d", + VLAN_IFACE, VLAN_ID); + SYS(fail, "ip link set dev %s up", VLAN_IFACE); + /* + * lower than the veth1 parent (1500): the skb-path mtu check uses the + * FIB result (VLAN) device, so mtu_result is this value, which the + * no-flag arm below pins + */ + SYS(fail, "ip link set dev %s mtu 1400", VLAN_IFACE); + SYS(fail, "ip addr add %s/24 dev %s", IPV4_VLAN_IFACE_ADDR, VLAN_IFACE); + SYS(fail, "ip addr add %s/64 dev %s nodad", IPV6_VLAN_IFACE_ADDR, VLAN_IFACE); + + /* + * stays down: the input flag must treat its tag the way real + * ingress treats a frame arriving on a down VLAN device (drop) + */ + SYS(fail, "ip link add link veth1 name %s type vlan id %d", + VLAN_IFACE_DOWN, VLAN_ID_DOWN); + + err = write_sysctl("/proc/sys/net/ipv4/conf/" VLAN_IFACE "/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.conf." VLAN_IFACE ".forwarding)")) + goto fail; + + err = write_sysctl("/proc/sys/net/ipv6/conf/" VLAN_IFACE "/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv6.conf." VLAN_IFACE ".forwarding)")) + goto fail; + + SYS(fail, "ip link add link veth1 name %s type vlan proto 802.1ad id 200", + QINQ_OUTER_IFACE); + SYS(fail, "ip link add link %s name %s type vlan id 300", + QINQ_OUTER_IFACE, QINQ_INNER_IFACE); + SYS(fail, "ip link set dev %s up", QINQ_OUTER_IFACE); + SYS(fail, "ip link set dev %s up", QINQ_INNER_IFACE); + SYS(fail, "ip route add %s/32 dev %s", IPV4_QINQ_DST, QINQ_INNER_IFACE); + + SYS(fail, "ip route add %s/32 via %s", IPV4_VLAN_DST, IPV4_GW1); + SYS(fail, "ip route add table %s %s/32 via %s", + VLAN_TABLE, IPV4_VLAN_DST, IPV4_VLAN_GW); + SYS(fail, "ip rule add prio 3 iif %s lookup %s", VLAN_IFACE, VLAN_TABLE); + SYS(fail, "ip -6 route add %s/128 via %s", IPV6_VLAN_DST, IPV6_GW1); + SYS(fail, "ip -6 route add table %s %s/128 via %s", + VLAN_TABLE, IPV6_VLAN_DST, IPV6_VLAN_GW); + SYS(fail, "ip -6 rule add prio 3 iif %s lookup %s", VLAN_IFACE, VLAN_TABLE); + + /* a bond with one port and a VLAN on the bond */ + SYS(fail, "ip link add %s type bond", BOND_IFACE); + SYS(fail, "ip link add %s type veth peer name %s", BOND_PORT, BOND_PORT_PEER); + SYS(fail, "ip link set %s master %s", BOND_PORT, BOND_IFACE); + SYS(fail, "ip link set dev %s up", BOND_IFACE); + SYS(fail, "ip link set dev %s up", BOND_PORT); + SYS(fail, "ip link add link %s name %s.%d type vlan id %d", + BOND_IFACE, BOND_IFACE, BOND_VLAN_ID, BOND_VLAN_ID); + SYS(fail, "ip link set dev %s.%d up", BOND_IFACE, BOND_VLAN_ID); + SYS(fail, "ip route add %s/32 dev %s.%d", + IPV4_BOND_VLAN_DST, BOND_IFACE, BOND_VLAN_ID); + + /* + * a VRF with its own dedicated subinterface (the iif rules above + * must not see it), for the table-selection-by-ingress cases + */ + SYS(fail, "ip link add %s type vrf table %s", VRF_IFACE, VRF_TABLE); + SYS(fail, "ip link set dev %s up", VRF_IFACE); + SYS(fail, "ip link add link veth1 name %s type vlan id %d", + VRF_VLAN_IFACE, VRF_VLAN_ID); + SYS(fail, "ip link set %s master %s", VRF_VLAN_IFACE, VRF_IFACE); + SYS(fail, "ip link set dev %s up", VRF_VLAN_IFACE); + SYS(fail, "ip addr add %s/24 dev %s", IPV4_VRF_IFACE_ADDR, VRF_VLAN_IFACE); + err = write_sysctl("/proc/sys/net/ipv4/conf/" VRF_VLAN_IFACE "/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.conf." VRF_VLAN_IFACE ".forwarding)")) + goto fail; + SYS(fail, "ip route add %s/32 via %s", IPV4_VRF_DST, IPV4_GW1); + SYS(fail, "ip route add table %s %s/32 via %s", + VRF_TABLE, IPV4_VRF_DST, IPV4_VRF_GW); + + /* neighbours on the VLAN subinterface for the non-SKIP_NEIGH cases */ + err = write_sysctl("/proc/sys/net/ipv4/neigh/" VLAN_IFACE "/gc_stale_time", "900"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.neigh." VLAN_IFACE ".gc_stale_time)")) + goto fail; + SYS(fail, "ip neigh add %s dev %s lladdr %s nud stale", + IPV4_VLAN_EGRESS_DST, VLAN_IFACE, DMAC); + SYS(fail, "ip neigh add %s dev %s lladdr %s nud stale", + IPV4_VLAN_GW, VLAN_IFACE, DMAC2); + + /* a VLAN on veth2 with a route in the tbid test table */ + SYS(fail, "ip link add link veth2 name %s type vlan id %d", + TBID_VLAN_IFACE, TBID_VLAN_ID); + SYS(fail, "ip link set dev %s up", TBID_VLAN_IFACE); + SYS(fail, "ip route add table 100 %s/32 dev %s", + IPV4_TBID_VLAN_DST, TBID_VLAN_IFACE); + + /* a locked-mtu route via the subinterface for the FRAG_NEEDED case */ + SYS(fail, "ip route add %s/32 dev %s mtu lock 1000", + IPV4_VLAN_MTU_DST, VLAN_IFACE); + return 0; fail: return -1; @@ -218,9 +588,16 @@ static int set_lookup_params(struct bpf_fib_lookup *params, memset(params, 0, sizeof(*params)); params->l4_protocol = IPPROTO_TCP; - params->ifindex = ifindex; + params->ifindex = test->iif ? if_nametoindex(test->iif) : ifindex; params->tbid = test->tbid; params->mark = test->mark; + params->tot_len = test->tot_len; + + /* h_vlan_proto/h_vlan_TCI union with tbid */ + if (test->lookup_flags & BPF_FIB_LOOKUP_VLAN_INPUT) { + params->h_vlan_proto = htons(test->vlan_proto); + params->h_vlan_TCI = htons(test->vlan_id); + } if (inet_pton(AF_INET6, test->daddr, params->ipv6_dst) == 1) { params->family = AF_INET6; @@ -298,7 +675,7 @@ void test_fib_lookup(void) struct nstoken *nstoken = NULL; struct __sk_buff skb = { }; struct fib_lookup *skel; - int prog_fd, err, ret, i; + int prog_fd, xdp_fd, err, ret, i; /* The test does not use the skb->data, so * use pkt_v6 for both v6 and v4 test. @@ -309,11 +686,16 @@ void test_fib_lookup(void) .ctx_in = &skb, .ctx_size_in = sizeof(skb), ); + LIBBPF_OPTS(bpf_test_run_opts, xdp_opts, + .data_in = &pkt_v6, + .data_size_in = sizeof(pkt_v6), + ); skel = fib_lookup__open_and_load(); if (!ASSERT_OK_PTR(skel, "skel open_and_load")) return; prog_fd = bpf_program__fd(skel->progs.fib_lookup); + xdp_fd = bpf_program__fd(skel->progs.fib_lookup_xdp); SYS(fail, "ip netns add %s", NS_TEST); @@ -343,6 +725,16 @@ void test_fib_lookup(void) if (!ASSERT_OK(err, "bpf_prog_test_run_opts")) continue; + /* + * BPF_FIB_LOOKUP_VLAN is XDP-only; the tc helper rejects it. + * These cases are exercised on the XDP path below. + */ + if (tests[i].lookup_flags & BPF_FIB_LOOKUP_VLAN) { + ASSERT_EQ(skel->bss->fib_lookup_ret, -EINVAL, + "tc rejects BPF_FIB_LOOKUP_VLAN"); + continue; + } + ASSERT_EQ(skel->bss->fib_lookup_ret, tests[i].expected_ret, "fib_lookup_ret"); @@ -352,6 +744,21 @@ void test_fib_lookup(void) if (tests[i].expected_dst) assert_dst_ip(fib_params, tests[i].expected_dst); + if (tests[i].expected_dev) + ASSERT_EQ(fib_params->ifindex, + if_nametoindex(tests[i].expected_dev), "ifindex"); + + if (tests[i].expected_mtu) + ASSERT_EQ(fib_params->mtu_result, tests[i].expected_mtu, + "mtu_result"); + + if (tests[i].check_vlan) { + ASSERT_EQ(fib_params->h_vlan_proto, + htons(tests[i].vlan_proto), "h_vlan_proto"); + ASSERT_EQ(fib_params->h_vlan_TCI, + htons(tests[i].vlan_id), "h_vlan_TCI"); + } + ret = memcmp(tests[i].dmac, fib_params->dmac, sizeof(tests[i].dmac)); if (!ASSERT_EQ(ret, 0, "dmac not match")) { char expected[18], actual[18]; @@ -361,17 +768,322 @@ void test_fib_lookup(void) printf("dmac expected %s actual %s ", expected, actual); } - // ensure tbid is zero'd out after fib lookup. - if (tests[i].lookup_flags & BPF_FIB_LOOKUP_DIRECT) { + /* + * ensure tbid is zero'd out after fib lookup. With + * BPF_FIB_LOOKUP_VLAN the union holds the packed vlan + * fields instead, so skip the check for those. + */ + if ((tests[i].lookup_flags & BPF_FIB_LOOKUP_DIRECT) && + !(tests[i].lookup_flags & BPF_FIB_LOOKUP_VLAN)) { if (!ASSERT_EQ(skel->bss->fib_params.tbid, 0, "expected fib_params.tbid to be zero")) goto fail; } } + /* + * Re-run the cases through bpf_xdp_fib_lookup(). test_run uses the + * current netns' loopback for ctx->rxq->dev, so dev_net() is NS_TEST + * and the lookup runs against its FIB. The path-independent results + * (return code, swapped ifindex, vlan tag, gateway) must match the skb + * path; the no-tot_len mtu_result is skb-specific and not rechecked. + */ + for (i = 0; i < ARRAY_SIZE(tests); i++) { + if (set_lookup_params(fib_params, &tests[i], skb.ifindex)) + continue; + + skel->bss->fib_lookup_ret = -1; + skel->bss->lookup_flags = tests[i].lookup_flags; + + err = bpf_prog_test_run_opts(xdp_fd, &xdp_opts); + if (!ASSERT_OK(err, "xdp test_run")) + continue; + + if (!ASSERT_EQ(skel->bss->fib_lookup_ret, tests[i].expected_ret, + "xdp fib_lookup_ret")) + printf("(xdp) %s\n", tests[i].desc); + + if (tests[i].expected_dev) + ASSERT_EQ(fib_params->ifindex, + if_nametoindex(tests[i].expected_dev), + "xdp ifindex"); + + if (tests[i].expected_dst) + assert_dst_ip(fib_params, tests[i].expected_dst); + + if (tests[i].check_vlan) { + ASSERT_EQ(fib_params->h_vlan_proto, + htons(tests[i].vlan_proto), "xdp h_vlan_proto"); + ASSERT_EQ(fib_params->h_vlan_TCI, + htons(tests[i].vlan_id), "xdp h_vlan_TCI"); + } + + ret = memcmp(tests[i].dmac, fib_params->dmac, sizeof(tests[i].dmac)); + ASSERT_EQ(ret, 0, "xdp dmac"); + + /* + * mtu_result from a tot_len lookup is the route mtu and is + * path-independent; the no-tot_len arm reads dev->mtu and is + * skb-only, so gate on tot_len + */ + if (tests[i].expected_mtu && tests[i].tot_len) + ASSERT_EQ(fib_params->mtu_result, tests[i].expected_mtu, + "xdp mtu_result"); + } + fail: if (nstoken) close_netns(nstoken); SYS_NOFAIL("ip netns del " NS_TEST); fib_lookup__destroy(skel); } + +#define NS_VLAN_A "fib_lookup_vlan_ns_a" +#define NS_VLAN_B "fib_lookup_vlan_ns_b" +#define IPV4_VLAN_NETNS_ADDR "10.66.0.1" +#define IPV4_VLAN_NETNS_DST "10.66.0.2" + +/* + * A VLAN device can be moved to another netns while staying registered + * on its parent. Neither direction may then cross the boundary: the + * egress flag must not publish the foreign parent's ifindex, and the + * input flag must fail closed rather than use a foreign ingress. + */ +void test_fib_lookup_vlan_netns(void) +{ + struct bpf_fib_lookup *fib_params; + struct nstoken *nstoken = NULL; + struct __sk_buff skb = { }; + struct fib_lookup *skel = NULL; + int prog_fd, xdp_fd, err, parent_idx, vlan_idx; + + LIBBPF_OPTS(bpf_test_run_opts, run_opts, + .data_in = &pkt_v6, + .data_size_in = sizeof(pkt_v6), + .ctx_in = &skb, + .ctx_size_in = sizeof(skb), + ); + LIBBPF_OPTS(bpf_test_run_opts, xdp_opts, + .data_in = &pkt_v6, + .data_size_in = sizeof(pkt_v6), + ); + + skel = fib_lookup__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel open_and_load")) + return; + prog_fd = bpf_program__fd(skel->progs.fib_lookup); + xdp_fd = bpf_program__fd(skel->progs.fib_lookup_xdp); + fib_params = &skel->bss->fib_params; + + SYS(fail, "ip netns add %s", NS_VLAN_A); + SYS(fail, "ip netns add %s", NS_VLAN_B); + + nstoken = open_netns(NS_VLAN_A); + if (!ASSERT_OK_PTR(nstoken, "open_netns(a)")) + goto fail; + + SYS(fail, "ip link add veth7 type veth peer name veth8"); + SYS(fail, "ip link set dev veth7 up"); + SYS(fail, "ip link add link veth7 name veth7.66 type vlan id 66"); + SYS(fail, "ip link set veth7.66 netns %s", NS_VLAN_B); + /* + * up it in B before the input lookup: the move closed it, and a + * down device fails the resolver on IFF_UP before reaching the + * netns check this subtest exists to pin + */ + SYS(fail, "ip -n %s link set dev veth7.66 up", NS_VLAN_B); + + parent_idx = if_nametoindex("veth7"); + if (!ASSERT_NEQ(parent_idx, 0, "if_nametoindex(veth7)")) + goto fail; + + /* + * input: the moved device is still in veth7's VLAN group, but it + * lives in another netns, so the lookup must fail closed + */ + skb.ifindex = parent_idx; + memset(fib_params, 0, sizeof(*fib_params)); + fib_params->family = AF_INET; + fib_params->l4_protocol = IPPROTO_TCP; + fib_params->ifindex = parent_idx; + fib_params->h_vlan_proto = htons(ETH_P_8021Q); + fib_params->h_vlan_TCI = htons(66); + if (!ASSERT_EQ(inet_pton(AF_INET, IPV4_VLAN_NETNS_DST, &fib_params->ipv4_dst), + 1, "inet_pton(dst)")) + goto fail; + + skel->bss->fib_lookup_ret = -1; + skel->bss->lookup_flags = BPF_FIB_LOOKUP_VLAN_INPUT | + BPF_FIB_LOOKUP_SKIP_NEIGH; + err = bpf_prog_test_run_opts(prog_fd, &run_opts); + if (!ASSERT_OK(err, "test_run(input)")) + goto fail; + ASSERT_EQ(skel->bss->fib_lookup_ret, BPF_FIB_LKUP_RET_NOT_FWDED, + "input across netns fails closed"); + ASSERT_EQ(fib_params->ifindex, parent_idx, "ifindex untouched"); + ASSERT_EQ(fib_params->h_vlan_TCI, htons(66), "tag untouched"); + + close_netns(nstoken); + nstoken = open_netns(NS_VLAN_B); + if (!ASSERT_OK_PTR(nstoken, "open_netns(b)")) + goto fail; + + /* + * egress: the fib result is the VLAN device here, but its parent + * is in the other netns, so the swap must not happen + */ + SYS(fail, "ip addr add %s/24 dev veth7.66", IPV4_VLAN_NETNS_ADDR); + err = write_sysctl("/proc/sys/net/ipv4/conf/veth7.66/forwarding", "1"); + if (!ASSERT_OK(err, "write_sysctl(forwarding)")) + goto fail; + + vlan_idx = if_nametoindex("veth7.66"); + if (!ASSERT_NEQ(vlan_idx, 0, "if_nametoindex(veth7.66)")) + goto fail; + + memset(fib_params, 0, sizeof(*fib_params)); + fib_params->family = AF_INET; + fib_params->l4_protocol = IPPROTO_TCP; + fib_params->ifindex = vlan_idx; + if (!ASSERT_EQ(inet_pton(AF_INET, IPV4_VLAN_NETNS_DST, &fib_params->ipv4_dst), + 1, "inet_pton(dst)") || + !ASSERT_EQ(inet_pton(AF_INET, IPV4_VLAN_NETNS_ADDR, &fib_params->ipv4_src), + 1, "inet_pton(src)")) + goto fail; + + skel->bss->fib_lookup_ret = -1; + skel->bss->lookup_flags = BPF_FIB_LOOKUP_VLAN | + BPF_FIB_LOOKUP_SKIP_NEIGH; + err = bpf_prog_test_run_opts(xdp_fd, &xdp_opts); + if (!ASSERT_OK(err, "test_run(egress)")) + goto fail; + ASSERT_EQ(skel->bss->fib_lookup_ret, BPF_FIB_LKUP_RET_VLAN_FAILURE, + "egress returns VLAN_FAILURE"); + ASSERT_EQ(fib_params->ifindex, vlan_idx, + "foreign parent not published"); + ASSERT_EQ(fib_params->h_vlan_TCI, 0, "vlan fields zero"); + +fail: + if (nstoken) + close_netns(nstoken); + SYS_NOFAIL("ip netns del " NS_VLAN_A); + SYS_NOFAIL("ip netns del " NS_VLAN_B); + fib_lookup__destroy(skel); +} + +#define REDIRECT_NPKTS 1000 +#define NS_REDIRECT "fib_lookup_redirect_ns" + +/* + * The egress flag exists so an XDP program can redirect to the physical + * parent. A redirect that lands on a VLAN device is dropped at + * xdp_do_flush(), because a VLAN device has no ndo_xdp_xmit. Drive real + * frames with BPF_F_TEST_XDP_LIVE_FRAMES, which runs the native + * xdp_do_redirect() + xdp_do_flush() path: a reducible VLAN egress + * resolves to veth1 and is delivered to its peer veth2, while a QinQ + * egress returns VLAN_FAILURE and is passed to the stack instead of + * redirected to a device that would silently drop it. + */ +void test_fib_lookup_vlan_redirect(void) +{ + int redirect_fd, err, veth1_idx, veth2_idx = -1; + struct bpf_fib_lookup *fib_params; + struct nstoken *nstoken = NULL; + struct fib_lookup *skel = NULL; + bool xdp_attached = false; + + LIBBPF_OPTS(bpf_test_run_opts, lf_opts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .flags = BPF_F_TEST_XDP_LIVE_FRAMES, + .repeat = REDIRECT_NPKTS, + ); + + skel = fib_lookup__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel open_and_load")) + return; + redirect_fd = bpf_program__fd(skel->progs.fib_lookup_redirect); + fib_params = &skel->bss->fib_params; + + SYS(fail, "ip netns add %s", NS_REDIRECT); + nstoken = open_netns(NS_REDIRECT); + if (!ASSERT_OK_PTR(nstoken, "open_netns")) + goto fail; + if (setup_netns()) + goto fail; + + veth1_idx = if_nametoindex("veth1"); + veth2_idx = if_nametoindex("veth2"); + if (!ASSERT_NEQ(veth1_idx, 0, "if_nametoindex(veth1)") || + !ASSERT_NEQ(veth2_idx, 0, "if_nametoindex(veth2)")) + goto fail; + + /* + * A redirect to veth1 is delivered to its peer veth2. veth_xdp_xmit() + * only accepts the frame if veth2's NAPI is up, which on veth means + * veth2 carries an XDP program; xdp_count tallies what arrives. + */ + err = bpf_xdp_attach(veth2_idx, bpf_program__fd(skel->progs.xdp_count), + XDP_FLAGS_DRV_MODE, NULL); + if (!ASSERT_OK(err, "attach xdp_count on veth2")) + goto fail; + xdp_attached = true; + + /* reducible VLAN egress: resolves to the physical parent veth1 */ + memset(fib_params, 0, sizeof(*fib_params)); + fib_params->family = AF_INET; + fib_params->l4_protocol = IPPROTO_TCP; + fib_params->ifindex = veth1_idx; + if (!ASSERT_EQ(inet_pton(AF_INET, IPV4_IFACE_ADDR, &fib_params->ipv4_src), + 1, "inet_pton(src)") || + !ASSERT_EQ(inet_pton(AF_INET, IPV4_VLAN_EGRESS_DST, &fib_params->ipv4_dst), + 1, "inet_pton(reducible dst)")) + goto fail; + skel->bss->lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH; + skel->bss->redirected = 0; + skel->bss->passed = 0; + skel->bss->delivered = 0; + + err = bpf_prog_test_run_opts(redirect_fd, &lf_opts); + if (!ASSERT_OK(err, "test_run(reducible egress)")) + goto fail; + ASSERT_EQ(skel->bss->redirected, REDIRECT_NPKTS, "reducible egress redirected"); + ASSERT_EQ(skel->bss->passed, 0, "reducible egress not passed"); + ASSERT_GT(skel->bss->delivered, 0, "reducible egress delivered to veth2"); + + /* + * QinQ egress: not reducible, so the lookup returns VLAN_FAILURE and + * the program passes the frame instead of redirecting to the inner + * VLAN device. redirected == 0 is the assertion that matters: the + * program did not redirect to a device that would drop the frame at + * xdp_do_flush(). veth2's delivered count is not checked here, since + * a passed frame can still reach veth2 through the stack's forwarding + * path, which is unrelated to the redirect under test. + */ + memset(fib_params, 0, sizeof(*fib_params)); + fib_params->family = AF_INET; + fib_params->l4_protocol = IPPROTO_TCP; + fib_params->ifindex = veth1_idx; + if (!ASSERT_EQ(inet_pton(AF_INET, IPV4_IFACE_ADDR, &fib_params->ipv4_src), + 1, "inet_pton(src)") || + !ASSERT_EQ(inet_pton(AF_INET, IPV4_QINQ_DST, &fib_params->ipv4_dst), + 1, "inet_pton(qinq dst)")) + goto fail; + skel->bss->lookup_flags = BPF_FIB_LOOKUP_VLAN | BPF_FIB_LOOKUP_SKIP_NEIGH; + skel->bss->redirected = 0; + skel->bss->passed = 0; + + err = bpf_prog_test_run_opts(redirect_fd, &lf_opts); + if (!ASSERT_OK(err, "test_run(qinq egress)")) + goto fail; + ASSERT_EQ(skel->bss->passed, REDIRECT_NPKTS, "qinq egress passed"); + ASSERT_EQ(skel->bss->redirected, 0, "qinq egress not redirected"); + +fail: + if (xdp_attached) + bpf_xdp_detach(veth2_idx, XDP_FLAGS_DRV_MODE, NULL); + if (nstoken) + close_netns(nstoken); + SYS_NOFAIL("ip netns del " NS_REDIRECT); + fib_lookup__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/fib_lookup.c b/tools/testing/selftests/bpf/progs/fib_lookup.c index 7b5dd2214ff4..36b7218d9ae2 100644 --- a/tools/testing/selftests/bpf/progs/fib_lookup.c +++ b/tools/testing/selftests/bpf/progs/fib_lookup.c @@ -4,7 +4,11 @@ #include #include #include +#include +#include +#include #include +#include struct bpf_fib_lookup fib_params = {}; int fib_lookup_ret = 0; @@ -19,4 +23,57 @@ int fib_lookup(struct __sk_buff *skb) return TC_ACT_SHOT; } +SEC("xdp") +int fib_lookup_xdp(struct xdp_md *ctx) +{ + fib_lookup_ret = bpf_fib_lookup(ctx, &fib_params, sizeof(fib_params), + lookup_flags); + + return XDP_DROP; +} + +int redirected = 0; +int passed = 0; +int delivered = 0; + +SEC("xdp") +int fib_lookup_redirect(struct xdp_md *ctx) +{ + struct bpf_fib_lookup params = fib_params; + long ret; + + ret = bpf_fib_lookup(ctx, ¶ms, sizeof(params), lookup_flags); + if (ret == BPF_FIB_LKUP_RET_SUCCESS) { + redirected++; + return bpf_redirect(params.ifindex, 0); + } + + passed++; + return XDP_PASS; +} + +SEC("xdp") +int xdp_count(struct xdp_md *ctx) +{ + void *data = (void *)(long)ctx->data; + void *data_end = (void *)(long)ctx->data_end; + struct ethhdr *eth = data; + struct iphdr *iph; + + /* + * count only the test's TCP frames: the netns has live + * link-local traffic (DAD, MLD) that would satisfy a bare + * counter + */ + if ((void *)(eth + 1) > data_end || + eth->h_proto != bpf_htons(ETH_P_IP)) + return XDP_DROP; + iph = (void *)(eth + 1); + if ((void *)(iph + 1) > data_end || iph->protocol != IPPROTO_TCP) + return XDP_DROP; + + delivered++; + return XDP_DROP; +} + char _license[] SEC("license") = "GPL"; From f68df52fbad0928f4686f734827e5fb6b20386ad Mon Sep 17 00:00:00 2001 From: Jianlin Shi Date: Mon, 13 Jul 2026 10:49:01 +0800 Subject: [PATCH 083/373] docs: bpf: Document BPF_RB_OVERWRITE_POS in bpf_ringbuf_query BPF_RB_OVERWRITE_POS is supported by bpf_ringbuf_query() but was missing from the helper documentation. Add it to the flags list in both the kernel UAPI header and its tools/ mirror. Signed-off-by: Jianlin Shi Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/tencent_22134645443B75ED907D2A85A47AD554A709@qq.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/uapi/linux/bpf.h | 1 + tools/include/uapi/linux/bpf.h | 1 + 2 files changed, 2 insertions(+) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 005038fbeea4..ffd96e8b920b 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -4735,6 +4735,7 @@ union bpf_attr { * * **BPF_RB_RING_SIZE**: The size of ring buffer. * * **BPF_RB_CONS_POS**: Consumer position (can wrap around). * * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around). + * * **BPF_RB_OVERWRITE_POS**: Overwrite position (can wrap around). * * Data returned is just a momentary snapshot of actual values * and could be inaccurate, so this facility should be used to diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 005038fbeea4..ffd96e8b920b 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -4735,6 +4735,7 @@ union bpf_attr { * * **BPF_RB_RING_SIZE**: The size of ring buffer. * * **BPF_RB_CONS_POS**: Consumer position (can wrap around). * * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around). + * * **BPF_RB_OVERWRITE_POS**: Overwrite position (can wrap around). * * Data returned is just a momentary snapshot of actual values * and could be inaccurate, so this facility should be used to From c28cbef2f8986c392faf6ca94bb2088548ea6964 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:41 -0700 Subject: [PATCH 084/373] bpf: Remove dynptr check in check_stack_range_initialized() For a MEM_UNINIT ("raw mode") helper argument, check_stack_range_initialized() open-coded a scan that rejected any STACK_DYNPTR slot in the range with "potential write to dynptr". This duplicated, and was stricter than, the handling that runs when the buffer is actually marked initialized. check_helper_call() later replays the write byte by byte via check_mem_access(), which goes through destroy_if_dynptr_stack_slot(), which rejects overwritting a referenced dynptr. Therefore drop the redundant scan and rely on check_mem_access(). Signed-off-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260715064047.1793790-2-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 25 ------------------- .../testing/selftests/bpf/progs/dynptr_fail.c | 4 +-- 2 files changed, 2 insertions(+), 27 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 03e2202cca13..7dd961ede88d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6733,31 +6733,6 @@ static int check_stack_range_initialized( } if (meta && meta->raw_mode) { - /* Ensure we won't be overwriting dynptrs when simulating byte - * by byte access in check_helper_call using meta.access_size. - * This would be a problem if we have a helper in the future - * which takes: - * - * helper(uninit_mem, len, dynptr) - * - * Now, uninint_mem may overlap with dynptr pointer. Hence, it - * may end up writing to dynptr itself when touching memory from - * arg 1. This can be relaxed on a case by case basis for known - * safe cases, but reject due to the possibilitiy of aliasing by - * default. - */ - for (i = min_off; i < max_off + access_size; i++) { - int stack_off = -i - 1; - - spi = bpf_get_spi(i); - /* raw_mode may write past allocated_stack */ - if (state->allocated_stack <= stack_off) - continue; - if (state->stack[spi].slot_type[stack_off % BPF_REG_SIZE] == STACK_DYNPTR) { - verbose(env, "potential write to dynptr at off=%d disallowed\n", i); - return -EACCES; - } - } meta->access_size = access_size; meta->regno = reg_from_argno(argno); return 0; diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c index 344fb2aa0813..94489ac64da8 100644 --- a/tools/testing/selftests/bpf/progs/dynptr_fail.c +++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c @@ -1112,7 +1112,7 @@ int dynptr_overwrite_ref(void *ctx) /* Reject writes to dynptr slot from bpf_dynptr_read */ SEC("?raw_tp") -__failure __msg("potential write to dynptr at off=-16") +__failure __msg("cannot overwrite referenced dynptr") int dynptr_read_into_slot(void *ctx) { union { @@ -1558,7 +1558,7 @@ int BPF_PROG(skb_invalid_ctx_fexit, void *skb) /* Reject writes to dynptr slot for uninit arg */ SEC("?raw_tp") -__failure __msg("potential write to dynptr at off=-16") +__failure __msg("cannot overwrite referenced dynptr") int uninit_write_into_slot(void *ctx) { struct { From 92ec8b1b6b24381611600162536b8b5b6a9e7323 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:42 -0700 Subject: [PATCH 085/373] bpf: Factor out raw_mode-related fields in bpf_call_arg_meta To prepare for unifying the helper and kfunc call_arg_meta, group the scattered MEM_UNINIT ("raw") memory argument fields (raw_mode, regno and access_size) into a new struct arg_raw_mem_desc. The intention is to make it clear about when these are set and used instead of fields with overly generic names. Identify the raw argument once, up front, in check_raw_mode_ok() (like check_proto_release_reg() does for release_regno), recording its regno. check_stack_range_initialized() now recognizes the raw buffer by matching that regno, so the separate raw_mode flag is no longer needed, and the per-argument "meta->raw_mode = arg_type & MEM_UNINIT" assignments in check_func_arg() go away with it. A raw memory argument can be tagged either ARG_PTR_TO_MEM | MEM_UNINIT or ARG_PTR_TO_MAP_VALUE | MEM_UNINIT (the output buffer of bpf_map_pop_elem() and bpf_map_peek_elem()). Either may be passed as a PTR_TO_STACK, which reaches check_stack_range_initialized() through check_helper_mem_access(), so both must be treated as raw. Extend arg_type_is_raw_mem() to match the map value case as well; otherwise check_raw_mode_ok() would not record the regno for it and an uninitialized stack buffer passed to those helpers would be wrongly rejected for programs without CAP_PERFMON. No functional change intended. This patch does not enable raw_mode memory access for kfunc (i.e., uninit stack will not be allowed to be passed to kfunc for unprivileged programs). Existing kfuncs with arguments tagged with __uninit are either priviledged or dynptr kfuncs, which take another path to make sure the access is checked by check_mem_access(). Signed-off-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260715064047.1793790-3-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 9 ++++++ kernel/bpf/verifier.c | 58 +++++++++++++++++------------------- 2 files changed, 36 insertions(+), 31 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 317e99b9acc0..d3592f5b8621 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1464,6 +1464,15 @@ struct ref_obj_desc { u8 cnt; }; +/* + * A memory argument a call fills in. The verifier allows the stack to be uninitialized if + * the range is a known constant. Stack slots are marked as STACK_MISC by check_mem_access(). + */ +struct arg_raw_mem_desc { + u8 regno; + int size; +}; + struct bpf_kfunc_call_arg_meta { /* In parameters */ struct btf *btf; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7dd961ede88d..1a41f99a9133 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -256,11 +256,9 @@ struct bpf_call_arg_meta { struct bpf_map_desc map; struct bpf_dynptr_desc dynptr; struct ref_obj_desc ref_obj; - bool raw_mode; + struct arg_raw_mem_desc arg_raw_mem; bool pkt_access; u8 release_regno; - int regno; - int access_size; int mem_size; u64 msize_max_value; int func_id; @@ -6690,6 +6688,8 @@ static int check_stack_range_initialized( * but BTF based global subprog validation isn't accurate enough. */ bool allow_poison = access_size < 0 || clobber; + /* The call will initialize the memory; uninitialized stack allowed */ + bool raw_mode = meta && meta->arg_raw_mem.regno == reg_from_argno(argno); access_size = abs(access_size); @@ -6725,16 +6725,14 @@ static int check_stack_range_initialized( * helper return since specific bounds are unknown what may * cause uninitialized stack leaking. */ - if (meta && meta->raw_mode) - meta = NULL; + raw_mode = false; min_off = reg_smin(reg) + off; max_off = reg_smax(reg) + off; } - if (meta && meta->raw_mode) { - meta->access_size = access_size; - meta->regno = reg_from_argno(argno); + if (raw_mode) { + meta->arg_raw_mem.size = access_size; return 0; } @@ -7727,7 +7725,13 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) static bool arg_type_is_raw_mem(enum bpf_arg_type type) { - return base_type(type) == ARG_PTR_TO_MEM && + /* + * A map value output buffer (e.g. bpf_map_pop_elem) is also a raw + * (uninitialized) memory argument, and like ARG_PTR_TO_MEM it may be + * passed as a PTR_TO_STACK that reaches check_stack_range_initialized(). + */ + return (base_type(type) == ARG_PTR_TO_MEM || + base_type(type) == ARG_PTR_TO_MAP_VALUE) && type & MEM_UNINIT; } @@ -8403,7 +8407,6 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, verifier_bug(env, "invalid map_ptr to access map->value"); return -EFAULT; } - meta->raw_mode = arg_type & MEM_UNINIT; err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size, arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); @@ -8446,7 +8449,6 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, /* The access to this pointer is only checked when we hit the * next is_mem_size argument below. */ - meta->raw_mode = arg_type & MEM_UNINIT; if (arg_type & MEM_FIXED_SIZE) { err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg], arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, @@ -8797,26 +8799,19 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, return -EINVAL; } -static bool check_raw_mode_ok(const struct bpf_func_proto *fn) +static bool check_raw_mode_ok(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) { - int count = 0; + int i; - if (arg_type_is_raw_mem(fn->arg1_type)) - count++; - if (arg_type_is_raw_mem(fn->arg2_type)) - count++; - if (arg_type_is_raw_mem(fn->arg3_type)) - count++; - if (arg_type_is_raw_mem(fn->arg4_type)) - count++; - if (arg_type_is_raw_mem(fn->arg5_type)) - count++; + for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { + if (!arg_type_is_raw_mem(fn->arg_type[i])) + continue; + if (meta->arg_raw_mem.regno) + return false; + meta->arg_raw_mem.regno = i + 1; + } - /* We only support one arg being in raw mode at the moment, - * which is sufficient for the helper functions we have - * right now. - */ - return count <= 1; + return true; } static bool check_args_pair_invalid(const struct bpf_func_proto *fn, int arg) @@ -8906,7 +8901,7 @@ static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_ static int check_func_proto(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) { - return check_raw_mode_ok(fn) && + return check_raw_mode_ok(fn, meta) && check_arg_pair_ok(fn) && check_mem_arg_rw_flag_ok(fn) && check_proto_release_reg(fn, meta) && @@ -10303,8 +10298,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn /* Mark slots with STACK_MISC in case of raw mode, stack offset * is inferred from register state. */ - for (i = 0; i < meta.access_size; i++) { - err = check_mem_access(env, insn_idx, regs + meta.regno, argno_from_reg(meta.regno), i, BPF_B, + for (i = 0; i < meta.arg_raw_mem.size; i++) { + err = check_mem_access(env, insn_idx, regs + meta.arg_raw_mem.regno, + argno_from_reg(meta.arg_raw_mem.regno), i, BPF_B, BPF_WRITE, -1, false, false); if (err) return err; From 77a4974c17493b0e0bcd5010dc2ec9ad749d1a07 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:43 -0700 Subject: [PATCH 086/373] bpf: Pass argno to callees in check_func_arg() instead of argno_from_reg(regno) check_func_arg() only ever handles register arguments (the caller loops over the first MAX_BPF_FUNC_REG_ARGS arguments), so a single argno_t built from the register number identifies the argument for every callee. Remove the duplicated argno_from_reg() calls to simplify check_func_arg(). 'regno' is still kept for the few places that need the raw register number directly (register reads, verbose R%d messages) and for referring to the neighbouring size/memory argument in the ARG_CONST_SIZE{,_OR_ZERO} cases. No functional change intended. Suggested-by: Eduard Zingerman Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260715064047.1793790-4-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1a41f99a9133..529ad0b4fcbd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8275,7 +8275,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, u32 regno = BPF_REG_1 + arg; struct bpf_reg_state *reg = reg_state(env, regno); enum bpf_arg_type arg_type = fn->arg_type[arg]; - argno_t argno = argno_from_arg(arg + 1); + argno_t argno = argno_from_reg(regno); enum bpf_reg_type type = reg->type; u32 *arg_btf_id = NULL; u32 key_size; @@ -8320,11 +8320,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) arg_btf_id = fn->arg_btf_id[arg]; - err = check_reg_type(env, reg, argno_from_reg(regno), arg_type, arg_btf_id, meta); + err = check_reg_type(env, reg, argno, arg_type, arg_btf_id, meta); if (err) return err; - err = check_func_arg_reg_off(env, reg, argno_from_reg(regno), arg_type); + err = check_func_arg_reg_off(env, reg, argno, arg_type); if (err) return err; @@ -8381,7 +8381,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EFAULT; } key_size = meta->map.ptr->key_size; - err = check_helper_mem_access(env, reg, argno_from_reg(regno), key_size, BPF_READ, false, NULL); + err = check_helper_mem_access(env, reg, argno, key_size, BPF_READ, false, NULL); if (err) return err; if (can_elide_value_nullness(meta->map.ptr)) { @@ -8407,7 +8407,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, verifier_bug(env, "invalid map_ptr to access map->value"); return -EFAULT; } - err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size, + err = check_helper_mem_access(env, reg, argno, meta->map.ptr->value_size, arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); break; @@ -8425,11 +8425,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EACCES; } if (meta->func_id == BPF_FUNC_spin_lock) { - err = process_spin_lock(env, reg, argno_from_reg(regno), PROCESS_SPIN_LOCK); + err = process_spin_lock(env, reg, argno, PROCESS_SPIN_LOCK); if (err) return err; } else if (meta->func_id == BPF_FUNC_spin_unlock) { - err = process_spin_lock(env, reg, argno_from_reg(regno), 0); + err = process_spin_lock(env, reg, argno, 0); if (err) return err; } else { @@ -8438,7 +8438,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, } break; case ARG_PTR_TO_TIMER: - err = process_timer_helper(env, reg, argno_from_reg(regno), meta); + err = process_timer_helper(env, reg, argno, meta); if (err) return err; break; @@ -8450,7 +8450,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, * next is_mem_size argument below. */ if (arg_type & MEM_FIXED_SIZE) { - err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg], + err = check_helper_mem_access(env, reg, argno, fn->arg_size[arg], arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); if (err) @@ -8460,21 +8460,19 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, } break; case ARG_CONST_SIZE: - err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), - argno_from_reg(regno), - fn->arg_type[arg - 1] & MEM_WRITE ? - BPF_WRITE : BPF_READ, + err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, + argno_from_reg(regno - 1), argno, + fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); break; case ARG_CONST_SIZE_OR_ZERO: - err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), - argno_from_reg(regno), - fn->arg_type[arg - 1] & MEM_WRITE ? - BPF_WRITE : BPF_READ, + err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, + argno_from_reg(regno - 1), argno, + fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, true, meta); break; case ARG_PTR_TO_DYNPTR: - err = process_dynptr_func(env, reg, argno_from_reg(regno), insn_idx, arg_type, &meta->ref_obj, + err = process_dynptr_func(env, reg, argno, insn_idx, arg_type, &meta->ref_obj, &meta->dynptr); if (err) return err; @@ -8492,7 +8490,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, break; case ARG_PTR_TO_CONST_STR: { - err = check_arg_const_str(env, reg, argno_from_reg(regno)); + err = check_arg_const_str(env, reg, argno); if (err) return err; break; From 8faaa93b9f6a279472cd5490030151f1635292d2 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:44 -0700 Subject: [PATCH 087/373] bpf: Unify helper and kfunc allocation-size argument handling The constant "size of the PTR_TO_MEM returned in R0" argument is handled by both helpers (ARG_CONST_ALLOC_SIZE_OR_ZERO) and kfuncs (__rdonly_buf_size / __rdwr_buf_size), each with its own meta field (meta->mem_size, meta->r0_size) and duplicated validation. Add struct arg_alloc_mem_desc and a shared process_const_alloc_mem_size(), and replace both fields with meta->arg_alloc_mem. The desc records presence with a 'found' flag instead of using a non-zero size as the sentinel. This also fixes a pre-existing bug on the kfunc return path: "no size argument" was tested as r0_size == 0, so an explicit __rdonly_buf_size/__rdwr_buf_size of 0 was treated as absent and fell through to btf_resolve_size(), giving R0 the size of the pointed-to return type instead of 0. With 'found', an explicit zero size is honored and btf_resolve_size() is used only when no size argument was passed. The size is stored in a u32, matching regs[R0].mem_size. The U32_MAX check now apply to both helper and kfunc through process_const_alloc_mem_size(). Fold bpf_session_cookie return size assignment into current kfunc return size resolution path. Note that verifier saves kfunc return size through r0_size instead of mem_size. The later has no active readers so remove it. Signed-off-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260715064047.1793790-5-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 9 +- kernel/bpf/verifier.c | 87 ++++++++++--------- .../selftests/bpf/prog_tests/kfunc_call.c | 2 +- 3 files changed, 55 insertions(+), 43 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index d3592f5b8621..e3eda72cbf67 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1473,6 +1473,12 @@ struct arg_raw_mem_desc { int size; }; +/* Size of PTR_TO_MEM returned, taken from a constant allocation-size argument */ +struct ret_mem_desc { + u32 size; + bool found; +}; + struct bpf_kfunc_call_arg_meta { /* In parameters */ struct btf *btf; @@ -1484,7 +1490,6 @@ struct bpf_kfunc_call_arg_meta { u8 release_regno; bool r0_rdonly; u32 ret_btf_id; - u64 r0_size; u32 subprogno; struct { u64 value; @@ -1519,7 +1524,7 @@ struct bpf_kfunc_call_arg_meta { struct bpf_map_desc map; struct bpf_dynptr_desc dynptr; struct ref_obj_desc ref_obj; - u64 mem_size; + struct ret_mem_desc ret_mem; }; int bpf_get_helper_proto(struct bpf_verifier_env *env, int func_id, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 529ad0b4fcbd..463b49df4ff9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -257,9 +257,9 @@ struct bpf_call_arg_meta { struct bpf_dynptr_desc dynptr; struct ref_obj_desc ref_obj; struct arg_raw_mem_desc arg_raw_mem; + struct ret_mem_desc ret_mem; bool pkt_access; u8 release_regno; - int mem_size; u64 msize_max_value; int func_id; struct btf *btf; @@ -6974,6 +6974,40 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg return err; } +static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf_reg_state *reg, + argno_t argno, struct ret_mem_desc *ret_mem) +{ + int regno = reg_from_argno(argno); + int err; + + if (ret_mem->found) { + verifier_bug(env, "only one allocation size argument permitted"); + return -EFAULT; + } + + if (!tnum_is_const(reg->var_off)) { + verbose(env, "%s is not a const\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + if (reg->var_off.value > U32_MAX) { + verbose(env, "%s allocation size exceeds u32 max\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + if (regno >= 0) + err = mark_chain_precision(env, regno); + else + err = mark_stack_arg_precision(env, arg_idx_from_argno(argno)); + if (err) + return err; + + ret_mem->size = reg->var_off.value; + ret_mem->found = true; + + return 0; +} + static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno) { @@ -8478,13 +8512,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return err; break; case ARG_CONST_ALLOC_SIZE_OR_ZERO: - if (!tnum_is_const(reg->var_off)) { - verbose(env, "R%d is not a known constant'\n", - regno); - return -EACCES; - } - meta->mem_size = reg->var_off.value; - err = mark_chain_precision(env, regno); + err = process_const_alloc_mem_size(env, reg, argno, &meta->ret_mem); if (err) return err; break; @@ -10516,7 +10544,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn case RET_PTR_TO_MEM: mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_MEM | ret_flag; - regs[BPF_REG_0].mem_size = meta.mem_size; + regs[BPF_REG_0].mem_size = meta.ret_mem.size; break; case RET_PTR_TO_MEM_OR_BTF_ID: { @@ -12066,28 +12094,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } if (is_ret_buf_sz) { - if (meta->r0_size) { - verbose(env, "2 or more rdonly/rdwr_buf_size parameters for kfunc"); - return -EINVAL; - } - - if (!tnum_is_const(reg->var_off)) { - verbose(env, "%s is not a const\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - - meta->r0_size = reg->var_off.value; - if (meta->r0_size > U32_MAX) { - verbose(env, "%s rdonly/rdwr_buf_size exceeds u32 max\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - if (regno >= 0) - ret = mark_chain_precision(env, regno); - else - ret = mark_stack_arg_precision(env, i); - if (ret) + ret = process_const_alloc_mem_size(env, reg, argno, &meta->ret_mem); + if (ret < 0) return ret; } continue; @@ -13066,11 +13074,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } - if (meta.func_id == special_kfunc_list[KF_bpf_session_cookie]) { - meta.r0_size = sizeof(u64); - meta.r0_rdonly = false; - } - if (is_bpf_wq_set_callback_kfunc(meta.func_id)) { err = push_callback_call(env, insn, insn_idx, meta.subprogno, set_timer_callback_state); @@ -13203,15 +13206,19 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, /* kfunc returning 'void *' is equivalent to returning scalar */ mark_reg_unknown(env, regs, BPF_REG_0); } else if (!__btf_type_is_struct(ptr_type)) { - if (!meta.r0_size) { + if (!meta.ret_mem.found) { __u32 sz; if (!IS_ERR(btf_resolve_size(desc_btf, ptr_type, &sz))) { - meta.r0_size = sz; + meta.ret_mem.found = true; + meta.ret_mem.size = sz; meta.r0_rdonly = true; } + + if (meta.func_id == special_kfunc_list[KF_bpf_session_cookie]) + meta.r0_rdonly = false; } - if (!meta.r0_size) { + if (!meta.ret_mem.found) { ptr_type_name = btf_name_by_offset(desc_btf, ptr_type->name_off); verbose(env, @@ -13224,7 +13231,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, mark_reg_known_zero(env, regs, BPF_REG_0); regs[BPF_REG_0].type = PTR_TO_MEM; - regs[BPF_REG_0].mem_size = meta.r0_size; + regs[BPF_REG_0].mem_size = meta.ret_mem.size; if (meta.r0_rdonly) regs[BPF_REG_0].type |= MEM_RDONLY; diff --git a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c index 67a30bf69509..c9fce95d220e 100644 --- a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c +++ b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c @@ -66,7 +66,7 @@ static struct kfunc_test_params kfunc_tests[] = { TC_FAIL(kfunc_call_test_get_mem_fail_rdonly, 0, "R0 cannot write into rdonly_mem"), TC_FAIL(kfunc_call_test_get_mem_fail_use_after_free, 0, "invalid mem access 'scalar'"), TC_FAIL(kfunc_call_test_get_mem_fail_oob, 0, "min value is outside of the allowed memory range"), - TC_FAIL(kfunc_call_test_get_mem_fail_oversized, 0, "rdonly/rdwr_buf_size exceeds u32 max"), + TC_FAIL(kfunc_call_test_get_mem_fail_oversized, 0, "allocation size exceeds u32 max"), TC_FAIL(kfunc_call_test_get_mem_fail_not_const, 0, "is not a const"), TC_FAIL(kfunc_call_test_mem_acquire_fail, 0, "acquire kernel function does not return PTR_TO_BTF_ID"), TC_FAIL(kfunc_call_test_pointer_arg_type_mismatch, 0, "R1 expected pointer to ctx, but got scalar"), From 1e63cd6be0557fa1fc57522ec1f681a1dde11078 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:45 -0700 Subject: [PATCH 088/373] selftests/bpf: Test kfunc returning zero-sized allocation buffer Add a test passing an explicit rdwr_buf_size of 0 to bpf_kfunc_call_test_get_rdwr_mem() and then reading the returned R0. R0 should be a zero-sized PTR_TO_MEM, so the access must be rejected with "min value is outside of the allowed memory range". This covers the pre-existing bug where a zero size argument was treated as "no size argument": the verifier fell through to btf_resolve_size() and sized R0 after the pointed-to return type, wrongly allowing the read. Suggested-by: Eduard Zingerman Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260715064047.1793790-6-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/kfunc_call.c | 1 + .../selftests/bpf/progs/kfunc_call_fail.c | 27 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c index c9fce95d220e..7af5560f2a08 100644 --- a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c +++ b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c @@ -66,6 +66,7 @@ static struct kfunc_test_params kfunc_tests[] = { TC_FAIL(kfunc_call_test_get_mem_fail_rdonly, 0, "R0 cannot write into rdonly_mem"), TC_FAIL(kfunc_call_test_get_mem_fail_use_after_free, 0, "invalid mem access 'scalar'"), TC_FAIL(kfunc_call_test_get_mem_fail_oob, 0, "min value is outside of the allowed memory range"), + TC_FAIL(kfunc_call_test_get_mem_fail_zero_size, 0, "min value is outside of the allowed memory range"), TC_FAIL(kfunc_call_test_get_mem_fail_oversized, 0, "allocation size exceeds u32 max"), TC_FAIL(kfunc_call_test_get_mem_fail_not_const, 0, "is not a const"), TC_FAIL(kfunc_call_test_mem_acquire_fail, 0, "acquire kernel function does not return PTR_TO_BTF_ID"), diff --git a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c index 6144ce3ff0b2..64b6a0b0ab1c 100644 --- a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c +++ b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c @@ -103,6 +103,33 @@ int kfunc_call_test_get_mem_fail_oob(struct __sk_buff *skb) return ret; } +SEC("?tc") +int kfunc_call_test_get_mem_fail_zero_size(struct __sk_buff *skb) +{ + struct prog_test_ref_kfunc *pt; + unsigned long s = 0; + int *p = NULL; + int ret = 0; + + pt = bpf_kfunc_call_test_acquire(&s); + if (pt) { + /* + * An explicit rdwr_buf_size of 0 gives R0 a zero-sized buffer, + * so any access is out of bounds, hence -EACCES. Previously the + * verifier treated a zero size as "no size argument" and sized + * R0 after the pointed-to return type, wrongly allowing the read. + */ + p = bpf_kfunc_call_test_get_rdwr_mem(pt, 0); + if (p) + ret = p[0]; + else + ret = -1; + + bpf_kfunc_call_test_release(pt); + } + return ret; +} + SEC("?tc") int kfunc_call_test_get_mem_fail_oversized(struct __sk_buff *skb) { From d55149ff8c856c88cedc36557dadc6b5363b428d Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:46 -0700 Subject: [PATCH 089/373] bpf: Drop redundant pkt_access from bpf_call_arg_meta meta->pkt_access is only ever a copy of fn->pkt_access, assigned once in check_helper_call() and read back in may_access_direct_pkt_data(). Have may_access_direct_pkt_data() take the bpf_func_proto and read fn->pkt_access directly, and drop the meta field along with its assignment. The only non-NULL caller, check_func_arg(), already has fn in scope; the remaining callers pass NULL and are unaffected. No functional change intended. Suggested-by: Eduard Zingerman Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260715064047.1793790-7-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 463b49df4ff9..37127ae1ff28 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -258,7 +258,6 @@ struct bpf_call_arg_meta { struct ref_obj_desc ref_obj; struct arg_raw_mem_desc arg_raw_mem; struct ret_mem_desc ret_mem; - bool pkt_access; u8 release_regno; u64 msize_max_value; int func_id; @@ -4673,7 +4672,7 @@ static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state * } static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, - const struct bpf_call_arg_meta *meta, + const struct bpf_func_proto *fn, enum bpf_access_type t) { enum bpf_prog_type prog_type = resolve_prog_type(env->prog); @@ -4697,8 +4696,8 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, case BPF_PROG_TYPE_LWT_XMIT: case BPF_PROG_TYPE_SK_SKB: case BPF_PROG_TYPE_SK_MSG: - if (meta) - return meta->pkt_access; + if (fn) + return fn->pkt_access; env->seen_direct_write = true; return true; @@ -8332,7 +8331,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, } if (type_is_pkt_pointer(type) && - !may_access_direct_pkt_data(env, meta, BPF_READ)) { + !may_access_direct_pkt_data(env, fn, BPF_READ)) { verbose(env, "helper access to the packet is not allowed\n"); return -EACCES; } @@ -10285,7 +10284,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn } memset(&meta, 0, sizeof(meta)); - meta.pkt_access = fn->pkt_access; err = check_func_proto(fn, &meta); if (err) { From bf9c1b911f4db6fa5fe088c32f1de7ee1650eee9 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Tue, 14 Jul 2026 23:40:47 -0700 Subject: [PATCH 090/373] bpf: Unify helper and kfunc call argument meta Helper and kfunc argument checking carried two separate meta structs: the verifier-local struct bpf_call_arg_meta and bpf_kfunc_call_arg_meta. Merge them into a single struct bpf_call_arg_meta. This is groundwork for sharing argument checking between helpers and kfuncs. While merging, drop the btf_id field from the helper meta since it is never used. No functional change. Signed-off-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260715064047.1793790-8-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 38 +++++++++------ kernel/bpf/cfg.c | 2 +- kernel/bpf/verifier.c | 94 +++++++++++++++--------------------- 3 files changed, 62 insertions(+), 72 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index e3eda72cbf67..682c2cd3b844 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1479,18 +1479,23 @@ struct ret_mem_desc { bool found; }; -struct bpf_kfunc_call_arg_meta { - /* In parameters */ +struct bpf_call_arg_meta { + /* Common */ struct btf *btf; u32 func_id; + u8 release_regno; + u32 ret_btf_id; + u32 subprogno; + struct bpf_map_desc map; + struct bpf_dynptr_desc dynptr; + struct ref_obj_desc ref_obj; + struct ret_mem_desc ret_mem; + + /* Only set by kfunc */ + bool r0_rdonly; u32 kfunc_flags; const struct btf_type *func_proto; const char *func_name; - /* Out parameters */ - u8 release_regno; - bool r0_rdonly; - u32 ret_btf_id; - u32 subprogno; struct { u64 value; bool found; @@ -1521,28 +1526,31 @@ struct bpf_kfunc_call_arg_meta { u8 spi; u8 frameno; } iter; - struct bpf_map_desc map; - struct bpf_dynptr_desc dynptr; - struct ref_obj_desc ref_obj; - struct ret_mem_desc ret_mem; + + /* Only set by helper */ + u64 msize_max_value; + s64 const_map_key; + struct btf *ret_btf; + struct btf_field *kptr_field; + struct arg_raw_mem_desc arg_raw_mem; }; int bpf_get_helper_proto(struct bpf_verifier_env *env, int func_id, const struct bpf_func_proto **ptr); int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env, s32 func_id, - s16 offset, struct bpf_kfunc_call_arg_meta *meta); + s16 offset, struct bpf_call_arg_meta *meta); bool bpf_is_async_callback_calling_insn(struct bpf_insn *insn); bool bpf_is_sync_callback_calling_insn(struct bpf_insn *insn); -static inline bool bpf_is_iter_next_kfunc(struct bpf_kfunc_call_arg_meta *meta) +static inline bool bpf_is_iter_next_kfunc(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ITER_NEXT; } -static inline bool bpf_is_kfunc_sleepable(struct bpf_kfunc_call_arg_meta *meta) +static inline bool bpf_is_kfunc_sleepable(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_SLEEPABLE; } -bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta); +bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta); struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem); int bpf_copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off); bool bpf_insn_is_cond_jump(u8 code); diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index 26d37066465f..db3416a7c904 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -491,7 +491,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env) return ret; } } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { - struct bpf_kfunc_call_arg_meta meta; + struct bpf_call_arg_meta meta; ret = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta); if (ret == 0 && bpf_is_iter_next_kfunc(&meta)) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 37127ae1ff28..de816063ae63 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -252,24 +252,6 @@ static int validate_ref_obj(struct bpf_verifier_env *env, struct ref_obj_desc *r return 0; } -struct bpf_call_arg_meta { - struct bpf_map_desc map; - struct bpf_dynptr_desc dynptr; - struct ref_obj_desc ref_obj; - struct arg_raw_mem_desc arg_raw_mem; - struct ret_mem_desc ret_mem; - u8 release_regno; - u64 msize_max_value; - int func_id; - struct btf *btf; - u32 btf_id; - struct btf *ret_btf; - u32 ret_btf_id; - u32 subprogno; - struct btf_field *kptr_field; - s64 const_map_key; -}; - struct bpf_kfunc_meta { struct btf *btf; const struct btf_type *proto; @@ -927,10 +909,10 @@ static void __mark_reg_known_zero(struct bpf_reg_state *reg); static bool in_rcu_cs(struct bpf_verifier_env *env); -static bool is_kfunc_rcu_protected(struct bpf_kfunc_call_arg_meta *meta); +static bool is_kfunc_rcu_protected(struct bpf_call_arg_meta *meta); static int mark_stack_slots_iter(struct bpf_verifier_env *env, - struct bpf_kfunc_call_arg_meta *meta, + struct bpf_call_arg_meta *meta, struct bpf_reg_state *reg, int insn_idx, struct btf *btf, u32 btf_id, int nr_slots) { @@ -1063,7 +1045,7 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx); static int release_irq_state(struct bpf_verifier_state *state, int id); static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, - struct bpf_kfunc_call_arg_meta *meta, + struct bpf_call_arg_meta *meta, struct bpf_reg_state *reg, int insn_idx, int kfunc_class) { @@ -7245,7 +7227,7 @@ static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_sta } static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { return process_timer_func(env, reg, argno, &meta->map); } @@ -7412,23 +7394,23 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat return err; } -static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta) +static bool is_iter_kfunc(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & (KF_ITER_NEW | KF_ITER_NEXT | KF_ITER_DESTROY); } -static bool is_iter_new_kfunc(struct bpf_kfunc_call_arg_meta *meta) +static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ITER_NEW; } -static bool is_iter_destroy_kfunc(struct bpf_kfunc_call_arg_meta *meta) +static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ITER_DESTROY; } -static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, +static bool is_kfunc_arg_iter(struct bpf_call_arg_meta *meta, int arg_idx, const struct btf_param *arg) { /* btf_check_iter_kfuncs() guarantees that first argument of any iter @@ -7442,7 +7424,7 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, } static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int insn_idx, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { struct bpf_func_state *state = bpf_func(env, reg); const struct btf_type *t; @@ -7609,7 +7591,7 @@ static int widen_imprecise_scalars(struct bpf_verifier_env *env, } static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_st, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { int iter_frameno = meta->iter.frameno; int iter_spi = meta->iter.spi; @@ -7696,7 +7678,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_ * bpf_iter_num_destroy(&it); */ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st; struct bpf_func_state *cur_fr = cur_st->frame[cur_st->curframe], *queued_fr; @@ -10750,27 +10732,27 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size); } -static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ACQUIRE; } -static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_release(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_RELEASE; } -static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_destructive(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_DESTRUCTIVE; } -static bool is_kfunc_rcu(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_rcu(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_RCU; } -static bool is_kfunc_rcu_protected(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_rcu_protected(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_RCU_PROTECTED; } @@ -10991,7 +10973,7 @@ static bool is_kfunc_arg_prog_aux(const struct btf *btf, const struct btf_param * To determine whether an argument is implicit, we compare its position * against the number of arguments in the prototype w/o implicit args. */ -static bool is_kfunc_arg_implicit(const struct bpf_kfunc_call_arg_meta *meta, u32 arg_idx) +static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_idx) { const struct btf_type *func, *func_proto; u32 argn; @@ -11290,7 +11272,7 @@ static bool is_task_work_add_kfunc(u32 func_id) func_id == special_kfunc_list[KF_bpf_task_work_schedule_resume]; } -static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_ret_null(struct bpf_call_arg_meta *meta) { if (is_bpf_refcount_acquire_kfunc(meta->func_id) && meta->arg_owning_ref) return false; @@ -11298,34 +11280,34 @@ static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) return meta->kfunc_flags & KF_RET_NULL; } -static bool is_kfunc_bpf_rcu_read_lock(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_bpf_rcu_read_lock(struct bpf_call_arg_meta *meta) { return meta->func_id == special_kfunc_list[KF_bpf_rcu_read_lock]; } -static bool is_kfunc_bpf_rcu_read_unlock(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_bpf_rcu_read_unlock(struct bpf_call_arg_meta *meta) { return meta->func_id == special_kfunc_list[KF_bpf_rcu_read_unlock]; } -static bool is_kfunc_bpf_preempt_disable(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_bpf_preempt_disable(struct bpf_call_arg_meta *meta) { return meta->func_id == special_kfunc_list[KF_bpf_preempt_disable]; } -static bool is_kfunc_bpf_preempt_enable(struct bpf_kfunc_call_arg_meta *meta) +static bool is_kfunc_bpf_preempt_enable(struct bpf_call_arg_meta *meta) { return meta->func_id == special_kfunc_list[KF_bpf_preempt_enable]; } -bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) +bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) { return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } static enum kfunc_ptr_arg_type get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller, - struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta, + struct bpf_reg_state *regs, struct bpf_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, int arg, int nargs, argno_t argno, struct bpf_reg_state *reg) @@ -11431,7 +11413,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, const struct btf_type *ref_t, const char *ref_tname, u32 ref_id, - struct bpf_kfunc_call_arg_meta *meta, + struct bpf_call_arg_meta *meta, int arg, argno_t argno) { const struct btf_type *reg_ref_t; @@ -11501,7 +11483,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, } static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { int err, spi, kfunc_class = IRQ_NATIVE_KFUNC; bool irq_save; @@ -11826,7 +11808,7 @@ static bool check_kfunc_is_graph_node_api(struct bpf_verifier_env *env, static int __process_kf_arg_ptr_to_graph_root(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta, + struct bpf_call_arg_meta *meta, enum btf_field_type head_field_type, struct btf_field **head_field) { @@ -11876,7 +11858,7 @@ __process_kf_arg_ptr_to_graph_root(struct bpf_verifier_env *env, static int process_kf_arg_ptr_to_list_head(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { return __process_kf_arg_ptr_to_graph_root(env, reg, argno, meta, BPF_LIST_HEAD, &meta->arg_list_head.field); @@ -11884,7 +11866,7 @@ static int process_kf_arg_ptr_to_list_head(struct bpf_verifier_env *env, static int process_kf_arg_ptr_to_rbtree_root(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { return __process_kf_arg_ptr_to_graph_root(env, reg, argno, meta, BPF_RB_ROOT, &meta->arg_rbtree_root.field); @@ -11893,7 +11875,7 @@ static int process_kf_arg_ptr_to_rbtree_root(struct bpf_verifier_env *env, static int __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta, + struct bpf_call_arg_meta *meta, enum btf_field_type head_field_type, enum btf_field_type node_field_type, struct btf_field **node_field) @@ -11958,7 +11940,7 @@ __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env, static int process_kf_arg_ptr_to_list_node(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { return __process_kf_arg_ptr_to_graph_node(env, reg, argno, meta, BPF_LIST_HEAD, BPF_LIST_NODE, @@ -11967,7 +11949,7 @@ static int process_kf_arg_ptr_to_list_node(struct bpf_verifier_env *env, static int process_kf_arg_ptr_to_rbtree_node(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { return __process_kf_arg_ptr_to_graph_node(env, reg, argno, meta, BPF_RB_ROOT, BPF_RB_NODE, @@ -11996,7 +11978,7 @@ static bool check_css_task_iter_allowlist(struct bpf_verifier_env *env) } } -static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta, +static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, int insn_idx) { const char *func_name = meta->func_name, *ref_tname; @@ -12575,7 +12557,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env, s32 func_id, s16 offset, - struct bpf_kfunc_call_arg_meta *meta) + struct bpf_call_arg_meta *meta) { struct bpf_kfunc_meta kfunc; int err; @@ -12734,7 +12716,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn * int arg, int insn_idx) { struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx]; - struct bpf_kfunc_call_arg_meta meta; + struct bpf_call_arg_meta meta; const struct btf_param *args; const struct btf_type *t, *ref_t; const struct btf *btf; @@ -12795,7 +12777,7 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn * * 0 - fall-through to 'else' branch * < 0 - not fall-through to 'else' branch, return error */ -static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta, +static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, struct bpf_reg_state *regs, struct bpf_insn_aux_data *insn_aux, const struct btf_type *ptr_type, struct btf *desc_btf) { @@ -12974,7 +12956,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_reg_state *regs = cur_regs(env); const char *func_name, *ptr_type_name; const struct btf_type *t, *ptr_type; - struct bpf_kfunc_call_arg_meta meta; + struct bpf_call_arg_meta meta; struct bpf_insn_aux_data *insn_aux; int err, insn_idx = *insn_idx_p; const struct btf_param *args; @@ -16728,7 +16710,7 @@ bool bpf_verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, struct bpf_call_summary *cs) { - struct bpf_kfunc_call_arg_meta meta; + struct bpf_call_arg_meta meta; const struct bpf_func_proto *fn; int i; From d1f4b56417a3dc1a0600f960b14f46bd25eda89d Mon Sep 17 00:00:00 2001 From: Avinash Duduskar Date: Wed, 15 Jul 2026 15:33:49 +0530 Subject: [PATCH 091/373] selftests/bpf: Fix fib_lookup VLAN tests on hosts with forwarding on The VLAN tests assume the test namespace starts with IPv4 forwarding off, but a new netns copies conf/all and conf/default from init_net (devinet_init_net(), with net.core.devconf_inherit_init_net at its default), so on a host with net.ipv4.conf.all.forwarding=1 the devices in the netns come up with forwarding already enabled. IPv6 uses compiled defaults at the same sysctl value, so only the IPv4 arms are affected. Two arms break as a result. The arms that expect BPF_FIB_LKUP_RET_FWD_DISABLED see the lookup pass the forwarding check and return SUCCESS instead, so fib_lookup fails: test_fib_lookup:FAIL:fib_lookup_ret unexpected fib_lookup_ret: actual 0 != expected 5 Pin forwarding off in setup_netns() before the devices are created; the existing per-device writes still enable it where the tests need it. The netns arm has the opposite problem. It checks that a VLAN device in another netns is not resolved and expects NOT_FWDED. The lookup runs against the caller's FIB, which had no route to the destination, so a kernel that resolved the moved device anyway also returned NOT_FWDED and the arm passed regardless of the namespace check. On a forwarding-on host, where the resolved device clears the forwarding gate, this makes the arm a tautology. Add a route so a resolved device returns SUCCESS and the arm can tell the two apart. Verified by deleting the netns check from bpf_fib_vlan_input_dev(): with the fix the arm fails on both a forwarding-off host (actual 5) and a forwarding-on host (actual 0), where before it passed on the latter. The real kernel passes the full suite on both. Fixes: e54a87872e34 ("selftests/bpf: Add bpf_fib_lookup() VLAN flag tests") Reported-by: sashiko-bot Closes: https://lore.kernel.org/all/20260713163826.D70201F000E9@smtp.kernel.org/ Signed-off-by: Avinash Duduskar Link: https://lore.kernel.org/bpf/20260715100349.2684391-1-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/fib_lookup.c | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/fib_lookup.c b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c index f7361f9a3459..8f4779dd802e 100644 --- a/tools/testing/selftests/bpf/prog_tests/fib_lookup.c +++ b/tools/testing/selftests/bpf/prog_tests/fib_lookup.c @@ -419,6 +419,19 @@ static int setup_netns(void) { int err; + /* + * a new netns copies the IPv4 conf from init_net, so on a host with + * forwarding enabled the arms that expect FWD_DISABLED would see the + * lookup succeed instead; pin it off here and enable it per device + */ + err = write_sysctl("/proc/sys/net/ipv4/conf/all/forwarding", "0"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.conf.all.forwarding)")) + goto fail; + + err = write_sysctl("/proc/sys/net/ipv4/conf/default/forwarding", "0"); + if (!ASSERT_OK(err, "write_sysctl(net.ipv4.conf.default.forwarding)")) + goto fail; + SYS(fail, "ip link add veth1 type veth peer name veth2"); SYS(fail, "ip link set dev veth1 up"); SYS(fail, "ip link set dev veth2 up"); @@ -897,6 +910,14 @@ void test_fib_lookup_vlan_netns(void) if (!ASSERT_NEQ(parent_idx, 0, "if_nametoindex(veth7)")) goto fail; + /* + * give this netns a route to the destination: the lookup below runs + * against this FIB, so without the route a kernel that resolved the + * moved device anyway would still return NOT_FWDED and the arm would + * pass for the wrong reason + */ + SYS(fail, "ip route add %s/32 dev veth7", IPV4_VLAN_NETNS_DST); + /* * input: the moved device is still in veth7's VLAN group, but it * lives in another netns, so the lookup must fail closed From 3513ea9dab6c1a3d2dc8e6160c41f690206948b6 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Thu, 16 Jul 2026 12:01:55 +0000 Subject: [PATCH 092/373] bpf: Sync tail_call_reachable with callee state on entry Currently in check_max_stack_depth_subprog, when the verifier enters a new callee branch, the local tail_call_reachable is not properly synchronized with the callee's state. Consider a main prog branching into multiple subprogs: subprog0 -> tailcall main < subprog1 -> subprog2 When the verifier finishes checking subprog0 and backtracks to main prog, the local tail_call_reachable state is left as true. As it proceeds to subprog1, this uncleared state leaks into the new branch, falsely marking subprog1 and subprog2 as tailcall reachable. Fix this by explicitly syncing tail_call_reachable with the callee's has_tail_call state on entry. The caller's state is safely preserved and restored via the existing backtracking logic. Fixes: ebf7d1f508a7 ("bpf, x64: rework pro/epilogue and tailcall handling in JIT") Reported-by: Sashiko Signed-off-by: Pu Lehui Link: https://patch.msgid.link/20260716120157.835937-2-pulehui@huaweicloud.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index de816063ae63..782d939c38cd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5263,8 +5263,8 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, if (!priv_stack_supported) subprog[idx].priv_stack_mode = NO_PRIV_STACK; - if (subprog[idx].has_tail_call) - tail_call_reachable = true; + /* sync tail_call_reachable with callee state on entry */ + tail_call_reachable = subprog[idx].has_tail_call; frame = bpf_subprog_is_global(env, idx) ? 0 : frame + 1; if (frame >= MAX_CALL_FRAMES) { From a41d0c30d764e086c62b049e806fd12df4f4acfc Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Thu, 16 Jul 2026 12:01:56 +0000 Subject: [PATCH 093/373] bpf: Reject callback subprogs invoke tailcall MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Some JIT compilers, such as x86_64, rely on a register to pass the TCC. When subprograms of synchronous callback invoke tailcall, C helpers invoking bpf callback clobber this register, and the corrupted TCC may bypass the TCC limit, leading to infinite tailcall. Fix this by rejecting tailcall inside all subprogs of sync callback. This also cleanly consolidates the existing async and exception callback checks into a single unified `is_cb` check. Reported-by: Sashiko Reported-by: Björn Töpel Signed-off-by: Pu Lehui Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260716120157.835937-3-pulehui@huaweicloud.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 782d939c38cd..62d46b4c9962 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5237,10 +5237,6 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, if (verifier_bug_if(sidx < 0, env, "callee not found at insn %d", next_insn)) return -EFAULT; if (subprog[sidx].is_async_cb) { - if (subprog[sidx].has_tail_call) { - verifier_bug(env, "subprog has tail_call and async cb"); - return -EFAULT; - } /* async callbacks don't increase bpf prog stack size unless called directly */ if (!bpf_pseudo_call(insn + i)) continue; @@ -5281,8 +5277,8 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, */ if (tail_call_reachable) { for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) { - if (subprog[tmp].is_exception_cb) { - verbose(env, "cannot tail call within exception cb\n"); + if (subprog[tmp].is_cb) { + verbose(env, "cannot tail call within callback\n"); return -EINVAL; } if (subprog[tmp].stack_arg_cnt) { From 42bfd21a8b70143e68ae6d017752abb3f04738f5 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Thu, 16 Jul 2026 12:01:57 +0000 Subject: [PATCH 094/373] selftests/bpf: Add testcases for callback with tailcall Add 2 testcases for callback with tailcall: 1. failure case: callback->subprog->tailcall. 2. success case: subprog with tailcall do not affect no-tailcall callback. Signed-off-by: Pu Lehui Link: https://patch.msgid.link/20260716120157.835937-4-pulehui@huaweicloud.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/tailcalls.c | 7 ++ .../selftests/bpf/progs/tailcall_callback.c | 81 +++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/tailcall_callback.c diff --git a/tools/testing/selftests/bpf/prog_tests/tailcalls.c b/tools/testing/selftests/bpf/prog_tests/tailcalls.c index a5a226d0104c..c66037162da5 100644 --- a/tools/testing/selftests/bpf/prog_tests/tailcalls.c +++ b/tools/testing/selftests/bpf/prog_tests/tailcalls.c @@ -12,6 +12,7 @@ #include "tailcall_cgrp_storage_no_storage.skel.h" #include "tailcall_cgrp_storage.skel.h" #include "tailcall_sleepable.skel.h" +#include "tailcall_callback.skel.h" /* test_tailcall_1 checks basic functionality by patching multiple locations * in a single program for a single tail call slot with nop->jmp, jmp->nop @@ -1901,6 +1902,11 @@ static void test_tailcall_sleepable(void) tailcall_sleepable__destroy(skel); } +static void test_tailcall_callback(void) +{ + RUN_TESTS(tailcall_callback); +} + void test_tailcalls(void) { if (test__start_subtest("tailcall_1")) @@ -1967,4 +1973,5 @@ void test_tailcalls(void) test_tailcall_cgrp_storage_no_storage_leaf(); if (test__start_subtest("tailcall_cgrp_storage_no_storage_bridge")) test_tailcall_cgrp_storage_no_storage_bridge(); + test_tailcall_callback(); } diff --git a/tools/testing/selftests/bpf/progs/tailcall_callback.c b/tools/testing/selftests/bpf/progs/tailcall_callback.c new file mode 100644 index 000000000000..c41632cf423b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tailcall_callback.c @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "bpf_misc.h" +#include "bpf_test_utils.h" + +int classifier_0(struct __sk_buff *skb); + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 1); + __uint(key_size, sizeof(__u32)); + __array(values, void (void)); +} jmp_table SEC(".maps") = { + .values = { + [0] = (void *) &classifier_0, + }, +}; + +__auxiliary +SEC("tc") +int classifier_0(struct __sk_buff *skb) +{ + return 0; +} + +static __noinline +int subprog_tail0(struct __sk_buff *skb) +{ + int ret = 0; + + bpf_tail_call_static(skb, &jmp_table, 0); + barrier_var(ret); + return ret; +} + +static __noinline +int callback_loop(int index, void **cb_ctx) +{ + int ret; + + ret = subprog_tail0(*cb_ctx); + barrier_var(ret); + return ret ? 1 : 0; +} + +static __noinline +int callback_empty(int index, void *data) +{ + return 0; +} + +/* callback involving subprog with tail call is rejected */ +SEC("tc") +__failure __msg("cannot tail call within callback") +int tailcall_callback_1(struct __sk_buff *skb) +{ + clobber_regs_stack(); + + bpf_loop(1, callback_loop, &skb, 0); + return 0; +} + +/* subprogs with tailcall do not affect no-tailcall callback */ +SEC("tc") +__success +__retval(0) +int tailcall_callback_2(struct __sk_buff *skb) +{ + int ret; + + clobber_regs_stack(); + + ret = subprog_tail0(skb); + __sink(ret); + + bpf_loop(1, callback_empty, NULL, 0); + return 0; +} + +char __license[] SEC("license") = "GPL"; From 918787e8f569d225c968af2c783962ae069b8ac8 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Wed, 15 Jul 2026 10:21:26 -0700 Subject: [PATCH 095/373] bpf: Disable raw mode for bloom filter map_peek For a bloom filter, the value argument of bpf_map_peek_elem() is always an input. Therefore, the verifier should not allow passing uninitialized stack memory to it to avoid information leak. bpf_map_peek_elem() tags its value argument ARG_PTR_TO_MAP_VALUE | MEM_UNINIT, telling the verifier the callee fills the buffer. This holds for queue/stack maps, but not for a bloom filter, which reads the buffer as an input to test set membership and never writes it. As a result, a program can pass an uninitialized stack buffer to bpf_map_peek_elem() on a bloom filter. The verifier accepts it and marks the buffer initialized on return, letting the program read back leftover kernel stack memory. Bloom maps require CAP_BPF to create, so this is a CAP_BPF-gated stack infoleak that bypasses the boundary CAP_BPF is meant to enforce (arbitrary kernel reads are gated behind CAP_PERFMON). Signed-off-by: Amery Hung Acked-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260715172127.2416388-2-ameryhung@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 62d46b4c9962..828a220647d6 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8418,6 +8418,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, verifier_bug(env, "invalid map_ptr to access map->value"); return -EFAULT; } + + /* + * Disable raw mode for bpf_map_peek_elem() on a bloom filter. The helper reads + * the value buffer as an input rather than filling it. + */ + if (meta->func_id == BPF_FUNC_map_peek_elem && + meta->map.ptr->map_type == BPF_MAP_TYPE_BLOOM_FILTER) + meta->arg_raw_mem.regno = 0; + err = check_helper_mem_access(env, reg, argno, meta->map.ptr->value_size, arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); From 79c9dc93fcae5e52bd1b4f96e138604d844ad759 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Wed, 15 Jul 2026 10:21:27 -0700 Subject: [PATCH 096/373] bpf: Zero kfunc arg meta before error paths can read it check_kfunc_call() reads meta.func_name when bpf_fetch_kfunc_arg_meta() returns -EACCES, but that error can come from fetch_kfunc_meta() (e.g. fd_array_get_btf() rejecting BTF binding for a signed program) before meta is memset(), leaving it uninitialized and risking a garbage deref in verbose(). Move the memset() to the start of bpf_fetch_kfunc_arg_meta() so meta is zeroed on every error return. The intended "not allowed" -EACCES path still sets func_name first, so its message is unchanged. Signed-off-by: Amery Hung Acked-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260715172127.2416388-3-ameryhung@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 828a220647d6..a78cdabf8560 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12567,11 +12567,12 @@ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env, struct bpf_kfunc_meta kfunc; int err; + memset(meta, 0, sizeof(*meta)); + err = fetch_kfunc_meta(env, func_id, offset, &kfunc); if (err) return err; - memset(meta, 0, sizeof(*meta)); meta->btf = kfunc.btf; meta->func_id = kfunc.id; meta->func_proto = kfunc.proto; From f8248ac8f044ad3e79279cf3355412456bf416b0 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Fri, 17 Jul 2026 19:37:01 +0800 Subject: [PATCH 097/373] bpf: Pass arena instead of scratch_page to the pte callbacks Replace the scratch_page field in the pte-callback data with the arena pointer; later patches use other arena fields from these callbacks. No functional change. Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717114117.350851-2-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/arena.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 80b7b8a69446..8dbc24460890 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -143,14 +143,14 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) } struct apply_range_data { + struct bpf_arena *arena; struct page **pages; - struct page *scratch_page; int i; }; struct clear_range_data { + struct bpf_arena *arena; struct llist_head *free_pages; - struct page *scratch_page; }; static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) @@ -180,7 +180,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) if (pte_none(old)) continue; - if (WARN_ON_ONCE(pte_page(old) != d->scratch_page)) + if (WARN_ON_ONCE(pte_page(old) != d->arena->scratch_page)) return -EBUSY; ptep_get_and_clear(&init_mm, addr, pte); flush_tlb_before_set(addr); @@ -227,7 +227,7 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data) * scratches its PTE. A later bpf_arena_free_pages() over that range walks * here. Without the skip, scratch_page would be freed. */ - if (page == d->scratch_page) + if (page == d->arena->scratch_page) return 0; __llist_add(&page->pcp_llist, d->free_pages); @@ -506,8 +506,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) if (ret) goto out_sigsegv_memcg; - struct apply_range_data data = { .pages = &page, .i = 0, - .scratch_page = arena->scratch_page }; + struct apply_range_data data = { .arena = arena, .pages = &page, .i = 0 }; /* Account into memcg of the process that created bpf_arena */ ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); if (ret) { @@ -696,8 +695,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt bpf_map_memcg_exit(old_memcg, new_memcg); return 0; } + data.arena = arena; data.pages = pages; - data.scratch_page = arena->scratch_page; if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) goto out_free_pages; @@ -873,8 +872,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, range_tree_set(&arena->rt, pgoff, page_cnt); init_llist_head(&free_pages); + cdata.arena = arena; cdata.free_pages = &free_pages; - cdata.scratch_page = arena->scratch_page; /* clear ptes and collect struct pages */ apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, apply_range_clear_cb, &cdata); @@ -981,8 +980,8 @@ static void arena_free_worker(struct work_struct *work) bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); init_llist_head(&free_pages); + cdata.arena = arena; cdata.free_pages = &free_pages; - cdata.scratch_page = arena->scratch_page; arena_vm_start = bpf_arena_get_kern_vm_start(arena); user_vm_start = bpf_arena_get_user_vm_start(arena); From 89318afb141437817a88fe3e5d8f6638c6ab3ed3 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Fri, 17 Jul 2026 19:37:02 +0800 Subject: [PATCH 098/373] bpf: Add memory usage for arena arena is the only map type whose map_mem_usage() still returns 0, so "bpftool map show" and fdinfo always showed 0 memlock for an arena no matter how many pages it had. Count the pages that are actually mapped into the arena: bump a counter in apply_range_set_cb() when a page goes in and drop it in apply_range_clear_cb() when a page goes out, both under the arena spinlock. map_mem_usage() then just returns nr_pages << PAGE_SHIFT. Only real data pages are counted, not the scratch page. Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717114117.350851-3-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/arena.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 8dbc24460890..f046e878f7ae 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -55,8 +55,10 @@ struct bpf_arena { struct vm_struct *kern_vm; struct page *scratch_page; struct range_tree rt; - /* protects rt */ + /* protects rt and nr_pages */ rqspinlock_t spinlock; + /* number of pages currently populated in the arena */ + u64 nr_pages; struct list_head vma_list; /* protects vma_list */ struct mutex lock; @@ -196,6 +198,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) set_pte_at(&init_mm, addr, pte, pteval); #endif d->i++; + WRITE_ONCE(d->arena->nr_pages, d->arena->nr_pages + 1); return 0; } @@ -231,6 +234,7 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data) return 0; __llist_add(&page->pcp_llist, d->free_pages); + WRITE_ONCE(d->arena->nr_pages, d->arena->nr_pages - 1); return 0; } @@ -413,7 +417,9 @@ static int arena_map_check_btf(struct bpf_map *map, const struct btf *btf, static u64 arena_map_mem_usage(const struct bpf_map *map) { - return 0; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + return (u64)READ_ONCE(arena->nr_pages) << PAGE_SHIFT; } struct vma_list { From a14f9ba080c1e9ba18df9385de0d062c827fba56 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Fri, 17 Jul 2026 19:37:03 +0800 Subject: [PATCH 099/373] selftests/bpf: Run arena tests serially Every arena reserves a 4GB kernel vmalloc region at creation, and on a KASAN build that eagerly populates ~512MB of shadow memory per arena. When several arena tests run in parallel their shadow adds up, exhausts the memory of a small CI VM and trips the OOM killer during map creation. Make the dedicated arena tests (arena_* and libarena*) serial so they no longer pile up in the parallel phase. That alone brings the peak number of live arenas back within the CI memory budget. A few other tests such as verifier_arena*, stream and compute_live_registers create an arena too, but they run inside shared RUN_TESTS()/RUN() suites, so turning them serial would drag many unrelated subtests along with them. Leave those as-is. Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717114117.350851-4-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/arena_atomics.c | 2 +- tools/testing/selftests/bpf/prog_tests/arena_direct_value.c | 2 +- tools/testing/selftests/bpf/prog_tests/arena_htab.c | 2 +- tools/testing/selftests/bpf/prog_tests/arena_list.c | 2 +- tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c | 2 +- tools/testing/selftests/bpf/prog_tests/arena_strsearch.c | 2 +- tools/testing/selftests/bpf/prog_tests/libarena.c | 2 +- tools/testing/selftests/bpf/prog_tests/libarena_asan.c | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c index d98577a6babc..1ad5d03d07ad 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c @@ -222,7 +222,7 @@ static void test_store_release(struct arena_atomics *skel) "store_release64_result"); } -void test_arena_atomics(void) +void serial_test_arena_atomics(void) { struct arena_atomics *skel; int err; diff --git a/tools/testing/selftests/bpf/prog_tests/arena_direct_value.c b/tools/testing/selftests/bpf/prog_tests/arena_direct_value.c index 4b4adb3f4b71..01fcf4965ea4 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_direct_value.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_direct_value.c @@ -66,7 +66,7 @@ static void test_arena_direct_value_one_past_end(void) close(map_fd); } -void test_arena_direct_value(void) +void serial_test_arena_direct_value(void) { if (test__start_subtest("one_past_end")) test_arena_direct_value_one_past_end(); diff --git a/tools/testing/selftests/bpf/prog_tests/arena_htab.c b/tools/testing/selftests/bpf/prog_tests/arena_htab.c index d69fd2465f53..91ccf0402980 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_htab.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_htab.c @@ -81,7 +81,7 @@ static void test_arena_htab_asm(void) arena_htab_asm__destroy(skel); } -void test_arena_htab(void) +void serial_test_arena_htab(void) { if (test__start_subtest("arena_htab_llvm")) test_arena_htab_llvm(); diff --git a/tools/testing/selftests/bpf/prog_tests/arena_list.c b/tools/testing/selftests/bpf/prog_tests/arena_list.c index 4f2866a615ce..2648e06f53d0 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_list.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_list.c @@ -68,7 +68,7 @@ static void test_arena_list_add_del(int cnt, bool nonsleepable) arena_list__destroy(skel); } -void test_arena_list(void) +void serial_test_arena_list(void) { if (test__start_subtest("arena_list_1")) test_arena_list_add_del(1, false); diff --git a/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c b/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c index acb9d53b5973..545b05d7a0aa 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_spin_lock.c @@ -101,7 +101,7 @@ static void test_arena_spin_lock_size(int size) return; } -void test_arena_spin_lock(void) +void serial_test_arena_spin_lock(void) { repeat = 1000; if (test__start_subtest("arena_spin_lock_1")) diff --git a/tools/testing/selftests/bpf/prog_tests/arena_strsearch.c b/tools/testing/selftests/bpf/prog_tests/arena_strsearch.c index f81a0c066505..0c1c6cbfa0f1 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_strsearch.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_strsearch.c @@ -23,7 +23,7 @@ static void test_arena_str(void) arena_strsearch__destroy(skel); } -void test_arena_strsearch(void) +void serial_test_arena_strsearch(void) { if (test__start_subtest("arena_strsearch")) test_arena_str(); diff --git a/tools/testing/selftests/bpf/prog_tests/libarena.c b/tools/testing/selftests/bpf/prog_tests/libarena.c index ba5a5a50f7c0..df7e4b8dc394 100644 --- a/tools/testing/selftests/bpf/prog_tests/libarena.c +++ b/tools/testing/selftests/bpf/prog_tests/libarena.c @@ -202,7 +202,7 @@ static void run_libarena_parallel_test(struct libarena *skel, struct bpf_program run_libarena_parallel_fini(skel, name, prefixlen); } -void test_libarena(void) +void serial_test_libarena(void) { struct arena_alloc_reserve_args args; struct libarena *skel; diff --git a/tools/testing/selftests/bpf/prog_tests/libarena_asan.c b/tools/testing/selftests/bpf/prog_tests/libarena_asan.c index f897405f701d..9c31b17dbf39 100644 --- a/tools/testing/selftests/bpf/prog_tests/libarena_asan.c +++ b/tools/testing/selftests/bpf/prog_tests/libarena_asan.c @@ -85,7 +85,7 @@ static void run_test(void) * Run the test depending on whether LLVM can compile arena ASAN * programs. */ -void test_libarena_asan(void) +void serial_test_libarena_asan(void) { #ifdef HAS_BPF_ARENA_ASAN run_test(); From 29e4bcf12604a8b217d0e3b86cb619f52c3a0a5f Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Fri, 17 Jul 2026 19:37:04 +0800 Subject: [PATCH 100/373] selftests/bpf: Add tests for memory usage for arena Allocate and free arena pages, both from BPF and via user-space fault-in, and check that the map's memlock in fdinfo tracks the number of pages that are actually populated. Like the other arena tests it runs serially. test: ./test_progs -a arena_mem_usage #5 arena_mem_usage:OK Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717114117.350851-5-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/prog_tests/arena_mem_usage.c | 122 ++++++++++++++++++ .../selftests/bpf/progs/arena_mem_usage.c | 40 ++++++ 2 files changed, 162 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_mem_usage.c create mode 100644 tools/testing/selftests/bpf/progs/arena_mem_usage.c diff --git a/tools/testing/selftests/bpf/prog_tests/arena_mem_usage.c b/tools/testing/selftests/bpf/prog_tests/arena_mem_usage.c new file mode 100644 index 000000000000..14c2d1a1d673 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_mem_usage.c @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#ifndef PAGE_SIZE /* on some archs it comes in sys/user.h */ +#include +#define PAGE_SIZE getpagesize() +#endif + +#include "arena_mem_usage.skel.h" + +/* + * arena_map_mem_usage() is surfaced to user space through the map's + * /proc//fdinfo/ "memlock:" line (the same value bpftool map show + * prints). Read it directly so the test has no external dependency. + */ +static long map_memlock(int map_fd) +{ + char path[64], line[128]; + long memlock = -1; + FILE *f; + + snprintf(path, sizeof(path), "/proc/self/fdinfo/%d", map_fd); + f = fopen(path, "r"); + if (!ASSERT_OK_PTR(f, "open_fdinfo")) + return -1; + while (fgets(line, sizeof(line), f)) { + if (sscanf(line, "memlock:\t%ld", &memlock) == 1) + break; + } + fclose(f); + ASSERT_NEQ(memlock, -1, "parse_memlock"); + return memlock; +} + +static int run(struct bpf_program *prog, const char *name) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + int err = bpf_prog_test_run_opts(bpf_program__fd(prog), &opts); + + if (!ASSERT_OK(err, name)) + return -1; + if (!ASSERT_OK(opts.retval, name)) + return -1; + return 0; +} + +void serial_test_arena_mem_usage(void) +{ + struct arena_mem_usage *skel; + const long ps = PAGE_SIZE; + char *base; + size_t sz; + int fd, i; + + skel = arena_mem_usage__open_and_load(); + if (!ASSERT_OK_PTR(skel, "open_load")) + return; + fd = bpf_map__fd(skel->maps.arena); + + /* Fresh arena: no data pages, and the scratch page is not counted. */ + ASSERT_EQ(map_memlock(fd), 0, "initial"); + + /* BPF-side allocation of 17 pages. */ + skel->bss->alloc_cnt = 17; + if (run(skel->progs.alloc, "alloc")) + goto out; + /* + * A NULL ptr means bpf_arena_alloc_pages() itself failed (e.g. the host + * is under memory pressure), not a miscount -- flag it distinctly so a + * red CI run is not mistaken for a counting bug. + */ + if (!ASSERT_OK_PTR(skel->bss->ptr, "arena_alloc_pages")) + goto out; + ASSERT_EQ(map_memlock(fd), 17 * ps, "after_alloc"); + + /* Free a single page (arena_free_pages page_cnt==1 path). */ + skel->bss->free_byte_off = 0; + skel->bss->free_cnt = 1; + if (run(skel->progs.free_pages, "free_one")) + goto out; + ASSERT_EQ(map_memlock(fd), 16 * ps, "after_free_one"); + + /* Free ten pages in one call (bulk path); only the freed pages count. */ + skel->bss->free_byte_off = 1 * ps; + skel->bss->free_cnt = 10; + if (run(skel->progs.free_pages, "free_bulk")) + goto out; + ASSERT_EQ(map_memlock(fd), 6 * ps, "after_free_bulk"); + + /* Free the remaining six -> arena empty again. */ + skel->bss->free_byte_off = 11 * ps; + skel->bss->free_cnt = 6; + if (run(skel->progs.free_pages, "free_rest")) + goto out; + ASSERT_EQ(map_memlock(fd), 0, "after_free_rest"); + + /* + * User-space fault-in: touching unallocated arena pages allocates them + * through arena_vm_fault(). libbpf mmap()s the arena at map_extra during + * load, so bpf_map__initial_value() hands back that base. + */ + base = bpf_map__initial_value(skel->maps.arena, &sz); + if (!ASSERT_OK_PTR(base, "arena_base")) + goto out; + for (i = 0; i < 8; i++) + base[i * ps] = 1; + ASSERT_EQ(map_memlock(fd), 8 * ps, "after_faultin"); + + /* + * Free the faulted-in pages from BPF. They are mapped into the user vma + * (elevated refcount), so this also exercises the zap path. + */ + skel->bss->ptr = base; + skel->bss->free_byte_off = 0; + skel->bss->free_cnt = 8; + if (run(skel->progs.free_pages, "free_faulted")) + goto out; + ASSERT_EQ(map_memlock(fd), 0, "after_free_faulted"); +out: + arena_mem_usage__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/arena_mem_usage.c b/tools/testing/selftests/bpf/progs/arena_mem_usage.c new file mode 100644 index 000000000000..455ecd669a5a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_mem_usage.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_arena_common.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1000); /* number of pages */ +#ifdef __TARGET_ARCH_arm64 + __ulong(map_extra, 0x1ull << 32); /* start of mmap() region */ +#else + __ulong(map_extra, 0x1ull << 44); /* start of mmap() region */ +#endif +} arena SEC(".maps"); + +void __arena *ptr; +int alloc_cnt; /* in: pages to allocate */ +long free_byte_off; /* in: byte offset within ptr to start freeing */ +int free_cnt; /* in: pages to free */ + +SEC("syscall") +int alloc(void *ctx) +{ + ptr = bpf_arena_alloc_pages(&arena, NULL, alloc_cnt, NUMA_NO_NODE, 0); + /* Success/failure is checked from user space via skel->bss->ptr. */ + return 0; +} + +SEC("syscall") +int free_pages(void *ctx) +{ + if (!ptr) + return 1; + bpf_arena_free_pages(&arena, (char __arena *)ptr + free_byte_off, free_cnt); + return 0; +} + +char _license[] SEC("license") = "GPL"; From 45bf95da6de8d7a7f51b41c138d41a4e2dba3664 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20Lothor=C3=A9=20=28eBPF=20Foundation=29?= Date: Thu, 9 Jul 2026 20:42:59 +0200 Subject: [PATCH 101/373] selftests/bpf: Remove redundant config option from selftests fragments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CONFIG_IPV6_SEG6_LWTUNNEL is currently enabled in each arch-specific config fragment for the BPF selftests. Commit 33a971d549d8 ("selftests/bpf: Add LWT encap tests for skb metadata") has enabled this config in the generic config as well, resulting in a small warning when configuring a kernel for selftests: $ cat tools/testing/selftests/bpf/{config,config.vm,config.x86_64}>.config $ make olddefconfig HOSTCC scripts/kconfig/conf.o HOSTCC scripts/kconfig/confdata.o HOSTLD scripts/kconfig/conf config:269:warning: override: reassigning to symbol IPV6_SEG6_LWTUNNEL # # configuration written to .config # Now that IPV6_SEG6_LWTUNNEL is set in the general config fragment, drop it from the arch-specific fragments. Signed-off-by: Alexis Lothoré (eBPF Foundation) Link: https://lore.kernel.org/bpf/20260709-testing-fragments-v1-1-65244b0650ff@bootlin.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/config.aarch64 | 1 - tools/testing/selftests/bpf/config.ppc64el | 1 - tools/testing/selftests/bpf/config.riscv64 | 1 - tools/testing/selftests/bpf/config.s390x | 1 - tools/testing/selftests/bpf/config.x86_64 | 1 - 5 files changed, 5 deletions(-) diff --git a/tools/testing/selftests/bpf/config.aarch64 b/tools/testing/selftests/bpf/config.aarch64 index 7efad36ceb26..fc85257701dc 100644 --- a/tools/testing/selftests/bpf/config.aarch64 +++ b/tools/testing/selftests/bpf/config.aarch64 @@ -71,7 +71,6 @@ CONFIG_INPUT_EVDEV=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTICAST=y CONFIG_IP_MULTIPLE_TABLES=y -CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_IPVLAN=y CONFIG_JUMP_LABEL=y CONFIG_KERNEL_UNCOMPRESSED=y diff --git a/tools/testing/selftests/bpf/config.ppc64el b/tools/testing/selftests/bpf/config.ppc64el index b53afb5e0b71..5685fa4ee82b 100644 --- a/tools/testing/selftests/bpf/config.ppc64el +++ b/tools/testing/selftests/bpf/config.ppc64el @@ -39,7 +39,6 @@ CONFIG_INET=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTICAST=y CONFIG_IP_MULTIPLE_TABLES=y -CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_JUMP_LABEL=y CONFIG_KALLSYMS_ALL=y CONFIG_KPROBES=y diff --git a/tools/testing/selftests/bpf/config.riscv64 b/tools/testing/selftests/bpf/config.riscv64 index 7bee24a79a71..655cb05a7689 100644 --- a/tools/testing/selftests/bpf/config.riscv64 +++ b/tools/testing/selftests/bpf/config.riscv64 @@ -30,7 +30,6 @@ CONFIG_HARDLOCKUP_DETECTOR=y CONFIG_HIGH_RES_TIMERS=y CONFIG_HUGETLBFS=y CONFIG_INET=y -CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTICAST=y CONFIG_IP_MULTIPLE_TABLES=y diff --git a/tools/testing/selftests/bpf/config.s390x b/tools/testing/selftests/bpf/config.s390x index db61878148e4..755d1cfcd9e0 100644 --- a/tools/testing/selftests/bpf/config.s390x +++ b/tools/testing/selftests/bpf/config.s390x @@ -56,7 +56,6 @@ CONFIG_INET=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTICAST=y CONFIG_IP_MULTIPLE_TABLES=y -CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_IPVLAN=y CONFIG_JUMP_LABEL=y CONFIG_KERNEL_UNCOMPRESSED=y diff --git a/tools/testing/selftests/bpf/config.x86_64 b/tools/testing/selftests/bpf/config.x86_64 index 42ad817b00ae..523e0d29bbd4 100644 --- a/tools/testing/selftests/bpf/config.x86_64 +++ b/tools/testing/selftests/bpf/config.x86_64 @@ -114,7 +114,6 @@ CONFIG_IP_ROUTE_VERBOSE=y CONFIG_IPV6_MIP6=y CONFIG_IPV6_ROUTE_INFO=y CONFIG_IPV6_ROUTER_PREF=y -CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_IPV6_SUBTREES=y CONFIG_IRQ_POLL=y CONFIG_JUMP_LABEL=y From b5a71cb2db6d84ac0042549dcec266b18429d41e Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Fri, 17 Jul 2026 12:53:47 +0000 Subject: [PATCH 102/373] bpf: Reject arena frees below the arena base bpf_arena_free_pages() accepts scalar arena addresses. The runtime masks the address to the low 32 bits and reconstructs a full user address from the arena base before returning the range to the arena free tree. When the scalar value is below the low 32 bits of the arena base, full_uaddr falls below user_vm_start. The existing upper-end clipping then turns this into an out-of-range free-tree offset. A later allocation can reuse that offset and return an address below the arena mapping. Reject such frees before computing the clipped range. Fixes: 317460317a02a ("bpf: Introduce bpf_arena.") Signed-off-by: Yiyang Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717-c10-031-public-bpf-next-v2-b4-v2-1-54b555443a7c@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/arena.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index f046e878f7ae..34f023a537fe 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -858,6 +858,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, uaddr &= PAGE_MASK; kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr; full_uaddr = clear_lo32(arena->user_vm_start) + uaddr; + if (full_uaddr < arena->user_vm_start) + return; uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT)); if (full_uaddr >= uaddr_end) return; From 770b62a6d38906e65759bf95be7ffb488b4af006 Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Fri, 17 Jul 2026 12:53:48 +0000 Subject: [PATCH 103/373] selftests/bpf: Cover scalar arena frees below the base Add a verifier_arena case that fills a two-page arena, calls bpf_arena_free_pages() with a scalar address one page below the arena base, and then verifies that another allocation is still rejected. Before the runtime guard, the invalid free can repopulate the free tree with an out-of-domain offset and the final allocation succeeds. Signed-off-by: Yiyang Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260717-c10-031-public-bpf-next-v2-b4-v2-2-54b555443a7c@mails.tsinghua.edu.cn Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_arena.c | 40 ++++++++++++++++--- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/verifier_arena.c b/tools/testing/selftests/bpf/progs/verifier_arena.c index df0e22d1a29b..b241bbcf54a8 100644 --- a/tools/testing/selftests/bpf/progs/verifier_arena.c +++ b/tools/testing/selftests/bpf/progs/verifier_arena.c @@ -12,15 +12,17 @@ #define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8))) +#ifdef __TARGET_ARCH_arm64 +#define ARENA_VM_START ((1ull << 32) | (~0u - __PAGE_SIZE * 2 + 1)) +#else +#define ARENA_VM_START ((1ull << 44) | (~0u - __PAGE_SIZE * 2 + 1)) +#endif + struct { __uint(type, BPF_MAP_TYPE_ARENA); __uint(map_flags, BPF_F_MMAPABLE); __uint(max_entries, 2); /* arena of two pages close to 32-bit boundary*/ -#ifdef __TARGET_ARCH_arm64 - __ulong(map_extra, (1ull << 32) | (~0u - __PAGE_SIZE * 2 + 1)); /* start of mmap() region */ -#else - __ulong(map_extra, (1ull << 44) | (~0u - __PAGE_SIZE * 2 + 1)); /* start of mmap() region */ -#endif + __ulong(map_extra, ARENA_VM_START); /* start of mmap() region */ } arena SEC(".maps"); SEC("socket") @@ -93,6 +95,34 @@ int basic_alloc1(void *ctx) return 0; } +SEC("syscall") +__success __retval(0) +int free_scalar_below_arena(void *ctx) +{ + void __arena *page1, *page2, *page3; + __u64 bad_addr = ARENA_VM_START - __PAGE_SIZE; + + page1 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page1) + return 1; + + page2 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page2) + return 2; + + page3 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (page3) + return 3; + + bpf_arena_free_pages(&arena, (void __arena *)bad_addr, 1); + + page3 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (page3) + return 4; + + return 0; +} + SEC("socket") __success __retval(0) int basic_alloc2_nosleep(void *ctx) From 34746b5a84ec37c0ea2bf6808c65c5ed8790eb51 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 15 Jul 2026 22:11:20 +0800 Subject: [PATCH 104/373] bpf: Disallow interpreter fallback for arena-related insns Since the interpreter does not support the arena-related insns, interpreter fallback should not be allowed for these insns in core.c::__bpf_prog_select_runtime(). Currently, when the interpreter executes the arena ST/LDX/STX insns, it would hit the BUG_ON() in ___bpf_prog_run() at run time. [ 2.579196] BPF interpreter: unknown opcode a2 (imm: 0x0) [ 2.579998] ------------[ cut here ]------------ [ 2.580652] kernel BUG at kernel/bpf/core.c:2349! [ 2.581314] Oops: invalid opcode: 0000 [#1] SMP PTI Set jit_required as true when arena map is used in the prog to disallow interpreter fallback for arena-related insns. Fixes: 6082b6c328b5 ("bpf: Recognize addr_space_cast instruction in the verifier.") Signed-off-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260715141122.15783-2-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4446f0bde88b..2d323f13da19 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17887,6 +17887,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, return -EOPNOTSUPP; } env->prog->aux->arena = (void *)map; + env->prog->jit_required = true; if (!bpf_arena_get_user_vm_start(env->prog->aux->arena)) { verbose(env, "arena's user address must be set via map_extra or mmap()\n"); return -EINVAL; From 905f716362e1186c1a23447ca279e6d21f795cdb Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 15 Jul 2026 22:11:21 +0800 Subject: [PATCH 105/373] bpf: Disallow interpreter fallback for gotox insn The interpreter does not recognize the BPF_JMP|BPF_JA|BPF_X insn, which is used for insn_array map. Thereafter, it would hit the BUG_ON() in ___bpf_prog_run() at run time. [ 2.563726] BPF interpreter: unknown opcode 0d (imm: 0x0) [ 2.564557] ------------[ cut here ]------------ [ 2.565206] kernel BUG at kernel/bpf/core.c:2349! [ 2.565882] Oops: invalid opcode: 0000 [#1] SMP PTI Set jit_required as true when insn_array map is used in the prog in order to disallow interpreter fallback for gotox insn in core.c::__bpf_prog_select_runtime(). Fixes: 493d9e0d6083 ("bpf, x86: add support for indirect jumps") Signed-off-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260715141122.15783-3-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2d323f13da19..52be0a118cce 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17941,6 +17941,7 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map) return err; } env->insn_array_maps[env->insn_array_map_cnt++] = map; + env->prog->jit_required = true; } return env->used_map_cnt - 1; From 7a0855e73757ee9cf25ba635a1c735018ecba742 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 15 Jul 2026 22:11:22 +0800 Subject: [PATCH 106/373] bpf: Disallow interpreter fallback for BPF_ADDR_PERCPU insn The BPF_MOV64_PERCPU_REG insn requires JIT to emit native code to for 'dst_reg = src_reg + '. However, the interpreter ignores the 'off' at its ALU64_MOV_X label. The 'off' indicates the insn is BPF_MOV64_PERCPU_REG insn. Then, when the interpreter loads memory from the register, it will hit a page fault. [ 2.545572] BUG: unable to handle page fault for address: ffffffffacaaf034 [ 2.546485] #PF: supervisor read access in kernel mode [ 2.547167] #PF: error_code(0x0000) - not-present page [ 2.547850] PGD 134e63067 P4D 134e63067 PUD 134e64063 PMD 10021c063 PTE 800ffffeca550062 [ 2.548912] Oops: Oops: 0000 [#1] SMP PTI Set jit_required as true in order to disallow interpreter fallback in core.c::__bpf_prog_select_runtime(), if any BPF_ADDR_PERCPU insn is patched to the prog. BTW, rename the helper bpf_map_supports_cpu_flags() to bpf_map_is_percpu_map(). Fixes: 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs") Signed-off-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260715141122.15783-4-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 4 ++-- kernel/bpf/fixups.c | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 31181e0c2b80..d9542127dfdf 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -4163,7 +4163,7 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image) } #endif -static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type) +static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type) { switch (map_type) { case BPF_MAP_TYPE_PERCPU_ARRAY: @@ -4190,7 +4190,7 @@ static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 all return -EINVAL; if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) { - if (!bpf_map_supports_cpu_flags(map->map_type)) + if (!bpf_map_is_percpu_map(map->map_type)) return -EINVAL; if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS)) return -EINVAL; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index d3be972714b2..a0bddada7964 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -2008,6 +2008,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) return -EFAULT; } + if (bpf_map_is_percpu_map(map_ptr->map_type)) + prog->jit_required = true; + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); if (!new_prog) @@ -2112,6 +2115,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) * way, it's fine to back out this inlining logic */ #ifdef CONFIG_SMP + prog->jit_required = true; insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_number); insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); @@ -2133,6 +2137,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) /* Implement bpf_get_current_task() and bpf_get_current_task_btf() inline. */ if ((insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf) && bpf_verifier_inlines_helper_call(env, insn->imm)) { + prog->jit_required = true; insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)¤t_task); insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); insn_buf[2] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0); From cfce77b63375dac81d53f2f85593c548415206b7 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Mon, 20 Jul 2026 15:15:18 +0800 Subject: [PATCH 107/373] bpftool: Skip prog/map that disappears while looking it up by name Looking up a prog or map by name walks the whole id space. There is a window between bpf_prog_get_next_id()/bpf_map_get_next_id() and getting an fd for that id in which an unrelated object can be freed, and the lookup then fails with ENOENT and aborts the whole command. Skip such ids and keep walking, the same way do_show() already does. Signed-off-by: Jiayuan Chen Link: https://lore.kernel.org/bpf/20260720071520.396363-1-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- tools/bpf/bpftool/common.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/bpf/bpftool/common.c b/tools/bpf/bpftool/common.c index 8bfcff9e2f63..ef366ccc9650 100644 --- a/tools/bpf/bpftool/common.c +++ b/tools/bpf/bpftool/common.c @@ -832,6 +832,8 @@ static int prog_fd_by_nametag(void *nametag, int **fds, bool tag) fd = bpf_prog_get_fd_by_id(id); if (fd < 0) { + if (errno == ENOENT) + continue; p_err("can't get prog by id (%u): %s", id, strerror(errno)); goto err_close_fds; @@ -996,6 +998,8 @@ static int map_fd_by_name(char *name, int **fds, opts_ro.open_flags = BPF_F_RDONLY; fd = bpf_map_get_fd_by_id_opts(id, &opts_ro); if (fd < 0) { + if (errno == ENOENT) + continue; p_err("can't get map by id (%u): %s", id, strerror(errno)); goto err_close_fds; From 40f986aed81ff4d137ec101c6babbbc642690eac Mon Sep 17 00:00:00 2001 From: Viktor Malik Date: Wed, 15 Jul 2026 13:22:00 +0200 Subject: [PATCH 108/373] selftests/bpf: Check malloc result with ASSERT_NEQ in test_loader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace ASSERT_OK_PTR by ASSERT_NEQ(res, NULL, ...) when checking the result of malloc. It is more accurate since malloc returns NULL, not an error code, on failure and it also prevents the following false GCC warning when compiling BPF selftests with -O2: In file included from test_loader.c:6: test_loader.c: In function ‘verify_stderr’: /bpf-next/tools/testing/selftests/bpf/test_progs.h:393:22: error: ‘buf’ may be used uninitialized [-Werror=maybe-uninitialized] 393 | int ___err = libbpf_get_error(___res); \ | ^~~~~~~~~~~~~~~~~~~~~~~~ test_loader.c:810:14: note: in expansion of macro ‘ASSERT_OK_PTR’ 810 | if (!ASSERT_OK_PTR(buf, "malloc")) | ^~~~~~~~~~~~~ In file included from /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/bpf.h:32, from /bpf-next/tools/testing/selftests/bpf/test_progs.h:37: /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/libbpf_legacy.h:113:17: note: by argument 1 of type ‘const void *’ to ‘libbpf_get_error’ declared here 113 | LIBBPF_API long libbpf_get_error(const void *ptr); | ^~~~~~~~~~~~~~~~ Fixes: 554e4eb9e4b7 ("selftests/bpf: Reuse stderr parsing for libarena ASAN tests") Signed-off-by: Viktor Malik Link: https://lore.kernel.org/bpf/e25d50805fbcb3632f24b488568ab5ba49b82094.1784112948.git.vmalik@redhat.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_loader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index 3ce32d134e2c..07807757b518 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -807,7 +807,7 @@ static void verify_stderr(int prog_fd, struct expected_msgs *msgs) return; buf = malloc(TEST_LOADER_LOG_BUF_SZ); - if (!ASSERT_OK_PTR(buf, "malloc")) + if (!ASSERT_NEQ(buf, NULL, "malloc")) return; ret = bpf_prog_stream_read(prog_fd, 2, buf, TEST_LOADER_LOG_BUF_SZ - 1, From eb5cd154f174f42079a45f4bd7ee8bc20f2ba6f3 Mon Sep 17 00:00:00 2001 From: Viktor Malik Date: Wed, 15 Jul 2026 13:22:01 +0200 Subject: [PATCH 109/373] selftests/bpf: Check malloc result with ASSERT_NEQ in test_sha256 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace ASSERT_OK_PTR by ASSERT_NEQ(res, NULL, ...) when checking the result of malloc. It is more accurate since malloc returns NULL, not an error code, on failure and it also prevents the following false GCC warning when compiling BPF selftests with -O2: In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c:4: /bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c: In function ‘test_sha256’: ./test_progs.h:393:22: error: ‘data’ may be used uninitialized [-Werror=maybe-uninitialized] 393 | int ___err = libbpf_get_error(___res); \ | ^~~~~~~~~~~~~~~~~~~~~~~~ /bpf-next/tools/testing/selftests/bpf/prog_tests/sha256.c:28:14: note: in expansion of macro ‘ASSERT_OK_PTR’ 28 | if (!ASSERT_OK_PTR(data, "malloc")) | ^~~~~~~~~~~~~ In file included from /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/bpf.h:32, from ./test_progs.h:37: /bpf-next/tools/testing/selftests/bpf/tools/include/bpf/libbpf_legacy.h:113:17: note: by argument 1 of type ‘const void *’ to ‘libbpf_get_error’ declared here 113 | LIBBPF_API long libbpf_get_error(const void *ptr); | ^~~~~~~~~~~~~~~~ Fixes: f09f57c74677 ("selftests/bpf: Add test for libbpf_sha256()") Signed-off-by: Viktor Malik Link: https://lore.kernel.org/bpf/f9dec09cca0c2aa5eeb4fdcd400a13aa19e2c073.1784112948.git.vmalik@redhat.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/sha256.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/sha256.c b/tools/testing/selftests/bpf/prog_tests/sha256.c index 604a0b1423d5..5edbc6194b07 100644 --- a/tools/testing/selftests/bpf/prog_tests/sha256.c +++ b/tools/testing/selftests/bpf/prog_tests/sha256.c @@ -25,10 +25,10 @@ void test_sha256(void) size_t i; data = malloc(MAX_LEN); - if (!ASSERT_OK_PTR(data, "malloc")) + if (!ASSERT_NEQ(data, NULL, "malloc")) goto out; digests = malloc((MAX_LEN + 1) * SHA256_DIGEST_LENGTH); - if (!ASSERT_OK_PTR(digests, "malloc")) + if (!ASSERT_NEQ(digests, NULL, "malloc")) goto out; /* Generate MAX_LEN bytes of "random" data deterministically. */ From dcd164ec67f89e0db5ee025ee9e91280052eb737 Mon Sep 17 00:00:00 2001 From: Viktor Malik Date: Wed, 15 Jul 2026 13:22:02 +0200 Subject: [PATCH 110/373] selftests/bpf: Silence array bounds warning in global_map_resize MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When compiling BPF selftests with -O2, GCC reports an array bounds violation warning in global_map_resize test: In function ‘global_map_resize_bss_subtest’, inlined from ‘test_global_map_resize’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:228:3: /bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:64:33: error: array subscript 1 is above array bounds of ‘int[1]’ [-Werror=array-bounds=] 64 | skel->bss->array[i] = 1; | ~~~~~~~~~~~~~~~~^~~ In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/global_map_resize.c:6: ./test_global_map_resize.skel.h: In function ‘test_global_map_resize’: ./test_global_map_resize.skel.h:44:21: note: while referencing ‘array’ 44 | int array[1]; | ^~~~~ This is a false positive because `array` (a BPF map) has been resized from within the BPF program. GCC doesn't know that so let us silence the warning by accessing the array via a plain pointer. Fixes: 08b089567573 ("libbpf: Selftests for resizing datasec maps") Signed-off-by: Viktor Malik Link: https://lore.kernel.org/bpf/57765bc465a27923c3c093eba222cc24d08d8c40.1784112948.git.vmalik@redhat.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/prog_tests/global_map_resize.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/global_map_resize.c b/tools/testing/selftests/bpf/prog_tests/global_map_resize.c index 56b5baef35c8..602ce30f1720 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_map_resize.c +++ b/tools/testing/selftests/bpf/prog_tests/global_map_resize.c @@ -23,6 +23,7 @@ static void global_map_resize_bss_subtest(void) struct bpf_map *map; const __u32 desired_sz = sizeof(skel->bss->sum) + sysconf(_SC_PAGE_SIZE) * 2; size_t array_len, actual_sz, new_sz; + int *array; skel = test_global_map_resize__open(); if (!ASSERT_OK_PTR(skel, "test_global_map_resize__open")) @@ -58,10 +59,13 @@ static void global_map_resize_bss_subtest(void) goto teardown; /* fill the newly resized array with ones, - * skipping the first element which was previously set + * skipping the first element which was previously set; + * access through a plain pointer to avoid -Warray-bounds + * since the array was resized beyond its declared length. */ + array = skel->bss->array; for (int i = 1; i < array_len; i++) - skel->bss->array[i] = 1; + array[i] = 1; /* set global const values before loading */ skel->rodata->pid = getpid(); From 5763790965eb3414148720f030b20fd5ccc438ca Mon Sep 17 00:00:00 2001 From: Viktor Malik Date: Wed, 15 Jul 2026 13:22:03 +0200 Subject: [PATCH 111/373] selftests/bpf: Silence maybe-uninitialized compiler warning in libarena MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When compiling BPF selftests with -O2, GCC reports a maybe-uninitialized warning in libarena code: In file included from /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:11: In function ‘libarena_asan_init’, inlined from ‘run_test’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:59:8, inlined from ‘test_libarena_asan’ at /bpf-next/tools/testing/selftests/bpf/prog_tests/libarena_asan.c:91:2: /bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h:126:14: error: ‘globals_pages’ may be used uninitialized [-Werror=maybe-uninitialized] 126 | args = (struct asan_init_args){ | ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~ 127 | .arena_all_pages = arena_all_pages, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 128 | .arena_globals_pages = globals_pages, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 129 | }; | ~ /bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h: In function ‘test_libarena_asan’: /bpf-next/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h:118:13: note: ‘globals_pages’ was declared here 118 | u64 globals_pages; | ^~~~~~~~~~~~~ Silence the warning by initializing globals_pages to 0. Fixes: cfc00618b9df ("selftests/bpf: Add ASAN support for libarena selftests") Signed-off-by: Viktor Malik Link: https://lore.kernel.org/bpf/9f77a5c05c3c731ab2655fd66716ab9de4478b15.1784112948.git.vmalik@redhat.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/libarena/include/libarena/userspace.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h b/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h index fc27a4bcf5d7..b6676dd67bc0 100644 --- a/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h +++ b/tools/testing/selftests/bpf/libarena/include/libarena/userspace.h @@ -115,7 +115,7 @@ static inline int libarena_asan_init(int arena_asan_init_fd, { LIBBPF_OPTS(bpf_test_run_opts, opts); struct asan_init_args args; - u64 globals_pages; + u64 globals_pages = 0; int ret; ret = libarena_get_globals_pages(arena_asan_init_fd, From 5f30ac94727ee308cad0b8894e8e01acac34398a Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 21 Jul 2026 03:59:20 -0700 Subject: [PATCH 112/373] bpf, arm64: Optimize cast_user code generation cast_user converts an arena offset into a user address by combining the low 32 bits of the pointer with the upper 32 bits of user_vm_start, while keeping a NULL pointer NULL. The current sequence always emits six instructions: it materializes user_vm_start >> 32 into a register, shifts it into place, and ORs in the offset. The upper half of user_vm_start is a constant, so it can be written directly onto the offset with MOVK. Move the 32-bit offset into dst (which also zeroes the upper 32 bits), then MOVK the non-zero halfwords of the upper address, branching over the MOVKs when the offset is zero so NULL is preserved. This emits at most four instructions, and only one when the upper half of user_vm_start is zero. The generated code is equivalent. Before: ; bpf_addr_space_cast(page1, 1, 0); 7c: mov w10, w8 80: mov w8, #1 84: lsl x8, x8, #32 88: cbz x10, 0xffff800087b80c20 8c: orr x10, x8, x10 90: mov x8, x10 After: ; bpf_addr_space_cast(page1, 1, 0); 7c: mov w8, w8 80: cbz w8, 0xffff800087b80c28 84: movk x8, #1, lsl #32 Signed-off-by: Puranjay Mohan Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260721105921.1070501-1-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index b0075ece4a6e..0f61662b900b 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1284,12 +1284,25 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn case BPF_ALU | BPF_MOV | BPF_X: case BPF_ALU64 | BPF_MOV | BPF_X: if (insn_is_cast_user(insn)) { - emit(A64_MOV(0, tmp, src), ctx); // 32-bit mov clears the upper 32 bits - emit_a64_mov_i(0, dst, ctx->user_vm_start >> 32, ctx); - emit(A64_LSL(1, dst, dst, 32), ctx); - emit(A64_CBZ(1, tmp, 2), ctx); - emit(A64_ORR(1, tmp, dst, tmp), ctx); - emit(A64_MOV(1, dst, tmp), ctx); + u32 upper = ctx->user_vm_start >> 32; + u16 upper_low = upper & 0xffff; + u16 upper_high = upper >> 16; + int nr_movk = !!upper_low + !!upper_high; + + /* + * Build the user address: the low 32 bits are the arena + * offset, the upper 32 bits come from user_vm_start. A + * zero offset must stay NULL, so branch over the MOVKs + * when it is zero. + */ + emit(A64_MOV(0, dst, src), ctx); /* 32-bit mov clears the upper 32 bits */ + if (nr_movk) { + emit(A64_CBZ(0, dst, nr_movk + 1), ctx); + if (upper_low) + emit(A64_MOVK(1, dst, upper_low, 32), ctx); + if (upper_high) + emit(A64_MOVK(1, dst, upper_high, 48), ctx); + } break; } else if (insn_is_mov_percpu_addr(insn)) { if (dst != src) From 7ac6e1ae41a09f1dd4baeeff1d028ae49ee01232 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 14:54:18 +0200 Subject: [PATCH 113/373] bpf: Zero queue and stack outputs on lock failure Queue and stack pop/peek helpers accept an uninitialized output buffer because the verifier expects the helper to initialize it. The empty-map error path clears the buffer, but a failed lock acquisition returns -EBUSY without writing it. Clear the output before returning -EBUSY so BPF programs cannot observe uninitialized stack contents after a failed helper call. Fixes: a34a9f1a19af ("bpf: Avoid deadlock when using queue and stack maps from NMI") Signed-off-by: Kumar Kartikeya Dwivedi Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260719125419.1782196-1-memxor@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/queue_stack_maps.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index 9a5f94371e50..c1c9dee4dcdd 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -99,8 +99,10 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete) int err = 0; void *ptr; - if (raw_res_spin_lock_irqsave(&qs->lock, flags)) + if (raw_res_spin_lock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } if (queue_stack_map_is_empty(qs)) { memset(value, 0, qs->map.value_size); @@ -130,8 +132,10 @@ static long __stack_map_get(struct bpf_map *map, void *value, bool delete) void *ptr; u32 index; - if (raw_res_spin_lock_irqsave(&qs->lock, flags)) + if (raw_res_spin_lock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } if (queue_stack_map_is_empty(qs)) { memset(value, 0, qs->map.value_size); From 0b236ac75d04a235f3574a2208941b22c1e7a965 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= Date: Mon, 20 Jul 2026 08:13:07 -0300 Subject: [PATCH 114/373] selftests/bpf: Fix make install target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After "make install", test_progs fails because two files end up in the wrong place: - bpftool: TEST_GEN_PROGS_EXTENDED flattens it into $(INSTALL_PATH), losing the tools/sbin/ prefix that detect_bpftool_path() expects. Remove it from TEST_GEN_PROGS_EXTENDED and install it explicitly under tools/sbin/ instead. - *.BTF: resolve_btfids writes resolve_btfids.test.o.BTF as a side-effect of the build but INSTALL_RULE never copies it over. Install all *.BTF files alongside the rest of the per-flavor output. Fixes: f21fae577446 ("selftests/bpf: Add a few helpers for bpftool testing") Fixes: 522397d05e7d ("resolve_btfids: Change in-place update with raw binary output") Signed-off-by: Ricardo B. Marlière Acked-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-1-b450eda93dfe@suse.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/Makefile | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index b642ee489ea6..55d394438705 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -324,8 +324,6 @@ TRUNNER_BPFTOOL := $(DEFAULT_BPFTOOL) USE_BOOTSTRAP := "bootstrap/" endif -TEST_GEN_PROGS_EXTENDED += $(TRUNNER_BPFTOOL) - $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED): $(BPFOBJ) TESTING_HELPERS := $(OUTPUT)/testing_helpers.o @@ -1055,10 +1053,13 @@ endif DEFAULT_INSTALL_RULE := $(INSTALL_RULE) override define INSTALL_RULE $(DEFAULT_INSTALL_RULE) + @mkdir -p $(INSTALL_PATH)/tools/sbin + @rsync -a $(if $(PERMISSIVE),--ignore-missing-args) $(TRUNNER_BPFTOOL) $(INSTALL_PATH)/tools/sbin/ + @rsync -a $(if $(PERMISSIVE),--ignore-missing-args) $(OUTPUT)/*.BTF $(INSTALL_PATH)/ @for DIR in $(TEST_INST_SUBDIRS); do \ mkdir -p $(INSTALL_PATH)/$$DIR; \ rsync -a $(if $(PERMISSIVE),--ignore-missing-args) \ - $(OUTPUT)/$$DIR/*.bpf.o \ + $(OUTPUT)/$$DIR/*.bpf.o $(OUTPUT)/$$DIR/*.BTF \ $(INSTALL_PATH)/$$DIR; \ done endef From 71cf3f3275e5f4d7f31bd540608c80535343b427 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= Date: Mon, 20 Jul 2026 08:13:08 -0300 Subject: [PATCH 115/373] selftests/bpf: Fix lsm_bdev dev_t encoding mismatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit progs/lsm_bdev.c keys its verity_devices hashmap with the raw kernel dev_t read straight off bdev->bd_dev, i.e. MKDEV(major, minor) = (major << 20) | minor. prog_tests/lsm_bdev.c instead builds its lookup key with dev_key = (__u32)st.st_rdev from stat(2), but the stat(2) syscall fills st_rdev via the kernel's new_encode_dev(), a different bit layout: (minor & 0xff) | (major << 8) | ((minor & ~0xff) << 12). For any device with a non-trivial major these two values differ, so the lookup can never find what the BPF program stored, and test_lsm_bdev() always fails with: test_lsm_bdev:FAIL:map lookup unexpected error: -2 (errno 2) Reconstruct the raw kernel dev_t from the decoded major/minor instead of casting st_rdev directly, restoring the layout the BPF program actually reads. Fixes: 96f4c251a087 ("selftests/bpf: add block device management selftests") Signed-off-by: Ricardo B. Marlière Acked-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-2-b450eda93dfe@suse.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/lsm_bdev.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/prog_tests/lsm_bdev.c b/tools/testing/selftests/bpf/prog_tests/lsm_bdev.c index a970798e1173..28bc4b117f41 100644 --- a/tools/testing/selftests/bpf/prog_tests/lsm_bdev.c +++ b/tools/testing/selftests/bpf/prog_tests/lsm_bdev.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include "lsm_bdev.skel.h" @@ -172,7 +173,7 @@ void test_lsm_bdev(void) if (!ASSERT_OK(stat(DM_DEV_PATH, &st), "stat dm dev")) goto remove_dm; - dev_key = (__u32)st.st_rdev; + dev_key = (major(st.st_rdev) << 20) | minor(st.st_rdev); /* Look up the device in the BPF map and verify. */ err = bpf_map__lookup_elem(skel->maps.verity_devices, From 7b5ae0481efdac040cea72b4fabd1398109f975b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20B=2E=20Marli=C3=A8re?= Date: Mon, 20 Jul 2026 08:13:09 -0300 Subject: [PATCH 116/373] libbpf: Search /lib64 and /lib in resolve_full_path() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit attach_probe/uprobe-lib and uprobe_autoattach selftests fail with "failed to resolve full path for libc.so.6" on older non-usrmerged distros, where libc.so.6 lives under a top-level /lib64 or /lib rather than /usr/lib64 or /usr/lib. Add /lib64:/lib to the search paths, alongside the existing /usr/lib64:/usr/lib and Debian multiarch entries. Fixes: 1ce3a60e3c28 ("libbpf: auto-resolve programs/libraries when necessary for uprobes") Signed-off-by: Ricardo B. Marlière Acked-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-3-b450eda93dfe@suse.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/libbpf.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index f88b7c8de304..514e4e9daa82 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -12969,13 +12969,14 @@ static const char *arch_specific_lib_paths(void) /* Get full path to program/shared library. */ static int resolve_full_path(const char *file, char *result, size_t result_sz) { - const char *search_paths[3] = {}; + const char *search_paths[4] = {}; int i, perm; if (str_has_sfx(file, ".so") || strstr(file, ".so.")) { search_paths[0] = getenv("LD_LIBRARY_PATH"); search_paths[1] = "/usr/lib64:/usr/lib"; search_paths[2] = arch_specific_lib_paths(); + search_paths[3] = "/lib64:/lib"; perm = R_OK; } else { search_paths[0] = getenv("PATH"); From 6bcecae4b65d80f4474c42a026a5ba229ded347e Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:30 +0000 Subject: [PATCH 117/373] bpf: Extract the is_struct_ops_tramp helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extract the is_struct_ops_tramp helper, and use it in riscv as the current checks are somewhat hacky. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-2-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 6 ------ arch/riscv/net/bpf_jit_comp64.c | 2 +- include/linux/bpf.h | 6 ++++++ 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 0f61662b900b..4cdc7dfb05ba 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2557,12 +2557,6 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs) } } -static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes) -{ - return fentry_nodes->nr_nodes == 1 && - fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS; -} - static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off) { emit_a64_mov_i64(A64_R(10), func_meta, ctx); diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index f9d5347ba966..406e774b4ac3 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1033,7 +1033,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; - bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; + bool is_struct_ops = is_struct_ops_tramp(fentry); void *orig_call = func_addr; bool save_ret; u64 func_meta; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index d9542127dfdf..e066f44a9c05 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2196,6 +2196,12 @@ static inline bool is_tracing_multi(enum bpf_attach_type type) type == BPF_TRACE_FSESSION_MULTI; } +static inline bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes) +{ + return fentry_nodes->nr_nodes == 1 && + fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS; +} + #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) /* This macro helps developer to register a struct_ops type and generate * type information correctly. Developers should use this macro to register From 369e4635d04801f394d5bd42556f21029e95ff93 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:31 +0000 Subject: [PATCH 118/373] riscv, bpf: Fix memory leak in bpf_jit_free MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When bpf_int_jit_compile() is called for subprograms, it returns early during the first pass (!prog->is_func || extra_pass is false), keeping ctx->offset alive for the subsequent extra pass. If JIT compilation fails for a later subprogram, the BPF core aborts and calls bpf_jit_free() to clean up the first subprogram. However, bpf_jit_free() fails to free jit_data->ctx.offset, which causes a memory leak of the JIT context offsets array. Fix this by adding the missing kfree(jit_data->ctx.offset) in bpf_jit_free(). Fixes: 48a8f78c50bd ("bpf, riscv: use prog pack allocator in the BPF JIT") Reported-by: Sashiko Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-3-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index ce3bd3762e08..7cce19118619 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -234,6 +234,7 @@ void bpf_jit_free(struct bpf_prog *prog) */ if (jit_data) { bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header); + kfree(jit_data->ctx.offset); kfree(jit_data); } hdr = bpf_jit_binary_pack_hdr(prog); From f3ab878594b57602aab2231d859f25ccc6b1a534 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:32 +0000 Subject: [PATCH 119/373] riscv, bpf: Using kvzalloc_objs to allocate cache buffer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It is unnecessary to allocate continuous physical memory for cache buffer, and when ebpf program is too large, it may cause memory allocation failure. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-4-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 4 ++-- arch/riscv/net/bpf_jit_core.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 406e774b4ac3..2f06752fb036 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1195,7 +1195,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, } if (fmod_ret->nr_nodes) { - branches_off = kzalloc_objs(int, fmod_ret->nr_nodes); + branches_off = kvzalloc_objs(int, fmod_ret->nr_nodes); if (!branches_off) return -ENOMEM; @@ -1300,7 +1300,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, ret = ctx->ninsns; out: - kfree(branches_off); + kvfree(branches_off); return ret; } diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 7cce19118619..cbfcd287ea16 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -72,7 +72,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); ctx->prog = prog; - ctx->offset = kzalloc_objs(int, prog->len); + ctx->offset = kvzalloc_objs(int, prog->len); if (!ctx->offset) goto out_offset; @@ -170,7 +170,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); out_offset: - kfree(ctx->offset); + kvfree(ctx->offset); kfree(jit_data); prog->aux->jit_data = NULL; } @@ -234,7 +234,7 @@ void bpf_jit_free(struct bpf_prog *prog) */ if (jit_data) { bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header); - kfree(jit_data->ctx.offset); + kvfree(jit_data->ctx.offset); kfree(jit_data); } hdr = bpf_jit_binary_pack_hdr(prog); From 52fb1756ea1d2759dfef2d86245be00b05dac3a2 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:33 +0000 Subject: [PATCH 120/373] riscv, bpf: Fix kernel stack corruption in tailcall with CFI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When CONFIG_CFI_CLANG is enabled, prog->bpf_func already skips the kcfi instruction during setup. Including it again in the tailcall jump offset causes it to jump over an extra 4 bytes, skipping the stack pointer adjustment, which will result in kernel stack corruption. Fixes: 30a59cc79754 ("riscv, bpf: Fix possible infinite tailcall when CONFIG_CFI_CLANG is enabled") Reported-by: Sashiko Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-5-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 2f06752fb036..cedf40927474 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -18,7 +18,6 @@ #define RV_MAX_REG_ARGS 8 #define RV_FENTRY_NINSNS 2 #define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4) -#define RV_KCFI_NINSNS (IS_ENABLED(CONFIG_CFI) ? 1 : 0) /* imm that allows emit_imm to emit max count insns */ #define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF @@ -272,8 +271,8 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) if (!is_tail_call) emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx); emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, - /* kcfi, fentry and TCC init insns will be skipped on tailcall */ - is_tail_call ? (RV_KCFI_NINSNS + RV_FENTRY_NINSNS + 1) * 4 : 0, + /* fentry and TCC init insns will be skipped on tailcall */ + is_tail_call ? (RV_FENTRY_NINSNS + 1) * 4 : 0, ctx); } @@ -2033,6 +2032,8 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) /* emit kcfi type preamble immediately before the first insn */ emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx); + /* bpf prog starts here as kcfi skipped during prog->bpf_func setup */ + /* nops reserved for auipc+jalr pair */ for (i = 0; i < RV_FENTRY_NINSNS; i++) emit(rv_nop(), ctx); From a21731f54cfe59733357d8e0dc932f6bf2fcee2f Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:34 +0000 Subject: [PATCH 121/373] riscv, bpf: Add RV_TAILCALL_OFFSET macro to format tailcall offset MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add RV_TAILCALL_OFFSET macro to format tailcall offset, and correct the relevant comments. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-6-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index cedf40927474..7c6304e0b846 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -20,6 +20,8 @@ #define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4) /* imm that allows emit_imm to emit max count insns */ #define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF +/* fentry and TCC init insns will be skipped on tailcall */ +#define RV_TAILCALL_OFFSET ((RV_FENTRY_NINSNS + 1) * 4) #define RV_REG_TCC RV_REG_A6 #define RV_REG_TCC_SAVED RV_REG_S6 /* Store A6 in S6 if program do calls */ @@ -271,9 +273,7 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) if (!is_tail_call) emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx); emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, - /* fentry and TCC init insns will be skipped on tailcall */ - is_tail_call ? (RV_FENTRY_NINSNS + 1) * 4 : 0, - ctx); + is_tail_call ? RV_TAILCALL_OFFSET : 0, ctx); } static void emit_bcc(u8 cond, u8 rd, u8 rs, int rvoff, @@ -393,7 +393,7 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) off = ninsns_rvoff(tc_ninsn - (ctx->ninsns - start_insn)); emit_branch(BPF_JEQ, RV_REG_T2, RV_REG_ZERO, off, ctx); - /* goto *(prog->bpf_func + 4); */ + /* goto *(prog->bpf_func + RV_TAILCALL_OFFSET); */ off = offsetof(struct bpf_prog, bpf_func); if (is_12b_check(off, insn)) return -1; From ec72848ca0add4aff1e690873c7cd8b5eef4bcfd Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:35 +0000 Subject: [PATCH 122/373] riscv, bpf: Mixing bpf2bpf and tailcalls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the current RV64 JIT, if we just don't initialize the TCC in subprog, the TCC can be propagated from the parent process to the subprocess, but the updated TCC of the parent process cannot be restored when the subprocess exits. Since the RV64 TCC is initialized before saving the callee saved registers into the stack, we cannot use the callee saved register to pass the TCC, otherwise the original value of the callee saved register will be destroyed. So we implemented mixing bpf2bpf and tailcalls similar to x86_64, i.e. using a non-callee saved register to transfer the TCC between functions, and saving that register to the stack to protect the TCC value. As for the tailcall hierarchy issue, inspired by the s390's low-overhead approach, we store TCC from RV_REG_TCC back to stack after calling bpf2bpf call or calling orig bpf func in bpf trampoline. Tests test_bpf.ko and test_verifier have passed, as well as the relative testcases of test_progs*. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-7-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 108 ++++++++++++++++---------------- 2 files changed, 54 insertions(+), 55 deletions(-) diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index da0271790244..419b9d795f2a 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -81,6 +81,7 @@ struct rv_jit_context { int ex_jmp_off; unsigned long flags; int stack_size; + int tcc_offset; u64 arena_vm_start; u64 user_vm_start; }; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 7c6304e0b846..823262ca47eb 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -24,7 +24,6 @@ #define RV_TAILCALL_OFFSET ((RV_FENTRY_NINSNS + 1) * 4) #define RV_REG_TCC RV_REG_A6 -#define RV_REG_TCC_SAVED RV_REG_S6 /* Store A6 in S6 if program do calls */ #define RV_REG_ARENA RV_REG_S7 /* For storing arena_vm_start */ static const int regmap[] = { @@ -58,14 +57,12 @@ static const int pt_regmap[] = { }; enum { - RV_CTX_F_SEEN_TAIL_CALL = 0, RV_CTX_F_SEEN_CALL = RV_REG_RA, RV_CTX_F_SEEN_S1 = RV_REG_S1, RV_CTX_F_SEEN_S2 = RV_REG_S2, RV_CTX_F_SEEN_S3 = RV_REG_S3, RV_CTX_F_SEEN_S4 = RV_REG_S4, RV_CTX_F_SEEN_S5 = RV_REG_S5, - RV_CTX_F_SEEN_S6 = RV_REG_S6, }; static u8 bpf_to_rv_reg(int bpf_reg, struct rv_jit_context *ctx) @@ -78,7 +75,6 @@ static u8 bpf_to_rv_reg(int bpf_reg, struct rv_jit_context *ctx) case RV_CTX_F_SEEN_S3: case RV_CTX_F_SEEN_S4: case RV_CTX_F_SEEN_S5: - case RV_CTX_F_SEEN_S6: __set_bit(reg, &ctx->flags); } return reg; @@ -93,7 +89,6 @@ static bool seen_reg(int reg, struct rv_jit_context *ctx) case RV_CTX_F_SEEN_S3: case RV_CTX_F_SEEN_S4: case RV_CTX_F_SEEN_S5: - case RV_CTX_F_SEEN_S6: return test_bit(reg, &ctx->flags); } return false; @@ -109,32 +104,6 @@ static void mark_call(struct rv_jit_context *ctx) __set_bit(RV_CTX_F_SEEN_CALL, &ctx->flags); } -static bool seen_call(struct rv_jit_context *ctx) -{ - return test_bit(RV_CTX_F_SEEN_CALL, &ctx->flags); -} - -static void mark_tail_call(struct rv_jit_context *ctx) -{ - __set_bit(RV_CTX_F_SEEN_TAIL_CALL, &ctx->flags); -} - -static bool seen_tail_call(struct rv_jit_context *ctx) -{ - return test_bit(RV_CTX_F_SEEN_TAIL_CALL, &ctx->flags); -} - -static u8 rv_tail_call_reg(struct rv_jit_context *ctx) -{ - mark_tail_call(ctx); - - if (seen_call(ctx)) { - __set_bit(RV_CTX_F_SEEN_S6, &ctx->flags); - return RV_REG_S6; - } - return RV_REG_A6; -} - static bool is_32b_int(s64 val) { return -(1L << 31) <= val && val < (1L << 31); @@ -259,15 +228,14 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) emit_ld(RV_REG_S5, store_offset, RV_REG_SP, ctx); store_offset -= 8; } - if (seen_reg(RV_REG_S6, ctx)) { - emit_ld(RV_REG_S6, store_offset, RV_REG_SP, ctx); - store_offset -= 8; - } if (ctx->arena_vm_start) { emit_ld(RV_REG_ARENA, store_offset, RV_REG_SP, ctx); store_offset -= 8; } + /* restore TCC from stack to RV_REG_TCC */ + emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx); + emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx); /* Set return value. */ if (!is_tail_call) @@ -354,7 +322,6 @@ static void emit_branch(u8 cond, u8 rd, u8 rs, int rvoff, static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) { int tc_ninsn, off, start_insn = ctx->ninsns; - u8 tcc = rv_tail_call_reg(ctx); /* a0: &ctx * a1: &array @@ -377,7 +344,8 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) /* if (--TCC < 0) * goto out; */ - emit_addi(RV_REG_TCC, tcc, -1, ctx); + emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx); + emit_addi(RV_REG_TCC, RV_REG_TCC, -1, ctx); off = ninsns_rvoff(tc_ninsn - (ctx->ninsns - start_insn)); emit_branch(BPF_JSLT, RV_REG_TCC, RV_REG_ZERO, off, ctx); @@ -393,6 +361,9 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) off = ninsns_rvoff(tc_ninsn - (ctx->ninsns - start_insn)); emit_branch(BPF_JEQ, RV_REG_T2, RV_REG_ZERO, off, ctx); + /* store updated TCC back to stack */ + emit_sd(RV_REG_SP, ctx->tcc_offset, RV_REG_TCC, ctx); + /* goto *(prog->bpf_func + RV_TAILCALL_OFFSET); */ off = offsetof(struct bpf_prog, bpf_func); if (is_12b_check(off, insn)) @@ -1027,7 +998,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, int i, ret, offset; int *branches_off = NULL; int stack_size = 0, nr_arg_slots = 0; - int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off; + int retval_off, args_off, func_meta_off, ip_off; + int run_ctx_off, sreg_off, stk_arg_off, tcc_off; int cookie_off, cookie_cnt; struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; @@ -1078,6 +1050,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, * * FP - sreg_off [ callee saved reg ] * + * FP - tcc_off [ tail call count ] BPF_TRAMP_F_TAIL_CALL_CTX + * * [ pads ] pads for 16 bytes alignment * * [ stack_argN ] @@ -1125,6 +1099,11 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, stack_size += 8; sreg_off = stack_size; + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { + stack_size += 8; + tcc_off = stack_size; + } + if ((flags & BPF_TRAMP_F_CALL_ORIG) && (nr_arg_slots - RV_MAX_REG_ARGS > 0)) stack_size += (nr_arg_slots - RV_MAX_REG_ARGS) * 8; @@ -1159,6 +1138,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, emit_addi(RV_REG_FP, RV_REG_SP, stack_size, ctx); } + /* store tail call count */ + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx); + /* callee saved register S1 to pass start time */ emit_sd(RV_REG_FP, -sreg_off, RV_REG_S1, ctx); @@ -1217,9 +1200,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, orig_call += RV_FENTRY_NINSNS * 4; restore_args(min_t(int, nr_arg_slots, RV_MAX_REG_ARGS), args_off, ctx); restore_stack_args(nr_arg_slots - RV_MAX_REG_ARGS, args_off, stk_arg_off, ctx); + /* restore TCC to RV_REG_TCC before calling the orig bpf func */ + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + emit_ld(RV_REG_TCC, -tcc_off, RV_REG_FP, ctx); ret = emit_call((const u64)orig_call, true, ctx); if (ret) goto out; + /* store updated TCC back to stack after calling the orig bpf func */ + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx); emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx); emit_sd(RV_REG_FP, -(retval_off - 8), regmap[BPF_REG_0], ctx); im->ip_after_call = ctx->ro_insns + ctx->ninsns; @@ -1272,6 +1261,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, emit_ld(RV_REG_S1, -sreg_off, RV_REG_FP, ctx); + /* restore TCC from stack to RV_REG_TCC */ + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + emit_ld(RV_REG_TCC, -tcc_off, RV_REG_FP, ctx); + if (!is_struct_ops) { /* trampoline called from function entry */ emit_ld(RV_REG_T0, stack_size - 8, RV_REG_SP, ctx); @@ -1836,10 +1829,18 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, } } + /* restore TCC to RV_REG_TCC before bpf2bpf call */ + if (aux->tail_call_reachable && insn->src_reg == BPF_PSEUDO_CALL) + emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx); + ret = emit_call(addr, fixed_addr, ctx); if (ret) return ret; + /* store updated TCC back to stack after bpf2bpf call */ + if (aux->tail_call_reachable && insn->src_reg == BPF_PSEUDO_CALL) + emit_sd(RV_REG_SP, ctx->tcc_offset, RV_REG_TCC, ctx); + if (insn->src_reg != BPF_PSEUDO_CALL) emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_A0, ctx); break; @@ -2019,10 +2020,9 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) stack_adjust += 8; if (seen_reg(RV_REG_S5, ctx)) stack_adjust += 8; - if (seen_reg(RV_REG_S6, ctx)) - stack_adjust += 8; if (ctx->arena_vm_start) stack_adjust += 8; + stack_adjust += 8; /* RV_REG_TCC */ stack_adjust = round_up(stack_adjust, STACK_ALIGN); stack_adjust += bpf_stack_adjust; @@ -2038,11 +2038,10 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) for (i = 0; i < RV_FENTRY_NINSNS; i++) emit(rv_nop(), ctx); - /* First instruction is always setting the tail-call-counter - * (TCC) register. This instruction is skipped for tail calls. - * Force using a 4-byte (non-compressed) instruction. - */ - emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx); + if (!is_subprog) + emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx); + + /* tailcall starts here, emit insn before it must be fixed */ emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx); @@ -2072,26 +2071,20 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx); store_offset -= 8; } - if (seen_reg(RV_REG_S6, ctx)) { - emit_sd(RV_REG_SP, store_offset, RV_REG_S6, ctx); - store_offset -= 8; - } if (ctx->arena_vm_start) { emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx); store_offset -= 8; } + /* store TCC from RV_REG_TCC to stack */ + emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx); + ctx->tcc_offset = store_offset; + emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx); if (bpf_stack_adjust) emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx); - /* Program contains calls and tail calls, so RV_REG_TCC need - * to be saved across calls. - */ - if (seen_tail_call(ctx) && seen_call(ctx)) - emit_mv(RV_REG_TCC_SAVED, RV_REG_TCC, ctx); - ctx->stack_size = stack_adjust; if (ctx->arena_vm_start) @@ -2158,3 +2151,8 @@ bool bpf_jit_supports_fsession(void) { return true; } + +bool bpf_jit_supports_subprog_tailcalls(void) +{ + return true; +} From 683ed8b78c1d03bdddb17188b45ea803761d6d5f Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Wed, 8 Jul 2026 06:44:36 +0000 Subject: [PATCH 123/373] selftests/bpf: Remove tailcalls tests from DENYLIST.riscv64 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The RV64 BPF JIT now supports mixing bpf2bpf and tailcalls. Therefore, the tailcall_bpf2bpf tests can be safely removed from the riscv64 denylist. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-8-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/DENYLIST.riscv64 | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/testing/selftests/bpf/DENYLIST.riscv64 b/tools/testing/selftests/bpf/DENYLIST.riscv64 index 4fc4dfdde293..ca1beae7fe8f 100644 --- a/tools/testing/selftests/bpf/DENYLIST.riscv64 +++ b/tools/testing/selftests/bpf/DENYLIST.riscv64 @@ -1,3 +1,2 @@ # riscv64 deny list for BPF CI and local vmtest exceptions # JIT does not support exceptions -tailcalls/tailcall_bpf2bpf* # JIT does not support mixing bpf2bpf and tailcalls From 5eb8921371c6fd117d4a328b6053dfda38707df8 Mon Sep 17 00:00:00 2001 From: Feng Jiang Date: Mon, 20 Jul 2026 06:42:57 +0000 Subject: [PATCH 124/373] bpf, riscv: Fix extable handling for arena load_acquire MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit emit_atomic_ld_st() returns 1 to have build_body() skip the zext after a sub-word load_acquire. The caller does "ret = ret ?: add_exception_handler(...)", which skips add_exception_handler() on any non-zero ret, so the extable entry is missing and a faulting PROBE_ATOMIC load_acquire oopses. REG_DONT_CLEAR_MARKER leaves rd stale on fault, and the verifier still thinks the load overwrote it, so a program can leak it through a map. Check ret >= 0 before calling add_exception_handler(), and pass rd for LOAD_ACQ so the fault zeroes rd like a PROBE_MEM load. Return ret unchanged for the zext skip. Fixes: fb7cefabae81 ("riscv, bpf: Add support arena atomics for RV64") Suggested-by: Pu Lehui Signed-off-by: Feng Jiang Reviewed-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260720-bpf-riscv-fix-extable-v4-1-165c0b3b07d5@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 823262ca47eb..ad089a9a4ea9 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1986,7 +1986,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, else ret = emit_atomic_rmw(rd, rs, insn, ctx); - ret = ret ?: add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx); + /* ret can be 1 (skip-zext); extable entry still needs to be added */ + if (ret >= 0) + ret = add_exception_handler(insn, + insn->imm == BPF_LOAD_ACQ ? rd : REG_DONT_CLEAR_MARKER, + ctx) ?: ret; + if (ret) return ret; break; From 2e0fa2389c50fd3a69bb738efa74e127b7516938 Mon Sep 17 00:00:00 2001 From: Kuan-Wei Chiu Date: Tue, 14 Jul 2026 00:24:49 +0000 Subject: [PATCH 125/373] riscv, bpf: Add support for BPF_SDIV and BPF_SMOD in RV32 JIT The current rv32 bpf jit compiler incorrectly treats BPF_SDIV and BPF_SMOD as unsigned operations. The BPF instruction set allows signed division and modulo by reusing the BPF_DIV and BPF_MOD opcodes with the instruction offset set to 1. Update the emit_alu_r32() function to accept an 'is_sdiv' variable and emit the correct div and rem instructions when the offset is 1. Before this patch: [ 44.161771] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times) [ 44.167385] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times) [ 44.171053] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times) [ 44.172081] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times) After this patch: [ 16.002192] test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 95 PASS [ 16.002983] test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 1059 PASS [ 16.017167] test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 136 PASS [ 16.023002] test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 109 PASS Signed-off-by: Kuan-Wei Chiu Reviewed-by: Pu Lehui Link: https://lore.kernel.org/bpf/20260714002451.4091139-2-visitorckw@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp32.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp32.c b/arch/riscv/net/bpf_jit_comp32.c index 592dd86fbf81..89153946a4e4 100644 --- a/arch/riscv/net/bpf_jit_comp32.c +++ b/arch/riscv/net/bpf_jit_comp32.c @@ -509,12 +509,15 @@ static void emit_alu_r64(const s8 *dst, const s8 *src, } static void emit_alu_r32(const s8 *dst, const s8 *src, - struct rv_jit_context *ctx, const u8 op) + struct rv_jit_context *ctx, + const struct bpf_insn *insn) { const s8 *tmp1 = bpf2rv32[TMP_REG_1]; const s8 *tmp2 = bpf2rv32[TMP_REG_2]; const s8 *rd = bpf_get_reg32(dst, tmp1, ctx); const s8 *rs = bpf_get_reg32(src, tmp2, ctx); + u8 op = BPF_OP(insn->code); + bool is_signed = insn->off == 1; switch (op) { case BPF_MOV: @@ -539,10 +542,12 @@ static void emit_alu_r32(const s8 *dst, const s8 *src, emit(rv_mul(lo(rd), lo(rd), lo(rs)), ctx); break; case BPF_DIV: - emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx); + emit(is_signed ? rv_div(lo(rd), lo(rd), lo(rs)) : + rv_divu(lo(rd), lo(rd), lo(rs)), ctx); break; case BPF_MOD: - emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx); + emit(is_signed ? rv_rem(lo(rd), lo(rd), lo(rs)) : + rv_remu(lo(rd), lo(rd), lo(rs)), ctx); break; case BPF_LSH: emit(rv_sll(lo(rd), lo(rd), lo(rs)), ctx); @@ -1041,7 +1046,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm32(tmp2, imm, ctx); src = tmp2; } - emit_alu_r32(dst, src, ctx, BPF_OP(code)); + emit_alu_r32(dst, src, ctx, insn); break; case BPF_ALU | BPF_MOV | BPF_K: @@ -1065,7 +1070,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, * src is ignored---choose tmp2 as a dummy register since it * is not on the stack. */ - emit_alu_r32(dst, tmp2, ctx, BPF_OP(code)); + emit_alu_r32(dst, tmp2, ctx, insn); break; case BPF_ALU | BPF_END | BPF_FROM_LE: From c6a08afdfe3a81b1a54e921921e9481dd6c78f8a Mon Sep 17 00:00:00 2001 From: Kuan-Wei Chiu Date: Tue, 14 Jul 2026 00:24:50 +0000 Subject: [PATCH 126/373] riscv, bpf: Add support for BPF_MOVSX in RV32 JIT The current rv32 bpf jit compiler incorrectly treats BPF_MOVSX as a standard zero-extended move operation. The bpf instruction set allows sign-extension moves by reusing the BPF_MOV opcode with the instruction offset set to 8, 16, or 32. Update the bpf_jit_emit_insn() function to check the offset field for both ALU and ALU64 MOV operations. If the offset is non-zero, emit the correct slli and srai instructions to perform the sign extension. Before this patch: [ 19.549705] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) [ 19.551354] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) [ 19.552576] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) [ 19.553542] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) [ 19.554807] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) After this patch: [ 17.931172] test_bpf: #82 ALU_MOVSX | BPF_B jited:1 125 PASS [ 17.932198] test_bpf: #83 ALU_MOVSX | BPF_H jited:1 124 PASS [ 17.933039] test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 124 PASS [ 17.933918] test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 124 PASS [ 17.934751] test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 122 PASS Signed-off-by: Kuan-Wei Chiu Reviewed-by: Pu Lehui Link: https://lore.kernel.org/bpf/20260714002451.4091139-3-visitorckw@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/riscv/net/bpf_jit_comp32.c b/arch/riscv/net/bpf_jit_comp32.c index 89153946a4e4..39e2b0b907dc 100644 --- a/arch/riscv/net/bpf_jit_comp32.c +++ b/arch/riscv/net/bpf_jit_comp32.c @@ -972,6 +972,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, switch (code) { case BPF_ALU64 | BPF_MOV | BPF_X: + if (insn->off != 0) { + const s8 *rd = bpf_get_reg64(dst, tmp1, ctx); + const s8 *rs = bpf_get_reg64(src, tmp2, ctx); + + if (insn->off == 8) { + emit(rv_slli(lo(rd), lo(rs), 24), ctx); + emit(rv_srai(lo(rd), lo(rd), 24), ctx); + } else if (insn->off == 16) { + emit(rv_slli(lo(rd), lo(rs), 16), ctx); + emit(rv_srai(lo(rd), lo(rd), 16), ctx); + } else { + emit(rv_addi(lo(rd), lo(rs), 0), ctx); + } + emit(rv_srai(hi(rd), lo(rd), 31), ctx); + bpf_put_reg64(dst, rd, ctx); + break; + } + fallthrough; case BPF_ALU64 | BPF_ADD | BPF_X: case BPF_ALU64 | BPF_ADD | BPF_K: @@ -1022,6 +1040,20 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_zext64(dst, ctx); break; } + if (insn->off != 0) { + const s8 *rd = bpf_get_reg32(dst, tmp1, ctx); + const s8 *rs = bpf_get_reg32(src, tmp2, ctx); + + if (insn->off == 8) { + emit(rv_slli(lo(rd), lo(rs), 24), ctx); + emit(rv_srai(lo(rd), lo(rd), 24), ctx); + } else if (insn->off == 16) { + emit(rv_slli(lo(rd), lo(rs), 16), ctx); + emit(rv_srai(lo(rd), lo(rd), 16), ctx); + } + bpf_put_reg32(dst, rd, ctx); + break; + } fallthrough; case BPF_ALU | BPF_ADD | BPF_X: From a1b37972efc0b8b01221caf09376654f7a401875 Mon Sep 17 00:00:00 2001 From: Kuan-Wei Chiu Date: Tue, 14 Jul 2026 00:24:51 +0000 Subject: [PATCH 127/373] riscv, bpf: Add 32 bit atomic operations to RV32 JIT The RV32 BPF JIT compiler currently only supports the BPF_ADD atomic operation. Other 32 bit atomic operations (and, or, xor, xchg) and their BPF_FETCH variants are not supported and gracefully fall back to the interpreter. Since the RISC-V A extension is required for Linux on RV32, we can natively support these 32-bit BPF atomic operations by mapping them directly to the corresponding RISC-V amo*.w instructions. Implement BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, and BPF_XCHG with and without BPF_FETCH. BPF_CMPXCHG requires a more complex lr.w/sc.w loop and is left to fall back to the interpreter. Before this patch: [ 138.862161] test_bpf: Summary: 1054 PASSED, 0 FAILED, [843/1042 JIT'ed] After this patch: [ 157.024124] test_bpf: Summary: 1054 PASSED, 0 FAILED, [902/1042 JIT'ed] Signed-off-by: Kuan-Wei Chiu Reviewed-by: Pu Lehui Link: https://lore.kernel.org/bpf/20260714002451.4091139-4-visitorckw@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp32.c | 64 +++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 11 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp32.c b/arch/riscv/net/bpf_jit_comp32.c index 39e2b0b907dc..a9e0bd5cc81d 100644 --- a/arch/riscv/net/bpf_jit_comp32.c +++ b/arch/riscv/net/bpf_jit_comp32.c @@ -874,14 +874,58 @@ static int emit_load_r64(const s8 *dst, const s8 *src, s16 off, return 0; } -static int emit_store_r64(const s8 *dst, const s8 *src, s16 off, - struct rv_jit_context *ctx, const u8 size, - const u8 mode) +static int emit_bpf_atomic(s8 dst, const s8 *src, const s8 *rs, + struct rv_jit_context *ctx, + const struct bpf_insn *insn) +{ + s32 imm = insn->imm; + bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG); + s8 fetch_reg = is_fetch ? lo(rs) : RV_REG_ZERO; + int aq = is_fetch ? 1 : 0; + int rl = is_fetch ? 1 : 0; + + switch (imm) { + case BPF_ADD: + case BPF_ADD | BPF_FETCH: + emit(rv_amoadd_w(fetch_reg, lo(rs), dst, aq, rl), ctx); + break; + case BPF_AND: + case BPF_AND | BPF_FETCH: + emit(rv_amoand_w(fetch_reg, lo(rs), dst, aq, rl), ctx); + break; + case BPF_OR: + case BPF_OR | BPF_FETCH: + emit(rv_amoor_w(fetch_reg, lo(rs), dst, aq, rl), ctx); + break; + case BPF_XOR: + case BPF_XOR | BPF_FETCH: + emit(rv_amoxor_w(fetch_reg, lo(rs), dst, aq, rl), ctx); + break; + case BPF_XCHG: + emit(rv_amoswap_w(fetch_reg, lo(rs), dst, aq, rl), ctx); + break; + default: + return -1; + } + + if (is_fetch) { + emit(rv_addi(hi(rs), RV_REG_ZERO, 0), ctx); + bpf_put_reg64(src, rs, ctx); + } + return 0; +} + +static int emit_store_r64(const s8 *dst, const s8 *src, + struct rv_jit_context *ctx, + const struct bpf_insn *insn) { const s8 *tmp1 = bpf2rv32[TMP_REG_1]; const s8 *tmp2 = bpf2rv32[TMP_REG_2]; const s8 *rd = bpf_get_reg64(dst, tmp1, ctx); const s8 *rs = bpf_get_reg64(src, tmp2, ctx); + u8 size = BPF_SIZE(insn->code); + u8 mode = BPF_MODE(insn->code); + s16 off = insn->off; if (mode == BPF_ATOMIC && size != BPF_W) return -1; @@ -901,9 +945,9 @@ static int emit_store_r64(const s8 *dst, const s8 *src, s16 off, case BPF_MEM: emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx); break; - case BPF_ATOMIC: /* Only BPF_ADD supported */ - emit(rv_amoadd_w(RV_REG_ZERO, lo(rs), RV_REG_T0, 0, 0), - ctx); + case BPF_ATOMIC: + if (emit_bpf_atomic(RV_REG_T0, src, rs, ctx, insn)) + return -1; break; } break; @@ -1303,21 +1347,19 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, src = tmp2; } - if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code), - BPF_MODE(code))) + if (emit_store_r64(dst, src, ctx, insn)) return -1; break; case BPF_STX | BPF_ATOMIC | BPF_W: - if (insn->imm != BPF_ADD) { + if (insn->imm == BPF_CMPXCHG) { pr_info_once( "bpf-jit: not supported: atomic operation %02x ***\n", insn->imm); return -EFAULT; } - if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code), - BPF_MODE(code))) + if (emit_store_r64(dst, src, ctx, insn)) return -1; break; From fcac2b4a3bdfbb6bd34ef671f0e77ccaa3051c5b Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Tue, 21 Jul 2026 10:21:04 -0700 Subject: [PATCH 128/373] MAINTAINERS: BPF: Add self as reviewer Add myself as a reviewer for the BPF areas where I've been active. Signed-off-by: Ihor Solodrai Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260721172104.3689982-1-ihor.solodrai@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- MAINTAINERS | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index a674e36529f7..e45807bf019e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4916,6 +4916,7 @@ R: Song Liu R: Yonghong Song R: Jiri Olsa R: Emil Tsalapatis +R: Ihor Solodrai L: bpf@vger.kernel.org S: Supported W: https://bpf.io/ @@ -4976,6 +4977,7 @@ F: net/unix/unix_bpf.c BPF [LIBRARY] (libbpf) M: Andrii Nakryiko M: Eduard Zingerman +R: Ihor Solodrai L: bpf@vger.kernel.org S: Maintained F: tools/lib/bpf/ @@ -5041,6 +5043,7 @@ F: security/bpf/ BPF [SELFTESTS] (Test Runners & Infrastructure) M: Andrii Nakryiko M: Eduard Zingerman +R: Ihor Solodrai L: bpf@vger.kernel.org S: Maintained F: tools/testing/selftests/bpf/ @@ -5055,6 +5058,7 @@ F: tools/bpf/bpftool/ BPF [TRACING] M: Song Liu R: Jiri Olsa +R: Ihor Solodrai L: bpf@vger.kernel.org S: Maintained F: kernel/bpf/stackmap.c From 04e19012efaec2bfd8c3b37fd8a6c3f1fe731ffc Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 17:36:28 +0200 Subject: [PATCH 129/373] bpf: Fix offset warn check for bpf_res_spin_lock Sashiko pointed out correctly that the case statement for BPF_RES_SPIN_LOCK incorrectly checks offset for BPF_SPIN_LOCK. Fix it by checking res_spin_lock_off instead. Fixes: 0de2046137f9 ("bpf: Implement verifier support for rqspinlock") Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260719153634.2908692-2-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/btf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index cbb1e49b9bcb..e7d4e9ba24e2 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -4168,7 +4168,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type rec->spin_lock_off = rec->fields[i].offset; break; case BPF_RES_SPIN_LOCK: - WARN_ON_ONCE(rec->spin_lock_off >= 0); + WARN_ON_ONCE(rec->res_spin_lock_off >= 0); /* Cache offset for faster lookup at runtime */ rec->res_spin_lock_off = rec->fields[i].offset; break; From f08619f060468076e4acbdc10e0713af20d60e65 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 17:36:29 +0200 Subject: [PATCH 130/373] bpf: Preserve unique-field state across nested structs btf_find_struct_field() initializes a fresh seen mask for every recursive descent. Unique special fields in different levels of the same aggregate therefore do not see one another. The duplicate fields can reach btf_parse_fields(), where they trigger an invariant WARN_ON_ONCE(). A crafted user BTF can consequently trigger the warning before map creation checks capabilities. Initialize the seen mask once in btf_find_field() and pass the same pointer through struct, datasec, and nested-struct walks. This gives the entire field traversal one shared uniqueness state. Fixes: 64e8ee814819 ("bpf: look into the types of the fields of a struct type recursively.") Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260719153634.2908692-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/btf.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index e7d4e9ba24e2..c577f00e9d88 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3751,7 +3751,7 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt, - u32 level); + u32 level, u32 *seen_mask); /* Find special fields in the struct type of a field. * @@ -3762,7 +3762,7 @@ static int btf_find_struct_field(const struct btf *btf, static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t, u32 off, u32 nelems, u32 field_mask, struct btf_field_info *info, - int info_cnt, u32 level) + int info_cnt, u32 level, u32 *seen_mask) { int ret, err, i; @@ -3770,7 +3770,7 @@ static int btf_find_nested_struct(const struct btf *btf, const struct btf_type * if (level >= MAX_RESOLVE_DEPTH) return -E2BIG; - ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level); + ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level, seen_mask); if (ret <= 0) return ret; @@ -3827,7 +3827,7 @@ static int btf_find_field_one(const struct btf *btf, if (expected_size && expected_size != sz * nelems) return 0; ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask, - &info[0], info_cnt, level); + &info[0], info_cnt, level, seen_mask); return ret; } @@ -3892,11 +3892,11 @@ static int btf_find_field_one(const struct btf *btf, static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt, - u32 level) + u32 level, u32 *seen_mask) { int ret, idx = 0; const struct btf_member *member; - u32 i, off, seen_mask = 0; + u32 i, off; for_each_member(i, t, member) { const struct btf_type *member_type = btf_type_by_id(btf, @@ -3910,7 +3910,7 @@ static int btf_find_struct_field(const struct btf *btf, ret = btf_find_field_one(btf, t, member_type, i, off, 0, - field_mask, &seen_mask, + field_mask, seen_mask, &info[idx], info_cnt - idx, level); if (ret < 0) return ret; @@ -3921,11 +3921,11 @@ static int btf_find_struct_field(const struct btf *btf, static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, - int info_cnt, u32 level) + int info_cnt, u32 level, u32 *seen_mask) { int ret, idx = 0; const struct btf_var_secinfo *vsi; - u32 i, off, seen_mask = 0; + u32 i, off; for_each_vsi(i, t, vsi) { const struct btf_type *var = btf_type_by_id(btf, vsi->type); @@ -3933,7 +3933,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, off = vsi->offset; ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size, - field_mask, &seen_mask, + field_mask, seen_mask, &info[idx], info_cnt - idx, level); if (ret < 0) @@ -3947,10 +3947,12 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt) { + u32 seen_mask = 0; + if (__btf_type_is_struct(t)) - return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0); + return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0, &seen_mask); else if (btf_type_is_datasec(t)) - return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0); + return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0, &seen_mask); return -EINVAL; } From 61e655391cb19c31f94ecd4354f624c81ce4cf75 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 17:36:30 +0200 Subject: [PATCH 131/373] bpf: Mark bpf_refcount field as unique BPF_REFCOUNT is not marked as a unique field, while it should be. Fix this oversight. Fixes: d54730b50bae ("bpf: Introduce opaque bpf_refcount struct and add btf_record plumbing") Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260719153634.2908692-4-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/btf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index c577f00e9d88..4eeeaeb69790 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3669,7 +3669,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_ { BPF_LIST_NODE, "bpf_list_node", false }, { BPF_RB_ROOT, "bpf_rb_root", false }, { BPF_RB_NODE, "bpf_rb_node", false }, - { BPF_REFCOUNT, "bpf_refcount", false }, + { BPF_REFCOUNT, "bpf_refcount", true }, }; int type = 0, i; const char *name = __btf_name_by_offset(btf, var_type->name_off); From 9f2afb635cffd670e71580a3070b6e5c68fb99c5 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 17:36:31 +0200 Subject: [PATCH 132/373] selftests/bpf: Test duplicate unique fields in nested structs Add a raw BTF test with a spin lock directly in a struct and another in a nested struct. The duplicate must now be rejected during BTF loading. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260719153634.2908692-5-memxor@gmail.com Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/prog_tests/btf.c | 27 ++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c index 66855cbd6b73..2100400d896b 100644 --- a/tools/testing/selftests/bpf/prog_tests/btf.c +++ b/tools/testing/selftests/bpf/prog_tests/btf.c @@ -4250,6 +4250,33 @@ static struct btf_raw_test raw_tests[] = { .max_entries = 1, }, +/* + * struct inner { + * struct bpf_spin_lock lock; + * }; + * + * struct value { + * struct bpf_spin_lock lock; + * struct inner nested; + * }; + */ +{ + .descr = "struct test duplicate nested unique fields", + .raw_types = { + BTF_TYPE_INT_ENC(NAME_TBD, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_STRUCT_ENC(NAME_TBD, 1, 4), /* [2] */ + BTF_MEMBER_ENC(NAME_TBD, 1, 0), + BTF_STRUCT_ENC(NAME_TBD, 1, 4), /* [3] */ + BTF_MEMBER_ENC(NAME_TBD, 2, 0), + BTF_STRUCT_ENC(NAME_TBD, 2, 8), /* [4] */ + BTF_MEMBER_ENC(NAME_TBD, 2, 0), + BTF_MEMBER_ENC(NAME_TBD, 3, 32), + BTF_END_RAW, + }, + BTF_STR_SEC("\0int\0bpf_spin_lock\0val\0inner\0lock\0value\0lock\0nested"), + .btf_load_err = true, +}, + { .descr = "struct test repeated fields count overflow", .raw_types = { From 643bd5af853f41cc16e8d0d2e0ae12d60844402d Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 17:36:32 +0200 Subject: [PATCH 133/373] selftests/bpf: Test duplicate bpf_refcount fields Add a raw BTF test with two bpf_refcount fields. The duplicate must be rejected during BTF loading instead of reaching the duplicate-field invariant in btf_parse_fields(). Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260719153634.2908692-6-memxor@gmail.com Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/prog_tests/btf.c | 21 ++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/btf.c b/tools/testing/selftests/bpf/prog_tests/btf.c index 2100400d896b..67b9015cbd98 100644 --- a/tools/testing/selftests/bpf/prog_tests/btf.c +++ b/tools/testing/selftests/bpf/prog_tests/btf.c @@ -4277,6 +4277,27 @@ static struct btf_raw_test raw_tests[] = { .btf_load_err = true, }, +/* + * struct value { + * struct bpf_refcount a; + * struct bpf_refcount b; + * }; + */ +{ + .descr = "struct test duplicate bpf_refcount fields", + .raw_types = { + BTF_TYPE_INT_ENC(NAME_TBD, BTF_INT_SIGNED, 0, 32, 4), /* [1] */ + BTF_STRUCT_ENC(NAME_TBD, 1, 4), /* [2] */ + BTF_MEMBER_ENC(NAME_TBD, 1, 0), + BTF_STRUCT_ENC(NAME_TBD, 2, 8), /* [3] */ + BTF_MEMBER_ENC(NAME_TBD, 2, 0), + BTF_MEMBER_ENC(NAME_TBD, 2, 32), + BTF_END_RAW, + }, + BTF_STR_SEC("\0int\0bpf_refcount\0refs\0value\0a\0b"), + .btf_load_err = true, +}, + { .descr = "struct test repeated fields count overflow", .raw_types = { From 810a273919df09b345cdee74869c030aadbaa891 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 16 Jul 2026 10:51:35 +0300 Subject: [PATCH 134/373] bpf: dispatcher: Allocate bpf_dispatcher->rw_image with vzalloc() bpf_dispatcher->rw_image is a temporary writable buffer that arch_prepare_bpf_dispatcher() fills and then copies into bpf_dispatcher->image using bpf_arch_text_copy(). The rel32 offsets emitted by emit_bpf_dispatcher() are calculated against ->image, so ->rw_image does not need to live in the module address range. Allocate ->rw_image with vzalloc() to avoid permissions dance when EXECMEM_BPF will be backed by ROX caches. Using vzalloc() rather than vmalloc() ensures that the memory that bpf_dispatcher_update() unconditionally copies into the executable buffer is zeroed, which is not ideal but still better than random memory returned by the existing bpf_jit_alloc_exec() or plain vmalloc(). Switching from bpf_jit_alloc_exec() to vzalloc() also saves a bit of space in the more scarce module address space. Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-1-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/dispatcher.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c index ea2d60dc1fee..79f0c222c583 100644 --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, d->image = bpf_prog_pack_alloc(PAGE_SIZE, bpf_jit_fill_hole_with_zero, false); if (!d->image) goto out; - d->rw_image = bpf_jit_alloc_exec(PAGE_SIZE); + /* d->rw_image doesn't need to be in module memory range, so we + * can use vzalloc. + */ + d->rw_image = vzalloc(PAGE_SIZE); if (!d->rw_image) { bpf_prog_pack_free(d->image, PAGE_SIZE); d->image = NULL; From 4946eb5d37cb6a260b9e0ec4b812c2b104fd6ea1 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 16 Jul 2026 10:51:36 +0300 Subject: [PATCH 135/373] bpf: Drop __weak from bpf_jit_alloc_exec() and bpf_jit_free_exec() bpf_jit_alloc_exec() and bpf_jit_free_exec() are wrappers for the corresponding execmem APIs. Architectures define the properties of the memory range needed by BPF in their initialization of execmem and don't need to override neither of them. Drop the __weak qualifier from bpf_jit_alloc_exec() and bpf_jit_free_exec(). Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-2-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 47fe047ad30b..fc75625dc951 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1116,12 +1116,12 @@ void bpf_jit_uncharge_modmem(u32 size) atomic_long_sub(size, &bpf_jit_current); } -void *__weak bpf_jit_alloc_exec(unsigned long size) +void *bpf_jit_alloc_exec(unsigned long size) { return execmem_alloc(EXECMEM_BPF, size); } -void __weak bpf_jit_free_exec(void *addr) +void bpf_jit_free_exec(void *addr) { execmem_free(addr); } From 7516714947da93eed4d7bbb88b3781c38233f2c6 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 16 Jul 2026 10:51:37 +0300 Subject: [PATCH 136/373] bpf: alloc_prog_pack(): Skip ROX management for already ROX memory execmem_alloc() can return ROX memory that is already filled with architecture defined trapping instructions. In preparation for enabling this mode for BPF on x86, make sure that there is no redundant management of the ROX memory. There is no need to fill allocated memory with trapping instructions, to request permissions reset on free and to set ROX permissions as this all is handled by execmem_alloc(). Add bpf_jit_mem_is_rox() wrapper for execmem_is_rox(), use it to check if execmem_alloc() returns ROX memory and skip the redundant steps in that case. Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-3-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/core.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index fc75625dc951..1b89c18cf246 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -916,6 +916,11 @@ static LIST_HEAD(pack_list); #define BPF_PROG_CHUNK_COUNT (BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE) +static bool bpf_jit_mem_is_rox(void) +{ + return execmem_is_rox(EXECMEM_BPF); +} + static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns) { struct bpf_prog_pack *pack; @@ -927,16 +932,18 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins pack->ptr = bpf_jit_alloc_exec(BPF_PROG_PACK_SIZE); if (!pack->ptr) goto out; - bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE); bitmap_zero(pack->bitmap, BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE); if (static_branch_unlikely(&bpf_pred_flush_enabled)) pack->arch_flush_needed = true; - set_vm_flush_reset_perms(pack->ptr); - err = set_memory_rox((unsigned long)pack->ptr, - BPF_PROG_PACK_SIZE / PAGE_SIZE); - if (err) - goto out; + if (!bpf_jit_mem_is_rox()) { + bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE); + set_vm_flush_reset_perms(pack->ptr); + err = set_memory_rox((unsigned long)pack->ptr, + BPF_PROG_PACK_SIZE / PAGE_SIZE); + if (err) + goto out; + } list_add_tail(&pack->list, &pack_list); return pack; @@ -965,7 +972,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool pr_warn_once("BPF: Predictors not flushed for allocations greater than BPF_PROG_PACK_SIZE\n"); size = round_up(size, PAGE_SIZE); ptr = bpf_jit_alloc_exec(size); - if (ptr) { + if (ptr && !bpf_jit_mem_is_rox()) { int err; bpf_fill_ill_insns(ptr, size); From 5bf02dbf39fa0ec64661827fa4a1b4d5c1d942c0 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 16 Jul 2026 10:51:38 +0300 Subject: [PATCH 137/373] bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable arch_bpf_trampoline_size() allocates a buffer to get actual size required for a trampoline. This buffer must be in the module address space because __arch_prepare_bpf_trampoline() calculates rel32 offsets relatively to that buffer. In preparation for enabling ROX mode for EXECMEM_BPF make sure that the allocated memory is writable. Add bpf_jit_alloc_exec_rw() wrapper for execmem_alloc_rw() and use it for buffer allocation in arch_bpf_trampoline_size(). Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-4-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 5 ++--- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index de7515ea1bea..b2feec81e231 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3703,13 +3703,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, int ret; /* Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). - * This will NOT cause fragmentation in direct map, as we do not - * call set_memory_*() on this buffer. * * We cannot use kvmalloc here, because we need image to be in * module memory range. + * Since it must be writable use bpf_jit_alloc_exec_rw(). */ - image = bpf_jit_alloc_exec(PAGE_SIZE); + image = bpf_jit_alloc_exec_rw(PAGE_SIZE); if (!image) return -ENOMEM; diff --git a/include/linux/filter.h b/include/linux/filter.h index 14acb2455746..32d5297c557e 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1333,6 +1333,7 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr, void bpf_jit_binary_free(struct bpf_binary_header *hdr); u64 bpf_jit_alloc_exec_limit(void); void *bpf_jit_alloc_exec(unsigned long size); +void *bpf_jit_alloc_exec_rw(unsigned long size); void bpf_jit_free_exec(void *addr); void bpf_jit_free(struct bpf_prog *fp); struct bpf_binary_header * diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 1b89c18cf246..e2076667b245 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1128,6 +1128,11 @@ void *bpf_jit_alloc_exec(unsigned long size) return execmem_alloc(EXECMEM_BPF, size); } +void *bpf_jit_alloc_exec_rw(unsigned long size) +{ + return execmem_alloc_rw(EXECMEM_BPF, size); +} + void bpf_jit_free_exec(void *addr) { execmem_free(addr); From 6b88aaec81db48a76162361e5f734d2c729a02b1 Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Thu, 16 Jul 2026 10:51:39 +0300 Subject: [PATCH 138/373] x86/bpf: Enable EXECMEM_ROX_CACHE for BPF allocations BPF core and x86 JIT use text poking and temporary writable buffers and thus can handle ROX memory. Enable ROX cache for EXECMEM_BPF when configuration and CPU features allow that. Signed-off-by: Mike Rapoport (Microsoft) Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260716-execmem-x86-rox-bpf-v0-v3-5-4e76158c01c5@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/mm/init.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index fb67217fddcd..079f8c7e9e3c 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1107,10 +1107,10 @@ struct execmem_info __init *execmem_arch_setup(void) .alignment = MODULE_ALIGN, }, [EXECMEM_BPF] = { - .flags = EXECMEM_KASAN_SHADOW, + .flags = flags, .start = start, .end = MODULES_END, - .pgprot = PAGE_KERNEL, + .pgprot = pgprot, .alignment = MODULE_ALIGN, }, [EXECMEM_MODULE_DATA] = { From 87267b89459813cb50ab5377e076b639c07b4491 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Wed, 22 Jul 2026 07:10:01 -0700 Subject: [PATCH 139/373] libarena: Use compiler load-acquire/store-release in bpf_atomic.h Teach libarena's BPF atomic primitives to use compiler builtins for load-acquire and store-release when Clang advertises __BPF_FEATURE_LOAD_ACQ_STORE_REL. Older compilers continue to use the existing barrier-based fallback. Notably, as BPF programs begin running on arm64, it is better to use the more appropriate variants since we can no longer rely on x86 TSO ordering. Commit 880442305a39 ("bpf: Introduce load-acquire and store-release instructions") introduced support, hence kernels from 6.15 onwards are needed when compiling with compilers supporting these instructions. We have relatively recent kernel version requirements in libarena anyway, and have not cut first release, hence declare such a dependency. Signed-off-by: Puranjay Mohan Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260722141003.2841007-1-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/libarena/include/bpf_atomic.h | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tools/testing/selftests/bpf/libarena/include/bpf_atomic.h b/tools/testing/selftests/bpf/libarena/include/bpf_atomic.h index b7b230431929..43c306e17f19 100644 --- a/tools/testing/selftests/bpf/libarena/include/bpf_atomic.h +++ b/tools/testing/selftests/bpf/libarena/include/bpf_atomic.h @@ -86,6 +86,25 @@ extern bool CONFIG_X86_64 __kconfig __weak; /* Control dependency provides LOAD->STORE, provide LOAD->LOAD */ #define smp_acquire__after_ctrl_dep() ({ smp_rmb(); }) +#if defined(__BPF_FEATURE_LOAD_ACQ_STORE_REL) +/* + * Clang advertises this feature when it can lower acquire/release atomic + * builtins to BPF_LOAD_ACQ/BPF_STORE_REL. Older compilers keep using the + * barrier-based fallback below. The generated instructions require kernel + * verifier/JIT support added in Linux 6.15; compile for an older BPF CPU to + * keep using the fallback when targeting older kernels. + */ +#define smp_load_acquire(p) \ + ({ \ + __unqual_typeof(*(p)) ___p1 = __atomic_load_n((p), __ATOMIC_ACQUIRE); \ + (typeof(*(p)))___p1; \ + }) + +#define smp_store_release(p, val) \ + ({ \ + __atomic_store_n((p), (val), __ATOMIC_RELEASE); \ + }) +#else #define smp_load_acquire(p) \ ({ \ __unqual_typeof(*(p)) __v = READ_ONCE(*(p)); \ @@ -102,6 +121,7 @@ extern bool CONFIG_X86_64 __kconfig __weak; barrier(); \ WRITE_ONCE(*(p), val); \ }) +#endif #define smp_cond_load_relaxed_label(p, cond_expr, label) \ ({ \ From 6ef8ff20c30b21b523fe1065713e8fabd0debca4 Mon Sep 17 00:00:00 2001 From: Feng Jiang Date: Thu, 23 Jul 2026 05:41:32 +0000 Subject: [PATCH 140/373] bpf, riscv: Add support for timed may_goto MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement arch_bpf_timed_may_goto() for the RV64 JIT. The argument and return value are carried in BPF_REG_AX, and BPF R0-R5 are preserved across the call to the generic bpf_check_timed_may_goto(). Enable bpf_jit_supports_timed_may_goto() so the verifier uses the timed expansion path. Signed-off-by: Feng Jiang Reviewed-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-1-86acb54e5642@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/Makefile | 2 +- arch/riscv/net/bpf_jit_comp64.c | 12 +++++++- arch/riscv/net/bpf_timed_may_goto.S | 47 +++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/net/bpf_timed_may_goto.S diff --git a/arch/riscv/net/Makefile b/arch/riscv/net/Makefile index 9a1e5f0a94e5..6458d4d51990 100644 --- a/arch/riscv/net/Makefile +++ b/arch/riscv/net/Makefile @@ -3,7 +3,7 @@ obj-$(CONFIG_BPF_JIT) += bpf_jit_core.o ifeq ($(CONFIG_ARCH_RV64I),y) - obj-$(CONFIG_BPF_JIT) += bpf_jit_comp64.o + obj-$(CONFIG_BPF_JIT) += bpf_jit_comp64.o bpf_timed_may_goto.o else obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o endif diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index ad089a9a4ea9..8fe8969fb8a0 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1841,7 +1841,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, if (aux->tail_call_reachable && insn->src_reg == BPF_PSEUDO_CALL) emit_sd(RV_REG_SP, ctx->tcc_offset, RV_REG_TCC, ctx); - if (insn->src_reg != BPF_PSEUDO_CALL) + /* + * arch_bpf_timed_may_goto() is emitted by the verifier and + * returns its result in BPF_REG_AX instead of BPF_REG_0, so + * skip the normal "move return register into R0". + */ + if (insn->src_reg != BPF_PSEUDO_CALL && addr != (u64)arch_bpf_timed_may_goto) emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_A0, ctx); break; } @@ -2161,3 +2166,8 @@ bool bpf_jit_supports_subprog_tailcalls(void) { return true; } + +bool bpf_jit_supports_timed_may_goto(void) +{ + return true; +} diff --git a/arch/riscv/net/bpf_timed_may_goto.S b/arch/riscv/net/bpf_timed_may_goto.S new file mode 100644 index 000000000000..02c637d87420 --- /dev/null +++ b/arch/riscv/net/bpf_timed_may_goto.S @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Feng Jiang */ + +#include +#include + +/* + * Trampoline for the BPF timed may_goto loop bound. Custom calling convention: + * - input: stack offset in BPF_REG_AX (t0) + * - output: updated count in BPF_REG_AX (t0) + * + * Calls bpf_check_timed_may_goto(ptr) with the standard RISC-V ABI, where + * ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (a5, a0-a4) are saved + * across the call; BPF_REG_FP (s5) is callee-saved and needs no saving. + */ + +SYM_FUNC_START(arch_bpf_timed_may_goto) + addi sp, sp, -(8*SZREG) + REG_S ra, 7*SZREG(sp) + REG_S s0, 6*SZREG(sp) + addi s0, sp, 8*SZREG + + /* Save BPF registers R0-R5 (a5, a0-a4) */ + REG_S a5, 5*SZREG(sp) + REG_S a0, 4*SZREG(sp) + REG_S a1, 3*SZREG(sp) + REG_S a2, 2*SZREG(sp) + REG_S a3, 1*SZREG(sp) + REG_S a4, 0*SZREG(sp) + + add a0, t0, s5 + call bpf_check_timed_may_goto + mv t0, a0 + + /* Restore BPF registers R0-R5 */ + REG_L a4, 0*SZREG(sp) + REG_L a3, 1*SZREG(sp) + REG_L a2, 2*SZREG(sp) + REG_L a1, 3*SZREG(sp) + REG_L a0, 4*SZREG(sp) + REG_L a5, 5*SZREG(sp) + + REG_L s0, 6*SZREG(sp) + REG_L ra, 7*SZREG(sp) + addi sp, sp, 8*SZREG + ret +SYM_FUNC_END(arch_bpf_timed_may_goto) From a0032241aa14d16f6e8b84ae68fbe2b8d8d014f1 Mon Sep 17 00:00:00 2001 From: Feng Jiang Date: Thu, 23 Jul 2026 05:41:33 +0000 Subject: [PATCH 141/373] selftests/bpf: Test timed may_goto preserves R0-R5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a test that checks R0-R5 are preserved across arch_bpf_timed_may_goto() calls. Use bpf_get_prandom_u32() to avoid the verifier removing the checks via DCE. Suggested-by: Björn Töpel Signed-off-by: Feng Jiang Reviewed-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-2-86acb54e5642@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_may_goto_1.c | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c index 4bdf4256a41e..cb1ce4d13cfc 100644 --- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c @@ -106,4 +106,62 @@ __naked void may_goto_batch_2(void) : __clobber_all); } +/* + * Use bpf_get_prandom_u32() to prevent DCE from removing the checks. + * retval: 0=all ok, 1-6=R0-R5 clobbered. + */ +SEC("syscall") +__description("timed may_goto preserves R0-R5") +__arch_x86_64 +__arch_s390x +__arch_arm64 +__arch_riscv64 +__success +__retval(0) +__naked void timed_may_goto_preserves_regs(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r6 = r0;" + "r0 = 0x1111;" + "r0 += r6;" + "r1 = 0x2222;" + "r1 += r6;" + "r2 = 0x3333;" + "r2 += r6;" + "r3 = 0x4444;" + "r3 += r6;" + "r4 = 0x5555;" + "r4 += r6;" + "r5 = 0x6666;" + "r5 += r6;" + ".8byte %[may_goto];" + ".8byte %[loop];" + "r0 -= r6;" + "r1 -= r6;" + "r2 -= r6;" + "r3 -= r6;" + "r4 -= r6;" + "r5 -= r6;" + "if r0 != 0x1111 goto 1f;" + "if r1 != 0x2222 goto 2f;" + "if r2 != 0x3333 goto 3f;" + "if r3 != 0x4444 goto 4f;" + "if r4 != 0x5555 goto 5f;" + "if r5 != 0x6666 goto 6f;" + "r0 = 0;" + "exit;" + "1: r0 = 1; exit;" + "2: r0 = 2; exit;" + "3: r0 = 3; exit;" + "4: r0 = 4; exit;" + "5: r0 = 5; exit;" + "6: r0 = 6; exit;" + : + : __imm(bpf_get_prandom_u32), + __imm_insn(may_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0)), + __imm_insn(loop, BPF_RAW_INSN(BPF_JMP | BPF_JA, 0, 0, -2, 0)) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; From 3c2be3cbeb1b6899d53ed1f2a01b5a279d4f222c Mon Sep 17 00:00:00 2001 From: Feng Jiang Date: Thu, 23 Jul 2026 05:41:34 +0000 Subject: [PATCH 142/373] selftests/bpf: Enable timed may_goto tests for riscv64 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enable verifier_may_goto_1 (raw instruction tests), stream_cond_break (250ms timeout path), and the may_goto_interaction fastcall test on riscv64 now that the JIT supports timed may_goto. Signed-off-by: Feng Jiang Reviewed-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260723-riscv-bpf-timed-may-goto-v5-3-86acb54e5642@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/progs/stream.c | 1 + tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_may_goto_1.c | 4 ++++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/progs/stream.c b/tools/testing/selftests/bpf/progs/stream.c index 92ba1d72e0ec..8d8e53d37266 100644 --- a/tools/testing/selftests/bpf/progs/stream.c +++ b/tools/testing/selftests/bpf/progs/stream.c @@ -64,6 +64,7 @@ SEC("syscall") __arch_x86_64 __arch_arm64 __arch_s390x +__arch_riscv64 __success __retval(0) __stderr("ERROR: Timeout detected for may_goto instruction") __stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}") diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index 8d7ff38e4c06..83707faea049 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -660,6 +660,7 @@ __naked void may_goto_interaction_x86_64(void) SEC("raw_tp") __arch_arm64 +__arch_riscv64 __log_level(4) __msg("stack depth 24") /* may_goto counter at -24 */ __xlated("0: *(u64 *)(r10 -24) =") @@ -679,7 +680,7 @@ __xlated("10: *(u64 *)(r10 -24) = r12") __xlated("11: *(u64 *)(r10 -8) = r1") __xlated("12: exit") __success -__naked void may_goto_interaction_arm64(void) +__naked void may_goto_interaction(void) { asm volatile ( "r1 = 1;" diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c index cb1ce4d13cfc..0e211f030d0d 100644 --- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c @@ -11,6 +11,7 @@ __description("may_goto 0") __arch_x86_64 __arch_s390x __arch_arm64 +__arch_riscv64 __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -31,6 +32,7 @@ __description("batch 2 of may_goto 0") __arch_x86_64 __arch_s390x __arch_arm64 +__arch_riscv64 __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -53,6 +55,7 @@ __description("may_goto batch with offsets 2/1/0") __arch_x86_64 __arch_s390x __arch_arm64 +__arch_riscv64 __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -79,6 +82,7 @@ __description("may_goto batch with offsets 2/0") __arch_x86_64 __arch_s390x __arch_arm64 +__arch_riscv64 __xlated("0: *(u64 *)(r10 -16) = 65535") __xlated("1: *(u64 *)(r10 -8) = 0") __xlated("2: r12 = *(u64 *)(r10 -16)") From 2805abd089576799b15092949420e3f8ba97fabd Mon Sep 17 00:00:00 2001 From: Mykyta Yatsenko Date: Fri, 24 Jul 2026 08:52:06 -0700 Subject: [PATCH 143/373] bpf: Fix CFI mismatch in task work callback BPF subprograms use the bpf_callback_t ABI, but task work invokes the callback through a three-argument function pointer. This trips kCFI. Store and invoke the callback as bpf_callback_t. Fixes: 38aa7003e369 ("bpf: task work scheduling kfuncs") Signed-off-by: Mykyta Yatsenko Link: https://lore.kernel.org/bpf/20260724-task_work_cfi-v1-1-2616691781ed@meta.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/helpers.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index c18f1e16edee..88b38db47de9 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -4388,7 +4388,7 @@ struct bpf_task_work_ctx { struct bpf_map *map; void *map_val; enum task_work_notify_mode mode; - bpf_task_work_callback_t callback_fn; + bpf_callback_t callback_fn; struct rcu_head rcu; } __aligned(8); @@ -4471,7 +4471,8 @@ static void bpf_task_work_callback(struct callback_head *cb) key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx); migrate_disable(); - ctx->callback_fn(ctx->map, key, ctx->map_val); + ctx->callback_fn((u64)(long)ctx->map, (u64)(long)key, + (u64)(long)ctx->map_val, 0, 0); migrate_enable(); bpf_task_work_ctx_reset(ctx); @@ -4594,7 +4595,7 @@ static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work } static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw, - struct bpf_map *map, bpf_task_work_callback_t callback_fn, + struct bpf_map *map, void *callback_fn, struct bpf_prog_aux *aux, enum task_work_notify_mode mode) { struct bpf_prog *prog; @@ -4619,7 +4620,7 @@ static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work } ctx->task = task; - ctx->callback_fn = callback_fn; + ctx->callback_fn = (bpf_callback_t)callback_fn; ctx->prog = prog; ctx->mode = mode; ctx->map = map; From 61aaa8782bec59ecffd22e030f54ef9351bcabf9 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 22 Jul 2026 23:19:08 +0800 Subject: [PATCH 144/373] bpf: Fix WARNING in bpf_tracing_link_release The trampoline could be corrupted by the blindly 'tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX' in verifier. 1. A fexit attached to a tail_call_reachable prog. 'tr->flags' became 'BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_TAIL_CALL_CTX'. And, the trampoline would poke the target prog's nop insn using jmp insn instead of call insn. 2. Another fexit loaded with the same tail_call_reachable prog target. 'tr->flags' became 'BPF_TRAMP_F_TAIL_CALL_CTX'. 3. Close the first fexit link. Due to no BPF_TRAMP_F_CALL_ORIG in 'tr->flags', the trampoline will fail to restore the prog's nop insn using call insn. [ 3.410719] WARNING: kernel/bpf/syscall.c:3551 at bpf_tracing_link_release+0x53/0x60, CPU#1: test_progs/98 ... [ 3.428793] bpf_link_free+0x58/0x130 [ 3.429293] bpf_link_release+0x23/0x30 Fix the warning by updating 'tr->flags' with '|=' and lock. Fixes: 2b5dcb31a19a ("bpf, x64: Fix tailcall infinite loop") Signed-off-by: Leon Hwang Reviewed-by: Pu Lehui Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260722151909.69142-2-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 2 ++ kernel/bpf/trampoline.c | 7 +++++++ kernel/bpf/verifier.c | 2 +- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e066f44a9c05..7bfc28673124 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1523,6 +1523,7 @@ int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, struct bpf_tracing_multi_link *link); int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link); +void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags); /* * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn @@ -1646,6 +1647,7 @@ static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog, { return -ENOTSUPP; } +static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {} #endif struct bpf_func_info_aux { diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 6eadf64f7ec9..129d07db117e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -670,6 +670,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size) return ERR_PTR(err); } +void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) +{ + trampoline_lock(tr); + tr->flags |= flags; + trampoline_unlock(tr); +} + static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex, const struct bpf_trampoline_ops *ops, void *data) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 52be0a118cce..66d8d9eaec05 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19523,7 +19523,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -ENOMEM; if (tgt_prog && tgt_prog->aux->tail_call_reachable) - tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX; + bpf_trampoline_set_flags(tr, BPF_TRAMP_F_TAIL_CALL_CTX); prog->aux->dst_trampoline = tr; return 0; From 09e074666b7dbdaeef9107d36fb4b104d56e911e Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 22 Jul 2026 23:19:09 +0800 Subject: [PATCH 145/373] selftests/bpf: Verify no warning when close fexit link Add a test to verify that there's no WARNING when detaching fexit link by following the repro steps of previous commit. Without the fix, the WARNING could be triggered by this test. Signed-off-by: Leon Hwang Reviewed-by: Pu Lehui Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260722151909.69142-3-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/tailcalls.c | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/tailcalls.c b/tools/testing/selftests/bpf/prog_tests/tailcalls.c index c66037162da5..c5c9d6c359bb 100644 --- a/tools/testing/selftests/bpf/prog_tests/tailcalls.c +++ b/tools/testing/selftests/bpf/prog_tests/tailcalls.c @@ -13,6 +13,8 @@ #include "tailcall_cgrp_storage.skel.h" #include "tailcall_sleepable.skel.h" #include "tailcall_callback.skel.h" +#include "tailcall_bpf2bpf2.skel.h" +#include "tailcall_bpf2bpf_fexit.skel.h" /* test_tailcall_1 checks basic functionality by patching multiple locations * in a single program for a single tail call slot with nop->jmp, jmp->nop @@ -1907,6 +1909,50 @@ static void test_tailcall_callback(void) RUN_TESTS(tailcall_callback); } +static void test_tailcall_bpf2bpf_fexit_links(void) +{ + struct tailcall_bpf2bpf_fexit *skel1 = NULL, *skel2 = NULL; + struct tailcall_bpf2bpf2 *skel_tc; + int err, prog_fd; + + skel_tc = tailcall_bpf2bpf2__open_and_load(); + if (!ASSERT_OK_PTR(skel_tc, "tailcall_bpf2bpf2__open_and_load")) + return; + + skel1 = tailcall_bpf2bpf_fexit__open(); + if (!ASSERT_OK_PTR(skel1, "tailcall_bpf2bpf_fexit__open")) + goto out; + + prog_fd = bpf_program__fd(skel_tc->progs.classifier_0); + err = bpf_program__set_attach_target(skel1->progs.fexit, prog_fd, "subprog_tail"); + if (!ASSERT_OK(err, "bpf_program__set_attach_target")) + goto out; + + err = tailcall_bpf2bpf_fexit__load(skel1); + if (!ASSERT_OK(err, "tailcall_bpf2bpf_fexit__load")) + goto out; + + skel1->links.fexit = bpf_program__attach_trace(skel1->progs.fexit); + if (!ASSERT_OK_PTR(skel1->links.fexit, "bpf_program__attach_trace")) + goto out; + + skel2 = tailcall_bpf2bpf_fexit__open(); + if (!ASSERT_OK_PTR(skel2, "tailcall_bpf2bpf_fexit__open")) + goto out; + + err = bpf_program__set_attach_target(skel2->progs.fexit, prog_fd, "subprog_tail"); + if (!ASSERT_OK(err, "bpf_program__set_attach_target")) + goto out; + + err = tailcall_bpf2bpf_fexit__load(skel2); + ASSERT_OK(err, "tailcall_bpf2bpf_fexit__load"); + +out: + tailcall_bpf2bpf_fexit__destroy(skel1); + tailcall_bpf2bpf_fexit__destroy(skel2); + tailcall_bpf2bpf2__destroy(skel_tc); +} + void test_tailcalls(void) { if (test__start_subtest("tailcall_1")) @@ -1974,4 +2020,6 @@ void test_tailcalls(void) if (test__start_subtest("tailcall_cgrp_storage_no_storage_bridge")) test_tailcall_cgrp_storage_no_storage_bridge(); test_tailcall_callback(); + if (test__start_subtest("tailcall_bpf2bpf_fexit_links")) + test_tailcall_bpf2bpf_fexit_links(); } From 9b19237a46e7e173f9bc91721d4ada11967f277c Mon Sep 17 00:00:00 2001 From: Tiezhu Yang Date: Wed, 22 Jul 2026 10:29:06 +0800 Subject: [PATCH 146/373] selftests/bpf: Add get_preempt_count() support for RISC-V Currently, there is no RISC-V support for get_preempt_count() and its fallback path always returns 0. Add it so that bpf_in_interrupt(), bpf_in_nmi(), bpf_in_hardirq(), bpf_in_serving_softirq(), and bpf_in_task() work for RISC-V as well. Given that RISC-V has supported CONFIG_THREAD_INFO_IN_TASK since its initial commit fbe934d69eb7 ("RISC-V: Build Infrastructure") in 2017, directly retrieve preempt_count from the thread_info embedded within task_struct via bpf_get_current_task_btf(). This aligns the implementation with arm64, powerpc, and loongarch. Tested on a RISC-V virtual machine. Before: $ sudo ./test_progs -t exe_ctx ... #114 exe_ctx:FAIL Summary: 0/0 PASSED, 0 SKIPPED, 1 FAILED After: $ sudo ./test_progs -t exe_ctx #114 exe_ctx:OK Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Tiezhu Yang Tested-by: Pu Lehui Reviewed-by: Pu Lehui Link: https://lore.kernel.org/bpf/20260722022906.8778-1-yangtiezhu@loongson.cn Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/bpf_experimental.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 67ff7882299e..ff37ae5a113d 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -425,6 +425,8 @@ static inline int get_preempt_count(void) return bpf_get_lowcore()->preempt_count; #elif defined(bpf_target_loongarch) return bpf_get_current_task_btf()->thread_info.preempt_count; +#elif defined(bpf_target_riscv) + return bpf_get_current_task_btf()->thread_info.preempt_count; #endif return 0; } @@ -436,6 +438,7 @@ static inline int get_preempt_count(void) * * powerpc64 * * s390x * * loongarch + * * riscv */ static inline int bpf_in_interrupt(void) { @@ -458,6 +461,7 @@ static inline int bpf_in_interrupt(void) * * powerpc64 * * s390x * * loongarch + * * riscv */ static inline int bpf_in_nmi(void) { @@ -471,6 +475,7 @@ static inline int bpf_in_nmi(void) * * powerpc64 * * s390x * * loongarch + * * riscv */ static inline int bpf_in_hardirq(void) { @@ -484,6 +489,7 @@ static inline int bpf_in_hardirq(void) * * powerpc64 * * s390x * * loongarch + * * riscv */ static inline int bpf_in_serving_softirq(void) { @@ -505,6 +511,7 @@ static inline int bpf_in_serving_softirq(void) * * powerpc64 * * s390x * * loongarch + * * riscv */ static inline int bpf_in_task(void) { From 2d7f576c9d8356b38cc45b1dbd86a2598d9d0bd6 Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Thu, 23 Jul 2026 16:03:49 +0200 Subject: [PATCH 147/373] s390/bpf: Add emit_ldx and emit_stx functions Add new functions for load and store to reuse them in the load-acquire and store-release logic. Signed-off-by: Maxim Khmelevskii Reviewed-by: Ilya Leoshkevich Link: https://lore.kernel.org/bpf/20260723140648.583055-6-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/s390/net/bpf_jit_comp.c | 167 +++++++++++++++++------------------ 1 file changed, 82 insertions(+), 85 deletions(-) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 9ddd89f71f28..20c9533f2368 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -830,6 +830,72 @@ static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp, return 0; } +static int emit_ldx(struct bpf_jit *jit, struct bpf_prog *fp, struct bpf_insn *insn) +{ + struct bpf_jit_probe probe; + + bpf_jit_probe_init(&probe); + bpf_jit_probe_load_pre(jit, insn, &probe); + + switch (BPF_SIZE(insn->code)) { + case BPF_B: /* dst = *(u8 *)(ul) (src + off) */ + /* llgc %dst,off(%src,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0090, insn->dst_reg, insn->src_reg, + probe.arena_reg, insn->off); + break; + case BPF_H: /* dst = *(u16 *)(ul) (src + off) */ + /* llgh %dst,off(%src,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0091, insn->dst_reg, insn->src_reg, + probe.arena_reg, insn->off); + break; + case BPF_W: /* dst = *(u32 *)(ul) (src + off) */ + /* llgf %dst,off(%src,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0016, insn->dst_reg, insn->src_reg, + probe.arena_reg, insn->off); + break; + case BPF_DW: /* dst = *(u64 *)(ul) (src + off) */ + /* lg %dst,off(%src,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0004, insn->dst_reg, insn->src_reg, + probe.arena_reg, insn->off); + break; + } + + return bpf_jit_probe_post(jit, fp, &probe); +} + +static int emit_stx(struct bpf_jit *jit, struct bpf_prog *fp, struct bpf_insn *insn) +{ + struct bpf_jit_probe probe; + + bpf_jit_probe_init(&probe); + bpf_jit_probe_store_pre(jit, insn, &probe); + + switch (BPF_SIZE(insn->code)) { + case BPF_B: /* *(u8 *)(dst + off) = src_reg */ + /* stcy %src,off(%dst,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0072, insn->src_reg, insn->dst_reg, + probe.arena_reg, insn->off); + break; + case BPF_H: /* (u16 *)(dst + off) = src */ + /* sthy %src,off(%dst,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0070, insn->src_reg, insn->dst_reg, + probe.arena_reg, insn->off); + break; + case BPF_W: /* *(u32 *)(dst + off) = src */ + /* sty %src,off(%dst,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0050, insn->src_reg, insn->dst_reg, + probe.arena_reg, insn->off); + break; + case BPF_DW: /* (u64 *)(dst + off) = src */ + /* stg %src,off(%dst,%arena) */ + EMIT6_DISP_LH(0xe3000000, 0x0024, insn->src_reg, insn->dst_reg, + probe.arena_reg, insn->off); + break; + } + + return bpf_jit_probe_post(jit, fp, &probe); +} + /* * Sign- or zero-extend the register if necessary */ @@ -1477,44 +1543,13 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, */ case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src_reg */ case BPF_STX | BPF_PROBE_MEM32 | BPF_B: - bpf_jit_probe_store_pre(jit, insn, &probe); - /* stcy %src,off(%dst,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0072, src_reg, dst_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - jit->seen |= SEEN_MEM; - break; case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */ case BPF_STX | BPF_PROBE_MEM32 | BPF_H: - bpf_jit_probe_store_pre(jit, insn, &probe); - /* sthy %src,off(%dst,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0070, src_reg, dst_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - jit->seen |= SEEN_MEM; - break; case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */ case BPF_STX | BPF_PROBE_MEM32 | BPF_W: - bpf_jit_probe_store_pre(jit, insn, &probe); - /* sty %src,off(%dst,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0050, src_reg, dst_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - jit->seen |= SEEN_MEM; - break; case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */ case BPF_STX | BPF_PROBE_MEM32 | BPF_DW: - bpf_jit_probe_store_pre(jit, insn, &probe); - /* stg %src,off(%dst,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0024, src_reg, dst_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); + err = emit_stx(jit, fp, insn); if (err < 0) return err; jit->seen |= SEEN_MEM; @@ -1574,8 +1609,12 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, /* * BPF_ATOMIC */ + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_DW: case BPF_STX | BPF_ATOMIC | BPF_W: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_B: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_H: case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW: case BPF_STX | BPF_PROBE_ATOMIC | BPF_W: { @@ -1687,15 +1726,20 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */ case BPF_LDX | BPF_PROBE_MEM | BPF_B: case BPF_LDX | BPF_PROBE_MEM32 | BPF_B: - bpf_jit_probe_load_pre(jit, insn, &probe); - /* llgc %dst,off(%src,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); + case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */ + case BPF_LDX | BPF_PROBE_MEM | BPF_H: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_H: + case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */ + case BPF_LDX | BPF_PROBE_MEM | BPF_W: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_W: + case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */ + case BPF_LDX | BPF_PROBE_MEM | BPF_DW: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW: + err = emit_ldx(jit, fp, insn); if (err < 0) return err; jit->seen |= SEEN_MEM; - if (insn_is_zext(&insn[1])) + if (BPF_SIZE(insn->code) != BPF_DW && insn_is_zext(&insn[1])) insn_count = 2; break; case BPF_LDX | BPF_MEMSX | BPF_B: /* dst = *(s8 *)(ul) (src + off) */ @@ -1708,20 +1752,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, return err; jit->seen |= SEEN_MEM; break; - case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */ - case BPF_LDX | BPF_PROBE_MEM | BPF_H: - case BPF_LDX | BPF_PROBE_MEM32 | BPF_H: - bpf_jit_probe_load_pre(jit, insn, &probe); - /* llgh %dst,off(%src,%arena) */ - EMIT6_DISP_LH(0xe3000000, 0x0091, dst_reg, src_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - jit->seen |= SEEN_MEM; - if (insn_is_zext(&insn[1])) - insn_count = 2; - break; case BPF_LDX | BPF_MEMSX | BPF_H: /* dst = *(s16 *)(ul) (src + off) */ case BPF_LDX | BPF_PROBE_MEMSX | BPF_H: bpf_jit_probe_load_pre(jit, insn, &probe); @@ -1732,20 +1762,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, return err; jit->seen |= SEEN_MEM; break; - case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */ - case BPF_LDX | BPF_PROBE_MEM | BPF_W: - case BPF_LDX | BPF_PROBE_MEM32 | BPF_W: - bpf_jit_probe_load_pre(jit, insn, &probe); - /* llgf %dst,off(%src) */ - jit->seen |= SEEN_MEM; - EMIT6_DISP_LH(0xe3000000, 0x0016, dst_reg, src_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - if (insn_is_zext(&insn[1])) - insn_count = 2; - break; case BPF_LDX | BPF_MEMSX | BPF_W: /* dst = *(s32 *)(ul) (src + off) */ case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: bpf_jit_probe_load_pre(jit, insn, &probe); @@ -1756,18 +1772,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, if (err < 0) return err; break; - case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */ - case BPF_LDX | BPF_PROBE_MEM | BPF_DW: - case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW: - bpf_jit_probe_load_pre(jit, insn, &probe); - /* lg %dst,off(%src,%arena) */ - jit->seen |= SEEN_MEM; - EMIT6_DISP_LH(0xe3000000, 0x0004, dst_reg, src_reg, - probe.arena_reg, off); - err = bpf_jit_probe_post(jit, fp, &probe); - if (err < 0) - return err; - break; /* * BPF_JMP / CALL */ @@ -3028,13 +3032,6 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) if (!in_arena) return true; switch (insn->code) { - case BPF_STX | BPF_ATOMIC | BPF_B: - case BPF_STX | BPF_ATOMIC | BPF_H: - case BPF_STX | BPF_ATOMIC | BPF_W: - case BPF_STX | BPF_ATOMIC | BPF_DW: - if (bpf_atomic_is_load_store(insn)) - return false; - break; case BPF_LDX | BPF_MEMSX | BPF_B: case BPF_LDX | BPF_MEMSX | BPF_H: case BPF_LDX | BPF_MEMSX | BPF_W: From 898343edde4797d58e3b6ced72e9b9497a1c82c7 Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Thu, 23 Jul 2026 16:03:50 +0200 Subject: [PATCH 148/373] s390/bpf: Support load-acquire and store-release instructions Support load-acquire (BPF_LOAD_ACQ) and store-release (BPF_STORE_REL) instructions. Since s390 has strong memory model, implement them as regular BPF_LDX/BPF_STX instructions. Tested with: ./test_progs -t verifier_load_acquire,verifier_store_release,atomics Signed-off-by: Maxim Khmelevskii Reviewed-by: Ilya Leoshkevich Link: https://lore.kernel.org/bpf/20260723140648.583055-7-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/s390/net/bpf_jit_comp.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 20c9533f2368..b60877478b45 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -743,10 +743,12 @@ static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn, { if (BPF_MODE(insn->code) != BPF_PROBE_MEM && BPF_MODE(insn->code) != BPF_PROBE_MEMSX && - BPF_MODE(insn->code) != BPF_PROBE_MEM32) + BPF_MODE(insn->code) != BPF_PROBE_MEM32 && + BPF_MODE(insn->code) != BPF_PROBE_ATOMIC) return; - if (BPF_MODE(insn->code) == BPF_PROBE_MEM32) { + if (BPF_MODE(insn->code) == BPF_PROBE_MEM32 || + BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) { /* lgrl %r1,kern_arena */ EMIT6_PCREL_RILB(0xc4080000, REG_W1, jit->kern_arena); probe->arena_reg = REG_W1; @@ -758,7 +760,8 @@ static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn, static void bpf_jit_probe_store_pre(struct bpf_jit *jit, struct bpf_insn *insn, struct bpf_jit_probe *probe) { - if (BPF_MODE(insn->code) != BPF_PROBE_MEM32) + if (BPF_MODE(insn->code) != BPF_PROBE_MEM32 && + BPF_MODE(insn->code) != BPF_PROBE_ATOMIC) return; /* lgrl %r1,kern_arena */ @@ -1621,11 +1624,11 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, bool is32 = BPF_SIZE(insn->code) == BPF_W; /* - * Unlike loads and stores, atomics have only a base register, - * but no index register. For the non-arena case, simply use - * %dst as a base. For the arena case, use the work register - * %r1: first, load the arena base into it, and then add %dst - * to it. + * Unlike loads and stores, s390 atomics have only a base + * register, but no index register. For the non-arena case, + * simply use %dst as a base. For the arena case, use the + * work register %r1: first, load the arena base into it, + * and then add %dst to it. */ probe.arena_reg = dst_reg; @@ -1712,6 +1715,18 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, if (err < 0) return err; break; + case BPF_LOAD_ACQ: + /* s390 has strong ordering, just use load */ + err = emit_ldx(jit, fp, insn); + if (err < 0) + return err; + break; + case BPF_STORE_REL: + /* s390 has strong ordering, just use store */ + err = emit_stx(jit, fp, insn); + if (err < 0) + return err; + break; default: pr_err("Unknown atomic operation %02x\n", insn->imm); return -1; From 2925f8deeadedfe42bcbac9b560babe8b1640e4b Mon Sep 17 00:00:00 2001 From: Maxim Khmelevskii Date: Thu, 23 Jul 2026 16:03:51 +0200 Subject: [PATCH 149/373] s390/bpf: Enable atomics tests for s390 Add s390 to the if statement, that defines CAN_USE_LOAD_ACQ_STORE_REL. Reuse CAN_USE_LOAD_ACQ_STORE_REL in arena_atomics selftest, to remove code duplication. Signed-off-by: Maxim Khmelevskii Reviewed-by: Ilya Leoshkevich Link: https://lore.kernel.org/bpf/20260723140648.583055-8-max@linux.ibm.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/arena_atomics.c | 18 ++++++++++++------ tools/testing/selftests/bpf/progs/bpf_misc.h | 9 ++++++--- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/arena_atomics.c b/tools/testing/selftests/bpf/progs/arena_atomics.c index 2e7751a85399..73bc2b835f3f 100644 --- a/tools/testing/selftests/bpf/progs/arena_atomics.c +++ b/tools/testing/selftests/bpf/progs/arena_atomics.c @@ -28,8 +28,10 @@ bool skip_all_tests = true; #if defined(ENABLE_ATOMICS_TESTS) && \ defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ - (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) + (defined(__TARGET_ARCH_arm64) || \ + defined(__TARGET_ARCH_x86) || \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \ + defined(__TARGET_ARCH_s390)) bool skip_lacq_srel_tests __attribute((__section__(".data"))) = false; #else bool skip_lacq_srel_tests = true; @@ -315,8 +317,10 @@ int load_acquire(const void *ctx) { #if defined(ENABLE_ATOMICS_TESTS) && \ defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ - (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) + (defined(__TARGET_ARCH_arm64) || \ + defined(__TARGET_ARCH_x86) || \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \ + defined(__TARGET_ARCH_s390)) #define LOAD_ACQUIRE_ARENA(SIZEOP, SIZE, SRC, DST) \ { asm volatile ( \ @@ -367,8 +371,10 @@ int store_release(const void *ctx) { #if defined(ENABLE_ATOMICS_TESTS) && \ defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ - (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)) + (defined(__TARGET_ARCH_arm64) || \ + defined(__TARGET_ARCH_x86) || \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \ + defined(__TARGET_ARCH_s390)) #define STORE_RELEASE_ARENA(SIZEOP, DST, VAL) \ { asm volatile ( \ diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h index b0c441384f20..5eacf1b43252 100644 --- a/tools/testing/selftests/bpf/progs/bpf_misc.h +++ b/tools/testing/selftests/bpf/progs/bpf_misc.h @@ -264,9 +264,12 @@ #endif #if __clang_major__ >= 18 && defined(ENABLE_ATOMICS_TESTS) && \ - (defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_x86) || \ - (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \ - defined(__TARGET_ARCH_powerpc) || defined(__TARGET_ARCH_loongarch)) + (defined(__TARGET_ARCH_arm64) || \ + defined(__TARGET_ARCH_x86) || \ + (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64) || \ + defined(__TARGET_ARCH_s390) || \ + defined(__TARGET_ARCH_powerpc) || \ + defined(__TARGET_ARCH_loongarch)) #define CAN_USE_LOAD_ACQ_STORE_REL #endif From 791841e038c40fb4c69c2888356650e554a3209a Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Thu, 23 Jul 2026 16:50:56 +0800 Subject: [PATCH 150/373] selftests/bpf: Fix extra free of subtest_state->name The name has already been freed in the free_subtest_state function and does not need to be freed again. The extra free is noop since the pointer was already set to NULL. Signed-off-by: Feng Yang Link: https://lore.kernel.org/bpf/20260723085100.482147-2-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_progs.c | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 7ba82974ee78..1d3caf996971 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -1886,7 +1886,6 @@ static int worker_main_send_subtests(int sock, struct test_state *state) worker_main_send_log(sock, subtest_state->log_buf, subtest_state->log_cnt); free_subtest_state(subtest_state); - free(subtest_state->name); } out: From b04b8d4e198aefc863e7b702ececb957845b0c25 Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Thu, 23 Jul 2026 16:50:57 +0800 Subject: [PATCH 151/373] selftests/bpf: Fix incorrect error checking for pthread_create pthread_create returns 0 on success and a positive error code on failure; it never returns a negative value. The current conditional branch can never be taken. Failures during thread creation are silently ignored, which will lead to invalid memory access when waiting on threads or dereferencing thread handles later. Fixes: 91b2c0afd00c ("selftests/bpf: Add parallelism to test_progs") Signed-off-by: Feng Yang Link: https://lore.kernel.org/bpf/20260723085100.482147-3-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_progs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 1d3caf996971..312743c4337f 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -1741,7 +1741,7 @@ static void server_main(void) data[i].worker_id = i; data[i].sock_fd = env.worker_socks[i]; rc = pthread_create(&dispatcher_threads[i], NULL, dispatch_thread, &data[i]); - if (rc < 0) { + if (rc) { perror("Failed to launch dispatcher thread"); exit(EXIT_ERR_SETUP_INFRA); } From 12b362b2f06b283b7d8a2450f702f9b6a0f94aed Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Thu, 23 Jul 2026 16:50:58 +0800 Subject: [PATCH 152/373] selftests/bpf: Fix missing allocation null checks in test_progs.c Add null checks after memory allocations to prevent potential segmentation faults. Fixes: 79b453501310 ("tools/bpf: add a test for bpf_get_stack with tracepoint prog") Fixes: 0925225956bb ("bpf/selftests: Add granular subtest output for prog_test") Signed-off-by: Feng Yang Link: https://lore.kernel.org/bpf/20260723085100.482147-4-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_progs.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 312743c4337f..301c6e11ceaf 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -730,11 +730,14 @@ int compare_map_keys(int map1_fd, int map2_fd) int compare_stack_ips(int smap_fd, int amap_fd, int stack_trace_len) { __u32 key, next_key, *cur_key_p, *next_key_p; - char *val_buf1, *val_buf2; - int i, err = 0; + char *val_buf1 = NULL, *val_buf2 = NULL; + int i, err = -ENOMEM; val_buf1 = malloc(stack_trace_len); val_buf2 = malloc(stack_trace_len); + if (!val_buf1 || !val_buf2) + goto out; + err = 0; cur_key_p = NULL; next_key_p = &key; while (bpf_map_get_next_key(smap_fd, cur_key_p, next_key_p) == 0) { @@ -1514,6 +1517,10 @@ static int dispatch_thread_send_subtests(int sock_fd, struct test_state *state) int subtest_num = state->subtest_num; state->subtest_states = malloc(subtest_num * sizeof(*subtest_state)); + if (!state->subtest_states) { + state->subtest_num = 0; + return -ENOMEM; + } for (int i = 0; i < subtest_num; i++) { subtest_state = &state->subtest_states[i]; From a813ad2185cd9f653a79b96b7c8a24240118c4e1 Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Thu, 23 Jul 2026 16:50:59 +0800 Subject: [PATCH 153/373] selftests/bpf: Use calloc to allocate subtest_states An early return triggered by read_prog_test_msg leaves uninitialized elements, which leads to memory corruption during free_test_states cleanup. Signed-off-by: Feng Yang Link: https://lore.kernel.org/bpf/20260723085100.482147-5-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_progs.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 301c6e11ceaf..07da45230c4b 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -1516,7 +1516,7 @@ static int dispatch_thread_send_subtests(int sock_fd, struct test_state *state) struct subtest_state *subtest_state; int subtest_num = state->subtest_num; - state->subtest_states = malloc(subtest_num * sizeof(*subtest_state)); + state->subtest_states = calloc(subtest_num, sizeof(*subtest_state)); if (!state->subtest_states) { state->subtest_num = 0; return -ENOMEM; @@ -1525,8 +1525,6 @@ static int dispatch_thread_send_subtests(int sock_fd, struct test_state *state) for (int i = 0; i < subtest_num; i++) { subtest_state = &state->subtest_states[i]; - memset(subtest_state, 0, sizeof(*subtest_state)); - if (read_prog_test_msg(sock_fd, &msg, MSG_SUBTEST_DONE)) return 1; From 06efb01c6530e9cfc247178cb96aa8adb3beaf61 Mon Sep 17 00:00:00 2001 From: Feng Yang Date: Thu, 23 Jul 2026 16:51:00 +0800 Subject: [PATCH 154/373] selftests/bpf: Fix memory leak on subtest_states reallocation Fix memory leak in subtest_states reallocation, and revert subtest_num if allocation fails. Fixes: 0925225956bb ("bpf/selftests: Add granular subtest output for prog_test") Signed-off-by: Feng Yang Link: https://lore.kernel.org/bpf/20260723085100.482147-6-yangfeng59949@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/test_progs.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 07da45230c4b..aa06bab30966 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -573,18 +573,19 @@ bool test__start_subtest_with_desc(const char *subtest_name, const char *subtest struct subtest_state *subtest_state; const char *subtest_display_name; size_t sub_state_size = sizeof(*subtest_state); + void *tmp; if (env.subtest_state) test__end_subtest(); state->subtest_num++; - state->subtest_states = - realloc(state->subtest_states, - state->subtest_num * sub_state_size); - if (!state->subtest_states) { + tmp = realloc(state->subtest_states, state->subtest_num * sub_state_size); + if (!tmp) { + state->subtest_num--; fprintf(stderr, "Not enough memory to allocate subtest result\n"); return false; } + state->subtest_states = tmp; subtest_state = &state->subtest_states[state->subtest_num - 1]; From 3c6e8a37eff15c3f834ba80c201932712e4b71d4 Mon Sep 17 00:00:00 2001 From: Weimin Xiong Date: Thu, 16 Jul 2026 10:51:03 +0800 Subject: [PATCH 155/373] unix: Use kvmalloc_array() for BPF iterator batches Use kvmalloc_array() instead of open-coding the element-size multiplication when allocating the Unix-domain BPF iterator batch. Signed-off-by: Weimin Xiong Signed-off-by: Andrii Nakryiko --- net/unix/af_unix.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index f7a9d55eee8a..6664ead46216 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -3659,8 +3659,8 @@ static int bpf_iter_unix_realloc_batch(struct bpf_unix_iter_state *iter, { struct sock **new_batch; - new_batch = kvmalloc(sizeof(*new_batch) * new_batch_sz, - GFP_USER | __GFP_NOWARN); + new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch), + GFP_USER | __GFP_NOWARN); if (!new_batch) return -ENOMEM; From fcf741584a562aaa89b0ace71f410b1c1a3e5193 Mon Sep 17 00:00:00 2001 From: Weimin Xiong Date: Thu, 16 Jul 2026 10:51:02 +0800 Subject: [PATCH 156/373] tcp: Use kvmalloc_array() for BPF iterator batches Use kvmalloc_array() instead of open-coding the element-size multiplication when allocating the TCP BPF iterator batch. Signed-off-by: Weimin Xiong Signed-off-by: Andrii Nakryiko --- net/ipv4/tcp_ipv4.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index b8887cdd66c5..127c90e6f561 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -2931,8 +2931,8 @@ static int bpf_iter_tcp_realloc_batch(struct bpf_tcp_iter_state *iter, { union bpf_tcp_iter_batch_item *new_batch; - new_batch = kvmalloc(sizeof(*new_batch) * new_batch_sz, - flags | __GFP_NOWARN); + new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch), + flags | __GFP_NOWARN); if (!new_batch) return -ENOMEM; From 59b9731addc7633a39d3a042ed968864f7b3aaf5 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 19 Jul 2026 13:35:49 +0200 Subject: [PATCH 157/373] bpf: Allow bpf_res_spin_lock() in all contexts There is no particular reason to keep bpf_res_spin_lock() disabled in tracing programs, since it is safe against reentrancy and deadlocks. Remove the restriction for tracing programs covered by the predicate is_tracing_prog_type(). This is a prerequisite before the definition of is_tracing_prog_type() is updated to include raw_tp, fentry, fexit, and fmod_ret. Existing tracing programs will be updated to use bpf_res_spin_lock() instead when it is available. Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260719113551.1294284-2-memxor@gmail.com --- kernel/bpf/verifier.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 641c3c62c1ec..e6f35f4e715b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17816,7 +17816,9 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, verbose(env, "socket filter progs cannot use bpf_spin_lock yet\n"); return -EINVAL; } + } + if (btf_record_has_field(map->record, BPF_SPIN_LOCK)) { if (is_tracing_prog_type(prog_type)) { verbose(env, "tracing progs cannot use bpf_spin_lock yet\n"); return -EINVAL; From fdec474c65fd35d5a6e1497ed50a9f98c07192f0 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Thu, 23 Jul 2026 14:13:37 +0800 Subject: [PATCH 158/373] selftests/bpf: Report the real error from libarena parallel workers Several CI runs failed in the libarena parallel tests with -4 (-EINTR) [1], which says nothing about what actually went wrong. Two workers can fail like this: worker 1: gives up, e.g. the rendezvous times out, sets test_abort and returns its own error (-ETIMEDOUT) worker 2: sees test_abort and returns -EINTR -EINTR only means "someone else already gave up", so it carries no information. Which of the two gets reported depends on the order pthread_join() collects them, because err = err ?: (long)thread_ret; keeps the first non-zero value and drops the rest. When the -EINTR worker comes first, the error describing the actual failure is lost. Skip -EINTR entirely: a worker only returns it once another worker has already reported the real error, so report and log only the real errors. It is still unclear whether the timeouts come from CI load or from a problem in the test itself. Report the error accurately first, so the next failure can be diagnosed. [1]: https://github.com/kernel-patches/bpf/actions/runs/29867905253/job/88764463566 https://github.com/kernel-patches/bpf/actions/runs/29878191901/job/88794845824 Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260723061347.398591-1-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/libarena.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/prog_tests/libarena.c b/tools/testing/selftests/bpf/prog_tests/libarena.c index df7e4b8dc394..daade4150af6 100644 --- a/tools/testing/selftests/bpf/prog_tests/libarena.c +++ b/tools/testing/selftests/bpf/prog_tests/libarena.c @@ -73,6 +73,7 @@ static int run_libarena_parallel_test_workers(struct libarena *skel, uint32_t nthreads; void *thread_ret; int ret, err = 0; + int worker_err; int i; for (nthreads = 0; nthreads < UINT_MAX; nthreads++) { @@ -118,7 +119,22 @@ static int run_libarena_parallel_test_workers(struct libarena *skel, continue; } - err = err ?: (long)thread_ret; + worker_err = (long)thread_ret; + + /* + * A worker that bails out because another one already gave up + * reports -EINTR. It is collateral damage that carries no + * information, so skip it entirely: never let it become the + * reported error, and don't log it either. + */ + if (!worker_err || worker_err == -EINTR) + continue; + + if (!err) + err = worker_err; + + fprintf(stdout, "%.*s__%d returned %d\n", (int)prefixlen, name, + i, worker_err); } free(threads); From 47d62db5504300bb0f85983e45726905be7166cb Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:11 -0700 Subject: [PATCH 159/373] resolve_btfids: Implement generic ensure_mem() to grow arrays push_*() helpers in resolve_btfids contain copy-pasted array growth logic. Factor it out into ensure_mem() - a simplified variant of libbpf_ensure_mem(), and use it in the helpers. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-2-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 55 ++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index f8a91fa7584f..183466ddc5f9 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -201,6 +201,35 @@ static int eprintf(int level, int var, const char *fmt, ...) #define pr_info(fmt, ...) \ eprintf(0, verbose, pr_fmt(fmt), ##__VA_ARGS__) +/* + * Grow *data so it can hold at least cnt elements of elem_sz bytes each. + * *cap is the capacity in elements and is updated on growth. + */ +static int __ensure_mem(void **data, u32 *cap, u32 cnt, size_t elem_sz) +{ + u32 new_cap, old_cap = *cap; + void *arr; + + if (cnt <= old_cap) + return 0; + + new_cap = max(old_cap + 256, old_cap * 2); + if (new_cap < cnt) + new_cap = cnt; + + arr = realloc(*data, elem_sz * new_cap); + if (!arr) + return -ENOMEM; + + *data = arr; + *cap = new_cap; + + return 0; +} + +#define ensure_mem(arr_ptr, cap_ptr, cnt) \ + __ensure_mem((void **)(arr_ptr), (cap_ptr), (cnt), sizeof(**(arr_ptr))) + static bool is_btf_id(const char *name) { return name && !strncmp(name, BTF_ID_PREFIX, sizeof(BTF_ID_PREFIX) - 1); @@ -890,17 +919,8 @@ static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf, s3 static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id) { - u32 *arr = ctx->decl_tags; - u32 cap = ctx->max_decl_tags; - - if (ctx->nr_decl_tags + 1 > cap) { - cap = max(cap + 256, cap * 2); - arr = realloc(arr, sizeof(u32) * cap); - if (!arr) - return -ENOMEM; - ctx->max_decl_tags = cap; - ctx->decl_tags = arr; - } + if (ensure_mem(&ctx->decl_tags, &ctx->max_decl_tags, ctx->nr_decl_tags + 1)) + return -ENOMEM; ctx->decl_tags[ctx->nr_decl_tags++] = decl_tag_id; @@ -909,17 +929,8 @@ static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id) static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc) { - struct kfunc *arr = ctx->kfuncs; - u32 cap = ctx->max_kfuncs; - - if (ctx->nr_kfuncs + 1 > cap) { - cap = max(cap + 256, cap * 2); - arr = realloc(arr, sizeof(struct kfunc) * cap); - if (!arr) - return -ENOMEM; - ctx->max_kfuncs = cap; - ctx->kfuncs = arr; - } + if (ensure_mem(&ctx->kfuncs, &ctx->max_kfuncs, ctx->nr_kfuncs + 1)) + return -ENOMEM; ctx->kfuncs[ctx->nr_kfuncs++] = *kfunc; From c5ab68a5fa9212c06f56d2ace6aa2c9249af33c4 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:12 -0700 Subject: [PATCH 160/373] resolve_btfids: Index BTF ID symbols by address Keep an address-sorted index of parsed .BTF_ids symbols so that the original BTF_ID symbol name can be recovered from an entry address. Use the index in find_kfunc_flags() to scan BTF_SET8_KFUNCS entries directly and match each entry back to the requested kfunc. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-3-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 94 +++++++++++++++++++++++++-------- 1 file changed, 71 insertions(+), 23 deletions(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 183466ddc5f9..3198198b03a7 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -119,6 +119,11 @@ struct btf_id { Elf64_Addr addr[ADDR_CNT]; }; +struct addr_sym { + Elf64_Addr addr; + const char *name; +}; + struct object { const char *path; const char *btf_path; @@ -150,6 +155,10 @@ struct object { int nr_structs; int nr_unions; int nr_typedefs; + + struct addr_sym *addr_syms; + u32 addr_syms_cnt; + u32 addr_syms_cap; }; #define KF_IMPLICIT_ARGS (1 << 16) @@ -509,6 +518,40 @@ static int elf_collect(struct object *obj) return 0; } +static int push_addr_sym(struct object *obj, Elf64_Addr addr, const char *name) +{ + if (ensure_mem(&obj->addr_syms, &obj->addr_syms_cap, obj->addr_syms_cnt + 1)) + return -ENOMEM; + + obj->addr_syms[obj->addr_syms_cnt++] = (struct addr_sym){ + .addr = addr, + .name = name, + }; + + return 0; +} + +static int cmp_addr_sym(const void *a, const void *b) +{ + Elf64_Addr aa = ((const struct addr_sym *)a)->addr; + Elf64_Addr ab = ((const struct addr_sym *)b)->addr; + + return (aa > ab) - (aa < ab); +} + +static const char *find_name_by_addr(struct object *obj, Elf64_Addr addr) +{ + struct addr_sym key = { .addr = addr }; + struct addr_sym *res; + + if (!obj->addr_syms_cnt) + return NULL; + + res = bsearch(&key, obj->addr_syms, obj->addr_syms_cnt, + sizeof(*obj->addr_syms), cmp_addr_sym); + return res ? res->name : NULL; +} + static int symbols_collect(struct object *obj) { Elf_Scn *scn = NULL; @@ -602,8 +645,15 @@ static int symbols_collect(struct object *obj) return -1; } id->addr[id->addr_cnt++] = sym.st_value; + + if (push_addr_sym(obj, sym.st_value, id->name)) + return -1; } + if (obj->addr_syms_cnt) + qsort(obj->addr_syms, obj->addr_syms_cnt, + sizeof(*obj->addr_syms), cmp_addr_sym); + return 0; } @@ -957,43 +1007,40 @@ static int collect_decl_tags(struct btf2btf_context *ctx) } /* - * To find the kfunc flags having its struct btf_id (with ELF addresses) - * we need to find the address that is in range of a set8. - * If a set8 is found, then the flags are located at addr + 4 bytes. + * To find kfunc flags, scan BTF_SET8_KFUNCS entries and use the entry + * address to recover the corresponding BTF_ID symbol name. * Return 0 (no flags!) if not found. */ static u32 find_kfunc_flags(struct object *obj, struct btf_id *kfunc_id) { - const u32 *elf_data_ptr = obj->efile.idlist->d_buf; - u64 set_lower_addr, set_upper_addr, addr; + Elf_Data *idlist = obj->efile.idlist; struct btf_id *set_id; struct rb_node *next; - u32 flags; - u64 idx; for (next = rb_first(&obj->sets); next; next = rb_next(next)) { + struct btf_id_set8 *set8; + u64 set_addr; + set_id = rb_entry(next, struct btf_id, rb_node); if (set_id->kind != BTF_ID_KIND_SET8 || set_id->addr_cnt != 1) continue; - set_lower_addr = set_id->addr[0]; - set_upper_addr = set_lower_addr + set_id->cnt * sizeof(u64); + set_addr = set_id->addr[0]; + set8 = idlist->d_buf + (set_addr - obj->efile.idlist_addr); + if (!(set8->flags & BTF_SET8_KFUNCS)) + continue; - for (u32 i = 0; i < kfunc_id->addr_cnt; i++) { - addr = kfunc_id->addr[i]; - /* - * Lower bound is exclusive to skip the 8-byte header of the set. - * Upper bound is inclusive to capture the last entry at offset 8*cnt. - */ - if (set_lower_addr < addr && addr <= set_upper_addr) { - pr_debug("found kfunc %s in BTF_ID_FLAGS %s\n", - kfunc_id->name, set_id->name); - idx = addr - obj->efile.idlist_addr; - idx = idx / sizeof(u32) + 1; - flags = elf_data_ptr[idx]; + for (u32 i = 0; i < set_id->cnt; i++) { + size_t off = (char *)&set8->pairs[i] - (char *)set8; + const char *name = find_name_by_addr(obj, set_addr + off); - return flags; - } + if (!name || strcmp(name, kfunc_id->name) != 0) + continue; + + pr_debug("found kfunc %s in BTF_ID_FLAGS %s\n", + kfunc_id->name, set_id->name); + + return set8->pairs[i].flags; } } @@ -1586,6 +1633,7 @@ int main(int argc, const char **argv) btf_id__free_all(&obj.typedefs); btf_id__free_all(&obj.funcs); btf_id__free_all(&obj.sets); + free(obj.addr_syms); if (obj.efile.elf) { elf_end(obj.efile.elf); close(obj.efile.fd); From e4293c31a32061b66bb7257ab2538cebfc61c84a Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:13 -0700 Subject: [PATCH 161/373] resolve_btfids: Keep collected kfuncs in a rbtree Store collected kfuncs in a rbtree keyed by BTF ID instead of a dynamically grown array. This allows for efficient deduplication for kfuncs declared in multiple sets, which is needed for subsequent patches [1]. [1] https://lore.kernel.org/bpf/CAEf4BzaLzX3mXvQzxv+gbmZOh84XvYofLjMSWFYghNjS-ohEZg@mail.gmail.com/ Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-4-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 50 +++++++++++++++++++++++++++------ 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 3198198b03a7..de4986e1bc3d 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -165,6 +165,7 @@ struct object { #define KF_IMPL_SUFFIX "_impl" struct kfunc { + struct rb_node rb_node; const char *name; u32 btf_id; u32 flags; @@ -175,9 +176,7 @@ struct btf2btf_context { u32 *decl_tags; u32 nr_decl_tags; u32 max_decl_tags; - struct kfunc *kfuncs; - u32 nr_kfuncs; - u32 max_kfuncs; + struct rb_root kfuncs; }; static int verbose; @@ -979,14 +978,48 @@ static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id) static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc) { - if (ensure_mem(&ctx->kfuncs, &ctx->max_kfuncs, ctx->nr_kfuncs + 1)) + struct rb_node **p = &ctx->kfuncs.rb_node; + struct rb_node *parent = NULL; + struct kfunc *k; + + /* Dedup by BTF ID: collecting the same kfunc twice is a no-op. */ + while (*p) { + parent = *p; + k = rb_entry(parent, struct kfunc, rb_node); + + if (kfunc->btf_id < k->btf_id) + p = &(*p)->rb_left; + else if (kfunc->btf_id > k->btf_id) + p = &(*p)->rb_right; + else + return 0; + } + + k = zalloc(sizeof(*k)); + if (!k) return -ENOMEM; - ctx->kfuncs[ctx->nr_kfuncs++] = *kfunc; + *k = *kfunc; + rb_link_node(&k->rb_node, parent, p); + rb_insert_color(&k->rb_node, &ctx->kfuncs); return 0; } +static void free_kfuncs(struct rb_root *root) +{ + struct rb_node *next; + struct kfunc *kfunc; + + next = rb_first(root); + while (next) { + kfunc = rb_entry(next, struct kfunc, rb_node); + next = rb_next(&kfunc->rb_node); + rb_erase(&kfunc->rb_node, root); + free(kfunc); + } +} + static int collect_decl_tags(struct btf2btf_context *ctx) { const u32 type_cnt = btf__type_cnt(ctx->btf); @@ -1272,14 +1305,15 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct static int btf2btf(struct object *obj) { struct btf2btf_context ctx = {}; + struct rb_node *next; int err; err = build_btf2btf_context(obj, &ctx); if (err) goto out; - for (u32 i = 0; i < ctx.nr_kfuncs; i++) { - struct kfunc *kfunc = &ctx.kfuncs[i]; + for (next = rb_first(&ctx.kfuncs); next; next = rb_next(next)) { + struct kfunc *kfunc = rb_entry(next, struct kfunc, rb_node); if (!(kfunc->flags & KF_IMPLICIT_ARGS)) continue; @@ -1292,7 +1326,7 @@ static int btf2btf(struct object *obj) err = 0; out: free(ctx.decl_tags); - free(ctx.kfuncs); + free_kfuncs(&ctx.kfuncs); return err; } From ef242559eb483466a959020af2e404bda77b0015 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:14 -0700 Subject: [PATCH 162/373] libbpf: Export btf__find_by_name_kind_own() btf__find_by_name_kind() searches the base BTF before the split BTF, so in case of a name collision between base and split it always returns a base type. Tools that process split BTF may need to restrict a lookup to the split's own types. The internal helper btf__find_by_name_kind_own() already does exactly that. Make it a public API. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-5-ihor.solodrai@linux.dev --- tools/lib/bpf/btf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + tools/lib/bpf/libbpf_internal.h | 2 -- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 1a31f2da947f..587172c0de08 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -172,6 +172,8 @@ LIBBPF_API __s32 btf__find_by_name(const struct btf *btf, const char *type_name); LIBBPF_API __s32 btf__find_by_name_kind(const struct btf *btf, const char *type_name, __u32 kind); +LIBBPF_API __s32 btf__find_by_name_kind_own(const struct btf *btf, + const char *type_name, __u32 kind); LIBBPF_API __u32 btf__type_cnt(const struct btf *btf); LIBBPF_API const struct btf *btf__base_btf(const struct btf *btf); LIBBPF_API const struct btf_type *btf__type_by_id(const struct btf *btf, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index b731df19ae69..08ab2ea881fb 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -460,5 +460,6 @@ LIBBPF_1.8.0 { global: bpf_program__attach_tracing_multi; bpf_program__clone; + btf__find_by_name_kind_own; btf__new_empty_opts; } LIBBPF_1.7.0; diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index d5b7db703b3f..7a74abb904f8 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -596,8 +596,6 @@ typedef int (*type_id_visit_fn)(__u32 *type_id, void *ctx); typedef int (*str_off_visit_fn)(__u32 *str_off, void *ctx); int btf_ext_visit_type_ids(struct btf_ext *btf_ext, type_id_visit_fn visit, void *ctx); int btf_ext_visit_str_offs(struct btf_ext *btf_ext, str_off_visit_fn visit, void *ctx); -__s32 btf__find_by_name_kind_own(const struct btf *btf, const char *type_name, - __u32 kind); /* handle direct returned errors */ static inline int libbpf_err(int ret) From 91efe50e3042ffe08eb7f0d77938cf237992241e Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:15 -0700 Subject: [PATCH 163/373] resolve_btfids: Fix the _impl lookup for module BTF process_kfunc_with_implicit_args() skips generating _impl function when one already exists for backwards compatibility. It uses btf__find_by_name_kind(), which searches the base BTF before the split BTF. When resolve_btfids processes a module, a same-named _impl in vmlinux would be found and the module's own counterpart would not be created. Fix by using btf__find_by_name_kind_own() for the lookup. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-6-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index de4986e1bc3d..ab3ab3045592 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -1232,7 +1232,7 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct return -E2BIG; } - if (btf__find_by_name_kind(btf, tmp_name, BTF_KIND_FUNC) > 0) { + if (btf__find_by_name_kind_own(btf, tmp_name, BTF_KIND_FUNC) > 0) { pr_debug("resolve_btfids: function %s already exists in BTF\n", tmp_name); goto add_new_proto; } From 5c4923172dae6f4cc97567cf86a08e6f250a641d Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:16 -0700 Subject: [PATCH 164/373] HID: bpf: Make syscall kfunc flags match the struct_ops set Update kfunc flags for hid_bpf_syscall_kfunc_ids set to exactly match hid_bpf_kfunc_ids set by adding KF_SLEEPABLE flag. The syscall set omitted the flag because syscall programs are always sleepable (the verifier rejects a non-sleepable syscall program). However the upcoming resolve_btfids change enforces per-kfunc flag consistency across BTF ID sets at build time, which is why this change is necessary. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-7-ihor.solodrai@linux.dev --- drivers/hid/bpf/hid_bpf_dispatch.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/hid/bpf/hid_bpf_dispatch.c b/drivers/hid/bpf/hid_bpf_dispatch.c index 536f6d01fd14..44671dbdeca8 100644 --- a/drivers/hid/bpf/hid_bpf_dispatch.c +++ b/drivers/hid/bpf/hid_bpf_dispatch.c @@ -590,11 +590,11 @@ static const struct btf_kfunc_id_set hid_bpf_kfunc_set = { /* for syscall HID-BPF */ BTF_KFUNCS_START(hid_bpf_syscall_kfunc_ids) -BTF_ID_FLAGS(func, hid_bpf_allocate_context, KF_ACQUIRE | KF_RET_NULL) -BTF_ID_FLAGS(func, hid_bpf_release_context, KF_RELEASE) -BTF_ID_FLAGS(func, hid_bpf_hw_request) -BTF_ID_FLAGS(func, hid_bpf_hw_output_report) -BTF_ID_FLAGS(func, hid_bpf_input_report) +BTF_ID_FLAGS(func, hid_bpf_allocate_context, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE) +BTF_ID_FLAGS(func, hid_bpf_release_context, KF_RELEASE | KF_SLEEPABLE) +BTF_ID_FLAGS(func, hid_bpf_hw_request, KF_SLEEPABLE) +BTF_ID_FLAGS(func, hid_bpf_hw_output_report, KF_SLEEPABLE) +BTF_ID_FLAGS(func, hid_bpf_input_report, KF_SLEEPABLE) BTF_KFUNCS_END(hid_bpf_syscall_kfunc_ids) static const struct btf_kfunc_id_set hid_bpf_syscall_kfunc_set = { From f9f60d41ba2c84bf74e42a3a09744561e770adc8 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:17 -0700 Subject: [PATCH 165/373] resolve_btfids: Discover kfuncs from BTF ID sets collect_kfuncs() currently uses bpf_kfunc decl tags to identify the list of kfuncs. The decl tags are generated by pahole, which makes current implementation implicitly rely on those tags being generated. The authoritative source, used by the the BPF verifier for kfunc registration, of functions being BPF kfuncs are BTF_KFUNCS_START()/END() declarations. These are BTF_ID_SET8 under the hood. Currently resolve_btfids reads kfunc flags from these sets, and populates them with BTF IDs. Implement kfunc discovery from BTF_ID_SET8 symbols in resolve_btfids, removing the dependency on pahole's emmission of decl tags. Walk BTF_ID_KIND_SET8 sets, and use the address-to-symbol index to look up set entry's BTF_ID symbol name (before .BTF_ids is patched), recording the paired flags directly. This makes find_kfunc_flags() helper unnecessary, so it's removed. Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-8-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 89 +++++++++++---------------------- 1 file changed, 29 insertions(+), 60 deletions(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index ab3ab3045592..338d0c0a8e58 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -1039,19 +1039,18 @@ static int collect_decl_tags(struct btf2btf_context *ctx) return 0; } -/* - * To find kfunc flags, scan BTF_SET8_KFUNCS entries and use the entry - * address to recover the corresponding BTF_ID symbol name. - * Return 0 (no flags!) if not found. - */ -static u32 find_kfunc_flags(struct object *obj, struct btf_id *kfunc_id) +static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx) { Elf_Data *idlist = obj->efile.idlist; - struct btf_id *set_id; + struct btf *btf = ctx->btf; struct rb_node *next; + if (!idlist || !idlist->d_buf) + return 0; + for (next = rb_first(&obj->sets); next; next = rb_next(next)) { struct btf_id_set8 *set8; + struct btf_id *set_id; u64 set_addr; set_id = rb_entry(next, struct btf_id, rb_node); @@ -1066,69 +1065,39 @@ static u32 find_kfunc_flags(struct object *obj, struct btf_id *kfunc_id) for (u32 i = 0; i < set_id->cnt; i++) { size_t off = (char *)&set8->pairs[i] - (char *)set8; const char *name = find_name_by_addr(obj, set_addr + off); + struct kfunc kfunc; + s32 func_id; + int err; - if (!name || strcmp(name, kfunc_id->name) != 0) + if (!name) { + pr_err("WARN: resolve_btfids: no BTF ID symbol for %s entry %u\n", + set_id->name, i); + warnings++; continue; + } - pr_debug("found kfunc %s in BTF_ID_FLAGS %s\n", - kfunc_id->name, set_id->name); + func_id = btf__find_by_name_kind_own(btf, name, BTF_KIND_FUNC); + if (func_id < 0) { + pr_err("WARN: resolve_btfids: no BTF func for kfunc %s in %s\n", + name, set_id->name); + warnings++; + continue; + } - return set8->pairs[i].flags; + pr_debug("found kfunc %s in %s\n", name, set_id->name); + + kfunc.name = name; + kfunc.btf_id = func_id; + kfunc.flags = set8->pairs[i].flags; + err = push_kfunc(ctx, &kfunc); + if (err) + return err; } } return 0; } -static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx) -{ - const char *tag_name, *func_name; - struct btf *btf = ctx->btf; - const struct btf_type *t; - u32 flags, func_id; - struct kfunc kfunc; - struct btf_id *id; - int err; - - if (ctx->nr_decl_tags == 0) - return 0; - - for (u32 i = 0; i < ctx->nr_decl_tags; i++) { - t = btf__type_by_id(btf, ctx->decl_tags[i]); - if (btf_kflag(t) || btf_decl_tag(t)->component_idx != -1) - continue; - - tag_name = btf__name_by_offset(btf, t->name_off); - if (strcmp(tag_name, "bpf_kfunc") != 0) - continue; - - func_id = t->type; - t = btf__type_by_id(btf, func_id); - if (!btf_is_func(t)) - continue; - - func_name = btf__name_by_offset(btf, t->name_off); - if (!func_name) - continue; - - id = btf_id__find(&obj->funcs, func_name); - if (!id || id->kind != BTF_ID_KIND_SYM) - continue; - - flags = find_kfunc_flags(obj, id); - - kfunc.name = id->name; - kfunc.btf_id = func_id; - kfunc.flags = flags; - - err = push_kfunc(ctx, &kfunc); - if (err) - return err; - } - - return 0; -} - static int build_btf2btf_context(struct object *obj, struct btf2btf_context *ctx) { int err; From 140a3479ef66507a7de06f4cd8bcefb86d2c640a Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Wed, 22 Jul 2026 16:35:18 -0700 Subject: [PATCH 166/373] resolve_btfids: Enforce consistent kfunc flags across BTF ID sets A kfunc may be listed in several BTF ID sets, which is expected because different kfuncs are available to BPF programs depending on their type. However kfunc flags across different BTF ID sets must be consistent [1]. The flags should be considered a part of the kfunc declaration, because they influence its BTF representation and verifier handling. Enforce the kfunc flag consistency in resolve_btifds by hard failing on error and blocking kernel (or module) build. [1] https://lore.kernel.org/bpf/9b2196dd-443b-4632-ae11-030cdbdc59b4@linux.dev/ Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260722233518.778854-9-ihor.solodrai@linux.dev --- tools/bpf/resolve_btfids/main.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 338d0c0a8e58..85488935909d 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -982,17 +982,26 @@ static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc) struct rb_node *parent = NULL; struct kfunc *k; - /* Dedup by BTF ID: collecting the same kfunc twice is a no-op. */ + /* + * Dedup by BTF ID: collecting the same kfunc twice is a no-op, + * UNLESS the kfunc flags are inconsistent, in which case we + * fail hard because it indicates a bug in a kfunc set declaration. + */ while (*p) { parent = *p; k = rb_entry(parent, struct kfunc, rb_node); - if (kfunc->btf_id < k->btf_id) + if (kfunc->btf_id < k->btf_id) { p = &(*p)->rb_left; - else if (kfunc->btf_id > k->btf_id) + } else if (kfunc->btf_id > k->btf_id) { p = &(*p)->rb_right; - else + } else if (k->flags == kfunc->flags) { return 0; + } else { + pr_err("ERROR: resolve_btfids: kfunc %s has inconsistent flags across BTF ID sets: 0x%x != 0x%x\n", + kfunc->name, k->flags, kfunc->flags); + return -EINVAL; + } } k = zalloc(sizeof(*k)); From 5c5997836381010fc5907b36bc17d3b19407e933 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Tue, 28 Jul 2026 02:32:59 +0000 Subject: [PATCH 167/373] bpf: Fix potential UAF in bpf_netns_link_update_prog In bpf_netns_link_update_prog, the checks for old_prog and prog type are currently performed locklessly before acquiring netns_bpf_mutex. This creates a race condition that can lead to a UAF issue. If two threads concurrently execute BPF_LINK_UPDATE on the same netns link, the following execution path can trigger a UAF: CPU0 CPU1 bpf_netns_link_update_prog if (old_prog && old_prog != link->prog) return -EPERM; bpf_netns_link_update_prog if (old_prog && old_prog != link->prog) ... old_prog = xchg(&link->prog, new_prog); bpf_prog_put(old_prog); if (new_prog->type != link->prog->type) <-- trigger UAF Fix this by moving the old_prog and prog->type checks inside the netns_bpf_mutex critical section. Meanwhile, use guard() to simplify lock management and avoid all the goto jumping. Fixes: 7f045a49fee0 ("bpf: Add link-based BPF program attachment to network namespace") Reported-by: Sashiko Signed-off-by: Pu Lehui Signed-off-by: Andrii Nakryiko Reviewed-by: Amery Hung Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0] Link: https://lore.kernel.org/bpf/20260728023259.2813482-1-pulehui@huaweicloud.com --- kernel/bpf/net_namespace.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/net_namespace.c b/kernel/bpf/net_namespace.c index 25f30f9edaef..81006a242618 100644 --- a/kernel/bpf/net_namespace.c +++ b/kernel/bpf/net_namespace.c @@ -171,33 +171,28 @@ static int bpf_netns_link_update_prog(struct bpf_link *link, struct net *net; int idx, ret; + guard(mutex)(&netns_bpf_mutex); + if (old_prog && old_prog != link->prog) return -EPERM; if (new_prog->type != link->prog->type) return -EINVAL; - mutex_lock(&netns_bpf_mutex); - net = net_link->net; - if (!net || !check_net(net)) { + if (!net || !check_net(net)) /* Link auto-detached or netns dying */ - ret = -ENOLINK; - goto out_unlock; - } + return -ENOLINK; run_array = rcu_dereference_protected(net->bpf.run_array[type], lockdep_is_held(&netns_bpf_mutex)); idx = link_index(net, type, net_link); ret = bpf_prog_array_update_at(run_array, idx, new_prog); if (ret) - goto out_unlock; + return ret; old_prog = xchg(&link->prog, new_prog); bpf_prog_put(old_prog); - -out_unlock: - mutex_unlock(&netns_bpf_mutex); - return ret; + return 0; } static int bpf_netns_link_fill_info(const struct bpf_link *link, From 863f3ddd0b8ac65abfb50d3be0869268ac0e277b Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Tue, 28 Jul 2026 02:54:57 +0000 Subject: [PATCH 168/373] bpf: Fix potential UAF when reading bpf link info In bpf_link_show_fdinfo and bpf_link_get_info_by_fd, link->prog is accessed without holding any locks. If the prog is concurrently replaced via bpf_link_update, the old prog can be freed, leading to a potential UAF issue. Fix this by accessing link->prog under RCU protection to safely fetch the pointer and guarantee its lifetime while reading its fields. Fixes: 0c991ebc8c69 ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link") Reported-by: Sashiko Signed-off-by: Pu Lehui Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Reviewed-by: Amery Hung Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [0] Link: https://lore.kernel.org/bpf/20260728025457.2814876-1-pulehui@huaweicloud.com --- kernel/bpf/syscall.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0ff9e3aa293d..0eb43ba76a8a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3392,9 +3392,10 @@ static const char *bpf_link_type_strs[] = { static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) { const struct bpf_link *link = filp->private_data; - const struct bpf_prog *prog = link->prog; + const struct bpf_prog *prog; enum bpf_link_type type = link->type; char prog_tag[sizeof(prog->tag) * 2 + 1] = { }; + u32 prog_id = 0; if (type < ARRAY_SIZE(bpf_link_type_strs) && bpf_link_type_strs[type]) { if (link->type == BPF_LINK_TYPE_KPROBE_MULTI) @@ -3411,13 +3412,20 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) } seq_printf(m, "link_id:\t%u\n", link->id); + rcu_read_lock(); + prog = READ_ONCE(link->prog); if (prog) { bin2hex(prog_tag, prog->tag, sizeof(prog->tag)); + prog_id = prog->aux->id; + } + rcu_read_unlock(); + + if (prog) { seq_printf(m, "prog_tag:\t%s\n" "prog_id:\t%u\n", prog_tag, - prog->aux->id); + prog_id); } if (link->ops->show_fdinfo) link->ops->show_fdinfo(link, m); @@ -5456,6 +5464,7 @@ static int bpf_link_get_info_by_fd(struct file *file, { struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info); struct bpf_link_info info; + const struct bpf_prog *prog; u32 info_len = attr->info.info_len; int err; @@ -5470,8 +5479,12 @@ static int bpf_link_get_info_by_fd(struct file *file, info.type = link->type; info.id = link->id; - if (link->prog) - info.prog_id = link->prog->aux->id; + + rcu_read_lock(); + prog = READ_ONCE(link->prog); + if (prog) + info.prog_id = prog->aux->id; + rcu_read_unlock(); if (link->ops->fill_link_info) { err = link->ops->fill_link_info(link, &info); From f0e80dee4e32fd11e6ee1b714b75f681c7cafd3e Mon Sep 17 00:00:00 2001 From: Xu Xin Date: Wed, 29 Jul 2026 14:11:59 +0800 Subject: [PATCH 169/373] bpf: Log error code on trampoline unlink failure Replace silent WARN_ON_ONCE with WARN_ONCE that prints the actual error code from bpf_trampoline_unlink_prog(). This aids debugging of race conditions during link teardown, while keeping the warning rate limited to avoid log flooding. This will be very helpful for speeding up trouble-shooting of some crash UAF due to bpf_trampoline_unlink_prog failures. No change to unlink behavior. Signed-off-by: Xu Xin Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260729141159128mEJmS_aujBKr-cBu1p_UI@zte.com.cn --- kernel/bpf/syscall.c | 8 +++++--- kernel/bpf/trampoline.c | 12 ++++++++---- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0eb43ba76a8a..94091130bcc5 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3555,10 +3555,12 @@ static void bpf_tracing_link_release(struct bpf_link *link) { struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); + int err; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node, - tr_link->trampoline, - tr_link->tgt_prog)); + err = bpf_trampoline_unlink_prog(&tr_link->link.node, + tr_link->trampoline, + tr_link->tgt_prog); + WARN_ONCE(err, "bpf_trampoline_unlink_prog failed: %d\n", err); bpf_trampoline_put(tr_link->trampoline); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 129d07db117e..ed7999ad6c66 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -1004,12 +1004,15 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) { struct bpf_shim_tramp_link *shim_link = container_of(link, struct bpf_shim_tramp_link, link.link); + int err; /* paired with 'shim_link->trampoline = tr' in bpf_trampoline_link_cgroup_shim */ if (!shim_link->trampoline) return; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL)); + err = bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL); + WARN_ONCE(err, "bpf_trampoline_unlink_prog failed: %d\n", err); + bpf_trampoline_put(shim_link->trampoline); } @@ -1720,15 +1723,16 @@ int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_ { struct bpf_tracing_multi_data *data = &link->data; struct bpf_tracing_multi_node *mnode; - int i; + int i, err; trampoline_lock_all(); for_each_mnode(mnode, link) { data->entry = &mnode->entry; bpf_trampoline_multi_attach_init(mnode->trampoline); - WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline, - NULL, &trampoline_multi_ops, data)); + err = __bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline, NULL, + &trampoline_multi_ops, data); + WARN_ONCE(err, "__bpf_trampoline_unlink_prog failed: %d\n", err); } if (ftrace_hash_count(data->unreg)) From c48796aa6c392cde93946e5d5a9a1f1b1cf72feb Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Tue, 28 Jul 2026 22:01:59 -0700 Subject: [PATCH 170/373] bpf: Reject >8 byte return values on return-reading trampoline paths btf_distill_func_proto() builds the function model used for the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. It has accepted a 16-byte __int128 return value since the trampoline was introduced: __get_type_size() returns the integer's type size, and the return-type check only rejected ret < 0. But the BPF trampoline preserves only 8 bytes of the return value (RAX on x86, i.e. R0). For an attach type that reads the target's return value the second half (RDX / R3) is neither saved nor restored, so a program attached to a function returning a 16-byte value corrupts the value seen by the real caller and itself observes only half of it. struct_ops trampolines have the same limitation. This affects the attach types that read the target's return value: fexit, fmod_ret and fsession (plus the _multi variants of fexit and fsession), and struct_ops. fentry/fentry_multi run before the target returns and are unaffected. Reject a >8 byte return value for these attach types in bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for struct_ops in bpf_struct_ops_desc_init(). Fixes: fec56f5890d9 ("bpf: Introduce BPF trampoline") Signed-off-by: Yonghong Song Reviewed-by: Eduard Zingerman Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260729050159.2585809-1-yonghong.song@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/bpf_struct_ops.c | 12 ++++++++++++ kernel/bpf/verifier.c | 25 +++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 51b16e5f5534..4e7a48c02be5 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -445,6 +445,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc, goto errout; } + /* + * A >8 byte return value is passed back in a register pair, + * which the struct_ops trampoline does not preserve (only + * 8 bytes of the return value are saved and restored). + */ + if (st_ops->func_models[i].ret_size > 8) { + pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n", + mname, st_ops->name); + err = -EOPNOTSUPP; + goto errout; + } + stub_func_addr = *(void **)(st_ops->cfi_stubs + moff); err = prepare_arg_info(btf, st_ops->name, mname, func_proto, stub_func_addr, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e6f35f4e715b..8d0635ee48c7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19027,6 +19027,20 @@ btf_attach_func_proto(struct bpf_verifier_log *log, struct btf *btf, u32 func_id return btf_type_by_id(btf, func->type); } +static bool attach_uses_trampoline_retval(enum bpf_attach_type type) +{ + switch (type) { + case BPF_MODIFY_RETURN: + case BPF_TRACE_FEXIT: + case BPF_TRACE_FEXIT_MULTI: + case BPF_TRACE_FSESSION: + case BPF_TRACE_FSESSION_MULTI: + return true; + default: + return false; + } +} + int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, @@ -19291,6 +19305,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, if (ret < 0) return ret; + if (tgt_info->fmodel.ret_size > 8 && + attach_uses_trampoline_retval(prog->expected_attach_type)) { + bpf_log(log, + "Attach to function %s with a >8 byte return value is not supported for this attach type\n", + tname); + return -EOPNOTSUPP; + } + /* * *.multi programs don't need an address during program * verification, we just take the module ref if needed. @@ -19565,6 +19587,9 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt err = btf_distill_func_proto(NULL, btf, t, tname, &tgt_info->fmodel); if (err < 0) return err; + if (tgt_info->fmodel.ret_size > 8 && + attach_uses_trampoline_retval(prog->expected_attach_type)) + return -EOPNOTSUPP; if (btf_is_module(btf)) { /* The bpf program already holds reference to module. */ if (WARN_ON_ONCE(!prog->aux->mod)) From 814cba835ef648e0c5eb79505c96c0493b29eea6 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Tue, 28 Jul 2026 22:02:04 -0700 Subject: [PATCH 171/373] bpf, x86: Fix trampoline stack size for 128-bit arguments btf_distill_func_proto() accepts a function argument up to 16 bytes, so a 128-bit scalar such as __int128 reaches the x86 trampoline with arg_size == 16. But the current implementation assumes an __int128 argument only needs one register, so the register save area is under-allocated and save_args() overwrites adjacent stack slots. Compute the register count from arg_size for all arguments to fix it. Fixes: a9c5ad31fbdc ("bpf: x86: Support in-register struct arguments in trampoline programs") Signed-off-by: Yonghong Song Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260729050204.2586457-1-yonghong.song@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index b2feec81e231..01e7ce569c1e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3369,11 +3369,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); - /* extra registers for struct arguments */ - for (i = 0; i < m->nr_args; i++) { - if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) - nr_regs += (m->arg_size[i] + 7) / 8 - 1; - } + for (i = 0; i < m->nr_args; i++) + nr_regs += (m->arg_size[i] + 7) / 8 - 1; /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 * are passed through regs, the remains are through stack. From 13cc6b788b1a361d6903c141a8e609bcae664301 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Tue, 28 Jul 2026 22:02:09 -0700 Subject: [PATCH 172/373] selftests/bpf: Add tests for >8 byte return value and 128-bit arguments The BPF trampoline preserves only 8 bytes of the target's return value (R0), so attaching an fexit/fmod_ret/fsession program to a function that returns a >8 byte value is now rejected by the verifier. Add a bpf_testmod function returning __int128 and an fexit program that targets it. The program is expected to fail to load with the "with a >8 byte return value is not supported for this attach type" message. A 128-bit __int128 argument is passed in a register pair and occupies two trampoline context slots. Add a bpf_testmod function taking a leading __int128 argument followed by an int and a long, and an fexit program that reads those two trailing arguments and the return value, verifying that the trampoline reserves enough stack for the 128-bit argument and places the following arguments and the return value at the right context slots. __int128 is only available on 64-bit targets (where the compiler defines __SIZEOF_INT128__). The argument test additionally depends on the calling convention: x86_64 and arm64 pass an __int128 in a register pair as the trampoline expects, while other architectures pass it differently (e.g. s390x passes larger arguments by reference), so that subtest runs only on x86_64 and arm64 and is skipped elsewhere. Signed-off-by: Yonghong Song Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260729050209.2587581-1-yonghong.song@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/prog_tests/tracing_failure.c | 20 +++++++++++ .../selftests/bpf/prog_tests/tracing_struct.c | 36 +++++++++++++++++++ .../selftests/bpf/progs/tracing_failure.c | 6 ++++ .../bpf/progs/tracing_struct_int128.c | 18 ++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 32 +++++++++++++++++ 5 files changed, 112 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/tracing_struct_int128.c diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_failure.c b/tools/testing/selftests/bpf/prog_tests/tracing_failure.c index f9f9e1cb87bf..eb585918f0d4 100644 --- a/tools/testing/selftests/bpf/prog_tests/tracing_failure.c +++ b/tools/testing/selftests/bpf/prog_tests/tracing_failure.c @@ -76,6 +76,24 @@ static void test_fexit_noreturns(void) "Attaching fexit/fsession/fmod_ret to __noreturn function 'do_exit' is rejected."); } +static void test_fexit_int128_ret(void) +{ + /* + * __int128 is returned in a register pair on x86_64 and arm64, so + * bpf_testmod_test_int128_ret() is BTF-encoded and attachable and the + * verifier can reject its >8 byte return value. Other architectures + * return a __int128 differently (e.g. s390x returns larger values by + * reference, which makes pahole skip BTF encoding of the function), so + * only exercise this on x86_64 and arm64. + */ +#if defined(__x86_64__) || defined(__aarch64__) + test_tracing_fail_prog("fexit_int128_ret", + "with a >8 byte return value is not supported for this attach type"); +#else + test__skip(); +#endif +} + void test_tracing_failure(void) { if (test__start_subtest("bpf_spin_lock")) @@ -86,4 +104,6 @@ void test_tracing_failure(void) test_tracing_deny(); if (test__start_subtest("fexit_noreturns")) test_fexit_noreturns(); + if (test__start_subtest("fexit_int128_ret")) + test_fexit_int128_ret(); } diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c index 6f8c0bfb0415..15b95d0235b5 100644 --- a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c +++ b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c @@ -4,6 +4,7 @@ #include #include "tracing_struct.skel.h" #include "tracing_struct_many_args.skel.h" +#include "tracing_struct_int128.skel.h" static void test_struct_args(void) { @@ -112,6 +113,39 @@ static void test_struct_many_args(void) tracing_struct_many_args__destroy(skel); } +static void test_int128_args(void) +{ + /* + * __int128 arguments are passed in a register pair on x86_64 and + * arm64, which the trampoline packs into two context slots. Other + * architectures pass a __int128 differently (e.g. s390x passes larger + * arguments by reference), so only exercise this on x86_64 and arm64. + */ +#if defined(__x86_64__) || defined(__aarch64__) + struct tracing_struct_int128 *skel; + int err; + + skel = tracing_struct_int128__open_and_load(); + if (!ASSERT_OK_PTR(skel, "tracing_struct_int128__open_and_load")) + return; + + err = tracing_struct_int128__attach(skel); + if (!ASSERT_OK(err, "tracing_struct_int128__attach")) + goto destroy_skel; + + ASSERT_OK(trigger_module_test_read(256), "trigger_read"); + + ASSERT_EQ(skel->bss->t_b, 2, "t:b"); + ASSERT_EQ(skel->bss->t_c, 3, "t:c"); + ASSERT_EQ(skel->bss->t_ret, 6, "t ret"); + +destroy_skel: + tracing_struct_int128__destroy(skel); +#else + test__skip(); +#endif +} + static void test_union_args(void) { struct tracing_struct *skel; @@ -145,6 +179,8 @@ void test_tracing_struct(void) test_struct_args(); if (test__start_subtest("struct_many_args")) test_struct_many_args(); + if (test__start_subtest("int128_args")) + test_int128_args(); if (test__start_subtest("union_args")) test_union_args(); } diff --git a/tools/testing/selftests/bpf/progs/tracing_failure.c b/tools/testing/selftests/bpf/progs/tracing_failure.c index 65e485c4468c..f7a095767679 100644 --- a/tools/testing/selftests/bpf/progs/tracing_failure.c +++ b/tools/testing/selftests/bpf/progs/tracing_failure.c @@ -30,3 +30,9 @@ int BPF_PROG(fexit_noreturns) { return 0; } + +SEC("?fexit/bpf_testmod_test_int128_ret") +int BPF_PROG(fexit_int128_ret) +{ + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128.c new file mode 100644 index 000000000000..4638dfec1f38 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +long t_b, t_c, t_ret; + +SEC("fexit/bpf_testmod_test_int128_arg") +int test_int128_arg_fexit(unsigned long long *ctx) +{ + t_b = (int)ctx[2]; + t_c = (long)ctx[3]; + t_ret = (long)ctx[4]; + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 30f1cd23093c..eb0f9b5e18d8 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -161,6 +161,33 @@ bpf_testmod_test_arg_ptr_to_struct(struct bpf_testmod_struct_arg_1 *a) { return bpf_testmod_test_struct_arg_result; } +#ifdef __SIZEOF_INT128__ +noinline __int128 +bpf_testmod_test_int128_ret(int a) +{ + bpf_testmod_test_struct_arg_result = a; + return (__int128)a; +} + +/* + * The __int128 'a' is the first argument on purpose. On arm64 a 16-byte + * argument must start in an even-numbered register pair, so placing it + * after a single-register scalar would leave a padding register (x1) + * unused. pahole maps parameters to registers positionally and would then + * see the following argument in an "unexpected" register and skip BTF + * encoding of the whole function, making it unattachable. Keeping the + * __int128 first (x0:x1) avoids the padding while still exercising the + * trampoline packing of a 128-bit argument together with the trailing + * int and long arguments. + */ +noinline long +bpf_testmod_test_int128_arg(__int128 a, int b, long c) +{ + bpf_testmod_test_struct_arg_result = (long)a + b + c; + return bpf_testmod_test_struct_arg_result; +} +#endif + __weak noinline void bpf_testmod_looooooooooooooooooooooooooooooong_name(void) { } @@ -514,6 +541,11 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, (void)bpf_testmod_test_arg_ptr_to_struct(&struct_arg1_2); +#ifdef __SIZEOF_INT128__ + (void)bpf_testmod_test_int128_ret(i); + (void)bpf_testmod_test_int128_arg((__int128)1, 2, 3); +#endif + (void)trace_bpf_testmod_test_raw_tp_null_tp(NULL); bpf_testmod_test_struct_ops3(); From 28e911d61d66b92a3bded8b54622ed3cd2795bf6 Mon Sep 17 00:00:00 2001 From: Matt Bobrowski Date: Sat, 1 Aug 2026 23:03:30 +1000 Subject: [PATCH 173/373] bpf: update BPF LSM maintainer list I've recently left Google, so my mattbobrowski@google.com mail address is now inactive. Update the BPF LSM maintainer entry with a mail address that I do still have access to and use. Note that I lost access to my mattbobrowski@google.com mail address before being able to make this MAINTAINER entry change from it. Signed-off-by: Matt Bobrowski Link: https://lore.kernel.org/bpf/am3uorMW7_UWA5An@lima-development Signed-off-by: Kumar Kartikeya Dwivedi --- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 32ca23e58f2b..e46c57f8493c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5030,7 +5030,7 @@ F: kernel/bpf/ringbuf.c BPF [SECURITY & LSM] (Security Audit and Enforcement using BPF) M: KP Singh -M: Matt Bobrowski +M: Matt Bobrowski L: bpf@vger.kernel.org S: Maintained F: Documentation/bpf/prog_lsm.rst From 70a841617aa8f228fc9de3b82cd1bdc4cb249497 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:16 -0700 Subject: [PATCH 174/373] bpf: Drop process_timer_func wrappers Drop process_timer_{helper,kfunc}() since bpf_call_arg_meta is now shared by helper and kfunc. Call process_timer_func() directly. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-2-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8d0635ee48c7..616194f25dfb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7224,18 +7224,6 @@ static int process_timer_func(struct bpf_verifier_env *env, struct bpf_reg_state return check_map_field_pointer(env, reg, argno, BPF_TIMER, map); } -static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_call_arg_meta *meta) -{ - return process_timer_func(env, reg, argno, &meta->map); -} - -static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - struct bpf_call_arg_meta *meta) -{ - return process_timer_func(env, reg, argno, &meta->map); -} - static int process_kptr_func(struct bpf_verifier_env *env, int regno, struct bpf_call_arg_meta *meta) { @@ -8466,7 +8454,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, } break; case ARG_PTR_TO_TIMER: - err = process_timer_helper(env, reg, argno, meta); + err = process_timer_func(env, reg, argno, &meta->map); if (err) return err; break; @@ -12515,7 +12503,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me reg_arg_name(env, argno)); return -EINVAL; } - ret = process_timer_kfunc(env, reg, argno, meta); + ret = process_timer_func(env, reg, argno, &meta->map); if (ret < 0) return ret; break; From b33d09b4d9141586706b6a99ce5b2f189adc7cdd Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:17 -0700 Subject: [PATCH 175/373] bpf: Unify const map ptr argument checking for helpers and kfuncs Both the helper ARG_CONST_MAP_PTR and the kfunc KF_ARG_PTR_TO_MAP recorded the map pointer in meta->map and, when a map was already bound by a preceding timer/workqueue/task_work argument, rejected a mismatching map. Factor the logic into a single process_map_ptr_arg() used by both paths. The bound-object name (timer, workqueue, or bpf_task_work) is derived from the bound map's btf_record, and the register numbers in the message are computed from the map argument position instead of being hard-coded. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-3-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 97 ++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 53 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 616194f25dfb..4a5b54219210 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8274,6 +8274,44 @@ static int get_constant_map_key(struct bpf_verifier_env *env, static bool can_elide_value_nullness(const struct bpf_map *map); +static int process_map_ptr_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, + argno_t argno, struct bpf_call_arg_meta *meta) +{ + /* Use map_uid (which is unique id of inner map) to reject: + * inner_map1 = bpf_map_lookup_elem(outer_map, key1) + * inner_map2 = bpf_map_lookup_elem(outer_map, key2) + * if (inner_map1 && inner_map2) { + * timer = bpf_map_lookup_elem(inner_map1); + * if (timer) + * // mismatch would have been allowed + * bpf_timer_init(timer, inner_map2); + * } + * + * Comparing map_ptr is enough to distinguish normal and outer maps. + */ + if (meta->map.ptr && + (meta->map.ptr != reg->map_ptr || meta->map.uid != reg->map_uid)) { + argno_t obj_argno = argno_from_reg(reg_from_argno(argno) - 1); + struct btf_record *rec = meta->map.ptr->record; + const char *obj_name = "workqueue"; + + if (rec->timer_off >= 0) + obj_name = "timer"; + else if (rec->task_work_off >= 0) + obj_name = "bpf_task_work"; + + verbose(env, "%s pointer in %s map_uid=%d ", + obj_name, reg_arg_name(env, obj_argno), meta->map.uid); + verbose(env, "doesn't match map pointer in %s map_uid=%d\n", + reg_arg_name(env, argno), reg->map_uid); + return -EINVAL; + } + + meta->map.ptr = reg->map_ptr; + meta->map.uid = reg->map_uid; + return 0; +} + static int check_func_arg(struct bpf_verifier_env *env, u32 arg, struct bpf_call_arg_meta *meta, const struct bpf_func_proto *fn, @@ -8349,29 +8387,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, switch (base_type(arg_type)) { case ARG_CONST_MAP_PTR: /* bpf_map_xxx(map_ptr) call: remember that map_ptr */ - if (meta->map.ptr) { - /* Use map_uid (which is unique id of inner map) to reject: - * inner_map1 = bpf_map_lookup_elem(outer_map, key1) - * inner_map2 = bpf_map_lookup_elem(outer_map, key2) - * if (inner_map1 && inner_map2) { - * timer = bpf_map_lookup_elem(inner_map1); - * if (timer) - * // mismatch would have been allowed - * bpf_timer_init(timer, inner_map2); - * } - * - * Comparing map_ptr is enough to distinguish normal and outer maps. - */ - if (meta->map.ptr != reg->map_ptr || - meta->map.uid != reg->map_uid) { - verbose(env, - "timer pointer in R1 map_uid=%d doesn't match map pointer in R2 map_uid=%d\n", - meta->map.uid, reg->map_uid); - return -EINVAL; - } - } - meta->map.ptr = reg->map_ptr; - meta->map.uid = reg->map_uid; + err = process_map_ptr_arg(env, reg, argno, meta); + if (err) + return err; break; case ARG_PTR_TO_MAP_KEY: /* bpf_map_xxx(..., map_ptr, ..., key) call: @@ -12123,36 +12141,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me reg_arg_name(env, argno)); return -EINVAL; } - if (meta->map.ptr && (reg->map_ptr->record->wq_off >= 0 || - reg->map_ptr->record->task_work_off >= 0)) { - /* Use map_uid (which is unique id of inner map) to reject: - * inner_map1 = bpf_map_lookup_elem(outer_map, key1) - * inner_map2 = bpf_map_lookup_elem(outer_map, key2) - * if (inner_map1 && inner_map2) { - * wq = bpf_map_lookup_elem(inner_map1); - * if (wq) - * // mismatch would have been allowed - * bpf_wq_init(wq, inner_map2); - * } - * - * Comparing map_ptr is enough to distinguish normal and outer maps. - */ - if (meta->map.ptr != reg->map_ptr || - meta->map.uid != reg->map_uid) { - if (reg->map_ptr->record->task_work_off >= 0) { - verbose(env, - "bpf_task_work pointer in R2 map_uid=%d doesn't match map pointer in R3 map_uid=%d\n", - meta->map.uid, reg->map_uid); - return -EINVAL; - } - verbose(env, - "workqueue pointer in R1 map_uid=%d doesn't match map pointer in R2 map_uid=%d\n", - meta->map.uid, reg->map_uid); - return -EINVAL; - } - } - meta->map.ptr = reg->map_ptr; - meta->map.uid = reg->map_uid; + ret = process_map_ptr_arg(env, reg, argno, meta); + if (ret < 0) + return ret; fallthrough; case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: From c82b998777b7102f3617a46201805b7e7b899524 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:18 -0700 Subject: [PATCH 176/373] bpf: Split kfunc map argument into __const_map and __map Kfuncs used a single '__map' suffix (KF_ARG_PTR_TO_MAP) for two different things: a verifier-known map matched by map_uid against a bound timer/wq/task_work object (bpf_wq_init, bpf_task_work_schedule*), and an opaque 'struct bpf_map *' used only at runtime (bpf_arena_*), which may be a map fd or a PTR_TO_BTF_ID struct bpf_map (e.g. a bpf_map iterator's ctx->map). That combined path only accepted the btf map form due to type confusion. The 'if (!reg->map_ptr)' check reads reg->map_ptr, which aliases reg->btf in the bpf_reg_state union. A PTR_TO_BTF_ID register always has a non-NULL reg->btf, so the guard silently passed and validation fell through to process_kf_arg_ptr_to_btf_id(). It also recorded PTR_TO_BTF_ID info in meta->map, which would be meaningless. Split the annotation to avoid such type confusion and to align with helper: - '__const_map' -> KF_ARG_CONST_MAP_PTR: verifier-known map, handled by process_map_ptr_arg() like helper ARG_CONST_MAP_PTR. - '__map' -> KF_ARG_PTR_TO_BTF_ID: opaque struct bpf_map, validated by process_kf_arg_ptr_to_btf_id(). A map fd still matches via reg2btf_ids[CONST_PTR_TO_MAP], so bpf_arena_alloc_pages(&map) keeps working. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-4-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- Documentation/bpf/kfuncs.rst | 30 +++++++++++++++++++++++- kernel/bpf/helpers.c | 16 ++++++------- kernel/bpf/verifier.c | 45 +++++++++++++++++++++--------------- 3 files changed, 64 insertions(+), 27 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index c801a330aece..cbde86d082cc 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -250,6 +250,34 @@ Or:: ... } +2.3.7 __const_map and __map Annotations +--------------------------------------- + +These annotations are used for ``struct bpf_map *`` arguments and distinguish a +verifier-known map from an opaque one. + +``__const_map`` indicates a map must be known at the verification time, i.e. a +concrete map fd the BPF program references directly. + +An example is given below:: + + __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, + unsigned int flags) + { + ... + } + +``__map`` indicates an opaque ``struct bpf_map *`` that may be resolved +at run time. The argument may take either a map fd or a ``PTR_TO_BTF_ID`` +``struct bpf_map`` pointer. + +An example is given below:: + + __bpf_kfunc void *bpf_arena_alloc_pages(void *p__map, ...) + { + ... + } + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function @@ -411,7 +439,7 @@ Example declaration: .. code-block:: c __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw, - void *map__map, bpf_task_work_callback_t callback, + void *map__const_map, bpf_task_work_callback_t callback, struct bpf_prog_aux *aux) { ... } Example usage in BPF program: diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 88b38db47de9..e472535bce85 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3404,10 +3404,10 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN(1, "A call to BPF exception callback should never return\n"); } -__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) +__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags) { struct bpf_async_kern *async = (struct bpf_async_kern *)wq; - struct bpf_map *map = p__map; + struct bpf_map *map = p__const_map; BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq)); BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq)); @@ -4643,17 +4643,17 @@ static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work * mode * @task: Task struct for which callback should be scheduled * @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping - * @map__map: bpf_map that embeds struct bpf_task_work in the values + * @map__const_map: bpf_map that embeds struct bpf_task_work in the values * @callback: pointer to BPF subprogram to call * @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier * * Return: 0 if task work has been scheduled successfully, negative error code otherwise */ __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw, - void *map__map, bpf_task_work_callback_t callback, + void *map__const_map, bpf_task_work_callback_t callback, struct bpf_prog_aux *aux) { - return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL); + return bpf_task_work_schedule(task, tw, map__const_map, callback, aux, TWA_SIGNAL); } /** @@ -4661,17 +4661,17 @@ __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct b * mode * @task: Task struct for which callback should be scheduled * @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping - * @map__map: bpf_map that embeds struct bpf_task_work in the values + * @map__const_map: bpf_map that embeds struct bpf_task_work in the values * @callback: pointer to BPF subprogram to call * @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier * * Return: 0 if task work has been scheduled successfully, negative error code otherwise */ __bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw, - void *map__map, bpf_task_work_callback_t callback, + void *map__const_map, bpf_task_work_callback_t callback, struct bpf_prog_aux *aux) { - return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME); + return bpf_task_work_schedule(task, tw, map__const_map, callback, aux, TWA_RESUME); } static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4a5b54219210..4f39e439973c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10818,6 +10818,11 @@ static bool is_kfunc_arg_map(const struct btf *btf, const struct btf_param *arg) return btf_param_match_suffix(btf, arg, "__map"); } +static bool is_kfunc_arg_const_map(const struct btf *btf, const struct btf_param *arg) +{ + return btf_param_match_suffix(btf, arg, "__const_map"); +} + static bool is_kfunc_arg_alloc_obj(const struct btf *btf, const struct btf_param *arg) { return btf_param_match_suffix(btf, arg, "__alloc"); @@ -11064,7 +11069,7 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_RB_NODE, KF_ARG_PTR_TO_NULL, KF_ARG_PTR_TO_CONST_STR, - KF_ARG_PTR_TO_MAP, + KF_ARG_CONST_MAP_PTR, KF_ARG_PTR_TO_TIMER, KF_ARG_PTR_TO_WORKQUEUE, KF_ARG_PTR_TO_IRQ_FLAG, @@ -11383,8 +11388,11 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *call if (is_kfunc_arg_const_str(meta->btf, &args[arg])) return KF_ARG_PTR_TO_CONST_STR; + if (is_kfunc_arg_const_map(meta->btf, &args[arg])) + return KF_ARG_CONST_MAP_PTR; + if (is_kfunc_arg_map(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_MAP; + return KF_ARG_PTR_TO_BTF_ID; if (is_kfunc_arg_wq(meta->btf, &args[arg])) return KF_ARG_PTR_TO_WORKQUEUE; @@ -12132,19 +12140,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (kf_arg_type < 0) return kf_arg_type; + if (is_kfunc_arg_map(btf, &args[i])) { + ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; + ref_t = btf_type_by_id(btf_vmlinux, ref_id); + ref_tname = btf_name_by_offset(btf, ref_t->name_off); + } + switch (kf_arg_type) { case KF_ARG_PTR_TO_NULL: continue; - case KF_ARG_PTR_TO_MAP: - if (!reg->map_ptr) { - verbose(env, "pointer in %s isn't map pointer\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - ret = process_map_ptr_arg(env, reg, argno, meta); - if (ret < 0) - return ret; - fallthrough; case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: if (!is_trusted_reg(env, reg)) { @@ -12160,6 +12164,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me } } fallthrough; + case KF_ARG_CONST_MAP_PTR: case KF_ARG_PTR_TO_ITER: case KF_ARG_PTR_TO_LIST_HEAD: case KF_ARG_PTR_TO_LIST_NODE: @@ -12366,12 +12371,16 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (ret < 0) return ret; break; - case KF_ARG_PTR_TO_MAP: - /* If argument has '__map' suffix expect 'struct bpf_map *' */ - ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; - ref_t = btf_type_by_id(btf_vmlinux, ref_id); - ref_tname = btf_name_by_offset(btf, ref_t->name_off); - fallthrough; + case KF_ARG_CONST_MAP_PTR: + if (base_type(reg->type) != CONST_PTR_TO_MAP) { + verbose(env, "pointer in %s isn't map pointer\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + ret = process_map_ptr_arg(env, reg, argno, meta); + if (ret < 0) + return ret; + break; case KF_ARG_PTR_TO_BTF_ID: /* Only base_type is checked, further checks are done here */ if ((base_type(reg->type) != PTR_TO_BTF_ID || From 9f52714dd89b489f7de5de2b60736edde07ddf5f Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:19 -0700 Subject: [PATCH 177/373] bpf: Pass kfunc meta to mem and mem_size check kfunc now shares the same bpf_call_arg_meta with helpers. Pass kfunc's own meta to check_mem_reg() and check_kfunc_mem_size() instead of NULL or a temporary meta on the stack. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-5-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4f39e439973c..3fec0afacb3f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6924,7 +6924,7 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, } static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, - argno_t argno, u32 mem_size) + argno_t argno, u32 mem_size, struct bpf_call_arg_meta *meta) { bool may_be_null = type_may_be_null(reg->type); struct bpf_reg_state saved_reg; @@ -6950,8 +6950,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; - err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, NULL); - err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, NULL); + err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta); + err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta); if (may_be_null) *reg = saved_reg; @@ -6994,22 +6994,20 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf } static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, - struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno) + struct bpf_reg_state *size_reg, argno_t mem_argno, + argno_t size_argno, struct bpf_call_arg_meta *meta) { bool may_be_null = type_may_be_null(mem_reg->type); struct bpf_reg_state saved_reg; - struct bpf_call_arg_meta meta; int err; - memset(&meta, 0, sizeof(meta)); - if (may_be_null) { saved_reg = *mem_reg; mark_ptr_not_null_reg(mem_reg); } - err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, &meta); - err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, &meta); + err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, meta); + err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, meta); if (may_be_null) *mem_reg = saved_reg; @@ -9258,7 +9256,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE); if (ret < 0) return ret; - if (check_mem_reg(env, reg, argno, arg->mem_size)) + if (check_mem_reg(env, reg, argno, arg->mem_size, NULL)) return -EINVAL; if (!(arg->arg_type & PTR_MAYBE_NULL) && (type_may_be_null(reg->type) || bpf_register_is_null(reg))) { @@ -12405,7 +12403,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname, PTR_ERR(resolve_ret)); return -EINVAL; } - ret = check_mem_reg(env, reg, argno, type_size); + ret = check_mem_reg(env, reg, argno, type_size, meta); if (ret < 0) return ret; break; @@ -12419,7 +12417,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (!bpf_register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) { ret = check_kfunc_mem_size_reg(env, buff_reg, size_reg, - argno, next_argno); + argno, next_argno, meta); if (ret < 0) { verbose(env, "%s and ", reg_arg_name(env, argno)); verbose(env, "%s memory, len pair leads to invalid memory access\n", From 341d227fa5db567ff2e3114b5be9b1051f336e34 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Sat, 1 Aug 2026 00:46:20 -0700 Subject: [PATCH 178/373] bpf: Resolve map lookup result type at lookup time bpf_map_lookup_elem() is typed to return PTR_TO_MAP_VALUE for every map, but for some map kinds the looked up value is actually a different object: an inner map, a socket or an xsk socket. Until now this reinterpretation happened once the pointer was converted from its NULL-able form to a concrete value. Such reinterpretation logic placement led to mark_ptr_not_null_reg() being called for a temporary register copy in check_mem_reg() and check_kfunc_mem_size_reg() (check_mem_size_reg() was buggy because of not calling it). The temporary copy was necessary to pass reinterpreted parameters as nullable helper and kfunc arguments. Avoid this complication by refining map lookup result type right away. The test case verifier_map_in_map/on_the_inner_map_pointer needs an update because the verifier now prints a concrete NULL-able type for the lookup. Signed-off-by: Eduard Zingerman Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-6-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 83 +++++++------------ .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- 2 files changed, 33 insertions(+), 52 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3fec0afacb3f..1b6ae6e5b995 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1854,32 +1854,34 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type ty reg->dynptr.first_slot = first_slot; } +/* + * Refine the return type of the bpf_map_lookup_elem() for special map types: + * map-in-map, xskmap, sockmap and sockhash. + */ +static void refine_map_lookup_value(struct bpf_reg_state *reg) +{ + enum bpf_type_flag maybe_null = reg->type & PTR_MAYBE_NULL; + const struct bpf_map *map = reg->map_ptr; + + if (map->inner_map_meta) { + reg->type = CONST_PTR_TO_MAP | maybe_null; + reg->map_ptr = map->inner_map_meta; + /* transfer reg's id which is unique for every map_lookup_elem + * as UID of the inner map. + */ + if (btf_record_has_field(map->inner_map_meta->record, + BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK)) + reg->map_uid = reg->id; + } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { + reg->type = PTR_TO_XDP_SOCK | maybe_null; + } else if (map->map_type == BPF_MAP_TYPE_SOCKMAP || + map->map_type == BPF_MAP_TYPE_SOCKHASH) { + reg->type = PTR_TO_SOCKET | maybe_null; + } +} + static void mark_ptr_not_null_reg(struct bpf_reg_state *reg) { - if (base_type(reg->type) == PTR_TO_MAP_VALUE) { - const struct bpf_map *map = reg->map_ptr; - - if (map->inner_map_meta) { - reg->type = CONST_PTR_TO_MAP; - reg->map_ptr = map->inner_map_meta; - /* transfer reg's id which is unique for every map_lookup_elem - * as UID of the inner map. - */ - if (btf_record_has_field(map->inner_map_meta->record, - BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK)) { - reg->map_uid = reg->id; - } - } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { - reg->type = PTR_TO_XDP_SOCK; - } else if (map->map_type == BPF_MAP_TYPE_SOCKMAP || - map->map_type == BPF_MAP_TYPE_SOCKHASH) { - reg->type = PTR_TO_SOCKET; - } else { - reg->type = PTR_TO_MAP_VALUE; - } - return; - } - reg->type &= ~PTR_MAYBE_NULL; } @@ -6926,8 +6928,6 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, u32 mem_size, struct bpf_call_arg_meta *meta) { - bool may_be_null = type_may_be_null(reg->type); - struct bpf_reg_state saved_reg; int err; if (bpf_register_is_null(reg)) @@ -6939,23 +6939,11 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg return -EACCES; } - /* Assuming that the register contains a value check if the memory - * access is safe. Temporarily save and restore the register's state as - * the conversion shouldn't be visible to a caller. - */ - if (may_be_null) { - saved_reg = *reg; - mark_ptr_not_null_reg(reg); - } - int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta); err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta); - if (may_be_null) - *reg = saved_reg; - return err; } @@ -6997,21 +6985,11 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno, struct bpf_call_arg_meta *meta) { - bool may_be_null = type_may_be_null(mem_reg->type); - struct bpf_reg_state saved_reg; int err; - if (may_be_null) { - saved_reg = *mem_reg; - mark_ptr_not_null_reg(mem_reg); - } - err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, meta); err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, meta); - if (may_be_null) - *mem_reg = saved_reg; - return err; } @@ -10522,10 +10500,12 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn regs[BPF_REG_0].map_ptr = meta.map.ptr; regs[BPF_REG_0].map_uid = meta.map.uid; regs[BPF_REG_0].type = PTR_TO_MAP_VALUE | ret_flag; - if (!type_may_be_null(ret_flag) && + if (type_may_be_null(ret_flag) || btf_record_has_field(meta.map.ptr->record, BPF_SPIN_LOCK | BPF_RES_SPIN_LOCK)) { regs[BPF_REG_0].id = ++env->id_gen; } + /* requires regs[BPF_REG_0].id to be set because of the map-in-map case */ + refine_map_lookup_value(®s[BPF_REG_0]); break; case RET_PTR_TO_SOCKET: mark_reg_known_zero(env, regs, BPF_REG_0); @@ -10623,7 +10603,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn return -EINVAL; } - if (type_may_be_null(regs[BPF_REG_0].type)) + if (type_may_be_null(regs[BPF_REG_0].type) && !regs[BPF_REG_0].id) regs[BPF_REG_0].id = ++env->id_gen; if (is_ptr_cast_function(func_id) && @@ -12370,7 +12350,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me return ret; break; case KF_ARG_CONST_MAP_PTR: - if (base_type(reg->type) != CONST_PTR_TO_MAP) { + if (base_type(reg->type) != CONST_PTR_TO_MAP || + type_may_be_null(reg->type)) { verbose(env, "pointer in %s isn't map pointer\n", reg_arg_name(env, argno)); return -EINVAL; diff --git a/tools/testing/selftests/bpf/progs/verifier_map_in_map.c b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c index b606b5dca734..7918646e5bfc 100644 --- a/tools/testing/selftests/bpf/progs/verifier_map_in_map.c +++ b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c @@ -154,7 +154,7 @@ l0_%=: r0 = 0; \ SEC("socket") __description("forgot null checking on the inner map pointer") -__failure __msg("R1 type=map_value_or_null expected=map_ptr") +__failure __msg("R1 type=map_ptr_or_null expected=map_ptr") __failure_unpriv __naked void on_the_inner_map_pointer(void) { From d4e7fb59c0d4c746cf95054190b7b30d91995ddc Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:21 -0700 Subject: [PATCH 179/373] bpf: Check helper and kfunc mem+size arguments identically Helper ARG_CONST_SIZE and kfunc KF_ARG_PTR_TO_MEM_SIZE memory arguments already share check_mem_size_reg(), but the kfunc path reached it through a thin wrapper, check_kfunc_mem_size_reg(). The wrapper existed only to invoke check_mem_size_reg() twice. Once for BPF_READ and once for BPF_WRITE because a kfunc mem argument may be both read and written, whereas a helper argument carries a single access direction. Let check_mem_size_reg() take a bitmask of access directions (widening access_type to u32) and perform each requested access, then pass BPF_READ | BPF_WRITE from the kfunc call site. This removes the check_kfunc_mem_size_reg() wrapper so helper and kfunc mem+size arguments run through exactly the same code. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-7-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1b6ae6e5b995..f61771e2a8b2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6871,11 +6871,11 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_ static int check_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, struct bpf_reg_state *size_reg, argno_t mem_argno, - argno_t size_argno, enum bpf_access_type access_type, + argno_t size_argno, u32 access_type, bool zero_size_allowed, struct bpf_call_arg_meta *meta) { - int err; + int err = 0; /* This is used to refine r0 return value bounds for helpers * that enforce this value as an upper bound on return values. @@ -6912,8 +6912,14 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, reg_arg_name(env, size_argno)); return -EACCES; } - err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), - access_type, zero_size_allowed, meta); + + if (access_type & BPF_READ) + err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), + BPF_READ, zero_size_allowed, meta); + if (!err && access_type & BPF_WRITE) + err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), + BPF_WRITE, zero_size_allowed, meta); + if (!err) { int regno = reg_from_argno(size_argno); @@ -6922,6 +6928,7 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, else err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno)); } + return err; } @@ -6981,18 +6988,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf return 0; } -static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, - struct bpf_reg_state *size_reg, argno_t mem_argno, - argno_t size_argno, struct bpf_call_arg_meta *meta) -{ - int err; - - err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, meta); - err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, meta); - - return err; -} - enum { PROCESS_SPIN_LOCK = (1 << 0), PROCESS_RES_LOCK = (1 << 1), @@ -12397,8 +12392,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me argno_t next_argno = argno_from_arg(i + 2); if (!bpf_register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) { - ret = check_kfunc_mem_size_reg(env, buff_reg, size_reg, - argno, next_argno, meta); + ret = check_mem_size_reg(env, buff_reg, size_reg, argno, next_argno, + BPF_READ | BPF_WRITE, true, meta); if (ret < 0) { verbose(env, "%s and ", reg_arg_name(env, argno)); verbose(env, "%s memory, len pair leads to invalid memory access\n", From e6d200dd1026cc2b6708ef0eb7f48fc064d4431c Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:22 -0700 Subject: [PATCH 180/373] selftests/bpf: Test map lookup result refinement A map-of-maps lookup value is refined to a map pointer (map_ptr_or_null) at lookup time by refine_map_lookup_value(). Test that it is rejected wherever a raw map value would be read as bytes, so the inner map descriptor cannot leak. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-8-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_map_lookup_refine.c | 73 +++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_map_lookup_refine.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index be97f6887f0e..41cd071d016a 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -61,6 +61,7 @@ #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h" #include "verifier_map_in_map.skel.h" +#include "verifier_map_lookup_refine.skel.h" #include "verifier_map_ptr.skel.h" #include "verifier_map_ptr_mixing.skel.h" #include "verifier_map_ret_val.skel.h" @@ -215,6 +216,7 @@ void test_verifier_liveness_exp(void) { RUN(verifier_liveness_exp); } void test_verifier_loops1(void) { RUN(verifier_loops1); } void test_verifier_lwt(void) { RUN(verifier_lwt); } void test_verifier_map_in_map(void) { RUN(verifier_map_in_map); } +void test_verifier_map_lookup_refine(void) { RUN(verifier_map_lookup_refine); } void test_verifier_map_ptr(void) { RUN(verifier_map_ptr); } void test_verifier_map_ptr_mixing(void) { RUN(verifier_map_ptr_mixing); } void test_verifier_map_ret_val(void) { RUN(verifier_map_ret_val); } diff --git a/tools/testing/selftests/bpf/progs/verifier_map_lookup_refine.c b/tools/testing/selftests/bpf/progs/verifier_map_lookup_refine.c new file mode 100644 index 000000000000..c01abf54923d --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_map_lookup_refine.c @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" +#include "bpf_kfuncs.h" + +char _license[] SEC("license") = "GPL"; + +struct inner_map { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} inner_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS); + __uint(max_entries, 1); + __type(key, int); + __array(values, struct inner_map); +} outer_map SEC(".maps") = { + .values = { [0] = &inner_map }, +}; + +SEC("?tc") +__failure __msg("type=map_ptr_or_null expected=fp") +int mapofmaps_value_as_kfunc_mem_buf(struct __sk_buff *skb) +{ + struct bpf_dynptr dptr; + __u32 key = 0; + void *inner; + char *p; + + inner = bpf_map_lookup_elem(&outer_map, &key); + /* intentionally NOT NULL-checked: type is map_ptr_or_null */ + + bpf_dynptr_from_skb(skb, 0, &dptr); + /* arg3 is mem+size */ + p = bpf_dynptr_slice(&dptr, 0, inner, 4); + if (p) + return p[0]; + return 0; +} + +SEC("?tc") +__failure __msg("type=map_ptr_or_null expected=fp") +int mapofmaps_value_as_helper_mem_buf(struct __sk_buff *skb) +{ + __u32 key = 0; + void *inner; + + inner = bpf_map_lookup_elem(&outer_map, &key); + /* intentionally NOT NULL-checked: type is map_ptr_or_null */ + + /* arg1 is mem+size */ + return bpf_csum_diff(inner, 4, NULL, 0, 0) + skb->len; +} + +SEC("?tc") +__failure __msg("type=map_ptr_or_null expected=fp") +int mapofmaps_value_as_helper_fixed_mem(struct __sk_buff *skb) +{ + char th[sizeof(struct tcphdr)] = {}; + __u32 key = 0; + void *inner; + + inner = bpf_map_lookup_elem(&outer_map, &key); + /* intentionally NOT NULL-checked: type is map_ptr_or_null */ + + /* arg1 is fixed-sized mem */ + return bpf_tcp_raw_check_syncookie_ipv4(inner, (void *)th); +} From e566701b9b0c1ec398152cb972d592992984e1a8 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:23 -0700 Subject: [PATCH 181/373] bpf: Check fixed-size mem args of helpers and kfuncs the same way Fixed-size memory arguments went through two paths: helpers called check_helper_mem_access() directly, while kfuncs and global subprogs used check_mem_reg(). Route the helper MEM_FIXED_SIZE case through check_mem_reg() too so all three share the same check. This also fixes a bug in the helper path. When passing a NULL to PTR_MAYBE_NULL | ARG_PTR_TO_FIXED_SIZE_MEM argument, the program would be falsely rejected by check_helper_mem_access(). This is not triggerable since there is no such kind of helper. Also, note that check_reg_type() still make sure NULL cannot be passed to an argument not marked with PTR_MAYBE_NULL. It also tightens the poisoned-stack-slot check. check_mem_reg() encoded "a STACK_POISON slot may be read" as a negative access size for any PTR_TO_STACK argument, but that is only sound for global subprogs, where static stack liveness proved the callee body does not read those slots (2cb27158adb3 ("bpf: poison dead stack slots")). Since check_mem_reg() is also used for kfuncs, kfuncs accidentally inherited it and could read a poisoned (dead, possibly uninitialized) stack slot. Restrict the negative size to global subprogs (meta == NULL) so kfuncs, like helpers, require the whole argument initialized. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-9-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f61771e2a8b2..9e0c7f0a15c1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6669,7 +6669,7 @@ static int check_stack_range_initialized( */ bool clobber = type == BPF_WRITE; /* - * Negative access_size signals global subprog/kfunc arg check where + * Negative access_size signals global subprog arg check where * STACK_POISON slots are acceptable. static stack liveness * might have determined that subprog doesn't read them, * but BTF based global subprog validation isn't accurate enough. @@ -6933,9 +6933,10 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, } static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, - argno_t argno, u32 mem_size, struct bpf_call_arg_meta *meta) + argno_t argno, u32 mem_size, enum bpf_access_type access_type, + struct bpf_call_arg_meta *meta) { - int err; + int size, err = 0; if (bpf_register_is_null(reg)) return 0; @@ -6946,10 +6947,16 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg return -EACCES; } - int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; + /* + * Only a global subprog (meta == NULL) may read poisoned stack slots: + * its static stack liveness proved the callee body skips them. + */ + size = (!meta && base_type(reg->type) == PTR_TO_STACK) ? -(int)mem_size : mem_size; - err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta); - err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta); + if (access_type & BPF_READ) + err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta); + if (!err && (access_type & BPF_WRITE)) + err = check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta); return err; } @@ -8455,9 +8462,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, * next is_mem_size argument below. */ if (arg_type & MEM_FIXED_SIZE) { - err = check_helper_mem_access(env, reg, argno, fn->arg_size[arg], - arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, - false, meta); + err = check_mem_reg(env, reg, argno_from_reg(regno), fn->arg_size[arg], + arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, meta); if (err) return err; if (arg_type & MEM_ALIGNED) @@ -9229,7 +9235,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE); if (ret < 0) return ret; - if (check_mem_reg(env, reg, argno, arg->mem_size, NULL)) + if (check_mem_reg(env, reg, argno, arg->mem_size, BPF_READ | BPF_WRITE, NULL)) return -EINVAL; if (!(arg->arg_type & PTR_MAYBE_NULL) && (type_may_be_null(reg->type) || bpf_register_is_null(reg))) { @@ -12379,7 +12385,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname, PTR_ERR(resolve_ret)); return -EINVAL; } - ret = check_mem_reg(env, reg, argno, type_size, meta); + ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, meta); if (ret < 0) return ret; break; From c0e091f30f0dd80d817c3a15d0a97962957e5786 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:24 -0700 Subject: [PATCH 182/373] bpf: Rename ARG_CONST_SIZE{,_OR_ZERO} to ARG_MEM_SIZE{,_OR_ZERO} ARG_CONST_SIZE does not require a constant: check_mem_size_reg() accepts any bounded scalar and verifies the memory access against its maximum (reg_umax). Rename ARG_CONST_SIZE and ARG_CONST_SIZE_OR_ZERO to ARG_MEM_SIZE and ARG_MEM_SIZE_OR_ZERO to reflect that. ARG_CONST_ALLOC_ SIZE_OR_ZERO, which does require a constant, is left unchanged. Pure rename, no functional change. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-10-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 4 +- include/linux/bpf_verifier.h | 2 +- kernel/bpf/backtrack.c | 2 +- kernel/bpf/bpf_lsm.c | 4 +- kernel/bpf/btf.c | 2 +- kernel/bpf/cgroup.c | 8 +- kernel/bpf/helpers.c | 26 ++-- kernel/bpf/ringbuf.c | 2 +- kernel/bpf/stackmap.c | 10 +- kernel/bpf/syscall.c | 4 +- kernel/bpf/verifier.c | 11 +- kernel/trace/bpf_trace.c | 62 +++++----- net/core/filter.c | 116 +++++++++--------- .../bpf/progs/mem_rdonly_untrusted.c | 2 +- .../selftests/bpf/progs/verifier_bounds.c | 2 +- .../progs/verifier_helper_access_var_len.c | 6 +- .../bpf/progs/verifier_helper_value_access.c | 2 +- 17 files changed, 132 insertions(+), 133 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7bfc28673124..be53655d1362 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -888,8 +888,8 @@ enum bpf_arg_type { ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */ ARG_PTR_TO_ARENA, - ARG_CONST_SIZE, /* number of bytes accessed from memory */ - ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ + ARG_MEM_SIZE, /* number of bytes accessed from memory */ + ARG_MEM_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ ARG_PTR_TO_CTX, /* pointer to context */ ARG_ANYTHING, /* any (initialized) argument is ok */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 682c2cd3b844..bb0d43814e90 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -15,7 +15,7 @@ * ensures that umax_value + (int)off + (int)size cannot overflow a u64. */ #define BPF_MAX_VAR_OFF (1 << 29) -/* Maximum variable size permitted for ARG_CONST_SIZE[_OR_ZERO]. This ensures +/* Maximum variable size permitted for ARG_MEM_SIZE[_OR_ZERO]. This ensures * that converting umax_value to int cannot overflow. */ #define BPF_MAX_VAR_SIZ (1 << 29) diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 2e4ae0ef0860..2f473ad4fd7c 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -636,7 +636,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, * r5 += 1 * ... * call bpf_perf_event_output#25 - * where .arg5_type = ARG_CONST_SIZE_OR_ZERO + * where .arg5_type = ARG_MEM_SIZE_OR_ZERO * * and this case: * r6 = 1 diff --git a/kernel/bpf/bpf_lsm.c b/kernel/bpf/bpf_lsm.c index 3983b4ce73c8..82c5988417a0 100644 --- a/kernel/bpf/bpf_lsm.c +++ b/kernel/bpf/bpf_lsm.c @@ -186,7 +186,7 @@ static const struct bpf_func_proto bpf_ima_inode_hash_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &bpf_ima_inode_hash_btf_ids[0], .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .allowed = bpf_ima_inode_hash_allowed, }; @@ -205,7 +205,7 @@ static const struct bpf_func_proto bpf_ima_file_hash_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &bpf_ima_file_hash_btf_ids[0], .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .allowed = bpf_ima_inode_hash_allowed, }; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 4eeeaeb69790..5e8ac45ce56a 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -8709,7 +8709,7 @@ const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 4355ccb78a9c..fb9357b64cad 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -2305,7 +2305,7 @@ static const struct bpf_func_proto bpf_sysctl_get_name_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -2347,7 +2347,7 @@ static const struct bpf_func_proto bpf_sysctl_get_current_value_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; BPF_CALL_3(bpf_sysctl_get_new_value, struct bpf_sysctl_kern *, ctx, char *, buf, @@ -2367,7 +2367,7 @@ static const struct bpf_func_proto bpf_sysctl_get_new_value_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, @@ -2393,7 +2393,7 @@ static const struct bpf_func_proto bpf_sysctl_set_new_value_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; static const struct bpf_func_proto * diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index e472535bce85..4709a5ad0474 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -278,7 +278,7 @@ const struct bpf_func_proto bpf_get_current_comm_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, }; #if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK) @@ -539,7 +539,7 @@ const struct bpf_func_proto bpf_strtol_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED, .arg4_size = sizeof(s64), @@ -567,7 +567,7 @@ const struct bpf_func_proto bpf_strtoul_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED, .arg4_size = sizeof(u64), @@ -583,7 +583,7 @@ static const struct bpf_func_proto bpf_strncmp_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_PTR_TO_CONST_STR, }; @@ -627,7 +627,7 @@ const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = { .arg1_type = ARG_ANYTHING, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = { @@ -653,7 +653,7 @@ const struct bpf_func_proto bpf_event_output_data_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size, @@ -675,7 +675,7 @@ const struct bpf_func_proto bpf_copy_from_user_proto = { .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -706,7 +706,7 @@ const struct bpf_func_proto bpf_copy_from_user_task_proto = { .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_BTF_ID, .arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK], @@ -1093,10 +1093,10 @@ const struct bpf_func_proto bpf_snprintf_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_PTR_TO_CONST_STR, .arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx) @@ -1888,7 +1888,7 @@ static const struct bpf_func_proto bpf_dynptr_from_mem_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE, }; @@ -1943,7 +1943,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_PTR_TO_DYNPTR, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, @@ -2004,7 +2004,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = { .arg1_type = ARG_PTR_TO_DYNPTR, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE_OR_ZERO, + .arg4_type = ARG_MEM_SIZE_OR_ZERO, .arg5_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 35ae64ade36b..c1bf7a197a96 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -634,7 +634,7 @@ const struct bpf_func_proto bpf_ringbuf_output_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_CONST_MAP_PTR, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 41fe87d7302f..463f94ba1cc4 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -781,7 +781,7 @@ const struct bpf_func_proto bpf_get_stack_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -797,7 +797,7 @@ const struct bpf_func_proto bpf_get_stack_sleepable_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -831,7 +831,7 @@ const struct bpf_func_proto bpf_get_task_stack_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK], .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -848,7 +848,7 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK], .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -911,7 +911,7 @@ const struct bpf_func_proto bpf_get_stack_proto_pe = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 94091130bcc5..d6be7f49433c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -6559,7 +6559,7 @@ static const struct bpf_func_proto bpf_sys_bpf_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; const struct bpf_func_proto * __weak @@ -6606,7 +6606,7 @@ static const struct bpf_func_proto bpf_kallsyms_lookup_name_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED, .arg4_size = sizeof(u64), diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9e0c7f0a15c1..89e72d0a2c46 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7704,8 +7704,7 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx, static bool arg_type_is_mem_size(enum bpf_arg_type type) { - return type == ARG_CONST_SIZE || - type == ARG_CONST_SIZE_OR_ZERO; + return type == ARG_MEM_SIZE || type == ARG_MEM_SIZE_OR_ZERO; } static bool arg_type_is_raw_mem(enum bpf_arg_type type) @@ -7851,8 +7850,8 @@ static const struct bpf_reg_types dynptr_types = { static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_MAP_KEY] = &mem_types, [ARG_PTR_TO_MAP_VALUE] = &mem_types, - [ARG_CONST_SIZE] = &scalar_types, - [ARG_CONST_SIZE_OR_ZERO] = &scalar_types, + [ARG_MEM_SIZE] = &scalar_types, + [ARG_MEM_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, @@ -8470,13 +8469,13 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, err = check_ptr_alignment(env, reg, 0, fn->arg_size[arg], true); } break; - case ARG_CONST_SIZE: + case ARG_MEM_SIZE: err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), argno, fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, false, meta); break; - case ARG_CONST_SIZE_OR_ZERO: + case ARG_MEM_SIZE_OR_ZERO: err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), argno, fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 76ab51deaa6b..891897f8a1b3 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -221,7 +221,7 @@ const struct bpf_func_proto bpf_probe_read_user_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -258,7 +258,7 @@ const struct bpf_func_proto bpf_probe_read_user_str_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -273,7 +273,7 @@ const struct bpf_func_proto bpf_probe_read_kernel_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -308,7 +308,7 @@ const struct bpf_func_proto bpf_probe_read_kernel_str_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -328,7 +328,7 @@ static const struct bpf_func_proto bpf_probe_read_compat_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; @@ -347,7 +347,7 @@ static const struct bpf_func_proto bpf_probe_read_compat_str_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_ANYTHING, }; #endif /* CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE */ @@ -383,7 +383,7 @@ static const struct bpf_func_proto bpf_probe_write_user_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; #define MAX_TRACE_PRINTK_VARARGS 3 @@ -418,7 +418,7 @@ static const struct bpf_func_proto bpf_trace_printk_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, }; static void __set_printk_clr_event(struct work_struct *work) @@ -474,9 +474,9 @@ static const struct bpf_func_proto bpf_trace_vprintk_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE_OR_ZERO, + .arg4_type = ARG_MEM_SIZE_OR_ZERO, }; const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void) @@ -518,9 +518,9 @@ static const struct bpf_func_proto bpf_seq_printf_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_seq_file_ids[0], .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len) @@ -535,7 +535,7 @@ static const struct bpf_func_proto bpf_seq_write_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_seq_file_ids[0], .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr, @@ -559,7 +559,7 @@ static const struct bpf_func_proto bpf_seq_printf_btf_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_seq_file_ids[0], .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -633,7 +633,7 @@ static const struct bpf_func_proto bpf_perf_event_read_value_proto = { .arg1_type = ARG_CONST_MAP_PTR, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void) @@ -730,7 +730,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; static DEFINE_PER_CPU(int, bpf_event_output_nest_level); @@ -996,7 +996,7 @@ static const struct bpf_func_proto bpf_d_path_proto = { .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &bpf_d_path_btf_ids[0], .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .allowed = bpf_d_path_allowed, }; @@ -1053,9 +1053,9 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_MEM_SIZE, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, .arg5_type = ARG_ANYTHING, }; @@ -1218,7 +1218,7 @@ const struct bpf_func_proto bpf_get_branch_snapshot_proto = { .gpl_only = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_3(get_func_arg, void *, ctx, u32, n, u64 *, value) @@ -1421,7 +1421,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_tp = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_3(bpf_get_stackid_tp, void *, tp_buff, struct bpf_map *, map, @@ -1462,7 +1462,7 @@ static const struct bpf_func_proto bpf_get_stack_proto_tp = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -1524,12 +1524,12 @@ BPF_CALL_3(bpf_perf_prog_read_value, struct bpf_perf_event_data_kern *, ctx, } static const struct bpf_func_proto bpf_perf_prog_read_value_proto = { - .func = bpf_perf_prog_read_value, - .gpl_only = true, - .ret_type = RET_INTEGER, - .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .func = bpf_perf_prog_read_value, + .gpl_only = true, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_PTR_TO_UNINIT_MEM, + .arg3_type = ARG_MEM_SIZE, }; BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx, @@ -1566,7 +1566,7 @@ static const struct bpf_func_proto bpf_read_branch_records_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -1646,7 +1646,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; extern const struct bpf_func_proto bpf_skb_output_proto; @@ -1701,7 +1701,7 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; diff --git a/net/core/filter.c b/net/core/filter.c index c21c1daecf9d..eb4d299b1fec 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -1747,7 +1747,7 @@ static const struct bpf_func_proto bpf_skb_store_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, .arg5_type = ARG_ANYTHING, }; @@ -1784,7 +1784,7 @@ static const struct bpf_func_proto bpf_skb_load_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; int __bpf_skb_load_bytes(const struct sk_buff *skb, u32 offset, void *to, u32 len) @@ -1823,7 +1823,7 @@ static const struct bpf_func_proto bpf_flow_dissector_load_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_skb_load_bytes_relative, const struct sk_buff *, skb, @@ -1867,7 +1867,7 @@ static const struct bpf_func_proto bpf_skb_load_bytes_relative_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, .arg5_type = ARG_ANYTHING, }; @@ -2063,9 +2063,9 @@ static const struct bpf_func_proto bpf_csum_diff_proto = { .pkt_access = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_MEM_SIZE_OR_ZERO, .arg3_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE_OR_ZERO, + .arg4_type = ARG_MEM_SIZE_OR_ZERO, .arg5_type = ARG_ANYTHING, }; @@ -2626,7 +2626,7 @@ static const struct bpf_func_proto bpf_redirect_neigh_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, .arg2_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, }; @@ -4199,7 +4199,7 @@ static const struct bpf_func_proto bpf_xdp_load_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; int __bpf_xdp_load_bytes(struct xdp_buff *xdp, u32 offset, void *buf, u32 len) @@ -4231,7 +4231,7 @@ static const struct bpf_func_proto bpf_xdp_store_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; int __bpf_xdp_store_bytes(struct xdp_buff *xdp, u32 offset, void *buf, u32 len) @@ -4794,7 +4794,7 @@ static const struct bpf_func_proto bpf_skb_event_output_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BTF_ID_LIST_SINGLE(bpf_skb_output_btf_ids, struct, sk_buff) @@ -4808,7 +4808,7 @@ const struct bpf_func_proto bpf_skb_output_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; static unsigned short bpf_tunnel_key_af(u64 flags) @@ -4891,7 +4891,7 @@ static const struct bpf_func_proto bpf_skb_get_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -4926,7 +4926,7 @@ static const struct bpf_func_proto bpf_skb_get_tunnel_opt_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; static struct metadata_dst __percpu *md_dst; @@ -5008,7 +5008,7 @@ static const struct bpf_func_proto bpf_skb_set_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -5036,7 +5036,7 @@ static const struct bpf_func_proto bpf_skb_set_tunnel_opt_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; static const struct bpf_func_proto * @@ -5208,7 +5208,7 @@ static const struct bpf_func_proto bpf_xdp_event_output_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BTF_ID_LIST_SINGLE(bpf_xdp_output_btf_ids, struct, xdp_buff) @@ -5222,7 +5222,7 @@ const struct bpf_func_proto bpf_xdp_output_proto = { .arg2_type = ARG_CONST_MAP_PTR, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE_OR_ZERO, + .arg5_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_1(bpf_get_socket_cookie, struct sk_buff *, skb) @@ -5769,7 +5769,7 @@ const struct bpf_func_proto bpf_sk_setsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sk_getsockopt, struct sock *, sk, int, level, @@ -5786,7 +5786,7 @@ const struct bpf_func_proto bpf_sk_getsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_UNINIT_MEM, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sk_setsockopt_nodelay, struct sock *, sk, int, level, @@ -5810,7 +5810,7 @@ const struct bpf_func_proto bpf_sk_setsockopt_nodelay_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_unlocked_sk_setsockopt, struct sock *, sk, int, level, @@ -5827,7 +5827,7 @@ const struct bpf_func_proto bpf_unlocked_sk_setsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_unlocked_sk_getsockopt, struct sock *, sk, int, level, @@ -5844,7 +5844,7 @@ const struct bpf_func_proto bpf_unlocked_sk_getsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_UNINIT_MEM, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sock_addr_setsockopt, struct bpf_sock_addr_kern *, ctx, @@ -5861,7 +5861,7 @@ static const struct bpf_func_proto bpf_sock_addr_setsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sock_addr_getsockopt, struct bpf_sock_addr_kern *, ctx, @@ -5878,7 +5878,7 @@ static const struct bpf_func_proto bpf_sock_addr_getsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_UNINIT_MEM, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; static int sk_bpf_set_get_bypass_prot_mem(struct sock *sk, @@ -5923,7 +5923,7 @@ static const struct bpf_func_proto bpf_sock_create_setsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sock_create_getsockopt, struct sock *, sk, int, level, @@ -5949,7 +5949,7 @@ static const struct bpf_func_proto bpf_sock_create_getsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_UNINIT_MEM, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_sock_ops_setsockopt, struct bpf_sock_ops_kern *, bpf_sock, @@ -5975,7 +5975,7 @@ static const struct bpf_func_proto bpf_sock_ops_setsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; static int bpf_sock_ops_get_syn(struct bpf_sock_ops_kern *bpf_sock, @@ -6085,7 +6085,7 @@ static const struct bpf_func_proto bpf_sock_ops_getsockopt_proto = { .arg2_type = ARG_ANYTHING, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_PTR_TO_UNINIT_MEM, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_2(bpf_sock_ops_cb_flags_set, struct bpf_sock_ops_kern *, bpf_sock, @@ -6152,7 +6152,7 @@ static const struct bpf_func_proto bpf_bind_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, }; #ifdef CONFIG_XFRM @@ -6205,7 +6205,7 @@ static const struct bpf_func_proto bpf_skb_get_xfrm_state_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, .arg5_type = ARG_ANYTHING, }; #endif @@ -6612,7 +6612,7 @@ static const struct bpf_func_proto bpf_xdp_fib_lookup_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -6672,7 +6672,7 @@ static const struct bpf_func_proto bpf_skb_fib_lookup_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -6870,7 +6870,7 @@ static const struct bpf_func_proto bpf_lwt_in_push_encap_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE + .arg4_type = ARG_MEM_SIZE }; static const struct bpf_func_proto bpf_lwt_xmit_push_encap_proto = { @@ -6880,7 +6880,7 @@ static const struct bpf_func_proto bpf_lwt_xmit_push_encap_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE + .arg4_type = ARG_MEM_SIZE }; #if IS_ENABLED(CONFIG_IPV6_SEG6_BPF) @@ -6924,7 +6924,7 @@ static const struct bpf_func_proto bpf_lwt_seg6_store_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE + .arg4_type = ARG_MEM_SIZE }; static void bpf_update_srh_state(struct sk_buff *skb) @@ -7013,7 +7013,7 @@ static const struct bpf_func_proto bpf_lwt_seg6_action_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg4_type = ARG_CONST_SIZE + .arg4_type = ARG_MEM_SIZE }; BPF_CALL_3(bpf_lwt_seg6_adjust_srh, struct sk_buff *, skb, u32, offset, @@ -7254,7 +7254,7 @@ static const struct bpf_func_proto bpf_skc_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7273,7 +7273,7 @@ static const struct bpf_func_proto bpf_sk_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7292,7 +7292,7 @@ static const struct bpf_func_proto bpf_sk_lookup_udp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7316,7 +7316,7 @@ static const struct bpf_func_proto bpf_tc_skc_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7340,7 +7340,7 @@ static const struct bpf_func_proto bpf_tc_sk_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7364,7 +7364,7 @@ static const struct bpf_func_proto bpf_tc_sk_lookup_udp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7402,7 +7402,7 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_udp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7426,7 +7426,7 @@ static const struct bpf_func_proto bpf_xdp_skc_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7450,7 +7450,7 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7470,7 +7470,7 @@ static const struct bpf_func_proto bpf_sock_addr_skc_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7489,7 +7489,7 @@ static const struct bpf_func_proto bpf_sock_addr_sk_lookup_tcp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7508,7 +7508,7 @@ static const struct bpf_func_proto bpf_sock_addr_sk_lookup_udp_proto = { .ret_type = RET_PTR_TO_SOCKET_OR_NULL, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, }; @@ -7828,9 +7828,9 @@ static const struct bpf_func_proto bpf_tcp_check_syncookie_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_5(bpf_tcp_gen_syncookie, struct sock *, sk, void *, iph, u32, iph_len, @@ -7897,9 +7897,9 @@ static const struct bpf_func_proto bpf_tcp_gen_syncookie_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg5_type = ARG_CONST_SIZE, + .arg5_type = ARG_MEM_SIZE, }; BPF_CALL_3(bpf_sk_assign, struct sk_buff *, skb, struct sock *, sk, u64, flags) @@ -8053,7 +8053,7 @@ static const struct bpf_func_proto bpf_sock_ops_load_hdr_opt_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -8131,7 +8131,7 @@ static const struct bpf_func_proto bpf_sock_ops_store_hdr_opt_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_MEM_SIZE, .arg4_type = ARG_ANYTHING, }; @@ -8226,7 +8226,7 @@ static const struct bpf_func_proto bpf_tcp_raw_gen_syncookie_ipv4_proto = { .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_RDONLY, .arg1_size = sizeof(struct iphdr), .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_3(bpf_tcp_raw_gen_syncookie_ipv6, struct ipv6hdr *, iph, @@ -8258,7 +8258,7 @@ static const struct bpf_func_proto bpf_tcp_raw_gen_syncookie_ipv6_proto = { .arg1_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_RDONLY, .arg1_size = sizeof(struct ipv6hdr), .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, - .arg3_type = ARG_CONST_SIZE_OR_ZERO, + .arg3_type = ARG_MEM_SIZE_OR_ZERO, }; BPF_CALL_2(bpf_tcp_raw_check_syncookie_ipv4, struct iphdr *, iph, @@ -11731,7 +11731,7 @@ static const struct bpf_func_proto sk_reuseport_load_bytes_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, }; BPF_CALL_5(sk_reuseport_load_bytes_relative, @@ -11749,7 +11749,7 @@ static const struct bpf_func_proto sk_reuseport_load_bytes_relative_proto = { .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_ANYTHING, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_MEM_SIZE, .arg5_type = ARG_ANYTHING, }; diff --git a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c index 5b4453747c23..0952d1ebf0f1 100644 --- a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c +++ b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c @@ -137,7 +137,7 @@ int helper_param_not_ok(void *ctx) p = bpf_rdonly_cast(0, 0); /* - * Any helper with ARG_CONST_SIZE_OR_ZERO constraint will do, + * Any helper with ARG_MEM_SIZE_OR_ZERO constraint will do, * the most permissive constraint */ bpf_copy_from_user(p, 0, (void *)42); diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c index bc038ac2df98..1a273e416fed 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bounds.c +++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c @@ -1195,7 +1195,7 @@ l0_%=: r1 = r6; \ r3 += -8; \ r5 = 0; \ /* The 4th argument of bpf_skb_store_bytes is defined as \ - * ARG_CONST_SIZE, so 0 is not allowed. The 'r4 != 0' \ + * ARG_MEM_SIZE, so 0 is not allowed. The 'r4 != 0' \ * is providing us this exclusion of zero from initial \ * [0, 7] range. \ */ \ diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_access_var_len.c b/tools/testing/selftests/bpf/progs/verifier_helper_access_var_len.c index f2c54e4d89eb..343fc08d9747 100644 --- a/tools/testing/selftests/bpf/progs/verifier_helper_access_var_len.c +++ b/tools/testing/selftests/bpf/progs/verifier_helper_access_var_len.c @@ -85,7 +85,7 @@ __naked void stack_bitwise_and_zero_included(void) r2 += -64; \ r4 = 0; \ /* Call bpf_ringbuf_output(), it is one of a few helper functions with\ - * ARG_CONST_SIZE_OR_ZERO parameter allowed in unpriv mode.\ + * ARG_MEM_SIZE_OR_ZERO parameter allowed in unpriv mode.\ * For unpriv this should signal an error, because memory at &fp[-64] is\ * not initialized. \ */ \ @@ -278,7 +278,7 @@ __naked void stack_jmp_no_min_check(void) r2 += -64; \ r4 = 0; \ /* Call bpf_ringbuf_output(), it is one of a few helper functions with\ - * ARG_CONST_SIZE_OR_ZERO parameter allowed in unpriv mode.\ + * ARG_MEM_SIZE_OR_ZERO parameter allowed in unpriv mode.\ * For unpriv this should signal an error, because memory at &fp[-64] is\ * not initialized. \ */ \ @@ -778,7 +778,7 @@ __naked void variable_memory_8_bytes_leak(void) r3 += 1; \ r4 = 0; \ /* Call bpf_ringbuf_output(), it is one of a few helper functions with\ - * ARG_CONST_SIZE_OR_ZERO parameter allowed in unpriv mode.\ + * ARG_MEM_SIZE_OR_ZERO parameter allowed in unpriv mode.\ * For unpriv this should signal an error, because memory region [1, 64]\ * at &fp[-64] is not fully initialized. \ */ \ diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c index 6d2a38597c34..c6603a118fdc 100644 --- a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c +++ b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c @@ -91,7 +91,7 @@ l0_%=: exit; \ /* Call a function taking a pointer and a size which doesn't allow the size to * be zero (i.e. bpf_trace_printk() declares the second argument to be - * ARG_CONST_SIZE, not ARG_CONST_SIZE_OR_ZERO). We attempt to pass zero for the + * ARG_MEM_SIZE, not ARG_MEM_SIZE_OR_ZERO). We attempt to pass zero for the * size and expect to fail. */ SEC("tracepoint") From f16e80c2c45175664ec6b0aaad6914fadbda395d Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:25 -0700 Subject: [PATCH 183/373] bpf: Fold __szk const size handling into the scalar arg path To align helper and kfunc pointer to memory argument handling, move kfunc constant memorry size argument handling to the kfunc scalar section. In addition, factor out constant scalar argument handling. The constant size argument (__szk) of a kfunc memory/size pair was recorded into meta->arg_constant by a dedicated block in the KF_ARG_PTR_TO_MEM_SIZE case, duplicating the "only one constant argument" and "must be a known constant" checks already in the generic scalar argument handling. That block also did an explicit i++ to skip the size argument. This also fixes a precision gap: the old dedicated block did not mark the size register precise, relying on check_mem_size_reg() for that. But check_mem_size_reg() is skipped when the buffer is a nullable arg passed as NULL (e.g. bpf_dynptr_slice(_rdwr) with a NULL buffer), so in that case the __szk value was recorded and used for regs[R0].mem_size without marking it precise. Routing the size through the scalar path marks it precise in all cases. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-11-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 11 +++--- kernel/bpf/verifier.c | 66 +++++++++++++++++------------------- 2 files changed, 39 insertions(+), 38 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index bb0d43814e90..b54c1a5c9b11 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1479,6 +1479,12 @@ struct ret_mem_desc { bool found; }; +/* A constant scalar argument; Populated by process_const_arg() */ +struct arg_constant_desc { + u64 value; + bool found; +}; + struct bpf_call_arg_meta { /* Common */ struct btf *btf; @@ -1496,10 +1502,7 @@ struct bpf_call_arg_meta { u32 kfunc_flags; const struct btf_type *func_proto; const char *func_name; - struct { - u64 value; - bool found; - } arg_constant; + struct arg_constant_desc arg_constant; /* arg_{btf,btf_id,owning_ref} are used by kfunc-specific handling, * generally to pass info about user-defined local kptr types to later diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 89e72d0a2c46..6fc22a4e38b2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6995,6 +6995,35 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf return 0; } +static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, + argno_t argno, struct bpf_call_arg_meta *meta) +{ + int regno = reg_from_argno(argno); + int err; + + if (meta->arg_constant.found) { + verifier_bug(env, "only one constant argument permitted"); + return -EFAULT; + } + + if (!tnum_is_const(reg->var_off)) { + verbose(env, "%s must be a known constant\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + if (regno >= 0) + err = mark_chain_precision(env, regno); + else + err = mark_stack_arg_precision(env, arg_idx_from_argno(argno)); + if (err < 0) + return err; + + meta->arg_constant.found = true; + meta->arg_constant.value = reg->var_off.value; + + return 0; +} + enum { PROCESS_SPIN_LOCK = (1 << 0), PROCESS_RES_LOCK = (1 << 1), @@ -12054,24 +12083,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me return -EINVAL; } - if (is_kfunc_arg_constant(meta->btf, &args[i])) { - if (meta->arg_constant.found) { - verifier_bug(env, "only one constant argument permitted"); - return -EFAULT; - } - if (!tnum_is_const(reg->var_off)) { - verbose(env, "%s must be a known constant\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - if (regno >= 0) - ret = mark_chain_precision(env, regno); - else - ret = mark_stack_arg_precision(env, i); + if (is_kfunc_arg_constant(meta->btf, &args[i]) || + is_kfunc_arg_const_mem_size(meta->btf, &args[i], reg)) { + ret = process_const_arg(env, reg, argno, meta); if (ret < 0) return ret; - meta->arg_constant.found = true; - meta->arg_constant.value = reg->var_off.value; } else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) { meta->r0_rdonly = true; is_ret_buf_sz = true; @@ -12393,7 +12409,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me struct bpf_reg_state *buff_reg = reg; const struct btf_param *buff_arg = &args[i]; struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1); - const struct btf_param *size_arg = &args[i + 1]; argno_t next_argno = argno_from_arg(i + 2); if (!bpf_register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) { @@ -12406,23 +12421,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me return ret; } } - - if (is_kfunc_arg_const_mem_size(meta->btf, size_arg, size_reg)) { - if (meta->arg_constant.found) { - verifier_bug(env, "only one constant argument permitted"); - return -EFAULT; - } - if (!tnum_is_const(size_reg->var_off)) { - verbose(env, "%s must be a known constant\n", - reg_arg_name(env, next_argno)); - return -EINVAL; - } - meta->arg_constant.found = true; - meta->arg_constant.value = size_reg->var_off.value; - } - - /* Skip next '__sz' or '__szk' argument */ - i++; break; } case KF_ARG_PTR_TO_CALLBACK: From f88caee62e450bba85ff60a18846f44fea43d3d3 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:26 -0700 Subject: [PATCH 184/373] selftests/bpf: Test __szk precision with a NULL nullable buffer When a nullable buffer is passed as NULL, check_mem_size_reg() is skipped, so the __szk memory size must be marked precise through the scalar argument path instead. Exercise this with bpf_dynptr_slice() and a NULL buffer. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-12-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/verifier.c | 2 ++ .../bpf/progs/verifier_mem_size_reg.c | 28 +++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 41cd071d016a..b79bafca68f7 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -68,6 +68,7 @@ #include "verifier_masking.skel.h" #include "verifier_may_goto_1.skel.h" #include "verifier_may_goto_2.skel.h" +#include "verifier_mem_size_reg.skel.h" #include "verifier_meta_access.skel.h" #include "verifier_movsx.skel.h" #include "verifier_mtu.skel.h" @@ -223,6 +224,7 @@ void test_verifier_map_ret_val(void) { RUN(verifier_map_ret_val); } void test_verifier_masking(void) { RUN(verifier_masking); } void test_verifier_may_goto_1(void) { RUN(verifier_may_goto_1); } void test_verifier_may_goto_2(void) { RUN(verifier_may_goto_2); } +void test_verifier_mem_size_reg(void) { RUN(verifier_mem_size_reg); } void test_verifier_meta_access(void) { RUN(verifier_meta_access); } void test_verifier_movsx(void) { RUN(verifier_movsx); } void test_verifier_mul(void) { RUN(verifier_mul); } diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c new file mode 100644 index 000000000000..7e24706a764e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" +#include "bpf_kfuncs.h" + +char _license[] SEC("license") = "GPL"; + +/* + * The __szk size of a kfunc memory/size pair must be marked precise even when + * the nullable buffer is passed as NULL. + */ +SEC("?tc") +__success __log_level(2) +__msg("mark_precise: frame0: regs=r4 stack= before") +int dynptr_slice_null_buf_size_precise(struct __sk_buff *skb) +{ + struct bpf_dynptr dptr; + char *p; + + bpf_dynptr_from_skb(skb, 0, &dptr); + + p = bpf_dynptr_slice(&dptr, 0, NULL, 8); + if (p) + return p[0]; + return 0; +} From ea5ab2389801ca82018024b7cfe156b050f69f1c Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:27 -0700 Subject: [PATCH 185/373] bpf: Classify kfunc mem_size args from BTF without register state check_kfunc_args() already makes sure a scalar value is passed to a scalar kfunc argument. Drop the check in is_kfunc_arg_mem_size() and is_kfunc_arg_const_mem_size() to further decouple get_kfunc_ptr_arg_type() from register state (a prerequisite for generating a helper-like prototype from kfunc's BTF). Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-13-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6fc22a4e38b2..d65da54f9c0f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10785,26 +10785,24 @@ static bool is_kfunc_rcu_protected(struct bpf_call_arg_meta *meta) } static bool is_kfunc_arg_mem_size(const struct btf *btf, - const struct btf_param *arg, - const struct bpf_reg_state *reg) + const struct btf_param *arg) { const struct btf_type *t; t = btf_type_skip_modifiers(btf, arg->type, NULL); - if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE) + if (!btf_type_is_scalar(t)) return false; return btf_param_match_suffix(btf, arg, "__sz"); } static bool is_kfunc_arg_const_mem_size(const struct btf *btf, - const struct btf_param *arg, - const struct bpf_reg_state *reg) + const struct btf_param *arg) { const struct btf_type *t; t = btf_type_skip_modifiers(btf, arg->type, NULL); - if (!btf_type_is_scalar(t) || reg->type != SCALAR_VALUE) + if (!btf_type_is_scalar(t)) return false; return btf_param_match_suffix(btf, arg, "__szk"); @@ -11338,7 +11336,7 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) } static enum kfunc_ptr_arg_type -get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller, +get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_reg_state *regs, struct bpf_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, @@ -11352,8 +11350,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *call return KF_ARG_PTR_TO_CTX; if (arg + 1 < nargs && - (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) || - is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)))) + (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || + is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) arg_mem_size = true; /* In this function, we verify the kfunc's BTF as per the argument type, @@ -12084,7 +12082,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me } if (is_kfunc_arg_constant(meta->btf, &args[i]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[i], reg)) { + is_kfunc_arg_const_mem_size(meta->btf, &args[i])) { ret = process_const_arg(env, reg, argno, meta); if (ret < 0) return ret; @@ -12129,7 +12127,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); ref_tname = btf_name_by_offset(btf, ref_t->name_off); - kf_arg_type = get_kfunc_ptr_arg_type(env, caller, regs, meta, t, ref_t, ref_tname, + kf_arg_type = get_kfunc_ptr_arg_type(env, regs, meta, t, ref_t, ref_tname, args, i, nargs, argno, reg); if (kf_arg_type < 0) return kf_arg_type; From 76fb08750481049796357f7636777bdf3c2be0a8 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:28 -0700 Subject: [PATCH 186/373] bpf: Handle NULL kfunc pointer args without a KF_ARG_PTR_TO_NULL type get_kfunc_ptr_arg_type() returned KF_ARG_PTR_TO_NULL when a nullable pointer argument was passed a NULL register. This folded a register-state decision (bpf_register_is_null()) into what is otherwise BTF-based argument classification, and it short-circuited before the BTF_ID/MEM resolution. Drop KF_ARG_PTR_TO_NULL and handle the NULL case in check_kfunc_args() instead: a nullable argument that is actually NULL is skipped. Note that it is okay to skip even when it is a mem+size pair because the size argument check has been moved to the scalar section. The skip is done before get_kfunc_ptr_arg_type() so that a NULL passed to a nullable non-scalar-struct argument is not newly rejected by the BTF_ID/MEM resolution. No functional change. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-14-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d65da54f9c0f..bf03ba2fc04f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11072,7 +11072,6 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_CALLBACK, KF_ARG_PTR_TO_RB_ROOT, KF_ARG_PTR_TO_RB_NODE, - KF_ARG_PTR_TO_NULL, KF_ARG_PTR_TO_CONST_STR, KF_ARG_CONST_MAP_PTR, KF_ARG_PTR_TO_TIMER, @@ -11349,11 +11348,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) return KF_ARG_PTR_TO_CTX; - if (arg + 1 < nargs && - (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) - arg_mem_size = true; - /* In this function, we verify the kfunc's BTF as per the argument type, * leaving the rest of the verification with respect to the register * type to our caller. When a set of conditions hold in the BTF type of @@ -11362,10 +11356,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg)) return KF_ARG_PTR_TO_CTX; - if (is_kfunc_arg_nullable(meta->btf, &args[arg]) && bpf_register_is_null(reg) && - !arg_mem_size) - return KF_ARG_PTR_TO_NULL; - if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg])) return KF_ARG_PTR_TO_ALLOC_BTF_ID; @@ -11427,6 +11417,11 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) return KF_ARG_PTR_TO_CALLBACK; + if (arg + 1 < nargs && + (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || + is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) + arg_mem_size = true; + /* This is the catch all argument type of register types supported by * check_helper_mem_access. However, we only allow when argument type is * pointer to scalar, or struct composed (recursively) of scalars. When @@ -12127,6 +12122,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); ref_tname = btf_name_by_offset(btf, ref_t->name_off); + if (is_kfunc_arg_nullable(meta->btf, &args[i]) && bpf_register_is_null(reg)) + continue; + kf_arg_type = get_kfunc_ptr_arg_type(env, regs, meta, t, ref_t, ref_tname, args, i, nargs, argno, reg); if (kf_arg_type < 0) @@ -12139,8 +12137,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me } switch (kf_arg_type) { - case KF_ARG_PTR_TO_NULL: - continue; case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: if (!is_trusted_reg(env, reg)) { @@ -12405,19 +12401,16 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me case KF_ARG_PTR_TO_MEM_SIZE: { struct bpf_reg_state *buff_reg = reg; - const struct btf_param *buff_arg = &args[i]; struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1); argno_t next_argno = argno_from_arg(i + 2); - if (!bpf_register_is_null(buff_reg) || !is_kfunc_arg_nullable(meta->btf, buff_arg)) { - ret = check_mem_size_reg(env, buff_reg, size_reg, argno, next_argno, - BPF_READ | BPF_WRITE, true, meta); - if (ret < 0) { - verbose(env, "%s and ", reg_arg_name(env, argno)); - verbose(env, "%s memory, len pair leads to invalid memory access\n", - reg_arg_name(env, next_argno)); - return ret; - } + ret = check_mem_size_reg(env, buff_reg, size_reg, argno, next_argno, + BPF_READ | BPF_WRITE, true, meta); + if (ret < 0) { + verbose(env, "%s and ", reg_arg_name(env, argno)); + verbose(env, "%s memory, len pair leads to invalid memory access\n", + reg_arg_name(env, next_argno)); + return ret; } break; } From 90990ee10be8952d55063990088b9ad4af344e34 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:29 -0700 Subject: [PATCH 187/373] bpf: Distinguish fixed- and variable-size kfunc mem args with MEM_FIXED_SIZE A kfunc memory-pointer argument comes in two flavors: a fixed-size buffer whose access size is derived from the pointed-to BTF type, and a variable-size buffer paired with a following __sz/__szk size argument. Both were represented by separate kfunc_ptr_arg_type values (KF_ARG_PTR_TO_MEM vs KF_ARG_PTR_TO_MEM_SIZE) with the pointer classified as the latter when a size argument followed. Mirror how helpers describe the same distinction: classify both as KF_ARG_PTR_TO_MEM and OR in MEM_FIXED_SIZE for the fixed-size case, just as helpers use ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The switches now key on base_type(kf_arg_type) so the flag rides along, and the KF_ARG_PTR_TO_MEM handler either resolves the size from BTF (MEM_FIXED_SIZE) or falls through to the mem/size-pair check, which validates the buffer against the following size register and skips it. No functional change. Currently, KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE are only reachable from ARG_PTR_TO_MEM fallthrough. A patch later will merge scalar checking into the same switch and remove the fallthrough. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-15-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bf03ba2fc04f..1ccf3b764c51 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11059,6 +11059,8 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, } enum kfunc_ptr_arg_type { + KF_ARG_CONST_MEM_SIZE, + KF_ARG_MEM_SIZE, KF_ARG_PTR_TO_CTX, KF_ARG_PTR_TO_ALLOC_BTF_ID, /* Allocated object */ KF_ARG_PTR_TO_REFCOUNTED_KPTR, /* Refcounted local kptr */ @@ -11068,7 +11070,6 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_LIST_NODE, KF_ARG_PTR_TO_BTF_ID, /* Also covers reg2btf_ids conversions */ KF_ARG_PTR_TO_MEM, - KF_ARG_PTR_TO_MEM_SIZE, /* Size derived from next argument, skip it */ KF_ARG_PTR_TO_CALLBACK, KF_ARG_PTR_TO_RB_ROOT, KF_ARG_PTR_TO_RB_NODE, @@ -11334,7 +11335,7 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) return meta->func_id == special_kfunc_list[KF_bpf_xdp_pull_data]; } -static enum kfunc_ptr_arg_type +static int get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_reg_state *regs, struct bpf_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, @@ -11434,7 +11435,7 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); return -EINVAL; } - return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM; + return arg_mem_size ? KF_ARG_PTR_TO_MEM : KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; } static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, @@ -12136,7 +12137,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname = btf_name_by_offset(btf, ref_t->name_off); } - switch (kf_arg_type) { + switch (base_type(kf_arg_type)) { case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: if (!is_trusted_reg(env, reg)) { @@ -12159,7 +12160,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me case KF_ARG_PTR_TO_RB_ROOT: case KF_ARG_PTR_TO_RB_NODE: case KF_ARG_PTR_TO_MEM: - case KF_ARG_PTR_TO_MEM_SIZE: case KF_ARG_PTR_TO_CALLBACK: case KF_ARG_PTR_TO_CONST_STR: case KF_ARG_PTR_TO_WORKQUEUE: @@ -12190,7 +12190,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (ret < 0) return ret; - switch (kf_arg_type) { + switch (base_type(kf_arg_type)) { case KF_ARG_PTR_TO_CTX: if (reg->type != PTR_TO_CTX) { verbose(env, "%s expected pointer to ctx, but got %s\n", @@ -12387,18 +12387,22 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me return ret; break; case KF_ARG_PTR_TO_MEM: - resolve_ret = btf_resolve_size(btf, ref_t, &type_size); - if (IS_ERR(resolve_ret)) { - verbose(env, "%s reference type('%s %s') size cannot be determined: %ld\n", - reg_arg_name(env, argno), btf_type_str(ref_t), - ref_tname, PTR_ERR(resolve_ret)); - return -EINVAL; + if (kf_arg_type & MEM_FIXED_SIZE) { + resolve_ret = btf_resolve_size(btf, ref_t, &type_size); + if (IS_ERR(resolve_ret)) { + verbose(env, "%s reference type('%s %s') size cannot be determined: %ld\n", + reg_arg_name(env, argno), btf_type_str(ref_t), + ref_tname, PTR_ERR(resolve_ret)); + return -EINVAL; + } + ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, meta); + if (ret < 0) + return ret; + break; } - ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, meta); - if (ret < 0) - return ret; - break; - case KF_ARG_PTR_TO_MEM_SIZE: + fallthrough; + case KF_ARG_CONST_MEM_SIZE: + case KF_ARG_MEM_SIZE: { struct bpf_reg_state *buff_reg = reg; struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1); From c9e995ba0d117119e7955f0bd3eacfb173fe636f Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:30 -0700 Subject: [PATCH 188/373] bpf: Classify kfunc pointer arguments from BTF, resolve type against the register get_kfunc_ptr_arg_type() decided part of a kfunc pointer argument's type from the caller's register: a PTR_TO_BTF_ID (or reg2btf_ids) register made the argument KF_ARG_PTR_TO_BTF_ID, otherwise it fell through to a memory buffer. Folding register state into argument classification prevents describing a kfunc's arguments from its BTF alone, which is a prerequisite for generating a helper-like prototype and eventually sharing the argument checking (check_func_arg()) between helpers and kfuncs. Classify pointer arguments from BTF only, and resolve them against the register in check_kfunc_args(): - A pointer to a struct that is not paired with a __sz/__szk size argument is classified KF_ARG_PTR_TO_BTF_ID and then checked against the register. A register carrying a BTF ID (PTR_TO_BTF_ID or a reg2btf_ids type) must be referenced or trusted and is matched against the expected type. The only relaxation is when the struct is composed of scalars, the register may be verified as a fixed-size memory buffer sized from the BTF type; anything else is rejected. - A pointer paired with a size argument is always a memory buffer and is never classified as BTF_ID, so the __sz/__szk case no longer detours through BTF_ID. The new design now accepts one previously rejected case: passing PTR_TO_BTF_ID to a pointer to scalar w/o a following __sz/__szk. The argument will be classified as KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE. The PTR_TO_BTF_ID register will go through check_mem_reg() -> check_helper_mem_access() -> check_ptr_to_btf_access(). For a pointer to scalar arg, a kernel btf id will be rejected unless explicitly granted by btf_struct_access(); a program allocated btf id will be allowed. The referenced-or-trusted check thus moves into the KF_ARG_PTR_TO_BTF_ID resolution, alongside the type match. get_kfunc_ptr_arg_type() no longer needs the register, so drop its regs and reg parameters; it is now a pure function of the kfunc's BTF. When a register cannot satisfy a BTF_ID argument, report the register type passed and, when the expected struct has a reg2btf_ids mapping, the register type that would be accepted, instead of a confusing "socket". Update the affected selftest messages accordingly. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-16-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 130 ++++++++++-------- .../selftests/bpf/progs/cgrp_kfunc_failure.c | 2 +- .../selftests/bpf/progs/task_kfunc_failure.c | 2 +- .../selftests/bpf/progs/verifier_vfs_reject.c | 6 +- tools/testing/selftests/bpf/verifier/calls.c | 6 +- 5 files changed, 84 insertions(+), 62 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1ccf3b764c51..9045369ba569 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4926,6 +4926,18 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { [CONST_PTR_TO_MAP] = btf_bpf_map_id, }; +static enum bpf_reg_type lookup_reg2btf_ids(u32 ref_id) +{ + enum bpf_reg_type type; + + for (type = 0; type < __BPF_REG_TYPE_MAX; type++) { + if (reg2btf_ids[type] && *reg2btf_ids[type] == ref_id) + return type; + } + + return NOT_INIT; +} + static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) { /* A referenced register is always trusted. */ @@ -11336,14 +11348,11 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) } static int -get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, - struct bpf_reg_state *regs, struct bpf_call_arg_meta *meta, +get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, const struct btf_type *t, const struct btf_type *ref_t, const char *ref_tname, const struct btf_param *args, - int arg, int nargs, argno_t argno, struct bpf_reg_state *reg) + int arg, int nargs, argno_t argno) { - bool arg_mem_size = false; - if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || meta->func_id == special_kfunc_list[KF_bpf_session_is_return] || meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) @@ -11405,37 +11414,37 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg])) return KF_ARG_PTR_TO_RES_SPIN_LOCK; - if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { - if (!btf_type_is_struct(ref_t)) { - verbose(env, "kernel function %s %s pointer type %s %s is not supported\n", - meta->func_name, reg_arg_name(env, argno), - btf_type_str(ref_t), ref_tname); - return -EINVAL; - } - return KF_ARG_PTR_TO_BTF_ID; - } - if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) return KF_ARG_PTR_TO_CALLBACK; if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) - arg_mem_size = true; + is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { + if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && + !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", + reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); + return -EINVAL; + } + return KF_ARG_PTR_TO_MEM; + } - /* This is the catch all argument type of register types supported by - * check_helper_mem_access. However, we only allow when argument type is - * pointer to scalar, or struct composed (recursively) of scalars. When - * arg_mem_size is true, the pointer can be void *. + /* A pointer to a struct without a size argument is classified as KF_ARG_PTR_TO_BTF_ID */ + if (btf_type_is_struct(ref_t)) + return KF_ARG_PTR_TO_BTF_ID; + + /* + * Otherwise this is a fixed-size memory buffer supported by + * check_helper_mem_access(): a pointer to a scalar or a struct of + * scalars. The access size is derived from the pointed-to BTF type. */ - if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) && - (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) { - verbose(env, "%s pointer type %s %s must point to %sscalar, or struct with scalar\n", - reg_arg_name(env, argno), - btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); + if (!btf_type_is_scalar(ref_t) && + !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", + reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; } - return arg_mem_size ? KF_ARG_PTR_TO_MEM : KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; + return KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; } static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, @@ -12126,8 +12135,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (is_kfunc_arg_nullable(meta->btf, &args[i]) && bpf_register_is_null(reg)) continue; - kf_arg_type = get_kfunc_ptr_arg_type(env, regs, meta, t, ref_t, ref_tname, - args, i, nargs, argno, reg); + kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, + args, i, nargs, argno); if (kf_arg_type < 0) return kf_arg_type; @@ -12140,19 +12149,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me switch (base_type(kf_arg_type)) { case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: - if (!is_trusted_reg(env, reg)) { - if (!is_kfunc_rcu(meta)) { - verbose(env, "%s must be referenced or trusted\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - if (!is_rcu_reg(reg)) { - verbose(env, "%s must be a rcu pointer\n", - reg_arg_name(env, argno)); - return -EINVAL; - } - } - fallthrough; case KF_ARG_CONST_MAP_PTR: case KF_ARG_PTR_TO_ITER: case KF_ARG_PTR_TO_LIST_HEAD: @@ -12372,20 +12368,46 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me break; case KF_ARG_PTR_TO_BTF_ID: /* Only base_type is checked, further checks are done here */ - if ((base_type(reg->type) != PTR_TO_BTF_ID || - (bpf_type_has_unsafe_modifiers(reg->type) && !is_rcu_reg(reg))) && - !reg2btf_ids[base_type(reg->type)]) { - verbose(env, "%s is %s ", reg_arg_name(env, argno), - reg_type_str(env, reg->type)); - verbose(env, "expected %s or socket\n", - reg_type_str(env, base_type(reg->type) | - (type_flag(reg->type) & BPF_REG_TRUSTED_MODIFIERS))); + if (base_type(reg->type) == PTR_TO_BTF_ID || + reg2btf_ids[base_type(reg->type)]) { + if (!is_trusted_reg(env, reg) || + bpf_type_has_unsafe_modifiers(reg->type)) { + if (!is_kfunc_rcu(meta)) { + verbose(env, "%s must be referenced or trusted\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + if (!is_rcu_reg(reg)) { + verbose(env, "%s must be a rcu pointer\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + } + + ret = process_kf_arg_ptr_to_btf_id(env, reg, ref_t, ref_tname, ref_id, meta, i, argno); + if (ret < 0) + return ret; + break; + } + + if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); + + verbose(env, "%s is %s expected %s %s", + reg_arg_name(env, argno), reg_type_str(env, reg->type), + btf_type_str(ref_t), ref_tname); + if (reg2btf_type != NOT_INIT) + verbose(env, " or %s", reg_type_str(env, reg2btf_type)); + verbose(env, "\n"); return -EINVAL; } - ret = process_kf_arg_ptr_to_btf_id(env, reg, ref_t, ref_tname, ref_id, meta, i, argno); - if (ret < 0) - return ret; - break; + + /* + * If the register does not contain btf id but the argument type is a pointer to + * scalar-only struct, allow verifying it as a fixed size memory. + */ + kf_arg_type = KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; + fallthrough; case KF_ARG_PTR_TO_MEM: if (kf_arg_type & MEM_FIXED_SIZE) { resolve_ret = btf_resolve_size(btf, ref_t, &type_size); diff --git a/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c b/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c index d0d65d6d450c..efe7bcae70f8 100644 --- a/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c +++ b/tools/testing/selftests/bpf/progs/cgrp_kfunc_failure.c @@ -64,7 +64,7 @@ int BPF_PROG(cgrp_kfunc_acquire_no_null_check, struct cgroup *cgrp, const char * } SEC("tp_btf/cgroup_mkdir") -__failure __msg("R1 pointer type STRUCT cgroup must point") +__failure __msg("R1 is fp expected STRUCT cgroup") int BPF_PROG(cgrp_kfunc_acquire_fp, struct cgroup *cgrp, const char *path) { struct cgroup *acquired, *stack_cgrp = (struct cgroup *)&path; diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c index 8942b5478129..5c99b1e6532b 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c +++ b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c @@ -50,7 +50,7 @@ int BPF_PROG(task_kfunc_acquire_untrusted, struct task_struct *task, u64 clone_f } SEC("tp_btf/task_newtask") -__failure __msg("R1 pointer type STRUCT task_struct must point") +__failure __msg("R1 is fp expected STRUCT task_struct") int BPF_PROG(task_kfunc_acquire_fp, struct task_struct *task, u64 clone_flags) { struct task_struct *acquired, *stack_task = (struct task_struct *)&clone_flags; diff --git a/tools/testing/selftests/bpf/progs/verifier_vfs_reject.c b/tools/testing/selftests/bpf/progs/verifier_vfs_reject.c index 2870738d93f7..8f0c45421f89 100644 --- a/tools/testing/selftests/bpf/progs/verifier_vfs_reject.c +++ b/tools/testing/selftests/bpf/progs/verifier_vfs_reject.c @@ -28,7 +28,7 @@ int BPF_PROG(get_task_exe_file_kfunc_null) } SEC("lsm.s/inode_getxattr") -__failure __msg("R1 pointer type STRUCT task_struct must point to scalar, or struct with scalar") +__failure __msg("R1 is fp expected STRUCT task_struct") int BPF_PROG(get_task_exe_file_kfunc_fp) { u64 x; @@ -98,7 +98,7 @@ int BPF_PROG(path_d_path_kfunc_null) } SEC("lsm.s/task_alloc") -__failure __msg("R1 must be referenced or trusted") +__failure __msg("dereference of modified untrusted_ptr_") int BPF_PROG(path_d_path_kfunc_untrusted_from_argument, struct task_struct *task) { struct path *root; @@ -112,7 +112,7 @@ int BPF_PROG(path_d_path_kfunc_untrusted_from_argument, struct task_struct *task } SEC("lsm.s/file_open") -__failure __msg("R1 must be referenced or trusted") +__failure __msg("dereference of modified untrusted_ptr_") int BPF_PROG(path_d_path_kfunc_untrusted_from_current) { struct path *pwd; diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c index 302d712e0d7e..8cd626e04551 100644 --- a/tools/testing/selftests/bpf/verifier/calls.c +++ b/tools/testing/selftests/bpf/verifier/calls.c @@ -31,7 +31,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "R1 pointer type STRUCT prog_test_fail1 must point to scalar", + .errstr = "R1 is fp expected STRUCT prog_test_fail1", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail1", 2 }, }, @@ -46,7 +46,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "max struct nesting depth exceeded\nR1 pointer type STRUCT prog_test_fail2", + .errstr = "max struct nesting depth exceeded\nR1 is fp expected STRUCT prog_test_fail2", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail2", 2 }, }, @@ -61,7 +61,7 @@ }, .prog_type = BPF_PROG_TYPE_SCHED_CLS, .result = REJECT, - .errstr = "R1 pointer type STRUCT prog_test_fail3 must point to scalar", + .errstr = "R1 is fp expected STRUCT prog_test_fail3", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_fail3", 2 }, }, From ba5b99470c4019ce936226e92d4255dd8ff4dd12 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:31 -0700 Subject: [PATCH 189/373] bpf: Tag nullable kfunc pointer args with PTR_MAYBE_NULL Now that get_kfunc_ptr_arg_type() classifies a kfunc pointer argument from its BTF alone, express a nullable argument by OR-ing PTR_MAYBE_NULL into the classified type, and resolve a NULL register after classification instead of before it. Previously check_kfunc_args() short-circuited a nullable argument passed a NULL register with a continue placed before get_kfunc_ptr_arg_type(), so the NULL never reached classification. That kept a register-state decision (bpf_register_is_null()) ahead of the BTF-based classification. This mirrors how helper arguments carry PTR_MAYBE_NULL in their bpf_arg_type and is a step toward describing kfuncs with a bpf_func_proto: the nullability now travels with the per-argument classification, so it is captured when the prototype is generated at add-call time. Signed-off-by: Amery Hung Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-17-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 147 +++++++++++++++++++----------------------- 1 file changed, 67 insertions(+), 80 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9045369ba569..32459f25f90b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11353,98 +11353,85 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *m const char *ref_tname, const struct btf_param *args, int arg, int nargs, argno_t argno) { - if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || - meta->func_id == special_kfunc_list[KF_bpf_session_is_return] || - meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) - return KF_ARG_PTR_TO_CTX; + int arg_type; /* In this function, we verify the kfunc's BTF as per the argument type, * leaving the rest of the verification with respect to the register * type to our caller. When a set of conditions hold in the BTF type of * arguments, we resolve it to a known kfunc_ptr_arg_type. */ - if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg)) - return KF_ARG_PTR_TO_CTX; - - if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_ALLOC_BTF_ID; - - if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_REFCOUNTED_KPTR; - - if (is_kfunc_arg_dynptr(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_DYNPTR; - - if (is_kfunc_arg_iter(meta, arg, &args[arg])) - return KF_ARG_PTR_TO_ITER; - - if (is_kfunc_arg_list_head(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_LIST_HEAD; - - if (is_kfunc_arg_list_node(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_LIST_NODE; - - if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_RB_ROOT; - - if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_RB_NODE; - - if (is_kfunc_arg_const_str(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_CONST_STR; - - if (is_kfunc_arg_const_map(meta->btf, &args[arg])) - return KF_ARG_CONST_MAP_PTR; - - if (is_kfunc_arg_map(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_BTF_ID; - - if (is_kfunc_arg_wq(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_WORKQUEUE; - - if (is_kfunc_arg_timer(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_TIMER; - - if (is_kfunc_arg_task_work(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_TASK_WORK; - - if (is_kfunc_arg_irq_flag(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_IRQ_FLAG; - - if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg])) - return KF_ARG_PTR_TO_RES_SPIN_LOCK; - - if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) - return KF_ARG_PTR_TO_CALLBACK; - - if (arg + 1 < nargs && - (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { + if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + meta->func_id == special_kfunc_list[KF_bpf_session_is_return] || + meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) + arg_type = KF_ARG_PTR_TO_CTX; + else if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg)) + arg_type = KF_ARG_PTR_TO_CTX; + else if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_ALLOC_BTF_ID; + else if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_REFCOUNTED_KPTR; + else if (is_kfunc_arg_dynptr(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_DYNPTR; + else if (is_kfunc_arg_iter(meta, arg, &args[arg])) + arg_type = KF_ARG_PTR_TO_ITER; + else if (is_kfunc_arg_list_head(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_LIST_HEAD; + else if (is_kfunc_arg_list_node(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_LIST_NODE; + else if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_RB_ROOT; + else if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_RB_NODE; + else if (is_kfunc_arg_const_str(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_CONST_STR; + else if (is_kfunc_arg_const_map(meta->btf, &args[arg])) + arg_type = KF_ARG_CONST_MAP_PTR; + else if (is_kfunc_arg_map(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_BTF_ID; + else if (is_kfunc_arg_wq(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_WORKQUEUE; + else if (is_kfunc_arg_timer(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_TIMER; + else if (is_kfunc_arg_task_work(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_TASK_WORK; + else if (is_kfunc_arg_irq_flag(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_IRQ_FLAG; + else if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_RES_SPIN_LOCK; + else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) + arg_type = KF_ARG_PTR_TO_CALLBACK; + else if (arg + 1 < nargs && + (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || + is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; } - return KF_ARG_PTR_TO_MEM; + arg_type = KF_ARG_PTR_TO_MEM; + } else if (btf_type_is_struct(ref_t)) + /* A pointer to a struct without a size argument is classified as KF_ARG_PTR_TO_BTF_ID */ + arg_type = KF_ARG_PTR_TO_BTF_ID; + else { + /* + * Otherwise this is a fixed-size memory buffer supported by + * check_helper_mem_access(): a pointer to a scalar or a struct of + * scalars. The access size is derived from the pointed-to BTF type. + */ + if (!btf_type_is_scalar(ref_t) && + !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", + reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); + return -EINVAL; + } + arg_type = KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; } - /* A pointer to a struct without a size argument is classified as KF_ARG_PTR_TO_BTF_ID */ - if (btf_type_is_struct(ref_t)) - return KF_ARG_PTR_TO_BTF_ID; + if (is_kfunc_arg_nullable(meta->btf, &args[arg])) + arg_type |= PTR_MAYBE_NULL; - /* - * Otherwise this is a fixed-size memory buffer supported by - * check_helper_mem_access(): a pointer to a scalar or a struct of - * scalars. The access size is derived from the pointed-to BTF type. - */ - if (!btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { - verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", - reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); - return -EINVAL; - } - return KF_ARG_PTR_TO_MEM | MEM_FIXED_SIZE; + return arg_type; } static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, @@ -12132,14 +12119,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); ref_tname = btf_name_by_offset(btf, ref_t->name_off); - if (is_kfunc_arg_nullable(meta->btf, &args[i]) && bpf_register_is_null(reg)) - continue; - kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs, argno); if (kf_arg_type < 0) return kf_arg_type; + if (bpf_register_is_null(reg) && type_may_be_null(kf_arg_type)) + continue; + if (is_kfunc_arg_map(btf, &args[i])) { ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; ref_t = btf_type_by_id(btf_vmlinux, ref_id); From 1690dcf27c7368d275cdcf73793aafe4d81007c9 Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:32 -0700 Subject: [PATCH 190/373] bpf: Classify scalar kfunc arguments from BTF Add kfunc scalar argument types, classify them in get_kfunc_arg_type() along side with pointer arguments and move scalar type verification into the main switch in check_kfunc_args(). This keeps BTF-based classification separate from register validation for every argument, paving the way for generating the kfunc argument prototype at add-call time. No functional change intended. KF_ARG_MEM_SIZE and KF_ARG_CONST_MEM_SIZE now are reachable. Therefore, remove the fallthrough from KF_ARG_PTR_TO_MEM case and adjust the register indexing. Signed-off-by: Amery Hung Reviewed-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260801074633.1595644-18-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 141 +++++++++++------- .../testing/selftests/bpf/progs/dynptr_fail.c | 2 +- 2 files changed, 92 insertions(+), 51 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 32459f25f90b..23a6355a38d3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11073,6 +11073,9 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, enum kfunc_ptr_arg_type { KF_ARG_CONST_MEM_SIZE, KF_ARG_MEM_SIZE, + KF_ARG_CONST, + KF_ARG_CONST_ALLOC_SIZE_OR_ZERO, + KF_ARG_ANYTHING, KF_ARG_PTR_TO_CTX, KF_ARG_PTR_TO_ALLOC_BTF_ID, /* Allocated object */ KF_ARG_PTR_TO_REFCOUNTED_KPTR, /* Refcounted local kptr */ @@ -11348,13 +11351,39 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) } static int -get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, - const struct btf_type *t, const struct btf_type *ref_t, - const char *ref_tname, const struct btf_param *args, - int arg, int nargs, argno_t argno) +get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, + const struct btf_param *args, int arg, int nargs) { + const struct btf_type *t, *ref_t = NULL; + argno_t argno = argno_from_arg(arg + 1); + const char *ref_tname = NULL; int arg_type; + t = btf_type_skip_modifiers(meta->btf, args[arg].type, NULL); + + /* Scalar arguments are classified from their BTF suffix/name alone. */ + if (btf_type_is_scalar(t)) { + if (is_kfunc_arg_constant(meta->btf, &args[arg])) + return KF_ARG_CONST; + if (is_kfunc_arg_const_mem_size(meta->btf, &args[arg])) + return KF_ARG_CONST_MEM_SIZE; + if (is_kfunc_arg_mem_size(meta->btf, &args[arg])) + return KF_ARG_MEM_SIZE; + if (is_kfunc_arg_scalar_with_name(meta->btf, &args[arg], "rdonly_buf_size") || + is_kfunc_arg_scalar_with_name(meta->btf, &args[arg], "rdwr_buf_size")) + return KF_ARG_CONST_ALLOC_SIZE_OR_ZERO; + return KF_ARG_ANYTHING; + } + + if (!btf_type_is_ptr(t)) { + verbose(env, "Unrecognized %s type %s\n", + reg_arg_name(env, argno), btf_type_str(t)); + return -EINVAL; + } + + ref_t = btf_type_skip_modifiers(meta->btf, t->type, NULL); + ref_tname = btf_name_by_offset(meta->btf, ref_t->name_off); + /* In this function, we verify the kfunc's BTF as per the argument type, * leaving the rest of the verification with respect to the register * type to our caller. When a set of conditions hold in the BTF type of @@ -12043,7 +12072,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me int regno = reg_from_argno(argno); bool btf_id_fixed_off_ok = true; u32 ref_id, type_size; - bool is_ret_buf_sz = false; int kf_arg_type; if (is_kfunc_arg_prog_aux(btf, &args[i])) { @@ -12067,39 +12095,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t = btf_type_skip_modifiers(btf, args[i].type, NULL); - if (btf_type_is_scalar(t)) { - if (reg->type != SCALAR_VALUE) { - verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); - return -EINVAL; - } - - if (is_kfunc_arg_constant(meta->btf, &args[i]) || - is_kfunc_arg_const_mem_size(meta->btf, &args[i])) { - ret = process_const_arg(env, reg, argno, meta); - if (ret < 0) - return ret; - } else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) { - meta->r0_rdonly = true; - is_ret_buf_sz = true; - } else if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdwr_buf_size")) { - is_ret_buf_sz = true; - } - - if (is_ret_buf_sz) { - ret = process_const_alloc_mem_size(env, reg, argno, &meta->ret_mem); - if (ret < 0) - return ret; - } - continue; - } - - if (!btf_type_is_ptr(t)) { - verbose(env, "Unrecognized %s type %s\n", - reg_arg_name(env, argno), btf_type_str(t)); - return -EINVAL; - } - - if ((bpf_register_is_null(reg) || type_may_be_null(reg->type)) && + if (btf_type_is_ptr(t) && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && !is_kfunc_arg_nullable(meta->btf, &args[i])) { verbose(env, "Possibly NULL pointer passed to trusted %s\n", reg_arg_name(env, argno)); @@ -12116,11 +12112,12 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (reg_is_referenced(env, reg)) update_ref_obj(&meta->ref_obj, reg); - ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); - ref_tname = btf_name_by_offset(btf, ref_t->name_off); + if (btf_type_is_ptr(t)) { + ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); + ref_tname = btf_name_by_offset(btf, ref_t->name_off); + } - kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, - args, i, nargs, argno); + kf_arg_type = get_kfunc_arg_type(env, meta, args, i, nargs); if (kf_arg_type < 0) return kf_arg_type; @@ -12134,6 +12131,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me } switch (base_type(kf_arg_type)) { + case KF_ARG_CONST: + case KF_ARG_CONST_MEM_SIZE: + case KF_ARG_MEM_SIZE: + case KF_ARG_ANYTHING: + case KF_ARG_CONST_ALLOC_SIZE_OR_ZERO: case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: case KF_ARG_CONST_MAP_PTR: @@ -12174,6 +12176,34 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me return ret; switch (base_type(kf_arg_type)) { + case KF_ARG_CONST: + if (reg->type != SCALAR_VALUE) { + verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + ret = process_const_arg(env, reg, argno, meta); + if (ret < 0) + return ret; + break; + case KF_ARG_ANYTHING: + if (reg->type != SCALAR_VALUE) { + verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + return -EINVAL; + } + break; + case KF_ARG_CONST_ALLOC_SIZE_OR_ZERO: + if (reg->type != SCALAR_VALUE) { + verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) + meta->r0_rdonly = true; + ret = process_const_alloc_mem_size(env, reg, argno, &meta->ret_mem); + if (ret < 0) + return ret; + break; case KF_ARG_PTR_TO_CTX: if (reg->type != PTR_TO_CTX) { verbose(env, "%s expected pointer to ctx, but got %s\n", @@ -12407,22 +12437,33 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, meta); if (ret < 0) return ret; - break; } - fallthrough; + break; case KF_ARG_CONST_MEM_SIZE: + ret = process_const_arg(env, reg, argno, meta); + if (ret < 0) + return ret; + fallthrough; case KF_ARG_MEM_SIZE: { - struct bpf_reg_state *buff_reg = reg; - struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1); - argno_t next_argno = argno_from_arg(i + 2); + struct bpf_reg_state *buff_reg = get_func_arg_reg(caller, regs, i - 1); + struct bpf_reg_state *size_reg = reg; + argno_t buff_argno = argno_from_arg(i); - ret = check_mem_size_reg(env, buff_reg, size_reg, argno, next_argno, + if (reg->type != SCALAR_VALUE) { + verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + return -EINVAL; + } + + if (bpf_register_is_null(buff_reg)) + break; + + ret = check_mem_size_reg(env, buff_reg, size_reg, buff_argno, argno, BPF_READ | BPF_WRITE, true, meta); if (ret < 0) { - verbose(env, "%s and ", reg_arg_name(env, argno)); + verbose(env, "%s and ", reg_arg_name(env, buff_argno)); verbose(env, "%s memory, len pair leads to invalid memory access\n", - reg_arg_name(env, next_argno)); + reg_arg_name(env, argno)); return ret; } break; diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c index 94489ac64da8..340bd7db79f0 100644 --- a/tools/testing/selftests/bpf/progs/dynptr_fail.c +++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c @@ -1589,7 +1589,7 @@ int xdp_invalid_ctx(void *ctx) __u32 hdr_size = sizeof(struct ethhdr); /* Can't pass in variable-sized len to bpf_dynptr_slice */ SEC("?tc") -__failure __msg("unbounded memory access") +__failure __msg("must be a known constant") int dynptr_slice_var_len1(struct __sk_buff *skb) { struct bpf_dynptr ptr; From a49b70400b9de06234eb99f87cf60217ed98cc0c Mon Sep 17 00:00:00 2001 From: Amery Hung Date: Sat, 1 Aug 2026 00:46:33 -0700 Subject: [PATCH 191/373] bpf: Generate kfunc argument prototype at add-call time Kfunc argument checking re-derives each argument's kfunc_ptr_arg_type from BTF on every verification of a call in check_kfunc_args(). Now that get_kfunc_arg_type() is a function of the kfunc's BTF alone, it no longer inspects register state. The classification can be computed once when the call is added and cached. This is a step toward describing kfuncs with a bpf_func_proto and sharing the helper argument-checking path. Generate the classification at bpf_add_kfunc_call() time: - Extend struct bpf_func_proto to be able to describe a kfunc: widen arg_type[] and the arg_btf_id[]/arg_size[] union from 5 to MAX_BPF_FUNC_ARGS, since a kfunc may take up to 12 arguments (5 in registers, 7 on the stack). - Embed a bpf_func_proto in struct bpf_kfunc_desc, populated by gen_kfunc_arg_proto() which runs get_kfunc_arg_type() for each argument and stores the result in proto.arg_type[]. Grow the descriptor table's descs[] as a flexible array to not waste memory. - check_kfunc_args() reads the cached classification from meta->fn The KF_ARG_PTR_TO_CTX classification depends on the resolved program type, and for BPF_PROG_TYPE_EXT that is the target program's type, which resolve_prog_type() reads from prog->aux->saved_dst_prog_type. That field is normally recorded later during verification in check_attach_btf_id(), after bpf_add_kfunc_call() has run. Record saved_dst_prog_type and saved_dst_attach_type from dst_prog at program load time in bpf_prog_load() so the resolved type is available at add-call time without reordering check_attach_btf_id(). This keeps e.g. an freplace of an XDP program calling bpf_xdp_metadata_rx_hash() classifying its struct xdp_md * argument as context. The classification result is unchanged; it is only computed earlier and cached. Signed-off-by: Amery Hung Link: https://lore.kernel.org/bpf/20260801074633.1595644-19-ameryhung@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 36 ++++++------- include/linux/bpf_verifier.h | 9 ++-- kernel/bpf/syscall.c | 4 ++ kernel/bpf/verifier.c | 98 +++++++++++++++++++++++++++++------- 4 files changed, 109 insertions(+), 38 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index be53655d1362..356884587ae1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -960,6 +960,21 @@ enum bpf_return_type { }; static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT); +/* The longest tracepoint has 12 args. + * See include/trace/bpf_probe.h + * + * Also reuse this macro for maximum number of arguments a BPF function + * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via + * stack arg slots. The JIT may map some stack arg slots to registers based + * on the native calling convention (e.g., arg 6 to R9 on x86-64). + */ +#define MAX_BPF_FUNC_ARGS 12 + +/* The maximum number of arguments passed through registers + * a single function may have. + */ +#define MAX_BPF_FUNC_REG_ARGS 5 + /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs * to in-kernel helper functions and for adjusting imm32 field in BPF_CALL * instructions after verifying @@ -984,7 +999,7 @@ struct bpf_func_proto { enum bpf_arg_type arg4_type; enum bpf_arg_type arg5_type; }; - enum bpf_arg_type arg_type[5]; + enum bpf_arg_type arg_type[MAX_BPF_FUNC_ARGS]; }; union { struct { @@ -994,7 +1009,7 @@ struct bpf_func_proto { u32 *arg4_btf_id; u32 *arg5_btf_id; }; - u32 *arg_btf_id[5]; + u32 *arg_btf_id[MAX_BPF_FUNC_ARGS]; struct { size_t arg1_size; size_t arg2_size; @@ -1002,7 +1017,7 @@ struct bpf_func_proto { size_t arg4_size; size_t arg5_size; }; - size_t arg_size[5]; + size_t arg_size[MAX_BPF_FUNC_ARGS]; }; int *ret_btf_id; /* return value btf_id */ bool (*allowed)(const struct bpf_prog *prog); @@ -1192,21 +1207,6 @@ struct bpf_prog_offload { u32 jited_len; }; -/* The longest tracepoint has 12 args. - * See include/trace/bpf_probe.h - * - * Also reuse this macro for maximum number of arguments a BPF function - * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via - * stack arg slots. The JIT may map some stack arg slots to registers based - * on the native calling convention (e.g., arg 6 to R9 on x86-64). - */ -#define MAX_BPF_FUNC_ARGS 12 - -/* The maximum number of arguments passed through registers - * a single function may have. - */ -#define MAX_BPF_FUNC_REG_ARGS 5 - /* The argument is a structure or a union. */ #define BTF_FMODEL_STRUCT_ARG BIT(0) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index b54c1a5c9b11..a2a40caca0a0 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1302,7 +1302,6 @@ static inline u32 type_flag(u32 type) return type & ~BPF_BASE_TYPE_MASK; } -/* only use after check_attach_btf_id() */ static inline enum bpf_prog_type resolve_prog_type(const struct bpf_prog *prog) { return (prog->type == BPF_PROG_TYPE_EXT && prog->aux->saved_dst_prog_type) ? @@ -1489,6 +1488,7 @@ struct bpf_call_arg_meta { /* Common */ struct btf *btf; u32 func_id; + const struct bpf_func_proto *fn; u8 release_regno; u32 ret_btf_id; u32 subprogno; @@ -1617,6 +1617,7 @@ enum bpf_reg_arg_type { struct bpf_kfunc_desc { struct btf_func_model func_model; + struct bpf_func_proto proto; u32 func_id; s32 imm; u16 offset; @@ -1624,13 +1625,15 @@ struct bpf_kfunc_desc { }; struct bpf_kfunc_desc_tab { + u32 nr_descs; /* Sorted by func_id (BTF ID) and offset (fd_array offset) during * verification. JITs do lookups by bpf_insn, where func_id may not be * available, therefore at the end of verification do_misc_fixups() * sorts this by imm and offset. + * + * Grown one entry at a time by bpf_add_kfunc_call(). */ - struct bpf_kfunc_desc descs[MAX_KFUNC_DESCS]; - u32 nr_descs; + struct bpf_kfunc_desc descs[]; }; /* Functions exported from verifier.c, used by fixups.c */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index d6be7f49433c..8d111da88655 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3043,6 +3043,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at prog->aux->attach_btf = attach_btf; prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id; prog->aux->dst_prog = dst_prog; + if (dst_prog) { + prog->aux->saved_dst_prog_type = dst_prog->type; + prog->aux->saved_dst_attach_type = dst_prog->expected_attach_type; + } prog->aux->dev_bound = !!attr->prog_ifindex; prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 23a6355a38d3..b274004fccfd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2721,8 +2721,12 @@ static int fetch_kfunc_meta(struct bpf_verifier_env *env, return 0; } +static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, + struct bpf_func_proto *proto); + int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) { + struct bpf_call_arg_meta meta; struct bpf_kfunc_btf_tab *btf_tab; struct btf_func_model func_model; struct bpf_kfunc_desc_tab *tab; @@ -2808,11 +2812,30 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) if (err) return err; - desc = &tab->descs[tab->nr_descs++]; + memset(&meta, 0, sizeof(meta)); + meta.btf = kfunc.btf; + meta.func_id = kfunc.id; + meta.func_proto = kfunc.proto; + meta.func_name = kfunc.name; + meta.kfunc_flags = kfunc.flags ? *kfunc.flags : 0; + + tab = krealloc(tab, struct_size(tab, descs, tab->nr_descs + 1), GFP_KERNEL_ACCOUNT); + if (!tab) + return -ENOMEM; + prog_aux->kfunc_tab = tab; + + desc = &tab->descs[tab->nr_descs]; + memset(desc, 0, sizeof(*desc)); + + err = gen_kfunc_arg_proto(env, &meta, &desc->proto); + if (err) + return err; + desc->func_id = func_id; desc->offset = offset; desc->addr = addr; desc->func_model = func_model; + tab->nr_descs++; sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off, NULL); return 0; @@ -8332,9 +8355,9 @@ static int process_map_ptr_arg(struct bpf_verifier_env *env, struct bpf_reg_stat static int check_func_arg(struct bpf_verifier_env *env, u32 arg, struct bpf_call_arg_meta *meta, - const struct bpf_func_proto *fn, int insn_idx) { + const struct bpf_func_proto *fn = meta->fn; u32 regno = BPF_REG_1 + arg; struct bpf_reg_state *reg = reg_state(env, regno); enum bpf_arg_type arg_type = fn->arg_type[arg]; @@ -8847,6 +8870,8 @@ static bool check_raw_mode_ok(const struct bpf_func_proto *fn, struct bpf_call_a int i; for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { + if (fn->arg_type[i] == ARG_DONTCARE) + break; if (!arg_type_is_raw_mem(fn->arg_type[i])) continue; if (meta->arg_raw_mem.regno) @@ -8895,6 +8920,8 @@ static bool check_btf_id_ok(const struct bpf_func_proto *fn) int i; for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { + if (fn->arg_type[i] == ARG_DONTCARE) + break; if (base_type(fn->arg_type[i]) == ARG_PTR_TO_BTF_ID) return !!fn->arg_btf_id[i]; if (base_type(fn->arg_type[i]) == ARG_PTR_TO_SPIN_LOCK) @@ -8916,6 +8943,8 @@ static bool check_mem_arg_rw_flag_ok(const struct bpf_func_proto *fn) for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { enum bpf_arg_type arg_type = fn->arg_type[i]; + if (arg_type == ARG_DONTCARE) + break; if (base_type(arg_type) != ARG_PTR_TO_MEM) continue; if (!(arg_type & (MEM_WRITE | MEM_RDONLY))) @@ -8932,6 +8961,8 @@ static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_ for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { enum bpf_arg_type arg_type = fn->arg_type[i]; + if (arg_type == ARG_DONTCARE) + break; if (arg_type_is_release(arg_type)) { if (meta->release_regno) return false; @@ -10321,9 +10352,10 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn env->insn_aux_data[insn_idx].non_sleepable = true; meta.func_id = func_id; + meta.fn = fn; /* check args */ for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++) { - err = check_func_arg(env, i, &meta, fn, insn_idx); + err = check_func_arg(env, i, &meta, insn_idx); if (err) return err; } @@ -11463,6 +11495,43 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, return arg_type; } +static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, + struct bpf_func_proto *proto) +{ + const struct btf *btf = meta->btf; + const struct btf_param *args; + u32 i, nargs; + int arg_type; + + args = (const struct btf_param *)(meta->func_proto + 1); + nargs = btf_type_vlen(meta->func_proto); + if (nargs > MAX_BPF_FUNC_ARGS) { + verbose(env, "Function %s has %d > %d args\n", meta->func_name, + nargs, MAX_BPF_FUNC_ARGS); + return -EINVAL; + } + if (nargs > MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args()) { + verbose(env, "JIT does not support kfunc %s() with %d args\n", + meta->func_name, nargs); + return -ENOTSUPP; + } + + for (i = 0; i < nargs; i++) { + if (is_kfunc_arg_prog_aux(btf, &args[i]) || + is_kfunc_arg_ignore(btf, &args[i]) || + is_kfunc_arg_implicit(meta, i)) + continue; + + arg_type = get_kfunc_arg_type(env, meta, args, i, nargs); + if (arg_type < 0) + return arg_type; + + proto->arg_type[i] = arg_type; + } + + return 0; +} + static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, const struct btf_type *ref_t, @@ -12046,16 +12115,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me args = (const struct btf_param *)(meta->func_proto + 1); nargs = btf_type_vlen(meta->func_proto); - if (nargs > MAX_BPF_FUNC_ARGS) { - verbose(env, "Function %s has %d > %d args\n", func_name, nargs, - MAX_BPF_FUNC_ARGS); - return -EINVAL; - } - if (nargs > MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args()) { - verbose(env, "JIT does not support kfunc %s() with %d args\n", - func_name, nargs); - return -ENOTSUPP; - } ret = check_outgoing_stack_args(env, caller, nargs); if (ret) @@ -12072,7 +12131,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me int regno = reg_from_argno(argno); bool btf_id_fixed_off_ok = true; u32 ref_id, type_size; - int kf_arg_type; + int kf_arg_type = meta->fn->arg_type[i]; if (is_kfunc_arg_prog_aux(btf, &args[i])) { /* Reject repeated use bpf_prog_aux */ @@ -12117,9 +12176,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname = btf_name_by_offset(btf, ref_t->name_off); } - kf_arg_type = get_kfunc_arg_type(env, meta, args, i, nargs); - if (kf_arg_type < 0) - return kf_arg_type; if (bpf_register_is_null(reg) && type_may_be_null(kf_arg_type)) continue; @@ -12986,6 +13042,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int err, insn_idx = *insn_idx_p; const struct btf_param *args; u32 i, nargs, ptr_type_id; + struct bpf_kfunc_desc *desc; struct btf *desc_btf; int id; @@ -13002,6 +13059,13 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, func_name = meta.func_name; insn_aux = &env->insn_aux_data[insn_idx]; + desc = find_kfunc_desc(env->prog, insn->imm, insn->off); + if (!desc) { + verifier_bug(env, "kfunc descriptor not found for func_id %u", insn->imm); + return -EFAULT; + } + meta.fn = &desc->proto; + insn_aux->is_iter_next = bpf_is_iter_next_kfunc(&meta); if (!insn->off && From 0b10b945479c954393d62ee3229d2f224a4ca91c Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 28 Jul 2026 14:05:16 +0800 Subject: [PATCH 192/373] bpf: Fix mmap_lock deadlock on arena lock failure Reported by the Sashiko AI review. arena_vm_fault() returns VM_FAULT_RETRY when it can't take arena->spinlock, but it never took mmap_lock. The fault path assumes a VM_FAULT_RETRY handler already dropped mmap_lock and re-takes it on the retry, so mmap_lock gets taken twice and can deadlock: do_user_addr_fault() { fault = handle_mm_fault(...); // calls arena_vm_fault() if (fault & VM_FAULT_RETRY) goto retry; // re-locks mmap_lock mmap_read_unlock(mm); } Return VM_FAULT_SIGBUS instead, for two reasons: 1. We could keep VM_FAULT_RETRY, but then we'd have to drop the fault lock first and cap the retry ourselves, the way __folio_lock_or_retry() does. 2. A failed raw_res_spin_lock_irqsave() already means a possible deadlock was detected, so retrying just hits the same lock again. So returning VM_FAULT_RETRY here is overkill. Fixes: b8467290edab ("bpf: arena: make arena kfuncs any context safe") Signed-off-by: Jiayuan Chen Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260728060517.95183-1-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/arena.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 34f023a537fe..555ee2531ef9 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -490,8 +490,12 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) kaddr = kbase + (u32)(vmf->address); if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) - /* Make a reasonable effort to address impossible case */ - return VM_FAULT_RETRY; + /* + * A failed lock means a possible deadlock was detected. Don't + * return VM_FAULT_RETRY: this handler never took mmap_lock, but + * the fault path would re-take it on retry and deadlock. Fail. + */ + return VM_FAULT_SIGBUS; page = vmalloc_to_page((void *)kaddr); if (page) { From dec58a70d4e7d7eb2764bb3cfcb8e6692b652f0c Mon Sep 17 00:00:00 2001 From: Shivaji Kant Date: Sat, 1 Aug 2026 05:12:59 +0000 Subject: [PATCH 193/373] bpf: Allow IP_TRANSPARENT and IPV6_TRANSPARENT in bpf_{set,get}sockopt() Currently, bpf_setsockopt() and bpf_getsockopt() for SOL_IP and SOL_IPV6 only allow a small subset of socket options (such as IP_TOS, IPV6_TCLASS, and IPV6_AUTOFLOWLABEL). Calling bpf_setsockopt() with IP_TRANSPARENT or IPV6_TRANSPARENT fails with -EINVAL. Transparent proxying (TPROXY) and related networking components often rely on IP_TRANSPARENT and IPV6_TRANSPARENT to enable binding sockets to non-local IP addresses. Allow IP_TRANSPARENT for SOL_IP in sol_ip_sockopt() and IPV6_TRANSPARENT for SOL_IPV6 in sol_ipv6_sockopt(). Signed-off-by: Shivaji Kant Tested-by: Anubhav Singh Reviewed-by: Kuniyuki Iwashima Link: https://lore.kernel.org/bpf/20260801051307.478469-1-shivajikant@google.com Signed-off-by: Kumar Kartikeya Dwivedi --- net/core/filter.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/core/filter.c b/net/core/filter.c index eb4d299b1fec..25bfd9d7df4b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -5645,6 +5645,7 @@ static int sol_ip_sockopt(struct sock *sk, int optname, switch (optname) { case IP_TOS: + case IP_TRANSPARENT: if (*optlen != sizeof(int)) return -EINVAL; break; @@ -5671,6 +5672,7 @@ static int sol_ipv6_sockopt(struct sock *sk, int optname, switch (optname) { case IPV6_TCLASS: case IPV6_AUTOFLOWLABEL: + case IPV6_TRANSPARENT: if (*optlen != sizeof(int)) return -EINVAL; break; From 60781269e26c786de2bb93fb1e697a5c32ccee48 Mon Sep 17 00:00:00 2001 From: Shivaji Kant Date: Sat, 1 Aug 2026 05:13:00 +0000 Subject: [PATCH 194/373] selftests/bpf: Add IP_TRANSPARENT and IPV6_TRANSPARENT to setget_sockopt Add test coverage for IP_TRANSPARENT and IPV6_TRANSPARENT socket options in the setget_sockopt BPF selftest to verify bpf_setsockopt() and bpf_getsockopt() helpers. Signed-off-by: Shivaji Kant Tested-by: Anubhav Singh Reviewed-by: Kuniyuki Iwashima Link: https://lore.kernel.org/bpf/20260801051307.478469-2-shivajikant@google.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/progs/bpf_tracing_net.h | 2 ++ tools/testing/selftests/bpf/progs/setget_sockopt.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h index c4b438854565..593b38f90417 100644 --- a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h +++ b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h @@ -31,10 +31,12 @@ #define __SO_ACCEPTCON (1 << 16) #define IP_TOS 1 +#define IP_TRANSPARENT 19 #define SOL_IPV6 41 #define IPV6_TCLASS 67 #define IPV6_AUTOFLOWLABEL 70 +#define IPV6_TRANSPARENT 75 #define TC_ACT_UNSPEC (-1) #define TC_ACT_OK 0 diff --git a/tools/testing/selftests/bpf/progs/setget_sockopt.c b/tools/testing/selftests/bpf/progs/setget_sockopt.c index 636a7cd8e2fa..d96e99b67aeb 100644 --- a/tools/testing/selftests/bpf/progs/setget_sockopt.c +++ b/tools/testing/selftests/bpf/progs/setget_sockopt.c @@ -69,12 +69,14 @@ static const struct sockopt_test sol_tcp_tests[] = { static const struct sockopt_test sol_ip_tests[] = { { .opt = IP_TOS, .new = 0xe1, .expected = 0xe1, .tcp_expected = 0xe0, }, + { .opt = IP_TRANSPARENT, .flip = 1, }, { .opt = 0, }, }; static const struct sockopt_test sol_ipv6_tests[] = { { .opt = IPV6_TCLASS, .new = 0xe1, .expected = 0xe1, .tcp_expected = 0xe0, }, { .opt = IPV6_AUTOFLOWLABEL, .flip = 1, }, + { .opt = IPV6_TRANSPARENT, .flip = 1, }, { .opt = 0, }, }; From 457d4ecb47aaf7a2cb46aaadd76e8c812e4f3c9e Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Sun, 2 Aug 2026 22:27:26 -0700 Subject: [PATCH 195/373] bpf: Remove unused BTF_FMODEL_STRUCT_ARG Commit 814cba835ef6 ("bpf, x86: Fix trampoline stack size for 128-bit arguments") changed the x86 trampoline to compute the number of registers from arg_size for every argument, which removed the last user of BTF_FMODEL_STRUCT_ARG. No other architecture or verifier code looks at the flag, so remove the macro and the code in __get_type_fmodel_flags() which sets it. Keep BTF_FMODEL_SIGNED_ARG at BIT(1) rather than renumbering it to BIT(0), so BIT(0) is available for a future flag. No functional change. Signed-off-by: Yonghong Song Signed-off-by: Daniel Borkmann Acked-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/20260803052726.2821447-1-yonghong.song@linux.dev --- include/linux/bpf.h | 3 --- kernel/bpf/btf.c | 2 -- 2 files changed, 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 356884587ae1..73bacfc6444d 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1207,9 +1207,6 @@ struct bpf_prog_offload { u32 jited_len; }; -/* The argument is a structure or a union. */ -#define BTF_FMODEL_STRUCT_ARG BIT(0) - /* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 5e8ac45ce56a..42414633cf26 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7533,8 +7533,6 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t) { u8 flags = 0; - if (btf_type_is_struct(t)) - flags |= BTF_FMODEL_STRUCT_ARG; if (btf_type_is_signed_int(t)) flags |= BTF_FMODEL_SIGNED_ARG; From 180c7000712db77063b3a26f4c97e7dd9038f449 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Mon, 3 Aug 2026 04:26:08 -0700 Subject: [PATCH 196/373] bpf: Invalidate RCU pointers after final spin unlock In a sleepable BPF program, a spin lock can provide the only RCU protection for a kptr. The final bpf_spin_unlock() ends that protection, but the verifier leaves the pointer valid. Another CPU can then free the object before the pointer is used. A capability-limited runtime PoC triggered a task_struct use-after-free in __bpf_get_task_stack(). Record whether the program is in an RCU-protected context before releasing the lock. Invalidate RCU-protected pointers only when the unlock leaves the final such context. This preserves valid pointers in non-sleepable programs and inside an explicit RCU read-side section. Fixes: 5861d1e8dbc4 ("bpf: Allow bpf_spin_{lock,unlock} in sleepable progs") Assisted-by: Codex:gpt-5.6-sol Assisted-by: ChatGPT:GPT-5.6-Pro Signed-off-by: Ning Ding Link: https://lore.kernel.org/bpf/20260803112615.3362122-2-dingning04@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b274004fccfd..7439afdc851a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -206,6 +206,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int par static int release_reference_nomark(struct bpf_verifier_state *state, int id); static int release_reference(struct bpf_verifier_env *env, int id); static void invalidate_non_owning_refs(struct bpf_verifier_env *env); +static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env); static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); static bool is_tracing_prog_type(enum bpf_prog_type type); static int ref_set_non_owning(struct bpf_verifier_env *env, @@ -7165,6 +7166,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state return err; } } else { + bool was_in_rcu_cs; void *ptr; int type; @@ -7192,10 +7194,13 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state verbose(env, "%s_unlock cannot be out of order\n", lock_str); return -EINVAL; } + was_in_rcu_cs = in_rcu_cs(env); if (release_lock_state(cur, type, reg->id, ptr)) { verbose(env, "%s_unlock of different lock\n", lock_str); return -EINVAL; } + if (was_in_rcu_cs && !in_rcu_cs(env)) + invalidate_rcu_protected_refs(env); invalidate_non_owning_refs(env); } From bb2df6fd891d6332cc180d198914e9f40ada50e8 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Mon, 3 Aug 2026 04:26:09 -0700 Subject: [PATCH 197/373] selftests/bpf: Test RCU pointer invalidation after spin unlock The verifier previously accepted a task kptr after the final spin unlock ended its RCU protection in a sleepable program. The pointer could then be used after the task was freed. Add a negative test for that case. Add positive controls showing that the pointer remains valid in a non-sleepable program and while an explicit RCU read-side section is still active. Assisted-by: Codex:gpt-5.6-sol Assisted-by: ChatGPT:GPT-5.6-Pro Signed-off-by: Ning Ding Link: https://lore.kernel.org/bpf/20260803112615.3362122-3-dingning04@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/task_kfunc.c | 2 + .../selftests/bpf/progs/task_kfunc_common.h | 12 +++++ .../selftests/bpf/progs/task_kfunc_failure.c | 24 ++++++++++ .../selftests/bpf/progs/task_kfunc_success.c | 48 +++++++++++++++++++ 4 files changed, 86 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/task_kfunc.c b/tools/testing/selftests/bpf/prog_tests/task_kfunc.c index e6e95c1416e6..fbd7855712c1 100644 --- a/tools/testing/selftests/bpf/prog_tests/task_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/task_kfunc.c @@ -176,6 +176,8 @@ static const char * const success_tests[] = { "test_task_from_pid_current", "test_task_from_pid_invalid", "task_kfunc_acquire_trusted_walked", + "task_kfunc_acquire_after_spin_unlock_non_sleepable", + "task_kfunc_acquire_after_spin_unlock_explicit_rcu", "test_task_kfunc_flavor_relo", "test_task_kfunc_flavor_relo_not_found", }; diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_common.h b/tools/testing/selftests/bpf/progs/task_kfunc_common.h index e9c4fea7a4bb..052c9d0e3e2a 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_common.h +++ b/tools/testing/selftests/bpf/progs/task_kfunc_common.h @@ -20,6 +20,18 @@ struct { __uint(max_entries, 1); } __tasks_kfunc_map SEC(".maps"); +struct task_kptr_lock_value { + struct bpf_spin_lock lock; + struct task_struct __kptr * task; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct task_kptr_lock_value); + __uint(max_entries, 1); +} task_kptr_lock_map SEC(".maps"); + struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym; void bpf_task_release(struct task_struct *p) __ksym; struct task_struct *bpf_task_from_pid(s32 pid) __ksym; diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c index 5c99b1e6532b..c0e7216b3419 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c +++ b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c @@ -378,3 +378,27 @@ int BPF_PROG(task_kfunc_release_in_map, struct task_struct *task, u64 clone_flag return 0; } + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("R1 must be a rcu pointer") +int BPF_PROG(task_kfunc_acquire_after_final_spin_unlock) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_spin_lock(&v->lock); + task = v->task; + bpf_spin_unlock(&v->lock); + if (!task) + return 0; + + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_success.c b/tools/testing/selftests/bpf/progs/task_kfunc_success.c index d63a79ee33dc..2bab7634c9df 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_success.c +++ b/tools/testing/selftests/bpf/progs/task_kfunc_success.c @@ -6,6 +6,7 @@ #include #include "../bpf_experimental.h" +#include "bpf_misc.h" #include "task_kfunc_common.h" char _license[] SEC("license") = "GPL"; @@ -366,6 +367,53 @@ int BPF_PROG(task_kfunc_acquire_trusted_walked, struct task_struct *task, u64 cl return 0; } +SEC("fentry/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_spin_unlock_non_sleepable) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_spin_lock(&v->lock); + task = v->task; + bpf_spin_unlock(&v->lock); + if (!task) + return 0; + + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_spin_unlock_explicit_rcu) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_rcu_read_lock(); + bpf_spin_lock(&v->lock); + task = v->task; + bpf_spin_unlock(&v->lock); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_rcu_read_unlock(); + return 0; +} + SEC("syscall") int test_task_from_vpid_current(const void *ctx) { From e2baf9cc374d5374e28702cb40e78551e82dd183 Mon Sep 17 00:00:00 2001 From: Tushar Vyavahare Date: Tue, 28 Jul 2026 11:50:36 +0000 Subject: [PATCH 198/373] selftests/xsk: Decouple xskxceiver and xdp apps from test_progs objects Build xskxceiver, xdp_hw_metadata, and xdp_features from explicit source lists instead of reusing helper objects produced by test_progs rules. Reusing shared objects such as network_helpers.o and xsk.o can pull in test_progs-only dependency chains and trigger unrelated libarena builds when invoking a single target. Keep these standalone binaries self-contained so each target builds only its own required sources and BPF skeleton dependencies. Signed-off-by: Tushar Vyavahare Signed-off-by: Andrii Nakryiko Tested-by: Maciej Fijalkowski Reviewed-by: Maciej Fijalkowski Link: https://lore.kernel.org/bpf/20260728115036.2049536-1-tushar.vyavahare@intel.com --- tools/testing/selftests/bpf/Makefile | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 55d394438705..2749b26fd4cd 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -932,17 +932,26 @@ $(OUTPUT)/test_verifier: test_verifier.c verifier/tests.h $(BPFOBJ) | $(OUTPUT) $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ -# Include find_bit.c to compile xskxceiver. -EXTRA_SRC := $(TOOLSDIR)/lib/find_bit.c prog_tests/test_xsk.c prog_tests/test_xsk.h -$(OUTPUT)/xskxceiver: $(EXTRA_SRC) xskxceiver.c xskxceiver.h $(OUTPUT)/network_helpers.o $(OUTPUT)/xsk.o $(OUTPUT)/xsk_xdp_progs.skel.h $(BPFOBJ) | $(OUTPUT) +# Keep xskxceiver independent from test_progs object dependencies. +$(OUTPUT)/xskxceiver: xskxceiver.c xsk.c network_helpers.c \ + $(TOOLSDIR)/lib/find_bit.c prog_tests/test_xsk.c \ + xskxceiver.h xsk.h network_helpers.h \ + prog_tests/test_xsk.h test_progs.h bpf_util.h \ + $(OUTPUT)/xsk_xdp_progs.skel.h $(BPFOBJ) | $(OUTPUT) $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ -$(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c $(OUTPUT)/network_helpers.o $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h | $(OUTPUT) +$(OUTPUT)/xdp_hw_metadata: xdp_hw_metadata.c xsk.c network_helpers.c \ + $(TOOLSDIR)/lib/find_bit.c xdp_metadata.h \ + xsk.h network_helpers.h test_progs.h bpf_util.h \ + $(OUTPUT)/xdp_hw_metadata.skel.h $(BPFOBJ) | $(OUTPUT) $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ -$(OUTPUT)/xdp_features: xdp_features.c $(OUTPUT)/network_helpers.o $(OUTPUT)/xdp_features.skel.h | $(OUTPUT) +$(OUTPUT)/xdp_features: xdp_features.c network_helpers.c xdp_features.h \ + network_helpers.h \ + test_progs.h bpf_util.h $(OUTPUT)/xdp_features.skel.h \ + $(BPFOBJ) | $(OUTPUT) $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(filter %.a %.o %.c,$^) $(LDLIBS) -o $@ From 6655c409707ec8ce9ce0850ffe4fe02331fd4d9c Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Mon, 3 Aug 2026 01:39:34 +0000 Subject: [PATCH 199/373] bpf, cgroup: Fix invalid storage access after __cgroup_bpf_attach failed A potential invalid storage access issue can occur after replacing a cgroup bpf prog. This occurs in the following scenario: 1. prog1 with storage is attached to a cgroup in multi-attach mode. 2. prog1 is replaced with prog2 using BPF_F_REPLACE in multi-attach mode, but fails midway (e.g. in bpf_trampoline_link_cgroup_shim or update_effective_progs). 3. A new prog3 is attached to the cgroup in multi-attach mode. The reason is that __cgroup_bpf_attach overwrites pl->storage with the new storage prior to attachment completion. When attachment fails midway, the cleanup path calls bpf_cgroup_storages_free(new_storage) to free the newly allocated storage, but fails to restore pl->storage back to old_storage. Consequently, the still-active prog1 holds invalid or dangling storage pointers, leading to an invalid memory access when prog1 executes and calls bpf_get_local_storage. Additionally, original pl->flags and cgrp->bpf.flags[atype] are left unrestored. Fix this by saving old_pl_flags, old_storage, and old_flags prior to the update, and properly restoring all of them in the cleanup path on error. Fixes: 7d9c3427894f ("bpf: Make cgroup storages shared between programs on the same cgroup") Reported-by: Sashiko Signed-off-by: Pu Lehui Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260803013934.4036646-1-pulehui@huaweicloud.com --- kernel/bpf/cgroup.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index fb9357b64cad..d2da5063d8f8 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -813,8 +813,10 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *old_prog = NULL; struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; struct bpf_cgroup_storage *new_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; + struct bpf_cgroup_storage *old_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; struct bpf_prog *new_prog = prog ? : link->link.prog; enum cgroup_bpf_attach_type atype; + u32 old_flags, old_pl_flags; struct bpf_prog_list *pl; struct hlist_head *progs; int err; @@ -865,6 +867,8 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, if (pl) { old_prog = pl->prog; + old_pl_flags = pl->flags; + bpf_cgroup_storages_assign(old_storage, pl->storage); } else { pl = kmalloc_obj(*pl); if (!pl) { @@ -884,6 +888,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, pl->link = link; pl->flags = flags; bpf_cgroup_storages_assign(pl->storage, storage); + old_flags = cgrp->bpf.flags[atype]; cgrp->bpf.flags[atype] = saved_flags; if (type == BPF_LSM_CGROUP) { @@ -915,12 +920,15 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, if (old_prog) { pl->prog = old_prog; pl->link = NULL; + pl->flags = old_pl_flags; + bpf_cgroup_storages_assign(pl->storage, old_storage); } bpf_cgroup_storages_free(new_storage); if (!old_prog) { hlist_del(&pl->node); kfree(pl); } + cgrp->bpf.flags[atype] = old_flags; return err; } From b87803391baa7e0bef60549d8841f12e549ad057 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 4 Aug 2026 22:19:16 +0200 Subject: [PATCH 200/373] bpf: Check load-acquire src ptr type before the load check_atomic_load() calls check_load_mem() before atomic_ptr_type_ok(). For a load-acquire that fetches into its own source register (dst_reg == src_reg), check_load_mem() overwrites src_reg's type with the type of the loaded value, so the subsequent atomic_ptr_type_ok() no longer sees the source pointer and fails to reject the disallowed types (ctx, pkt, flow_keys, sock). Since bpf_convert_ctx_accesses() does not rewrite atomic loads, the raw access to the underlying kernel object is left in place. The destination type is taken from the ctx access itself, so a load-acquire of the sk field of struct __sk_buff for example leaves the register typed as PTR_TO_SOCK_COMMON_OR_NULL, which type_is_sk_pointer() does not match either, while it actually holds unconverted struct sk_buff bytes. Once the NULL check has passed this is a type confusion, not just a leak of kernel data. Validate src_reg with check_reg_arg() and check the source pointer type with atomic_ptr_type_ok() before the load again, mirroring check_atomic_rmw(). Out-of-range register numbers are already rejected earlier by check_and_resolve_insns() (commit 503d21ef8eac ("bpf: Do register range validation early")), and the only exemption there, is_stack_arg_ldx(), requires BPF_LDX | BPF_MEM | BPF_DW and thus never matches a BPF_ATOMIC insn. atomic_ptr_type_ok() can therefore not dereference register state out of bounds, that is, the out-of-bounds read addressed by the Fixes commit below does not reappear (as proven also via selftest). Fixes: c03bb2fa327e ("bpf: Fix out-of-bounds read in check_atomic_load/store()") Reported-by: STAR Labs SG Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260804201917.253491-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7439afdc851a..09588b7b08b0 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6617,7 +6617,7 @@ static int check_atomic_load(struct bpf_verifier_env *env, { int err; - err = check_load_mem(env, insn, true, false, false, "atomic_load"); + err = check_reg_arg(env, insn->src_reg, SRC_OP); if (err) return err; @@ -6628,7 +6628,7 @@ static int check_atomic_load(struct bpf_verifier_env *env, return -EACCES; } - return 0; + return check_load_mem(env, insn, true, false, false, "atomic_load"); } static int check_atomic_store(struct bpf_verifier_env *env, From 363b15d8551ef6749fac410377a8422325008dae Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 4 Aug 2026 22:19:17 +0200 Subject: [PATCH 201/373] selftests/bpf: Add load-acquire test for dst_reg == src_reg from ctx Add a verifier test that a load-acquire fetching into its own source register (dst_reg == src_reg) from a ctx pointer is rejected. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_load_acquire [...] #614/1 verifier_load_acquire/load-acquire, 8-bit:OK #614/2 verifier_load_acquire/load-acquire, 8-bit @unpriv:OK #614/3 verifier_load_acquire/load-acquire, 16-bit:OK #614/4 verifier_load_acquire/load-acquire, 16-bit @unpriv:OK #614/5 verifier_load_acquire/load-acquire, 32-bit:OK #614/6 verifier_load_acquire/load-acquire, 32-bit @unpriv:OK #614/7 verifier_load_acquire/load-acquire, 64-bit:OK #614/8 verifier_load_acquire/load-acquire, 64-bit @unpriv:OK #614/9 verifier_load_acquire/load-acquire with uninitialized src_reg:OK #614/10 verifier_load_acquire/load-acquire with uninitialized src_reg @unpriv:OK #614/11 verifier_load_acquire/load-acquire with non-pointer src_reg:OK #614/12 verifier_load_acquire/load-acquire with non-pointer src_reg @unpriv:OK #614/13 verifier_load_acquire/misaligned load-acquire:OK #614/14 verifier_load_acquire/misaligned load-acquire @unpriv:OK #614/15 verifier_load_acquire/load-acquire from ctx pointer:OK #614/16 verifier_load_acquire/load-acquire from ctx pointer @unpriv:OK #614/17 verifier_load_acquire/load-acquire from ctx pointer, same dst and src register:OK #614/18 verifier_load_acquire/load-acquire from ctx pointer, same dst and src register @unpriv:OK #614/19 verifier_load_acquire/load-acquire with invalid register R15:OK #614/20 verifier_load_acquire/load-acquire with invalid register R15 @unpriv:OK #614/21 verifier_load_acquire/load-acquire from pkt pointer:OK #614/22 verifier_load_acquire/load-acquire from flow_keys pointer:OK #614/23 verifier_load_acquire/load-acquire from sock pointer:OK #614 verifier_load_acquire:OK Summary: 1/23 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260804201917.253491-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/verifier_load_acquire.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c index 74f4f19c10b8..ae1dab1b0cbb 100644 --- a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c +++ b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c @@ -148,6 +148,22 @@ __naked void load_acquire_from_ctx_pointer(void) : __clobber_all); } +SEC("socket") +__description("load-acquire from ctx pointer, same dst and src register") +__failure __failure_unpriv __msg("BPF_ATOMIC loads from R6 ctx is not allowed") +__naked void load_acquire_ctx_same_dst_src(void) +{ + asm volatile ( + "r6 = r1;" + ".8byte %[load_acquire_insn];" // w6 = load_acquire((u32 *)(r6 + 0)); + "r0 = 0;" + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_W, BPF_LOAD_ACQ, BPF_REG_6, BPF_REG_6, 0)) + : __clobber_all); +} + SEC("xdp") __description("load-acquire from pkt pointer") __failure __msg("BPF_ATOMIC loads from R2 pkt is not allowed") From 8efd87051c3a2a054519955ca401229f4e84b310 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:53 -0700 Subject: [PATCH 202/373] bpf: Correct the overflow check comment in bpf_iter_num_next() The comment on the s->cur + 1 >= s->end check claims the (s64) cast is needed to avoid overflow when s->cur == s->end == INT_MAX. It isn't: s->cur + 1 is computed in int and wraps before the cast, so the cast changes nothing (INT_MAX + 1 compares the same either way). The wraparound is the point. bpf_iter_num_new() sets s->cur = start - 1, which wraps to INT_MAX for start == INT_MIN, and the wrapping s->cur + 1 brings it back to start. (s64)s->cur + 1 would instead break iterators starting at INT_MIN. Drop the cast and reword the comment. No functional change; the wrap is well-defined under -fno-strict-overflow. Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-2-puranjay@kernel.org --- kernel/bpf/bpf_iter.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c index f5eaeb2493d4..b235e117e206 100644 --- a/kernel/bpf/bpf_iter.c +++ b/kernel/bpf/bpf_iter.c @@ -802,12 +802,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) { struct bpf_iter_num_kern *s = (void *)it; - /* check failed initialization or if we are done (same behavior); - * need to be careful about overflow, so convert to s64 for checks, - * e.g., if s->cur == s->end == INT_MAX, we can't just do - * s->cur + 1 >= s->end + /* + * s->cur < s->end while iterating, else s->cur == s->end == 0; the signed + * s->cur + 1 >= s->end holds even when s->cur + 1 wraps (start == INT_MIN). */ - if ((s64)(s->cur + 1) >= s->end) { + if (s->cur + 1 >= s->end) { s->cur = s->end = 0; return NULL; } From f8f2b567d56035ddd62ec82f2c35dcee8c516624 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:54 -0700 Subject: [PATCH 203/373] bpf: Inline bpf_iter_num_new() kfunc bpf_for() expands to the bpf_iter_num_{new,next,destroy}() kfuncs, which the verifier emits as regular calls. They are tiny and only touch the 8-byte on-stack iterator state, so open-code them in bpf_fixup_kfunc_call() like the other special kfuncs there. Start with bpf_iter_num_new(): R1 points to the iterator, R2/R3 hold start/end. The inlined sequence mirrors the kfunc and returns the same -EINVAL / -E2BIG / 0. start > end is rejected first, so end - start fits in a u32; range-check it as u32 on both sides ((u32)(end - start) in the kfunc). A movsx-based check would emit a cpuv4 instruction that some JITs (x86-32, mips32, sparc64) decode as a plain move and get wrong. The emitted instructions are plain BPF, so the interpreter path stays correct and no jit_required marking is needed. Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-3-puranjay@kernel.org --- kernel/bpf/bpf_iter.c | 4 ++-- kernel/bpf/verifier.c | 24 ++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c index b235e117e206..d19f1b2861d2 100644 --- a/kernel/bpf/bpf_iter.c +++ b/kernel/bpf/bpf_iter.c @@ -782,8 +782,8 @@ __bpf_kfunc int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) return -EINVAL; } - /* avoid overflows, e.g., if start == INT_MIN and end == INT_MAX */ - if ((s64)end - (s64)start > BPF_MAX_LOOPS) { + /* start <= end here, so end - start fits in a u32 without overflow */ + if ((u32)(end - start) > BPF_MAX_LOOPS) { s->cur = s->end = 0; return -E2BIG; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 09588b7b08b0..8401077ed8fc 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20006,6 +20006,30 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[4] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1); insn_buf[5] = BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0); *cnt = 6; + } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) { + /* inline bpf_iter_num_new(&it, start, end); R1=&it, R2=start, R3=end */ + int i = 0; + + /* if (start > end) goto einval; */ + insn_buf[i++] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8); + /* r0 = (u32)end - (u32)start; if (r0 > BPF_MAX_LOOPS) goto e2big; */ + insn_buf[i++] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3); + insn_buf[i++] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2); + insn_buf[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, BPF_MAX_LOOPS, 8); + /* s->cur = start - 1; s->end = end; return 0; */ + insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1); + insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0); + insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4); + insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0); + insn_buf[i++] = BPF_JMP_A(5); + /* einval: s->cur = s->end = 0; return -EINVAL; */ + insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0); + insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL); + insn_buf[i++] = BPF_JMP_A(2); + /* e2big: s->cur = s->end = 0; return -E2BIG; */ + insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0); + insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG); + *cnt = i; } if (env->insn_aux_data[insn_idx].arg_prog) { From e93347704878aa8a54b3154114d13e57e8923e27 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:55 -0700 Subject: [PATCH 204/373] bpf: Inline bpf_iter_num_next() kfunc bpf_iter_num_next() runs on every bpf_for() iteration, so inlining it drops a call from the loop body. R1 points to the iterator; the returned pointer to s->cur is R1 itself, since s->cur is first. s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end is a signed 32-bit compare and the inlined code needs no sign extension. Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-4-puranjay@kernel.org --- kernel/bpf/verifier.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8401077ed8fc..80c3cb654b89 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20030,6 +20030,23 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0); insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG); *cnt = i; + } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) { + /* inline bpf_iter_num_next(&it); R1=&it, returns &s->cur or NULL */ + int i = 0; + + /* r0 = s->cur + 1; if ((s32)r0 >= s->end) goto done; */ + insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0); + insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1); + insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4); + insn_buf[i++] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3); + /* s->cur = r0; return &s->cur; */ + insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0); + insn_buf[i++] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); + insn_buf[i++] = BPF_JMP_A(2); + /* done: s->cur = s->end = 0; return NULL; */ + insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0); + insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0); + *cnt = i; } if (env->insn_aux_data[insn_idx].arg_prog) { From 39f047682fe3ccc272dbebb6d1ecba8fc1e0d8b4 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:56 -0700 Subject: [PATCH 205/373] bpf: Inline bpf_iter_num_destroy() as a no-op Once destroy() returns the stack slot is no longer tracked as iterator state, so zeroing it is dead work. Make the kfunc a no-op and inline the call to a single BPF_JA 0 (the fixup can't drop the instruction outright, so emit a nop; the JITs elide it). Suggested-by: Andrii Nakryiko Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-5-puranjay@kernel.org --- kernel/bpf/bpf_iter.c | 4 +--- kernel/bpf/verifier.c | 4 ++++ 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c index d19f1b2861d2..14a5fdfa0421 100644 --- a/kernel/bpf/bpf_iter.c +++ b/kernel/bpf/bpf_iter.c @@ -818,9 +818,7 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) __bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it) { - struct bpf_iter_num_kern *s = (void *)it; - - s->cur = s->end = 0; + /* no-op */ } __bpf_kfunc_end_defs(); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 80c3cb654b89..4db151e24355 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20047,6 +20047,10 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0); insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0); *cnt = i; + } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) { + /* bpf_iter_num_destroy() is a no-op; emit a nop to drop the call */ + insn_buf[0] = BPF_JMP_A(0); + *cnt = 1; } if (env->insn_aux_data[insn_idx].arg_prog) { From b829bc167705b72ed6af084e193665f7c21f9135 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:57 -0700 Subject: [PATCH 206/373] selftests/bpf: Verify inlined numeric iterator shape with __xlated Add an __xlated test pinning the inlined bpf_iter_num_{new,next,destroy}() shapes. The program is __naked, so there is no compiler glue and the whole sequence is matched instruction for instruction. Gate it to x86_64 and arm64 (bpf_jit_needs_zext() == false); elsewhere the verifier interleaves "wN = wN" zero-extensions that would not match. The inlining is arch independent, so these two are enough. Suggested-by: Eduard Zingerman Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-6-puranjay@kernel.org --- tools/testing/selftests/bpf/progs/iters.c | 83 +++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c index 0fa70b133d93..62d7df9e80be 100644 --- a/tools/testing/selftests/bpf/progs/iters.c +++ b/tools/testing/selftests/bpf/progs/iters.c @@ -88,6 +88,89 @@ int iter_err_unsafe_asm_loop(const void *ctx) return 0; } +/* + * Naked function, so there is no compiler-generated glue and the whole inlined program can be + * matched. Pinned to arches whose JITs zero-extend 32-bit writes implicitly + * (bpf_jit_needs_zext() == false); on arches that need explicit zero-extension the verifier + * interleaves "wN = wN" insns and the fixed shape below would not match. The inlining itself is + * arch independent, so checking it on these arches is sufficient. + * + * bpf_iter_num_new() emits the full range check (distance computation and both the -EINVAL and + * -E2BIG error paths); bpf_iter_num_next() and bpf_iter_num_destroy() are inlined too. + */ +SEC("raw_tp") +__arch_x86_64 +__arch_arm64 +__success +__xlated("r6 = r10") +__xlated("r6 += -8") +__xlated("call unknown") +__xlated("r3 = r0") +__xlated("r3 &= 65535") +__xlated("r1 = r6") +__xlated("r2 = 0") +/* bpf_iter_num_new(&it, 0, ) with the range check kept */ +__xlated("if w2 s> w3 goto pc+8") +__xlated("w0 = w3") +__xlated("w0 -= w2") +__xlated("if r0 > 0x800000 goto pc+8") +__xlated("w2 += -1") +__xlated("*(u32 *)(r1 +0) = r2") +__xlated("*(u32 *)(r1 +4) = r3") +__xlated("r0 = 0") +__xlated("goto pc+5") +__xlated("*(u64 *)(r1 +0) = 0") +__xlated("r0 = -22") +__xlated("goto pc+2") +__xlated("*(u64 *)(r1 +0) = 0") +__xlated("r0 = -7") +__xlated("r1 = r6") +/* bpf_iter_num_next(&it) */ +__xlated("r0 = *(u32 *)(r1 +0)") +__xlated("w0 += 1") +__xlated("r2 = *(u32 *)(r1 +4)") +__xlated("if w0 s>= w2 goto pc+3") +__xlated("*(u32 *)(r1 +0) = r0") +__xlated("r0 = r1") +__xlated("goto pc+2") +__xlated("*(u64 *)(r1 +0) = 0") +__xlated("r0 = 0") +__xlated("if r0 != 0x0 goto pc-11") +__xlated("r1 = r6") +/* bpf_iter_num_destroy(&it) is inlined to a nop */ +__xlated("goto pc+0") +__xlated("r0 = 0") +__xlated("exit") +int __naked iter_num_new_inlined(void) +{ + asm volatile ( + /* r6 points to struct bpf_iter_num on the stack */ + "r6 = r10;" + "r6 += -8;" + /* non-constant end so the range checks are kept */ + "call %[bpf_get_prandom_u32];" + "r3 = r0;" + "r3 &= 0xffff;" + "r1 = r6;" + "r2 = 0;" + "call %[bpf_iter_num_new];" + "1:" + "r1 = r6;" + "call %[bpf_iter_num_next];" + "if r0 != 0 goto 1b;" + "r1 = r6;" + "call %[bpf_iter_num_destroy];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_iter_num_new), + __imm(bpf_iter_num_next), + __imm(bpf_iter_num_destroy) + : __clobber_common, "r6" + ); +} + SEC("raw_tp") __success int iter_while_loop(const void *ctx) From 5a1de41147b0b604e49144671937e9178bf5b5c1 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Tue, 4 Aug 2026 06:45:58 -0700 Subject: [PATCH 207/373] selftests/bpf: Add bpf_for() benchmark Add a bpf_for() benchmark modelled on bench_bpf_loop so the per-iteration iterator cost can be measured and compared against bpf_loop. It runs an empty bpf_for(i, 0, nr_loops) loop 1000 times per trigger and accounts nr_loops hits per outer iteration: $ ./bench -p 1 --nr_loops 1000 bpf-for Signed-off-by: Puranjay Mohan Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260804134601.2305303-7-puranjay@kernel.org --- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 4 + .../selftests/bpf/benchs/bench_bpf_for.c | 104 ++++++++++++++++++ .../selftests/bpf/benchs/run_bench_bpf_for.sh | 15 +++ .../selftests/bpf/progs/bpf_for_bench.c | 32 ++++++ 5 files changed, 157 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 2749b26fd4cd..d3655a706482 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -974,6 +974,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \ $(OUTPUT)/perfbuf_bench.skel.h $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h +$(OUTPUT)/bench_bpf_for.o: $(OUTPUT)/bpf_for_bench.skel.h $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h @@ -999,6 +1000,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_ringbufs.o \ $(OUTPUT)/bench_bloom_filter_map.o \ $(OUTPUT)/bench_bpf_loop.o \ + $(OUTPUT)/bench_bpf_for.o \ $(OUTPUT)/bench_strncmp.o \ $(OUTPUT)/bench_bpf_hashmap_full_update.o \ $(OUTPUT)/bench_local_storage.o \ diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c index 3d9d2cd7764b..b86b73456d3c 100644 --- a/tools/testing/selftests/bpf/bench.c +++ b/tools/testing/selftests/bpf/bench.c @@ -276,6 +276,7 @@ static const struct argp_option opts[] = { extern struct argp bench_ringbufs_argp; extern struct argp bench_bloom_map_argp; extern struct argp bench_bpf_loop_argp; +extern struct argp bench_bpf_for_argp; extern struct argp bench_local_storage_argp; extern struct argp bench_local_storage_rcu_tasks_trace_argp; extern struct argp bench_strncmp_argp; @@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 }, { &bench_bloom_map_argp, 0, "Bloom filter map benchmark", 0 }, { &bench_bpf_loop_argp, 0, "bpf_loop helper benchmark", 0 }, + { &bench_bpf_for_argp, 0, "bpf_for loop benchmark", 0 }, { &bench_local_storage_argp, 0, "local_storage benchmark", 0 }, { &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 }, { &bench_local_storage_rcu_tasks_trace_argp, 0, @@ -557,6 +559,7 @@ extern const struct bench bench_bloom_false_positive; extern const struct bench bench_hashmap_without_bloom; extern const struct bench bench_hashmap_with_bloom; extern const struct bench bench_bpf_loop; +extern const struct bench bench_bpf_for; extern const struct bench bench_strncmp_no_helper; extern const struct bench bench_strncmp_helper; extern const struct bench bench_bpf_hashmap_full_update; @@ -640,6 +643,7 @@ static const struct bench *benchs[] = { &bench_hashmap_without_bloom, &bench_hashmap_with_bloom, &bench_bpf_loop, + &bench_bpf_for, &bench_strncmp_no_helper, &bench_strncmp_helper, &bench_bpf_hashmap_full_update, diff --git a/tools/testing/selftests/bpf/benchs/bench_bpf_for.c b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file mode 100644 index 000000000000..730c51ad2dec --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/bench_bpf_for.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include +#include "bench.h" +#include "bpf_for_bench.skel.h" + +/* BPF triggering benchmarks */ +static struct ctx { + struct bpf_for_bench *skel; +} ctx; + +static struct { + __u32 nr_loops; +} args = { + /* + * Default to a large loop count so the per-iteration bpf_iter_num_next() cost dominates + * the one-time bpf_iter_num_new()/destroy() setup and teardown. + */ + .nr_loops = 1000, +}; + +enum { + ARG_NR_LOOPS = 4000, +}; + +static const struct argp_option opts[] = { + { "nr_loops", ARG_NR_LOOPS, "nr_loops", 0, + "Set number of iterations for the bpf_for() loop"}, + {}, +}; + +static error_t parse_arg(int key, char *arg, struct argp_state *state) +{ + switch (key) { + case ARG_NR_LOOPS: + args.nr_loops = strtol(arg, NULL, 10); + break; + default: + return ARGP_ERR_UNKNOWN; + } + + return 0; +} + +/* exported into benchmark runner */ +const struct argp bench_bpf_for_argp = { + .options = opts, + .parser = parse_arg, +}; + +static void validate(void) +{ + if (env.consumer_cnt != 0) { + fprintf(stderr, "benchmark doesn't support consumer!\n"); + exit(1); + } +} + +static void *producer(void *input) +{ + while (true) + /* trigger the bpf program */ + syscall(__NR_getpgid); + + return NULL; +} + +static void measure(struct bench_res *res) +{ + res->hits = atomic_swap(&ctx.skel->bss->hits, 0); +} + +static void setup(void) +{ + struct bpf_link *link; + + setup_libbpf(); + + ctx.skel = bpf_for_bench__open_and_load(); + if (!ctx.skel) { + fprintf(stderr, "failed to open skeleton\n"); + exit(1); + } + + link = bpf_program__attach(ctx.skel->progs.benchmark); + if (!link) { + fprintf(stderr, "failed to attach program!\n"); + exit(1); + } + + ctx.skel->bss->nr_loops = args.nr_loops; +} + +const struct bench bench_bpf_for = { + .name = "bpf-for", + .argp = &bench_bpf_for_argp, + .validate = validate, + .setup = setup, + .producer_thread = producer, + .measure = measure, + .report_progress = ops_report_progress, + .report_final = ops_report_final, +}; diff --git a/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh new file mode 100755 index 000000000000..7da6453920da --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh @@ -0,0 +1,15 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for t in 1 4 8 12 16; do +for i in 10 100 500 1000 5000 10000 50000 100000 500000 1000000; do +subtitle "nr_loops: $i, nr_threads: $t" + summarize_ops "bpf_for: " \ + "$($RUN_BENCH -p $t --nr_loops $i bpf-for)" + printf "\n" +done +done diff --git a/tools/testing/selftests/bpf/progs/bpf_for_bench.c b/tools/testing/selftests/bpf/progs/bpf_for_bench.c new file mode 100644 index 000000000000..f9c723051fc7 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bpf_for_bench.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +int nr_loops; +long hits; + +static int outer_loop(__u32 index, void *data) +{ + int i; + + /* + * Empty body: the work being measured is the open-coded numeric iterator itself + * (bpf_iter_num_new/next/destroy behind bpf_for()). + */ + bpf_for(i, 0, nr_loops) + ; + __sync_add_and_fetch(&hits, nr_loops); + return 0; +} + +SEC("fentry/" SYS_PREFIX "sys_getpgid") +int benchmark(void *ctx) +{ + bpf_loop(1000, outer_loop, NULL, 0); + return 0; +} From 15b837759a97237d647962f9943afe0d55af615a Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:38 +0200 Subject: [PATCH 208/373] bpf: Factor stackid_init function from __bpf_get_stackid The new stackid_init function stores all the necessary bits for stackid trace and it will be used by other functions in following changes. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-2-jolsa@kernel.org --- kernel/bpf/stackmap.c | 95 +++++++++++++++++++++++++++---------------- 1 file changed, 59 insertions(+), 36 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 463f94ba1cc4..19f9ea605a3e 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -504,33 +504,54 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) #endif } -static long __bpf_get_stackid(struct bpf_map *map, - struct perf_callchain_entry *trace, u64 flags) +struct stackid { + struct stack_map_bucket *bucket; + u64 *ips; + u32 nr; + u32 len; + u32 hash; + u32 id; +}; + +static int stackid_init(struct stackid *stackid, struct bpf_map *map, + struct perf_callchain_entry *trace, u64 flags) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); - struct stack_map_bucket *bucket, *new_bucket, *old_bucket; - u32 hash, id, trace_nr, trace_len, i, max_depth; u32 skip = flags & BPF_F_SKIP_FIELD_MASK; - bool user = flags & BPF_F_USER_STACK; - u64 *ips; - bool hash_matches; + u32 max_depth; if (trace->nr <= skip) /* skipping more than usable stack trace */ return -EFAULT; max_depth = stack_map_calculate_max_depth(map->value_size, stack_map_data_size(map), flags); - trace_nr = min_t(u32, trace->nr - skip, max_depth - skip); - trace_len = trace_nr * sizeof(u64); - ips = trace->ip + skip; - hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0); - id = hash & (smap->n_buckets - 1); - bucket = READ_ONCE(smap->buckets[id]); + stackid->nr = min_t(u32, trace->nr - skip, max_depth - skip); + stackid->len = stackid->nr * sizeof(u64); + stackid->ips = trace->ip + skip; + stackid->hash = jhash2((u32 *)stackid->ips, stackid->len / sizeof(u32), 0); + stackid->id = stackid->hash & (smap->n_buckets - 1); + stackid->bucket = READ_ONCE(smap->buckets[stackid->id]); + return 0; +} - hash_matches = bucket && bucket->hash == hash; +static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, + struct perf_callchain_entry *trace, u64 flags) +{ + struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); + struct stack_map_bucket *new_bucket, *old_bucket; + bool user = flags & BPF_F_USER_STACK; + bool hash_matches; + u32 trace_len, i; + int err; + + err = stackid_init(stackid, map, trace, flags); + if (err) + return err; + + hash_matches = stackid->bucket && stackid->bucket->hash == stackid->hash; /* fast cmp */ if (hash_matches && flags & BPF_F_FAST_STACK_CMP) - return id; + return stackid->id; if (stack_map_use_build_id(map)) { struct bpf_stack_build_id *id_offs; @@ -540,42 +561,42 @@ static long __bpf_get_stackid(struct bpf_map *map, pcpu_freelist_pop(&smap->freelist); if (unlikely(!new_bucket)) return -ENOMEM; - new_bucket->nr = trace_nr; + new_bucket->nr = stackid->nr; id_offs = (struct bpf_stack_build_id *)new_bucket->data; - for (i = 0; i < trace_nr; i++) - id_offs[i].ip = ips[i]; - stack_map_get_build_id_offset(id_offs, trace_nr, user, false /* !may_fault */); - trace_len = trace_nr * sizeof(struct bpf_stack_build_id); - if (hash_matches && bucket->nr == trace_nr && - memcmp(bucket->data, new_bucket->data, trace_len) == 0) { + for (i = 0; i < stackid->nr; i++) + id_offs[i].ip = stackid->ips[i]; + stack_map_get_build_id_offset(id_offs, stackid->nr, user, false /* !may_fault */); + trace_len = stackid->nr * sizeof(struct bpf_stack_build_id); + if (hash_matches && stackid->bucket->nr == stackid->nr && + memcmp(stackid->bucket->data, new_bucket->data, trace_len) == 0) { pcpu_freelist_push(&smap->freelist, &new_bucket->fnode); - return id; + return stackid->id; } - if (bucket && !(flags & BPF_F_REUSE_STACKID)) { + if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID)) { pcpu_freelist_push(&smap->freelist, &new_bucket->fnode); return -EEXIST; } } else { - if (hash_matches && bucket->nr == trace_nr && - memcmp(bucket->data, ips, trace_len) == 0) - return id; - if (bucket && !(flags & BPF_F_REUSE_STACKID)) + if (hash_matches && stackid->bucket->nr == stackid->nr && + memcmp(stackid->bucket->data, stackid->ips, stackid->len) == 0) + return stackid->id; + if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID)) return -EEXIST; new_bucket = (struct stack_map_bucket *) pcpu_freelist_pop(&smap->freelist); if (unlikely(!new_bucket)) return -ENOMEM; - memcpy(new_bucket->data, ips, trace_len); + memcpy(new_bucket->data, stackid->ips, stackid->len); } - new_bucket->hash = hash; - new_bucket->nr = trace_nr; + new_bucket->hash = stackid->hash; + new_bucket->nr = stackid->nr; - old_bucket = xchg(&smap->buckets[id], new_bucket); + old_bucket = xchg(&smap->buckets[stackid->id], new_bucket); if (old_bucket) pcpu_freelist_push(&smap->freelist, &old_bucket->fnode); - return id; + return stackid->id; } BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, @@ -584,6 +605,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u32 elem_size = stack_map_data_size(map); bool user = flags & BPF_F_USER_STACK; struct perf_callchain_entry *trace; + struct stackid stackid; bool kernel = !user; u32 max_depth; @@ -599,7 +621,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, /* couldn't fetch the stack trace */ return -EFAULT; - return __bpf_get_stackid(map, trace, flags); + return __bpf_get_stackid(&stackid, map, trace, flags); } const struct bpf_func_proto bpf_get_stackid_proto = { @@ -628,6 +650,7 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, { struct perf_event *event = ctx->event; struct perf_callchain_entry *trace; + struct stackid stackid; bool kernel, user; __u64 nr_kernel; int ret; @@ -653,7 +676,7 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, if (kernel) { trace->nr = nr_kernel; - ret = __bpf_get_stackid(map, trace, flags); + ret = __bpf_get_stackid(&stackid, map, trace, flags); } else { /* user */ u64 skip = flags & BPF_F_SKIP_FIELD_MASK; @@ -662,7 +685,7 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, return -EFAULT; flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; - ret = __bpf_get_stackid(map, trace, flags); + ret = __bpf_get_stackid(&stackid, map, trace, flags); } /* restore nr */ From 0ca56befcffec3a6c9d1842eae06c74e1cf41f11 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:39 +0200 Subject: [PATCH 209/373] bpf: Factor stackid_fastpath function from __bpf_get_stackid The new stackid_fastpath does the fast stack hash and trace check, that does not need new bucket allocation. It covers both just-ip and buildid code paths. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-3-jolsa@kernel.org --- kernel/bpf/stackmap.c | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 19f9ea605a3e..27210b5d16fc 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -511,6 +511,7 @@ struct stackid { u32 len; u32 hash; u32 id; + bool hash_matches; }; static int stackid_init(struct stackid *stackid, struct bpf_map *map, @@ -531,28 +532,46 @@ static int stackid_init(struct stackid *stackid, struct bpf_map *map, stackid->hash = jhash2((u32 *)stackid->ips, stackid->len / sizeof(u32), 0); stackid->id = stackid->hash & (smap->n_buckets - 1); stackid->bucket = READ_ONCE(smap->buckets[stackid->id]); + stackid->hash_matches = stackid->bucket && stackid->bucket->hash == stackid->hash; return 0; } +static int stackid_fastpath(struct stackid *stackid, struct bpf_map *map, + struct perf_callchain_entry *trace, u64 flags) +{ + int err; + + err = stackid_init(stackid, map, trace, flags); + if (err) + return err; + + /* fast cmp */ + if (stackid->hash_matches && flags & BPF_F_FAST_STACK_CMP) + return stackid->id; + + if (stack_map_use_build_id(map)) + return -ENOENT; + if (stackid->hash_matches && stackid->bucket->nr == stackid->nr && + memcmp(stackid->bucket->data, stackid->ips, stackid->len) == 0) + return stackid->id; + if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID)) + return -EEXIST; + return -ENOENT; +} + static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, struct perf_callchain_entry *trace, u64 flags) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); struct stack_map_bucket *new_bucket, *old_bucket; bool user = flags & BPF_F_USER_STACK; - bool hash_matches; u32 trace_len, i; int err; - err = stackid_init(stackid, map, trace, flags); - if (err) + err = stackid_fastpath(stackid, map, trace, flags); + if (err != -ENOENT) return err; - hash_matches = stackid->bucket && stackid->bucket->hash == stackid->hash; - /* fast cmp */ - if (hash_matches && flags & BPF_F_FAST_STACK_CMP) - return stackid->id; - if (stack_map_use_build_id(map)) { struct bpf_stack_build_id *id_offs; @@ -567,7 +586,7 @@ static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, id_offs[i].ip = stackid->ips[i]; stack_map_get_build_id_offset(id_offs, stackid->nr, user, false /* !may_fault */); trace_len = stackid->nr * sizeof(struct bpf_stack_build_id); - if (hash_matches && stackid->bucket->nr == stackid->nr && + if (stackid->hash_matches && stackid->bucket->nr == stackid->nr && memcmp(stackid->bucket->data, new_bucket->data, trace_len) == 0) { pcpu_freelist_push(&smap->freelist, &new_bucket->fnode); return stackid->id; @@ -577,12 +596,6 @@ static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, return -EEXIST; } } else { - if (hash_matches && stackid->bucket->nr == stackid->nr && - memcmp(stackid->bucket->data, stackid->ips, stackid->len) == 0) - return stackid->id; - if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID)) - return -EEXIST; - new_bucket = (struct stack_map_bucket *) pcpu_freelist_pop(&smap->freelist); if (unlikely(!new_bucket)) From bb4e6f4e1b68fe60c04ca04c564c6624e837dbf4 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:40 +0200 Subject: [PATCH 210/373] bpf: Factor stackid_new_bucket from __bpf_get_stackid The new stackid_new_bucket allocates the new bucket and initializes it with the trace data. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-4-jolsa@kernel.org --- kernel/bpf/stackmap.c | 48 +++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 27210b5d16fc..d930d9754d7d 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -559,31 +559,52 @@ static int stackid_fastpath(struct stackid *stackid, struct bpf_map *map, return -ENOENT; } +static struct stack_map_bucket * +stackid_new_bucket(struct stackid *stackid, struct bpf_map *map) +{ + struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); + struct bpf_stack_build_id *id_offs; + struct stack_map_bucket *bucket; + u32 i; + + bucket = (struct stack_map_bucket *) pcpu_freelist_pop(&smap->freelist); + if (unlikely(!bucket)) + return NULL; + + if (stack_map_use_build_id(map)) { + id_offs = (struct bpf_stack_build_id *)bucket->data; + for (i = 0; i < stackid->nr; i++) + id_offs[i].ip = stackid->ips[i]; + } else { + memcpy(bucket->data, stackid->ips, stackid->len); + } + + bucket->hash = stackid->hash; + bucket->nr = stackid->nr; + return bucket; +} + static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, struct perf_callchain_entry *trace, u64 flags) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); struct stack_map_bucket *new_bucket, *old_bucket; bool user = flags & BPF_F_USER_STACK; - u32 trace_len, i; + u32 trace_len; int err; err = stackid_fastpath(stackid, map, trace, flags); if (err != -ENOENT) return err; + new_bucket = stackid_new_bucket(stackid, map); + if (!new_bucket) + return -ENOMEM; + if (stack_map_use_build_id(map)) { struct bpf_stack_build_id *id_offs; - /* for build_id+offset, pop a bucket before slow cmp */ - new_bucket = (struct stack_map_bucket *) - pcpu_freelist_pop(&smap->freelist); - if (unlikely(!new_bucket)) - return -ENOMEM; - new_bucket->nr = stackid->nr; id_offs = (struct bpf_stack_build_id *)new_bucket->data; - for (i = 0; i < stackid->nr; i++) - id_offs[i].ip = stackid->ips[i]; stack_map_get_build_id_offset(id_offs, stackid->nr, user, false /* !may_fault */); trace_len = stackid->nr * sizeof(struct bpf_stack_build_id); if (stackid->hash_matches && stackid->bucket->nr == stackid->nr && @@ -595,17 +616,8 @@ static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, pcpu_freelist_push(&smap->freelist, &new_bucket->fnode); return -EEXIST; } - } else { - new_bucket = (struct stack_map_bucket *) - pcpu_freelist_pop(&smap->freelist); - if (unlikely(!new_bucket)) - return -ENOMEM; - memcpy(new_bucket->data, stackid->ips, stackid->len); } - new_bucket->hash = stackid->hash; - new_bucket->nr = stackid->nr; - old_bucket = xchg(&smap->buckets[stackid->id], new_bucket); if (old_bucket) pcpu_freelist_push(&smap->freelist, &old_bucket->fnode); From 09b3fd6caa0b57f8a39254ee5db3af30bdd53c18 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:41 +0200 Subject: [PATCH 211/373] bpf: Use stack id functions instead of __bpf_get_stackid Replacing __bpf_get_stackid calls with sequence of following functions: stackid_fastpath stackid_new_bucket stackid_install This makes code more structured and allows us to easily disable preemption only in bpf_get_stackid in following changes. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-5-jolsa@kernel.org --- kernel/bpf/stackmap.c | 43 +++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index d930d9754d7d..3ee0034daf52 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -584,22 +584,13 @@ stackid_new_bucket(struct stackid *stackid, struct bpf_map *map) return bucket; } -static long __bpf_get_stackid(struct stackid *stackid, struct bpf_map *map, - struct perf_callchain_entry *trace, u64 flags) +static long stackid_install(struct stackid *stackid, struct bpf_map *map, + struct stack_map_bucket *new_bucket, u64 flags) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); - struct stack_map_bucket *new_bucket, *old_bucket; bool user = flags & BPF_F_USER_STACK; + struct stack_map_bucket *old_bucket; u32 trace_len; - int err; - - err = stackid_fastpath(stackid, map, trace, flags); - if (err != -ENOENT) - return err; - - new_bucket = stackid_new_bucket(stackid, map); - if (!new_bucket) - return -ENOMEM; if (stack_map_use_build_id(map)) { struct bpf_stack_build_id *id_offs; @@ -629,10 +620,12 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, { u32 elem_size = stack_map_data_size(map); bool user = flags & BPF_F_USER_STACK; + struct stack_map_bucket *new_bucket; struct perf_callchain_entry *trace; struct stackid stackid; bool kernel = !user; u32 max_depth; + int err; if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK | BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID))) @@ -646,7 +639,15 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, /* couldn't fetch the stack trace */ return -EFAULT; - return __bpf_get_stackid(&stackid, map, trace, flags); + err = stackid_fastpath(&stackid, map, trace, flags); + if (err != -ENOENT) + return err; + + new_bucket = stackid_new_bucket(&stackid, map); + if (!new_bucket) + return -ENOMEM; + + return stackid_install(&stackid, map, new_bucket, flags); } const struct bpf_func_proto bpf_get_stackid_proto = { @@ -674,6 +675,7 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, struct bpf_map *, map, u64, flags) { struct perf_event *event = ctx->event; + struct stack_map_bucket *new_bucket; struct perf_callchain_entry *trace; struct stackid stackid; bool kernel, user; @@ -701,7 +703,6 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, if (kernel) { trace->nr = nr_kernel; - ret = __bpf_get_stackid(&stackid, map, trace, flags); } else { /* user */ u64 skip = flags & BPF_F_SKIP_FIELD_MASK; @@ -710,12 +711,22 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, return -EFAULT; flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; - ret = __bpf_get_stackid(&stackid, map, trace, flags); } + ret = stackid_fastpath(&stackid, map, trace, flags); + if (ret != -ENOENT) + goto out; + + new_bucket = stackid_new_bucket(&stackid, map); + if (new_bucket) { + trace->nr = nr; + return stackid_install(&stackid, map, new_bucket, flags); + } + ret = -ENOMEM; + +out: /* restore nr */ trace->nr = nr; - return ret; } From 15f1bd8574662f1b7b26aaa2e23ebf4066f0117d Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:42 +0200 Subject: [PATCH 212/373] bpf: Disable preemption in bpf_get_stackid The get_perf_callchain call needs disabled preemption plus we need it disabled as long as we access its returned trace entries buffer. Note the bpf_get_stackid_pe function is executed already with preemption disabled. Fixes: d5a3b1f69186 ("bpf: introduce BPF_MAP_TYPE_STACK_TRACE") Reported-by: Tao Chen Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260803210149.296496-6-jolsa@kernel.org Closes: https://lore.kernel.org/bpf/20260206090653.1336687-2-chen.dylane@linux.dev/ --- kernel/bpf/stackmap.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 3ee0034daf52..5b18d728f4b8 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -632,20 +632,22 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, return -EINVAL; max_depth = stack_map_calculate_max_depth(map->value_size, elem_size, flags); - trace = get_perf_callchain(regs, kernel, user, max_depth, - false, false, 0); - if (unlikely(!trace)) - /* couldn't fetch the stack trace */ - return -EFAULT; + scoped_guard(preempt) { + trace = get_perf_callchain(regs, kernel, user, max_depth, + false, false, 0); + if (unlikely(!trace)) + /* couldn't fetch the stack trace */ + return -EFAULT; - err = stackid_fastpath(&stackid, map, trace, flags); - if (err != -ENOENT) - return err; + err = stackid_fastpath(&stackid, map, trace, flags); + if (err != -ENOENT) + return err; - new_bucket = stackid_new_bucket(&stackid, map); - if (!new_bucket) - return -ENOMEM; + new_bucket = stackid_new_bucket(&stackid, map); + if (!new_bucket) + return -ENOMEM; + } return stackid_install(&stackid, map, new_bucket, flags); } From cbb99938e7935c9f62c891573a4b09690e5e22de Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:43 +0200 Subject: [PATCH 213/373] bpf: Factor callchain_store function from __bpf_get_stack The new callchain_store function stores trace entries buffer into user supplied buffer. It covers both just-ip and buildid data. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-7-jolsa@kernel.org --- kernel/bpf/stackmap.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 5b18d728f4b8..ee9905d67b8e 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -741,6 +741,29 @@ const struct bpf_func_proto bpf_get_stackid_proto_pe = { .arg3_type = ARG_ANYTHING, }; +static u32 callchain_store(struct perf_callchain_entry *trace, void *buf, + u32 elem_size, u64 flags) +{ + bool user_build_id = flags & BPF_F_USER_BUILD_ID; + u32 skip = flags & BPF_F_SKIP_FIELD_MASK; + u32 trace_nr, copy_len; + u64 *ips; + + trace_nr = trace->nr - skip; + copy_len = trace_nr * elem_size; + + ips = trace->ip + skip; + if (user_build_id) { + struct bpf_stack_build_id *id_offs = buf; + + for (u32 i = 0; i < trace_nr; i++) + id_offs[i].ip = ips[i]; + } else { + memcpy(buf, ips, copy_len); + } + return trace_nr; +} + static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, struct perf_callchain_entry *trace_in, void *buf, u32 size, u64 flags, bool may_fault) @@ -753,7 +776,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, struct perf_callchain_entry *trace; bool kernel = !user; int err = -EINVAL; - u64 *ips; if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK | BPF_F_USER_BUILD_ID))) @@ -798,21 +820,10 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, goto err_fault; } - trace_nr = trace->nr - skip; + trace_nr = callchain_store(trace, buf, elem_size, flags); copy_len = trace_nr * elem_size; - ips = trace->ip + skip; - if (user_build_id) { - struct bpf_stack_build_id *id_offs = buf; - u32 i; - - for (i = 0; i < trace_nr; i++) - id_offs[i].ip = ips[i]; - } else { - memcpy(buf, ips, copy_len); - } - - /* trace/ips should not be dereferenced after this point */ + /* trace should not be dereferenced after this point */ if (may_fault) rcu_read_unlock(); From 014fbe5902dccdaef69114abdcdb15e3dbe55e34 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:44 +0200 Subject: [PATCH 214/373] bpf: Factor callchain_finalize function from __bpf_get_stack The new callchain_finalize function calls the build-id retrieval (if needed) and zeroes the buffer. This makes things easier for preemption fix in following change. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-8-jolsa@kernel.org --- kernel/bpf/stackmap.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index ee9905d67b8e..cdeb6c2e50da 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -764,16 +764,31 @@ static u32 callchain_store(struct perf_callchain_entry *trace, void *buf, return trace_nr; } +static long callchain_finalize(void *buf, u32 size, u32 trace_nr, u32 elem_size, + u64 flags, bool may_fault) +{ + bool user_build_id = flags & BPF_F_USER_BUILD_ID; + bool user = flags & BPF_F_USER_STACK; + u32 copy_len = trace_nr * elem_size; + + if (user_build_id) + stack_map_get_build_id_offset(buf, trace_nr, user, may_fault); + + if (size > copy_len) + memset(buf + copy_len, 0, size - copy_len); + return copy_len; +} + static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, struct perf_callchain_entry *trace_in, void *buf, u32 size, u64 flags, bool may_fault) { - u32 trace_nr, copy_len, elem_size, max_depth; bool user_build_id = flags & BPF_F_USER_BUILD_ID; bool crosstask = task && task != current; u32 skip = flags & BPF_F_SKIP_FIELD_MASK; bool user = flags & BPF_F_USER_STACK; struct perf_callchain_entry *trace; + u32 trace_nr, elem_size, max_depth; bool kernel = !user; int err = -EINVAL; @@ -821,18 +836,12 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, } trace_nr = callchain_store(trace, buf, elem_size, flags); - copy_len = trace_nr * elem_size; /* trace should not be dereferenced after this point */ if (may_fault) rcu_read_unlock(); - if (user_build_id) - stack_map_get_build_id_offset(buf, trace_nr, user, may_fault); - - if (size > copy_len) - memset(buf + copy_len, 0, size - copy_len); - return copy_len; + return callchain_finalize(buf, size, trace_nr, elem_size, flags, may_fault); err_fault: err = -EFAULT; From 58cfc2201d964163fe9c4a703136eb64db799f08 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:45 +0200 Subject: [PATCH 215/373] bpf: Remove trace_in argument from __bpf_get_stack Now with the new callchain_* helper functions we can process trace_in case directly in bpf_get_stack_pe function and remove it from __bpf_get_stack which makes things easier for preemption fix in following change. Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-9-jolsa@kernel.org --- kernel/bpf/stackmap.c | 49 ++++++++++++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index cdeb6c2e50da..976c4e4c1af6 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -780,7 +780,6 @@ static long callchain_finalize(void *buf, u32 size, u32 trace_nr, u32 elem_size, } static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, - struct perf_callchain_entry *trace_in, void *buf, u32 size, u64 flags, bool may_fault) { bool user_build_id = flags & BPF_F_USER_BUILD_ID; @@ -819,10 +818,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, if (may_fault) rcu_read_lock(); /* need RCU for perf's callchain below */ - if (trace_in) { - trace = trace_in; - trace->nr = min_t(u32, trace->nr, max_depth); - } else if (kernel && task) { + if (kernel && task) { trace = get_callchain_entry_for_task(task, max_depth); } else { trace = get_perf_callchain(regs, kernel, user, max_depth, @@ -853,7 +849,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size, u64, flags) { - return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */); + return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */); } const struct bpf_func_proto bpf_get_stack_proto = { @@ -869,7 +865,7 @@ const struct bpf_func_proto bpf_get_stack_proto = { BPF_CALL_4(bpf_get_stack_sleepable, struct pt_regs *, regs, void *, buf, u32, size, u64, flags) { - return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, true /* may_fault */); + return __bpf_get_stack(regs, NULL, buf, size, flags, true /* may_fault */); } const struct bpf_func_proto bpf_get_stack_sleepable_proto = { @@ -893,7 +889,7 @@ static long __bpf_get_task_stack(struct task_struct *task, void *buf, u32 size, regs = task_pt_regs(task); if (regs) - res = __bpf_get_stack(regs, task, NULL, buf, size, flags, may_fault); + res = __bpf_get_stack(regs, task, buf, size, flags, may_fault); put_task_stack(task); return res; @@ -933,6 +929,32 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = { .arg4_type = ARG_ANYTHING, }; +static int __bpf_get_stack_pe(struct perf_callchain_entry *trace, void *buf, u32 size, + u64 flags) +{ + bool user_build_id = flags & BPF_F_USER_BUILD_ID; + u64 skip = flags & BPF_F_SKIP_FIELD_MASK; + bool user = flags & BPF_F_USER_STACK; + u32 elem_size, max_depth, nr_trace; + bool kernel = !user; + + if (kernel && user_build_id) + return -EINVAL; + + elem_size = user_build_id ? sizeof(struct bpf_stack_build_id) : sizeof(u64); + if (unlikely(size % elem_size)) + return -EINVAL; + + max_depth = stack_map_calculate_max_depth(size, elem_size, flags); + trace->nr = min_t(u32, trace->nr, max_depth); + + if (trace->nr < skip) + return -EFAULT; + + nr_trace = callchain_store(trace, buf, elem_size, flags); + return callchain_finalize(buf, size, nr_trace, elem_size, flags, false /* !may_fault */); +} + BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, void *, buf, u32, size, u64, flags) { @@ -944,7 +966,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, __u64 nr_kernel; if (!(event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)) - return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */); + return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */); if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK | BPF_F_USER_BUILD_ID))) @@ -964,7 +986,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, __u64 nr = trace->nr; trace->nr = nr_kernel; - err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */); + err = __bpf_get_stack_pe(trace, buf, size, flags); /* restore nr */ trace->nr = nr; @@ -974,14 +996,13 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, skip += nr_kernel; if (skip > BPF_F_SKIP_FIELD_MASK) goto clear; - flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; - err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */); + err = __bpf_get_stack_pe(trace, buf, size, flags); } - return err; clear: - memset(buf, 0, size); + if (err < 0) + memset(buf, 0, size); return err; } From f5d242825ca417bb6afe35fde6e8880f97ca43fb Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:46 +0200 Subject: [PATCH 216/373] bpf: Clear buf on error in __bpf_get_task_stack Both bpf_get_task_stack and bpf_get_task_stack_sleepable helpers that use __bpf_get_task_stack have buf defined as ARG_PTR_TO_UNINIT_MEM argument and we should initialize the buf on every return path. Adding missing buf memset for __bpf_get_task_stack fail paths. This provides deterministic buffer contents, which is useful when the buffer is used directly as a map key. Fixes: 06ab134ce8ec ("bpf: Refcount task stack in bpf_get_task_stack") Fixes: b992f01e6615 ("bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()") Reported-by: Sashiko Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-10-jolsa@kernel.org --- kernel/bpf/stackmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 976c4e4c1af6..7f728d319a65 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -884,14 +884,17 @@ static long __bpf_get_task_stack(struct task_struct *task, void *buf, u32 size, struct pt_regs *regs; long res = -EINVAL; - if (!try_get_task_stack(task)) + if (!try_get_task_stack(task)) { + memset(buf, 0, size); return -EFAULT; + } regs = task_pt_regs(task); if (regs) res = __bpf_get_stack(regs, task, buf, size, flags, may_fault); + else + memset(buf, 0, size); put_task_stack(task); - return res; } From b1a47b2708d4e95dbd23aee2ec83752190897b3f Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Mon, 3 Aug 2026 23:01:47 +0200 Subject: [PATCH 217/373] bpf: Disable preemption in __bpf_get_stack get_perf_callchain() returns a per-CPU perf_callchain_entry buffer and releases its recursion slot via put_callchain_entry() before returning, so nothing keeps the entry reserved while __bpf_get_stack() consumes it below. A preemptible BPF program (e.g. a non-sleepable raw tracepoint program on a PREEMPT kernel, which runs under migrate_disable() but not preempt_disable()) can be scheduled out between obtaining the entry and the copy. Another task scheduled on the same CPU then reuses the same per-CPU buffer and overwrites trace->nr with a larger value. copy_len is then computed from the inflated trace->nr and can exceed the caller's buffer, causing an out-of-bounds write in the memcpy() and in the build_id path. The rcu_read_lock() taken here alone does not prevent this. It is only taken on the may_fault path, and under CONFIG_PREEMPT_RCU it does not disable preemption; it merely keeps perf's callchain buffer array alive (freed via call_rcu()) and does nothing to stop another task from reusing the entry. Disable preemption around obtaining the callchain entry and copying it into the caller's buffer, so the entry cannot be reused underneath us and trace->nr stays bounded by max_depth. Build ID resolution may fault and is therefore deferred until after preemption is re-enabled; by then the instruction pointers have already been copied into buf, so it operates only on that private copy. Note, preempt_disable() also subsumes the buffer-lifetime guarantee the rcu_read_lock() provided, since a preempt-disabled section is an RCU read-side critical section for the callchain buffers' call_rcu() reclaim. Fixes: c195651e565a ("bpf: add bpf_get_stack helper") Reported-by: Tao Chen Reported-by: STAR Labs SG Signed-off-by: Daniel Borkmann Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260803210149.296496-11-jolsa@kernel.org Closes: https://lore.kernel.org/bpf/20260206090653.1336687-1-chen.dylane@linux.dev/ [ changed Fixes: commit ] --- kernel/bpf/stackmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 7f728d319a65..121caa90707b 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -815,6 +815,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, max_depth = stack_map_calculate_max_depth(size, elem_size, flags); + preempt_disable(); if (may_fault) rcu_read_lock(); /* need RCU for perf's callchain below */ @@ -828,6 +829,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, if (unlikely(!trace) || trace->nr < skip) { if (may_fault) rcu_read_unlock(); + preempt_enable(); goto err_fault; } @@ -836,6 +838,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, /* trace should not be dereferenced after this point */ if (may_fault) rcu_read_unlock(); + preempt_enable(); return callchain_finalize(buf, size, trace_nr, elem_size, flags, may_fault); From 347c1d722e3ea4ffa2850585f5c76c6e551eac83 Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:48 +0200 Subject: [PATCH 218/373] bpf: Avoid changing callchain in bpf_get_stack_pe There's no need to modify the trace object bpf_get_stack_pe, we just need to pass the needed callchain length in separate argument. This way we can have callchain pointers const and remove the trace->nr modification and restoration. Assisted-by: Codex:GPT-5.5 Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-12-jolsa@kernel.org --- kernel/bpf/stackmap.c | 36 +++++++++++++++--------------------- 1 file changed, 15 insertions(+), 21 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 121caa90707b..57a25b56d0ac 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -661,7 +661,7 @@ const struct bpf_func_proto bpf_get_stackid_proto = { .arg3_type = ARG_ANYTHING, }; -static __u64 count_kernel_ip(struct perf_callchain_entry *trace) +static __u64 count_kernel_ip(const struct perf_callchain_entry *trace) { __u64 nr_kernel = 0; @@ -741,15 +741,15 @@ const struct bpf_func_proto bpf_get_stackid_proto_pe = { .arg3_type = ARG_ANYTHING, }; -static u32 callchain_store(struct perf_callchain_entry *trace, void *buf, - u32 elem_size, u64 flags) +static u32 callchain_store(const struct perf_callchain_entry *trace, u32 trace_nr, + void *buf, u32 elem_size, u64 flags) { bool user_build_id = flags & BPF_F_USER_BUILD_ID; u32 skip = flags & BPF_F_SKIP_FIELD_MASK; - u32 trace_nr, copy_len; - u64 *ips; + const u64 *ips; + u32 copy_len; - trace_nr = trace->nr - skip; + trace_nr = trace_nr - skip; copy_len = trace_nr * elem_size; ips = trace->ip + skip; @@ -833,7 +833,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, goto err_fault; } - trace_nr = callchain_store(trace, buf, elem_size, flags); + trace_nr = callchain_store(trace, trace->nr, buf, elem_size, flags); /* trace should not be dereferenced after this point */ if (may_fault) @@ -935,8 +935,8 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = { .arg4_type = ARG_ANYTHING, }; -static int __bpf_get_stack_pe(struct perf_callchain_entry *trace, void *buf, u32 size, - u64 flags) +static int __bpf_get_stack_pe(const struct perf_callchain_entry *trace, u32 trace_nr, + void *buf, u32 size, u64 flags) { bool user_build_id = flags & BPF_F_USER_BUILD_ID; u64 skip = flags & BPF_F_SKIP_FIELD_MASK; @@ -952,12 +952,12 @@ static int __bpf_get_stack_pe(struct perf_callchain_entry *trace, void *buf, u32 return -EINVAL; max_depth = stack_map_calculate_max_depth(size, elem_size, flags); - trace->nr = min_t(u32, trace->nr, max_depth); + trace_nr = min_t(u32, trace_nr, max_depth); - if (trace->nr < skip) + if (trace_nr < skip) return -EFAULT; - nr_trace = callchain_store(trace, buf, elem_size, flags); + nr_trace = callchain_store(trace, trace_nr, buf, elem_size, flags); return callchain_finalize(buf, size, nr_trace, elem_size, flags, false /* !may_fault */); } @@ -965,8 +965,8 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, void *, buf, u32, size, u64, flags) { struct pt_regs *regs = (struct pt_regs *)(ctx->regs); + const struct perf_callchain_entry *trace; struct perf_event *event = ctx->event; - struct perf_callchain_entry *trace; bool kernel, user; int err = -EINVAL; __u64 nr_kernel; @@ -989,13 +989,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, nr_kernel = count_kernel_ip(trace); if (kernel) { - __u64 nr = trace->nr; - - trace->nr = nr_kernel; - err = __bpf_get_stack_pe(trace, buf, size, flags); - - /* restore nr */ - trace->nr = nr; + err = __bpf_get_stack_pe(trace, nr_kernel, buf, size, flags); } else { /* user */ u64 skip = flags & BPF_F_SKIP_FIELD_MASK; @@ -1003,7 +997,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx, if (skip > BPF_F_SKIP_FIELD_MASK) goto clear; flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; - err = __bpf_get_stack_pe(trace, buf, size, flags); + err = __bpf_get_stack_pe(trace, trace->nr, buf, size, flags); } clear: From a74594607a0b310c1533e713b9097cc1cbbb85bf Mon Sep 17 00:00:00 2001 From: Jiri Olsa Date: Mon, 3 Aug 2026 23:01:49 +0200 Subject: [PATCH 219/373] bpf: Avoid changing callchain in bpf_get_stackid_pe There's no need to modify the trace object bpf_get_stackid_pe, we just need to pass the needed callchain length in separate argument. This way we can have callchain pointers const and remove the trace->nr modification and restoration. Assisted-by: Codex:GPT-5.5 Signed-off-by: Jiri Olsa Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260803210149.296496-13-jolsa@kernel.org --- kernel/bpf/stackmap.c | 39 +++++++++++++++++---------------------- 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 57a25b56d0ac..8f0f3ff1a869 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -506,7 +506,7 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) struct stackid { struct stack_map_bucket *bucket; - u64 *ips; + const u64 *ips; u32 nr; u32 len; u32 hash; @@ -515,21 +515,21 @@ struct stackid { }; static int stackid_init(struct stackid *stackid, struct bpf_map *map, - struct perf_callchain_entry *trace, u64 flags) + const struct perf_callchain_entry *trace, u32 trace_nr, u64 flags) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); u32 skip = flags & BPF_F_SKIP_FIELD_MASK; u32 max_depth; - if (trace->nr <= skip) + if (trace_nr <= skip) /* skipping more than usable stack trace */ return -EFAULT; max_depth = stack_map_calculate_max_depth(map->value_size, stack_map_data_size(map), flags); - stackid->nr = min_t(u32, trace->nr - skip, max_depth - skip); + stackid->nr = min_t(u32, trace_nr - skip, max_depth - skip); stackid->len = stackid->nr * sizeof(u64); stackid->ips = trace->ip + skip; - stackid->hash = jhash2((u32 *)stackid->ips, stackid->len / sizeof(u32), 0); + stackid->hash = jhash2((const u32 *)stackid->ips, stackid->len / sizeof(u32), 0); stackid->id = stackid->hash & (smap->n_buckets - 1); stackid->bucket = READ_ONCE(smap->buckets[stackid->id]); stackid->hash_matches = stackid->bucket && stackid->bucket->hash == stackid->hash; @@ -537,11 +537,12 @@ static int stackid_init(struct stackid *stackid, struct bpf_map *map, } static int stackid_fastpath(struct stackid *stackid, struct bpf_map *map, - struct perf_callchain_entry *trace, u64 flags) + const struct perf_callchain_entry *trace, u32 trace_nr, + u64 flags) { int err; - err = stackid_init(stackid, map, trace, flags); + err = stackid_init(stackid, map, trace, trace_nr, flags); if (err) return err; @@ -640,7 +641,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, /* couldn't fetch the stack trace */ return -EFAULT; - err = stackid_fastpath(&stackid, map, trace, flags); + err = stackid_fastpath(&stackid, map, trace, trace->nr, flags); if (err != -ENOENT) return err; @@ -676,12 +677,13 @@ static __u64 count_kernel_ip(const struct perf_callchain_entry *trace) BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, struct bpf_map *, map, u64, flags) { + const struct perf_callchain_entry *trace; struct perf_event *event = ctx->event; struct stack_map_bucket *new_bucket; - struct perf_callchain_entry *trace; struct stackid stackid; bool kernel, user; __u64 nr_kernel; + u32 trace_nr; int ret; /* perf_sample_data doesn't have callchain, use bpf_get_stackid */ @@ -701,13 +703,13 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, return -EFAULT; nr_kernel = count_kernel_ip(trace); - __u64 nr = trace->nr; /* save original */ if (kernel) { - trace->nr = nr_kernel; + trace_nr = nr_kernel; } else { /* user */ u64 skip = flags & BPF_F_SKIP_FIELD_MASK; + trace_nr = trace->nr; skip += nr_kernel; if (skip > BPF_F_SKIP_FIELD_MASK) return -EFAULT; @@ -715,21 +717,14 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; } - ret = stackid_fastpath(&stackid, map, trace, flags); + ret = stackid_fastpath(&stackid, map, trace, trace_nr, flags); if (ret != -ENOENT) - goto out; + return ret; new_bucket = stackid_new_bucket(&stackid, map); - if (new_bucket) { - trace->nr = nr; + if (new_bucket) return stackid_install(&stackid, map, new_bucket, flags); - } - ret = -ENOMEM; - -out: - /* restore nr */ - trace->nr = nr; - return ret; + return -ENOMEM; } const struct bpf_func_proto bpf_get_stackid_proto_pe = { From 00244bdaa423d93f4571f3f6854378ce3365e524 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 5 Aug 2026 23:08:09 +0800 Subject: [PATCH 220/373] bpf: Fix sleepable check for tracing/lsm prog When CONFIG_FUNCTION_ERROR_INJECTION is disabled, a sleepable tracing prog is allowed to attach to '__x64_'-alike prefix symbols. It is because the verifier does not verify whether the symbol is a kernel function or a bpf prog. That said, a sleepable tracing prog is allowed to attach to a bpf prog target whose name has '__x64_'-alike prefix. For example, a sleepable fentry prog attaches to a '__x64_sys_nop' XDP prog, and copies buffer from a user pointer with bpf_copy_from_user() helper. After attaching the XDP prog to lo interface, the kernel BUG could be triggered by 'ping -c 1 -W 1 127.0.0.1': [ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324 Fix it by disallowing sleepable prog always when its target btf is not a kernel's btf. Fixes: 16d9c5660692 ("bpf: Always allow sleepable programs on syscalls") Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Acked-by: Viktor Malik Link: https://lore.kernel.org/bpf/20260805150810.34907-2-leon.hwang@linux.dev --- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4db151e24355..d925197c2e5f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19022,6 +19022,9 @@ static int btf_id_allow_sleepable(u32 btf_id, unsigned long addr, const struct b const struct btf_type *t; const char *tname; + if (!btf_is_kernel(btf)) + return -EINVAL; + switch (prog->type) { case BPF_PROG_TYPE_TRACING: t = btf_type_by_id(btf, btf_id); From 045809795751897ebbb82ab775edd8a3d040d66f Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Wed, 5 Aug 2026 23:08:10 +0800 Subject: [PATCH 221/373] selftests/bpf: Verify rejection of sleepable tracing prog Add a test to verify that the sleepable tracing prog cannot attach to a '__x64_sys' prefix prog target. When CONFIG_FUNCTION_ERROR_INJECTION is disabled, without the fix, the test would trigger the BUG: [ 3.460756] BUG: sleeping function called from invalid context at kernel/bpf/trampoline.c:1324 Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260805150810.34907-3-leon.hwang@linux.dev --- .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 57 +++++++++++++++++++ .../selftests/bpf/progs/fentry_sleepable.c | 18 ++++++ tools/testing/selftests/bpf/progs/xdp_dummy.c | 6 ++ 3 files changed, 81 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/fentry_sleepable.c diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 4a87d7163c8c..2523c07a16c6 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -5,6 +5,7 @@ #include #include "bind4_prog.skel.h" #include "freplace_progmap.skel.h" +#include "fentry_sleepable.skel.h" #include "xdp_dummy.skel.h" typedef int (*test_cb)(struct bpf_object *obj); @@ -576,6 +577,60 @@ static void test_func_replace_progmap(void) freplace_progmap__destroy(skel); } +static void test_sleepable_fentry_to_xdp(void) +{ + struct fentry_sleepable *skel = NULL; + struct xdp_dummy *skel_xdp = NULL; + int ifindex, prog_fd, err; + char buff[64] = {}; + +#ifndef __x86_64__ + test__skip(); + return; +#endif + + ifindex = if_nametoindex("lo"); + if (!ASSERT_GT(ifindex, 0, "if_nametoindex")) + return; + + skel_xdp = xdp_dummy__open_and_load(); + if (!ASSERT_OK_PTR(skel_xdp, "xdp_dummy__open_and_load")) + return; + + skel = fentry_sleepable__open(); + if (!ASSERT_OK_PTR(skel, "fentry_sleepable__open")) + goto out; + + skel->bss->user_ptr = buff; + + prog_fd = bpf_program__fd(skel_xdp->progs.__x64_sys_nop); + err = bpf_program__set_attach_target(skel->progs.fentry_xdp, prog_fd, "__x64_sys_nop"); + if (!ASSERT_OK(err, "bpf_program__set_attach_target")) + goto out; + + err = fentry_sleepable__load(skel); + ASSERT_ERR(err, "fentry_sleepable__load"); + if (err) + goto out; + + skel->links.fentry_xdp = bpf_program__attach_trace(skel->progs.fentry_xdp); + if (!ASSERT_OK_PTR(skel->links.fentry_xdp, "bpf_program__attach_trace")) + goto out; + + skel_xdp->links.__x64_sys_nop = bpf_program__attach_xdp(skel_xdp->progs.__x64_sys_nop, + ifindex); + if (!ASSERT_OK_PTR(skel_xdp->links.__x64_sys_nop, "bpf_program__attach_xdp")) + goto out; + + err = system("ping -q -c 1 -W 1 127.0.0.1 > /dev/null"); + ASSERT_OK(err, "ping"); + ASSERT_ERR(skel->bss->retval, "retval"); + +out: + fentry_sleepable__destroy(skel); + xdp_dummy__destroy(skel_xdp); +} + /* NOTE: affect other tests, must run in serial mode */ void serial_test_fexit_bpf2bpf(void) { @@ -607,4 +662,6 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace_int_with_void(); if (test__start_subtest("freplace_void")) test_func_replace_void(); + if (test__start_subtest("sleepable_fentry_to_xdp")) + test_sleepable_fentry_to_xdp(); } diff --git a/tools/testing/selftests/bpf/progs/fentry_sleepable.c b/tools/testing/selftests/bpf/progs/fentry_sleepable.c new file mode 100644 index 000000000000..8c0fc691d329 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/fentry_sleepable.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "vmlinux.h" +#include +#include + +char LICENSE[] SEC("license") = "GPL"; + +void *user_ptr; +int retval; + +SEC("fentry.s") +int BPF_PROG(fentry_xdp) +{ + char buff[64]; + + retval = bpf_copy_from_user(buff, sizeof(buff), user_ptr); + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/xdp_dummy.c b/tools/testing/selftests/bpf/progs/xdp_dummy.c index d988b2e0cee8..5f1e0771021d 100644 --- a/tools/testing/selftests/bpf/progs/xdp_dummy.c +++ b/tools/testing/selftests/bpf/progs/xdp_dummy.c @@ -10,4 +10,10 @@ int xdp_dummy_prog(struct xdp_md *ctx) return XDP_PASS; } +SEC("xdp") +int __x64_sys_nop(struct xdp_md *ctx) +{ + return XDP_PASS; +} + char _license[] SEC("license") = "GPL"; From 11c1e836710dcba03e50454a4eedfdbaf8d3050e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=A9r=C3=A9my=20Jean?= Date: Wed, 5 Aug 2026 06:02:28 +0000 Subject: [PATCH 222/373] bpf: Harden bloom filter sizing and indexing on 32-bit kernels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bloom_map_alloc() has two 32-bit-specific problems when the computed bitmap reaches the U32_MAX fallback case. First, BITS_TO_BYTES(U32_MAX) is evaluated with 32-bit arithmetic. The addition performed by DIV_ROUND_UP wraps, so the map allocates only the fixed-size bloom filter object while keeping bitset_mask == U32_MAX. Subsequent updates can then write past the allocated object. Second, fixing only the allocation size is not sufficient. The bloom hash is a u32, but set_bit() takes a signed long bit number and x86 test_bit() eventually feeds the index to variable_test_bit(long, ...). On 32-bit kernels, hashes in [0x80000000, U32_MAX] therefore become negative bit offsets. x86 bt/bts with a memory operand interpret those offsets relative to the supplied base, so a map with bitset_mask == U32_MAX can read or write before bloom->bitset even after allocating the full 512 MiB bitmap. Keep the U32_MAX fallback, but split each hash into a word pointer and an in-word bit number before calling test_bit() or set_bit(). The bitops argument is then always in [0, BITS_PER_LONG - 1], while BIT_WORD(h) still selects the intended word in the full bitmap. Compute the bitset size from (u64)bitset_mask + 1 before passing the final size to bpf_map_area_alloc(). This fixes the original under-allocation and keeps the allocated storage consistent with the addressable bitset. Exploitation note: local privilege escalation is possible on a 32-bit x86 kernel using the under-allocation bug from a binary with CAP_BPF. Fixes: 9330986c0300 ("bpf: Add bloom filter map implementation") Signed-off-by: Jérémy Jean Signed-off-by: Andrii Nakryiko Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260805060228.2703051-1-Jeremy.Jean@oss.cyber.gouv.fr Assisted-by: Codex:gpt-5 --- kernel/bpf/bloom_filter.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index b73336c976b7..c6e7559b07de 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -41,7 +41,7 @@ static long bloom_map_peek_elem(struct bpf_map *map, void *value) for (i = 0; i < bloom->nr_hash_funcs; i++) { h = hash(bloom, value, map->value_size, i); - if (!test_bit(h, bloom->bitset)) + if (!test_bit(h % BITS_PER_LONG, bloom->bitset + BIT_WORD(h))) return -ENOENT; } @@ -57,9 +57,13 @@ static long bloom_map_push_elem(struct bpf_map *map, void *value, u64 flags) if (flags != BPF_ANY) return -EINVAL; + /* + * On 32-bit architectures, hashes larger than INT_MAX would be + * treated as negative by set_bit(). + */ for (i = 0; i < bloom->nr_hash_funcs; i++) { h = hash(bloom, value, map->value_size, i); - set_bit(h, bloom->bitset); + set_bit(h % BITS_PER_LONG, bloom->bitset + BIT_WORD(h)); } return 0; @@ -94,9 +98,10 @@ static int bloom_map_alloc_check(union bpf_attr *attr) static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) { - u32 bitset_bytes, bitset_mask, nr_hash_funcs, nr_bits; + u32 bitset_mask, nr_hash_funcs, nr_bits; int numa_node = bpf_map_attr_numa_node(attr); struct bpf_bloom_filter *bloom; + u64 bitset_bytes; if (attr->key_size != 0 || attr->value_size == 0 || attr->max_entries == 0 || @@ -127,22 +132,16 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) if (check_mul_overflow(attr->max_entries, nr_hash_funcs, &nr_bits) || check_mul_overflow(nr_bits / 5, (u32)7, &nr_bits) || nr_bits > (1UL << 31)) { - /* The bit array size is 2^32 bits but to avoid overflowing the - * u32, we use U32_MAX, which will round up to the equivalent - * number of bytes - */ - bitset_bytes = BITS_TO_BYTES(U32_MAX); bitset_mask = U32_MAX; } else { if (nr_bits <= BITS_PER_LONG) nr_bits = BITS_PER_LONG; else nr_bits = roundup_pow_of_two(nr_bits); - bitset_bytes = BITS_TO_BYTES(nr_bits); bitset_mask = nr_bits - 1; } - bitset_bytes = roundup(bitset_bytes, sizeof(unsigned long)); + bitset_bytes = BITS_TO_LONGS((u64)bitset_mask + 1) * sizeof(unsigned long); bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node); if (!bloom) From ed3b3093b6242bdb2c4acfb932d4b15db4e33948 Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Wed, 5 Aug 2026 23:33:38 +0800 Subject: [PATCH 223/373] bpf: Add KF_SPINLOCK_SAFE flag for kfuncs under bpf_spin_lock Introduce the KF_SPINLOCK_SAFE kfunc metadata flag in BTF so kfuncs may be explicitly marked as safe to call while holding bpf_spin_lock. Allow kfuncs defined in kernel modules to be marked with KF_SPINLOCK_SAFE. Example: BTF_ID_FLAGS(func, $kfunc_name, KF_SPINLOCK_SAFE) Signed-off-by: Kaitao Cheng Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260805153340.34776-2-kaitao.cheng@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/btf.h | 1 + kernel/bpf/verifier.c | 20 +++++++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/include/linux/btf.h b/include/linux/btf.h index c09b7994de4e..3f5255d095a2 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -79,6 +79,7 @@ #define KF_ARENA_ARG1 (1 << 14) /* kfunc takes an arena pointer as its first argument */ #define KF_ARENA_ARG2 (1 << 15) /* kfunc takes an arena pointer as its second argument */ #define KF_IMPLICIT_ARGS (1 << 16) /* kfunc has implicit arguments supplied by the verifier */ +#define KF_SPINLOCK_SAFE (1 << 17) /* kfunc is allowed inside bpf_spin_lock-ed region */ /* * Tag marking a kernel function as a kfunc. This is meant to minimize the diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d925197c2e5f..8daba32306be 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11837,11 +11837,21 @@ static bool is_bpf_stream_kfunc(u32 btf_id) btf_id == special_kfunc_list[KF_bpf_stream_print_stack]; } -static bool kfunc_spin_allowed(u32 btf_id) +static bool kfunc_spin_allowed(struct bpf_verifier_env *env, s32 func_id, s16 offset) { - return is_bpf_graph_api_kfunc(btf_id) || is_bpf_iter_num_api_kfunc(btf_id) || - is_bpf_res_spin_lock_kfunc(btf_id) || is_bpf_arena_kfunc(btf_id) || - is_bpf_stream_kfunc(btf_id); + struct bpf_kfunc_meta kfunc; + int err; + + if (is_bpf_graph_api_kfunc(func_id) || is_bpf_iter_num_api_kfunc(func_id) || + is_bpf_res_spin_lock_kfunc(func_id) || is_bpf_arena_kfunc(func_id) || + is_bpf_stream_kfunc(func_id)) + return true; + + err = fetch_kfunc_meta(env, func_id, offset, &kfunc); + if (err || !kfunc.flags) + return false; + + return *kfunc.flags & KF_SPINLOCK_SAFE; } static bool is_sync_callback_calling_kfunc(u32 btf_id) @@ -17420,7 +17430,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state) insn->imm != BPF_FUNC_spin_unlock && insn->imm != BPF_FUNC_kptr_xchg) || (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && - (insn->off != 0 || !kfunc_spin_allowed(insn->imm)))) { + !kfunc_spin_allowed(env, insn->imm, insn->off))) { verbose(env, "function calls are not allowed while holding a lock\n"); return -EINVAL; From 7619a0ee9340b3cef114b1c7aae42c0835cf2bff Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Wed, 5 Aug 2026 23:33:39 +0800 Subject: [PATCH 224/373] bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE The verifier currently keeps a hard-coded list of kfuncs that may be called while holding a bpf_spin_lock. With KF_SPINLOCK_SAFE available, retaining this list creates two sources of truth and requires verifier changes whenever another lock-safe kfunc is added. Mark every kfunc currently accepted by kfunc_spin_allowed() with KF_SPINLOCK_SAFE. This covers the graph, numeric iterator, resource spin lock, arena, and stream kfuncs. Remove the obsolete category checks and make kfunc_spin_allowed() rely solely on the kfunc registration metadata. This preserves the behavior of existing kfuncs while using the same mechanism for built-in and module kfuncs. Signed-off-by: Kaitao Cheng Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260805153340.34776-3-kaitao.cheng@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/arena.c | 6 ++--- kernel/bpf/helpers.c | 56 +++++++++++++++++++++-------------------- kernel/bpf/rqspinlock.c | 8 +++--- kernel/bpf/verifier.c | 32 ----------------------- 4 files changed, 36 insertions(+), 66 deletions(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 555ee2531ef9..7b6847200b43 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -1118,9 +1118,9 @@ __bpf_kfunc int bpf_arena_reserve_pages(void *p__map, void *ptr__ign, u32 page_c __bpf_kfunc_end_defs(); BTF_KFUNCS_START(arena_kfuncs) -BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_ARENA_RET | KF_ARENA_ARG2) -BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2) -BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2) +BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_ARENA_RET | KF_ARENA_ARG2 | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE) BTF_KFUNCS_END(arena_kfuncs) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 4709a5ad0474..6388b6b23e49 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -4812,30 +4812,32 @@ BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE) BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE) -BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU) -BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_list_push_front_impl) -BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_list_push_back_impl) -BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_list_is_first) -BTF_ID_FLAGS(func, bpf_list_is_last) -BTF_ID_FLAGS(func, bpf_list_empty) +BTF_ID_FLAGS(func, bpf_refcount_acquire, + KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, + KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_push_front_impl, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_push_back_impl, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_is_first, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_is_last, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_list_empty, KF_SPINLOCK_SAFE) BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) -BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_rbtree_add_impl) -BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_add_impl, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL | KF_SPINLOCK_SAFE) #ifdef CONFIG_CGROUPS BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) @@ -4885,9 +4887,9 @@ BTF_ID_FLAGS(func, bpf_rcu_read_lock) BTF_ID_FLAGS(func, bpf_rcu_read_unlock) BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL) BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW) -BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY) +BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY | KF_SPINLOCK_SAFE) BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU) BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY) @@ -4962,8 +4964,8 @@ BTF_ID_FLAGS(func, bpf_strncasestr); #if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS) BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU) #endif -BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS) -BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS) +BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE) BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_dynptr_from_file) diff --git a/kernel/bpf/rqspinlock.c b/kernel/bpf/rqspinlock.c index e4e338cdb437..e527cb425cf4 100644 --- a/kernel/bpf/rqspinlock.c +++ b/kernel/bpf/rqspinlock.c @@ -744,10 +744,10 @@ __bpf_kfunc void bpf_res_spin_unlock_irqrestore(struct bpf_res_spin_lock *lock, __bpf_kfunc_end_defs(); BTF_KFUNCS_START(rqspinlock_kfunc_ids) -BTF_ID_FLAGS(func, bpf_res_spin_lock, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_res_spin_unlock) -BTF_ID_FLAGS(func, bpf_res_spin_lock_irqsave, KF_RET_NULL) -BTF_ID_FLAGS(func, bpf_res_spin_unlock_irqrestore) +BTF_ID_FLAGS(func, bpf_res_spin_lock, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_res_spin_unlock, KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_res_spin_lock_irqsave, KF_RET_NULL | KF_SPINLOCK_SAFE) +BTF_ID_FLAGS(func, bpf_res_spin_unlock_irqrestore, KF_SPINLOCK_SAFE) BTF_KFUNCS_END(rqspinlock_kfunc_ids) static const struct btf_kfunc_id_set rqspinlock_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8daba32306be..d952bd95cbb7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11802,20 +11802,6 @@ static bool is_bpf_rbtree_api_kfunc(u32 btf_id) btf_id == special_kfunc_list[KF_bpf_rbtree_right]; } -static bool is_bpf_iter_num_api_kfunc(u32 btf_id) -{ - return btf_id == special_kfunc_list[KF_bpf_iter_num_new] || - btf_id == special_kfunc_list[KF_bpf_iter_num_next] || - btf_id == special_kfunc_list[KF_bpf_iter_num_destroy]; -} - -static bool is_bpf_graph_api_kfunc(u32 btf_id) -{ - return is_bpf_list_api_kfunc(btf_id) || - is_bpf_rbtree_api_kfunc(btf_id) || - is_bpf_refcount_acquire_kfunc(btf_id); -} - static bool is_bpf_res_spin_lock_kfunc(u32 btf_id) { return btf_id == special_kfunc_list[KF_bpf_res_spin_lock] || @@ -11824,29 +11810,11 @@ static bool is_bpf_res_spin_lock_kfunc(u32 btf_id) btf_id == special_kfunc_list[KF_bpf_res_spin_unlock_irqrestore]; } -static bool is_bpf_arena_kfunc(u32 btf_id) -{ - return btf_id == special_kfunc_list[KF_bpf_arena_alloc_pages] || - btf_id == special_kfunc_list[KF_bpf_arena_free_pages] || - btf_id == special_kfunc_list[KF_bpf_arena_reserve_pages]; -} - -static bool is_bpf_stream_kfunc(u32 btf_id) -{ - return btf_id == special_kfunc_list[KF_bpf_stream_vprintk] || - btf_id == special_kfunc_list[KF_bpf_stream_print_stack]; -} - static bool kfunc_spin_allowed(struct bpf_verifier_env *env, s32 func_id, s16 offset) { struct bpf_kfunc_meta kfunc; int err; - if (is_bpf_graph_api_kfunc(func_id) || is_bpf_iter_num_api_kfunc(func_id) || - is_bpf_res_spin_lock_kfunc(func_id) || is_bpf_arena_kfunc(func_id) || - is_bpf_stream_kfunc(func_id)) - return true; - err = fetch_kfunc_meta(env, func_id, offset, &kfunc); if (err || !kfunc.flags) return false; From bca83aa31f15275b8ac4fc3fb7659f5d70492902 Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Wed, 5 Aug 2026 23:33:40 +0800 Subject: [PATCH 225/373] selftests/bpf: Test module kfunc calls under spin lock The verifier uses kfunc registration flags to decide whether a kfunc may be called while a BPF program holds a bpf_spin_lock. Mark bpf_testmod_test_mod_kfunc() as KF_SPINLOCK_SAFE and verify that it can be called while holding a bpf_spin_lock. Also attempt to call the unmarked bpf_kfunc_trigger_ctx_check() under the lock and verify that the program is rejected. Signed-off-by: Kaitao Cheng Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260805153340.34776-4-kaitao.cheng@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/kfunc_call.c | 2 ++ tools/testing/selftests/bpf/progs/kfunc_call_fail.c | 12 ++++++++++++ tools/testing/selftests/bpf/progs/kfunc_call_test.c | 12 ++++++++++++ tools/testing/selftests/bpf/test_kmods/bpf_testmod.c | 2 +- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c index 7af5560f2a08..2b39cc1b09f9 100644 --- a/tools/testing/selftests/bpf/prog_tests/kfunc_call.c +++ b/tools/testing/selftests/bpf/prog_tests/kfunc_call.c @@ -71,8 +71,10 @@ static struct kfunc_test_params kfunc_tests[] = { TC_FAIL(kfunc_call_test_get_mem_fail_not_const, 0, "is not a const"), TC_FAIL(kfunc_call_test_mem_acquire_fail, 0, "acquire kernel function does not return PTR_TO_BTF_ID"), TC_FAIL(kfunc_call_test_pointer_arg_type_mismatch, 0, "R1 expected pointer to ctx, but got scalar"), + TC_FAIL(kfunc_call_test_spin_lock_unsafe, 0, "function calls are not allowed while holding a lock"), /* success cases */ + TC_TEST(kfunc_call_test_spin_lock_safe, 0), TC_TEST(kfunc_call_test1, 12), TC_TEST(kfunc_call_test2, 3), TC_TEST(kfunc_call_test4, -1234), diff --git a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c index 64b6a0b0ab1c..7e93f7fb1329 100644 --- a/tools/testing/selftests/bpf/progs/kfunc_call_fail.c +++ b/tools/testing/selftests/bpf/progs/kfunc_call_fail.c @@ -4,6 +4,18 @@ #include #include "../test_kmods/bpf_testmod_kfunc.h" +static struct bpf_spin_lock kfunc_call_lock SEC(".data.A"); + +SEC("?tc") +int kfunc_call_test_spin_lock_unsafe(struct __sk_buff *skb) +{ + bpf_spin_lock(&kfunc_call_lock); + bpf_kfunc_trigger_ctx_check(); + bpf_spin_unlock(&kfunc_call_lock); + + return 0; +} + struct syscall_test_args { __u8 data[16]; size_t size; diff --git a/tools/testing/selftests/bpf/progs/kfunc_call_test.c b/tools/testing/selftests/bpf/progs/kfunc_call_test.c index 5edc51564f71..8e6560c31e78 100644 --- a/tools/testing/selftests/bpf/progs/kfunc_call_test.c +++ b/tools/testing/selftests/bpf/progs/kfunc_call_test.c @@ -5,6 +5,18 @@ #include "bpf_misc.h" #include "../test_kmods/bpf_testmod_kfunc.h" +static struct bpf_spin_lock kfunc_call_lock SEC(".data.A"); + +SEC("tc") +int kfunc_call_test_spin_lock_safe(struct __sk_buff *skb) +{ + bpf_spin_lock(&kfunc_call_lock); + bpf_testmod_test_mod_kfunc(42); + bpf_spin_unlock(&kfunc_call_lock); + + return 0; +} + SEC("tc") int kfunc_call_test5(struct __sk_buff *skb) { diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index eb0f9b5e18d8..0585794606ed 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -1384,7 +1384,7 @@ __bpf_kfunc void bpf_kfunc_trigger_ctx_check(void) } BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) -BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) +BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc, KF_SPINLOCK_SAFE) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) From d65739bf93be5160c1e0af00064874bbe262d5b6 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Wed, 5 Aug 2026 16:39:33 -0700 Subject: [PATCH 226/373] bpf: Account for preempt and IRQ state in RCU protection Disabling preemption or local IRQs keeps the current CPU in an RCU read-side critical section, but in_rcu_cs() does not account for either state. The verifier therefore rejects safe kptr accesses and invalidates pointers when another RCU source ends. Include preemption-disabled and IRQ-disabled state in in_rcu_cs(). Invalidate RCU-protected pointers on RCU unlock, preempt enable, or IRQ restore only after the final protection ends. Signed-off-by: Ning Ding Link: https://lore.kernel.org/bpf/20260805233940.3966981-2-dingning04@gmail.com [ kkd: Simplify was_in_rcu_cs on spin unlock and adjust the selftest. ] Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 14 ++++++++++---- .../testing/selftests/bpf/progs/cpumask_failure.c | 8 ++++++-- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d952bd95cbb7..e6233c0081d1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4452,7 +4452,9 @@ static bool in_sleepable(struct bpf_verifier_env *env) static bool in_rcu_cs(struct bpf_verifier_env *env) { return env->cur_state->active_rcu_locks || + env->cur_state->active_preempt_locks || env->cur_state->active_locks || + env->cur_state->active_irq_id || !in_sleepable(env); } @@ -7166,7 +7168,6 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state return err; } } else { - bool was_in_rcu_cs; void *ptr; int type; @@ -7194,12 +7195,11 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state verbose(env, "%s_unlock cannot be out of order\n", lock_str); return -EINVAL; } - was_in_rcu_cs = in_rcu_cs(env); if (release_lock_state(cur, type, reg->id, ptr)) { verbose(env, "%s_unlock of different lock\n", lock_str); return -EINVAL; } - if (was_in_rcu_cs && !in_rcu_cs(env)) + if (!in_rcu_cs(env)) invalidate_rcu_protected_refs(env); invalidate_non_owning_refs(env); @@ -11663,6 +11663,9 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state * err = unmark_stack_slot_irq_flag(env, reg, kfunc_class); if (err) return err; + + if (!in_rcu_cs(env)) + invalidate_rcu_protected_refs(env); } return 0; } @@ -13159,7 +13162,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, verbose(env, "unmatched rcu read unlock (kernel function %s)\n", func_name); return -EINVAL; } - if (--env->cur_state->active_rcu_locks == 0) + env->cur_state->active_rcu_locks--; + if (!in_rcu_cs(env)) invalidate_rcu_protected_refs(env); } else if (preempt_disable) { env->cur_state->active_preempt_locks++; @@ -13169,6 +13173,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EINVAL; } env->cur_state->active_preempt_locks--; + if (!in_rcu_cs(env)) + invalidate_rcu_protected_refs(env); } if (sleepable && !in_sleepable_context(env)) { diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c index 74b4cd4bcdbb..4628feb53d86 100644 --- a/tools/testing/selftests/bpf/progs/cpumask_failure.c +++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c @@ -116,9 +116,9 @@ int BPF_PROG(test_cpumask_null, struct task_struct *task, u64 clone_flags) return 0; } -SEC("tp_btf/task_newtask") +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") __failure __msg("R2 must be a rcu pointer") -int BPF_PROG(test_global_mask_out_of_rcu, struct task_struct *task, u64 clone_flags) +int BPF_PROG(test_global_mask_out_of_rcu) { struct bpf_cpumask *local, *prev; @@ -133,6 +133,10 @@ int BPF_PROG(test_global_mask_out_of_rcu, struct task_struct *task, u64 clone_fl return 0; } + /* + * Use a sleepable program so explicit RCU is the only source of RCU + * protection. + */ bpf_rcu_read_lock(); local = global_mask; if (!local) { From a7f62506df941a506a138caa3849f48c11af22ec Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Wed, 5 Aug 2026 16:39:34 -0700 Subject: [PATCH 227/373] selftests/bpf: Test overlapping RCU protection Add task kptr tests that keep RCU protection active after a spin or RCU unlock when preemption or IRQs remain disabled. Also test the reverse order with explicit RCU. Verify that task kptrs are rejected after leaving the final preemption-disabled or IRQ-disabled region. Signed-off-by: Ning Ding Link: https://lore.kernel.org/bpf/20260805233940.3966981-3-dingning04@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/task_kfunc.c | 6 + .../selftests/bpf/progs/task_kfunc_common.h | 2 + .../selftests/bpf/progs/task_kfunc_failure.c | 49 ++++++ .../selftests/bpf/progs/task_kfunc_success.c | 147 ++++++++++++++++++ 4 files changed, 204 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/task_kfunc.c b/tools/testing/selftests/bpf/prog_tests/task_kfunc.c index fbd7855712c1..30d403028f98 100644 --- a/tools/testing/selftests/bpf/prog_tests/task_kfunc.c +++ b/tools/testing/selftests/bpf/prog_tests/task_kfunc.c @@ -178,6 +178,12 @@ static const char * const success_tests[] = { "task_kfunc_acquire_trusted_walked", "task_kfunc_acquire_after_spin_unlock_non_sleepable", "task_kfunc_acquire_after_spin_unlock_explicit_rcu", + "task_kfunc_acquire_after_spin_unlock_preempt_disabled", + "task_kfunc_acquire_after_spin_unlock_irq_disabled", + "task_kfunc_acquire_after_rcu_unlock_preempt_disabled", + "task_kfunc_acquire_after_rcu_unlock_irq_disabled", + "task_kfunc_acquire_after_preempt_enable_explicit_rcu", + "task_kfunc_acquire_after_irq_restore_explicit_rcu", "test_task_kfunc_flavor_relo", "test_task_kfunc_flavor_relo_not_found", }; diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_common.h b/tools/testing/selftests/bpf/progs/task_kfunc_common.h index 052c9d0e3e2a..a0c599b58c29 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_common.h +++ b/tools/testing/selftests/bpf/progs/task_kfunc_common.h @@ -38,6 +38,8 @@ struct task_struct *bpf_task_from_pid(s32 pid) __ksym; struct task_struct *bpf_task_from_vpid(s32 vpid) __ksym; void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym; +void bpf_local_irq_save(unsigned long *flags) __weak __ksym; +void bpf_local_irq_restore(unsigned long *flags) __weak __ksym; static inline struct __tasks_kfunc_map_value *tasks_kfunc_map_value_lookup(struct task_struct *p) { diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c index c0e7216b3419..f96b0c13ed1a 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_failure.c +++ b/tools/testing/selftests/bpf/progs/task_kfunc_failure.c @@ -402,3 +402,52 @@ int BPF_PROG(task_kfunc_acquire_after_final_spin_unlock) bpf_task_release(acquired); return 0; } + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("R1 must be a rcu pointer") +int BPF_PROG(task_kfunc_acquire_after_preempt_enable) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_preempt_disable(); + task = v->task; + bpf_preempt_enable(); + if (!task) + return 0; + + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + return 0; +} + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("R1 must be a rcu pointer") +int BPF_PROG(task_kfunc_acquire_after_irq_restore) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + unsigned long flags; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_local_irq_save(&flags); + task = v->task; + bpf_local_irq_restore(&flags); + if (!task) + return 0; + + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/task_kfunc_success.c b/tools/testing/selftests/bpf/progs/task_kfunc_success.c index 2bab7634c9df..6545b124dee1 100644 --- a/tools/testing/selftests/bpf/progs/task_kfunc_success.c +++ b/tools/testing/selftests/bpf/progs/task_kfunc_success.c @@ -414,6 +414,153 @@ int BPF_PROG(task_kfunc_acquire_after_spin_unlock_explicit_rcu) return 0; } +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_spin_unlock_preempt_disabled) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_preempt_disable(); + bpf_spin_lock(&v->lock); + task = v->task; + bpf_spin_unlock(&v->lock); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_preempt_enable(); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_spin_unlock_irq_disabled) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + unsigned long flags; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_local_irq_save(&flags); + bpf_spin_lock(&v->lock); + task = v->task; + bpf_spin_unlock(&v->lock); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_local_irq_restore(&flags); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_rcu_unlock_preempt_disabled) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_preempt_disable(); + bpf_rcu_read_lock(); + task = v->task; + bpf_rcu_read_unlock(); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_preempt_enable(); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_rcu_unlock_irq_disabled) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + unsigned long flags; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_local_irq_save(&flags); + bpf_rcu_read_lock(); + task = v->task; + bpf_rcu_read_unlock(); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_local_irq_restore(&flags); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_preempt_enable_explicit_rcu) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_preempt_disable(); + task = v->task; + bpf_rcu_read_lock(); + bpf_preempt_enable(); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_rcu_read_unlock(); + return 0; +} + +SEC("fentry.s/" SYS_PREFIX "sys_getpgid") +int BPF_PROG(task_kfunc_acquire_after_irq_restore_explicit_rcu) +{ + struct task_kptr_lock_value *v; + struct task_struct *task, *acquired; + unsigned long flags; + int key = 0; + + v = bpf_map_lookup_elem(&task_kptr_lock_map, &key); + if (!v) + return 0; + + bpf_local_irq_save(&flags); + task = v->task; + bpf_rcu_read_lock(); + bpf_local_irq_restore(&flags); + if (task) { + acquired = bpf_task_acquire(task); + if (acquired) + bpf_task_release(acquired); + } + bpf_rcu_read_unlock(); + return 0; +} + SEC("syscall") int test_task_from_vpid_current(const void *ctx) { From 3d72aca40b83040aae08c3a3a3dfc5a42c26abe9 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:24 -0700 Subject: [PATCH 228/373] resolve_btfids: Deduplicate BTF after btf2btf transformations btf2btf() adds new types to the BTF: the KF_IMPLICIT_ARGS transform synthesizes an _impl FUNC together with its FUNC_PROTO and copies of the kfunc's decl tags. Nothing deduplicates them afterwards. pahole runs btf__dedup() on its own output, but that happens before resolve_btfids sees the BTF, so any type the tool itself creates is emitted as-is, even when a structurally identical type is already present. Call btf__dedup() at the start of finalize_btf(), so that base distillation and the by-name sort both operate on the canonical set of types. On an x86_64 build with the BPF selftests config this removes 17 duplicate FUNC_PROTOs from vmlinux BTF. The dedup call increases runtime of resolve_btfids on vmlinux by 30-40%. The performance hit is an acceptable cost to keep kernel BTF deduped [1]. [1] https://lore.kernel.org/bpf/986e6f4e-4b51-4440-a37c-9624906d7370@linux.dev/ Signed-off-by: Ihor Solodrai Link: https://patch.msgid.link/20260807032029.78092-2-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- tools/bpf/resolve_btfids/main.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 85488935909d..5d168c2a5ff5 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -1379,6 +1379,12 @@ static int finalize_btf(struct object *obj) struct btf *base_btf = obj->base_btf, *btf = obj->btf; int err; + err = btf__dedup(obj->btf, NULL); + if (err) { + pr_err("FAILED to dedup BTF: %s\n", strerror(errno)); + goto out_err; + } + if (obj->base_btf && obj->distill_base) { err = btf__distill_base(obj->btf, &base_btf, &btf); if (err) { From 27a78c2e7eeea8397ad3c28171084e695d4c12af Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:25 -0700 Subject: [PATCH 229/373] resolve_btfids: Process KF_ARENA_* flags in resolve_btfids For kfuncs flagged with KF_ARENA_RET, KF_ARENA_ARG1 or KF_ARENA_ARG2, the address_space(1) attribute (a type tag with kflag=1) must be emitted for the corresponding type in BTF. This was previously done by pahole via the "attributes" BTF feature [1]. Implement the emission of the arena attributes in resolve_btfids: for flagged kfuncs create a new function prototype with updated BTF types. The original proto may be shared with sibling FUNCs, so it is not modified in place. Emission is unconditional: kbuild controls the pahole flags, so the input BTF is expected to not have these attributes. Invalid declarations are reported as errors. Drop the "attributes" pahole feature from scripts/Makefile.btf resolve_btfids now emits them for all supported pahole versions. [1] https://lore.kernel.org/dwarves/20250228194654.1022535-1-ihor.solodrai@linux.dev/ Signed-off-by: Ihor Solodrai Link: https://patch.msgid.link/20260807032029.78092-3-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- scripts/Makefile.btf | 2 - tools/bpf/resolve_btfids/main.c | 139 ++++++++++++++++++++++++++++++-- 2 files changed, 134 insertions(+), 7 deletions(-) diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf index e66e13e79653..8f73c093d27a 100644 --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -16,8 +16,6 @@ else # Switch to using --btf_features for v1.26 and later. pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs -pahole-flags-$(call test-ge, $(pahole-ver), 130) += --btf_features=attributes - pahole-flags-$(call test-ge, $(pahole-ver), 131) += --btf_features=layout endif diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 5d168c2a5ff5..53d9045ed5c8 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -161,8 +161,12 @@ struct object { u32 addr_syms_cap; }; +#define KF_ARENA_RET (1 << 13) +#define KF_ARENA_ARG1 (1 << 14) +#define KF_ARENA_ARG2 (1 << 15) #define KF_IMPLICIT_ARGS (1 << 16) #define KF_IMPL_SUFFIX "_impl" +#define TYPE_ATTR_ARENA "address_space(1)" struct kfunc { struct rb_node rb_node; @@ -1280,6 +1284,126 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct return 0; } +static bool is_arena_arg(struct kfunc *kfunc, u32 idx) +{ + switch (idx) { + case 0: + return kfunc->flags & KF_ARENA_ARG1; + case 1: + return kfunc->flags & KF_ARENA_ARG2; + default: + return false; + } +} + +static s32 arena_tag_ptr(struct btf *btf, u32 ptr_id, struct kfunc *kfunc) +{ + const struct btf_type *ptr = btf__type_by_id(btf, ptr_id); + s32 tag_id, new_ptr_id; + + if (!btf_is_ptr(ptr)) { + pr_err("ERROR: resolve_btfids: kfunc %s: arena type is not a pointer\n", + kfunc->name); + return -EINVAL; + } + + tag_id = btf__add_type_attr(btf, TYPE_ATTR_ARENA, ptr->type); + if (tag_id < 0) { + pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a type attr to BTF: %d\n", + kfunc->name, tag_id); + return tag_id; + } + + new_ptr_id = btf__add_ptr(btf, tag_id); + if (new_ptr_id < 0) { + pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a pointer to BTF: %d\n", + kfunc->name, new_ptr_id); + } + + return new_ptr_id; +} + +/* + * Add a FUNC_PROTO for @kfunc with each relevant pointer tagged with + * an "address_space(1)" attribute. The original proto may be shared + * with other FUNCs, so it is never modified in place. + */ +static s32 add_arena_tagged_proto(struct btf *btf, struct kfunc *kfunc) +{ + const struct btf_type *func = btf__type_by_id(btf, kfunc->btf_id); + u32 proto_id = func->type; + const struct btf_type *proto = btf__type_by_id(btf, proto_id); + u32 nr_params = btf_vlen(proto); + s32 ret_type_id = proto->type; + const struct btf_type *t; + struct btf_param *params; + s32 new_proto_id, id; + const char *name; + int err, i; + + if (kfunc->flags & KF_ARENA_RET) { + ret_type_id = arena_tag_ptr(btf, ret_type_id, kfunc); + if (ret_type_id < 0) + return ret_type_id; + } + + new_proto_id = btf__add_func_proto(btf, ret_type_id); + if (new_proto_id < 0) { + pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a func proto to BTF: %d\n", + kfunc->name, new_proto_id); + return new_proto_id; + } + + for (i = 0; i < nr_params; i++) { + /* btf__add_func_param() below may move the proto, re-fetch */ + proto = btf__type_by_id(btf, proto_id); + name = btf__name_by_offset(btf, btf_params(proto)[i].name_off); + + err = btf__add_func_param(btf, name ?: "", btf_params(proto)[i].type); + if (err < 0) { + pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a proto param to BTF: %d\n", + kfunc->name, err); + return err; + } + } + + for (i = 0; i < nr_params; i++) { + if (!is_arena_arg(kfunc, i)) + continue; + + t = btf__type_by_id(btf, new_proto_id); + params = btf_params(t); + + id = arena_tag_ptr(btf, params[i].type, kfunc); + if (id < 0) + return id; + + t = btf__type_by_id(btf, new_proto_id); + params = btf_params(t); + params[i].type = id; + } + + pr_debug("added arena-tagged proto for kfunc %s: %d\n", kfunc->name, new_proto_id); + + return new_proto_id; +} + +static int process_kfunc_with_arena_flags(struct btf2btf_context *ctx, + struct kfunc *kfunc) +{ + struct btf_type *t; + s32 proto_id; + + proto_id = add_arena_tagged_proto(ctx->btf, kfunc); + if (proto_id < 0) + return proto_id; + + t = (struct btf_type *)btf__type_by_id(ctx->btf, kfunc->btf_id); + t->type = proto_id; + + return 0; +} + static int btf2btf(struct object *obj) { struct btf2btf_context ctx = {}; @@ -1293,12 +1417,17 @@ static int btf2btf(struct object *obj) for (next = rb_first(&ctx.kfuncs); next; next = rb_next(next)) { struct kfunc *kfunc = rb_entry(next, struct kfunc, rb_node); - if (!(kfunc->flags & KF_IMPLICIT_ARGS)) - continue; + if (kfunc->flags & KF_IMPLICIT_ARGS) { + err = process_kfunc_with_implicit_args(&ctx, kfunc); + if (err) + goto out; + } - err = process_kfunc_with_implicit_args(&ctx, kfunc); - if (err) - goto out; + if (kfunc->flags & (KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2)) { + err = process_kfunc_with_arena_flags(&ctx, kfunc); + if (err) + goto out; + } } err = 0; From ef77140f4b3ad1802ef6d84a2f803d1171a086ae Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:26 -0700 Subject: [PATCH 230/373] selftests/bpf: Verify arena type tags in resolve_btfids test Extend test_resolve_btfids() to assert that resolve_btfids emits the address_space(1) type attribute (a BTF_KIND_TYPE_TAG with kflag=1) on the return type and/or arguments of kfuncs marked KF_ARENA_RET, KF_ARENA_ARG1 or KF_ARENA_ARG2. Signed-off-by: Ihor Solodrai Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260807032029.78092-4-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/resolve_btfids.c | 66 +++++++++++++++++++ tools/testing/selftests/bpf/progs/btf_data.c | 10 +++ 2 files changed, 76 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index ac51fd454821..8482f00046d4 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -12,9 +12,20 @@ #define BTF_DATA_FILE "resolve_btfids.test.o.BTF" +#define TYPE_ATTR_ARENA "address_space(1)" + #ifndef KF_FASTCALL #define KF_FASTCALL (1 << 12) #endif +#ifndef KF_ARENA_RET +#define KF_ARENA_RET (1 << 13) +#endif +#ifndef KF_ARENA_ARG1 +#define KF_ARENA_ARG1 (1 << 14) +#endif +#ifndef KF_ARENA_ARG2 +#define KF_ARENA_ARG2 (1 << 15) +#endif struct symbol { const char *name; @@ -41,6 +52,8 @@ struct kfunc_symbol { static struct kfunc_symbol kfunc_symbols[] = { { "kfunc_a", -1, 0 }, { "kfunc_b", -1, KF_FASTCALL }, + { "kfunc_c", -1, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2 }, + { "kfunc_d", -1, KF_ARENA_ARG2 }, }; /* Align the .BTF_ids section to 4 bytes */ @@ -88,6 +101,8 @@ BTF_SET_END(test_set) BTF_KFUNCS_START(test_kfunc_set) BTF_ID_FLAGS(func, kfunc_a) BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) +BTF_ID_FLAGS(func, kfunc_c, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2) +BTF_ID_FLAGS(func, kfunc_d, KF_ARENA_ARG2) BTF_KFUNCS_END(test_kfunc_set) /* @@ -95,6 +110,8 @@ BTF_KFUNCS_END(test_kfunc_set) * actually sort at least one of the two sets. */ BTF_KFUNCS_START(test_kfunc_set_rev) +BTF_ID_FLAGS(func, kfunc_d, KF_ARENA_ARG2) +BTF_ID_FLAGS(func, kfunc_c, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2) BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) BTF_ID_FLAGS(func, kfunc_a) BTF_KFUNCS_END(test_kfunc_set_rev) @@ -184,6 +201,22 @@ static void check_kfunc_set(struct btf_id_set8 *set) } } +/* True if @id is PTR -> TYPE_TAG(kflag=1, "address_space(1)") -> pointee */ +static bool is_arena_tagged_ptr(struct btf *btf, __u32 id) +{ + const struct btf_type *ptr, *tag; + const char *name; + + ptr = btf__type_by_id(btf, id); + if (!btf_is_ptr(ptr)) + return false; + tag = btf__type_by_id(btf, ptr->type); + if (!btf_is_type_tag(tag) || !btf_kflag(tag)) + return false; + name = btf__name_by_offset(btf, tag->name_off); + return strcmp(name, TYPE_ATTR_ARENA) == 0; +} + void test_resolve_btfids(void) { __u32 *test_list, *test_lists[] = { test_list_local, test_list_global }; @@ -227,6 +260,39 @@ void test_resolve_btfids(void) check_kfunc_set(&test_kfunc_set); check_kfunc_set(&test_kfunc_set_rev); + /* + * Check resolve_btfids wrapped exactly the arena-flagged return/args + * with the address_space(1) type attribute, and left other + * pointers/returns untouched. + */ + for (i = 0; i < ARRAY_SIZE(kfunc_symbols); i++) { + const struct btf_type *fn, *proto; + const struct btf_param *params; + const char *name = kfunc_symbols[i].name; + u32 fl = kfunc_symbols[i].flags; + __u32 nr; + + fn = btf__type_by_id(btf, kfunc_symbols[i].id); + if (!ASSERT_TRUE(btf_is_func(fn), name)) + continue; + proto = btf__type_by_id(btf, fn->type); + if (!ASSERT_TRUE(btf_is_func_proto(proto), name)) + continue; + params = btf_params(proto); + nr = btf_vlen(proto); + + ASSERT_EQ(is_arena_tagged_ptr(btf, proto->type), + !!(fl & KF_ARENA_RET), name); + if (nr > 0) { + ASSERT_EQ(is_arena_tagged_ptr(btf, params[0].type), + !!(fl & KF_ARENA_ARG1), name); + } + if (nr > 1) { + ASSERT_EQ(is_arena_tagged_ptr(btf, params[1].type), + !!(fl & KF_ARENA_ARG2), name); + } + } + out: btf__free(btf); } diff --git a/tools/testing/selftests/bpf/progs/btf_data.c b/tools/testing/selftests/bpf/progs/btf_data.c index 8587658012c3..ec34f7a6e038 100644 --- a/tools/testing/selftests/bpf/progs/btf_data.c +++ b/tools/testing/selftests/bpf/progs/btf_data.c @@ -58,3 +58,13 @@ int kfunc_b(struct root_struct *root) { return 0; } + +struct root_struct *kfunc_c(struct root_struct *a, struct root_struct *b) +{ + return a; +} + +int kfunc_d(struct root_struct *a, struct root_struct *b) +{ + return 0; +} From 692393104909ee14686488995ad5009a618fb0c7 Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:27 -0700 Subject: [PATCH 231/373] resolve_btfids: Emit bpf_kfunc and bpf_fastcall decl tags Emit the bpf_kfunc decl tag for every discovered kfunc, and bpf_fastcall for kfuncs flagged KF_FASTCALL. These were previously produced by pahole under --btf_features=decl_tag_kfuncs. resolve_btfids now discovers kfuncs from the BTF ID sets [1] and becomes the source of truth for their annotations. Drop decl_tag_kfuncs pahole feature flag from scripts/Makefile.btf [1] https://lore.kernel.org/all/20260722233518.778854-1-ihor.solodrai@linux.dev/ Signed-off-by: Ihor Solodrai Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260807032029.78092-5-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- scripts/Makefile.btf | 2 +- tools/bpf/resolve_btfids/main.c | 31 ++++++++++++++++++++++++++++++- 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf index 8f73c093d27a..a1812985a61a 100644 --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -14,7 +14,7 @@ pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsis else # Switch to using --btf_features for v1.26 and later. -pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs +pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func pahole-flags-$(call test-ge, $(pahole-ver), 131) += --btf_features=layout diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index 53d9045ed5c8..a1b921e86ef0 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -161,6 +161,10 @@ struct object { u32 addr_syms_cap; }; +#define DECL_TAG_FASTCALL "bpf_fastcall" +#define DECL_TAG_KFUNC "bpf_kfunc" + +#define KF_FASTCALL (1 << 12) #define KF_ARENA_RET (1 << 13) #define KF_ARENA_ARG1 (1 << 14) #define KF_ARENA_ARG2 (1 << 15) @@ -1233,7 +1237,7 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct continue; tag_name = btf__name_by_offset(btf, t->name_off); - if (strcmp(tag_name, "bpf_kfunc") == 0) + if (strcmp(tag_name, DECL_TAG_KFUNC) == 0) continue; idx = btf_decl_tag(t)->component_idx; @@ -1404,6 +1408,21 @@ static int process_kfunc_with_arena_flags(struct btf2btf_context *ctx, return 0; } +static int add_decl_tag(struct btf2btf_context *ctx, const char *tag_name, + u32 target_btf_id, int component_idx) +{ + s32 new_id; + + new_id = btf__add_decl_tag(ctx->btf, tag_name, target_btf_id, component_idx); + if (new_id < 0) { + pr_err("ERROR: resolve_btfids: failed to add '%s' decl tag for BTF id %u: %d\n", + tag_name, target_btf_id, new_id); + return new_id; + } + + return push_decl_tag_id(ctx, new_id); +} + static int btf2btf(struct object *obj) { struct btf2btf_context ctx = {}; @@ -1417,6 +1436,16 @@ static int btf2btf(struct object *obj) for (next = rb_first(&ctx.kfuncs); next; next = rb_next(next)) { struct kfunc *kfunc = rb_entry(next, struct kfunc, rb_node); + err = add_decl_tag(&ctx, DECL_TAG_KFUNC, kfunc->btf_id, -1); + if (err) + goto out; + + if (kfunc->flags & KF_FASTCALL) { + err = add_decl_tag(&ctx, DECL_TAG_FASTCALL, kfunc->btf_id, -1); + if (err) + goto out; + } + if (kfunc->flags & KF_IMPLICIT_ARGS) { err = process_kfunc_with_implicit_args(&ctx, kfunc); if (err) From 9f3881ecad886bbcd7b69355bc14a6deb071672c Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:28 -0700 Subject: [PATCH 232/373] selftests/bpf: Verify decl tags emission in resolve_btfids test Extend test_resolve_btfids() to assert that resolve_btfids emits a BTF_KIND_DECL_TAG named "bpf_kfunc" for every kfunc, and "bpf_fastcall" for kfuncs marked KF_FASTCALL. Add a btf_has_decl_tag() helper that scans the output BTF for a decl tag matching name and target. Signed-off-by: Ihor Solodrai Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260807032029.78092-6-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/resolve_btfids.c | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index 8482f00046d4..732cfed35e1c 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -12,6 +12,8 @@ #define BTF_DATA_FILE "resolve_btfids.test.o.BTF" +#define DECL_TAG_FASTCALL "bpf_fastcall" +#define DECL_TAG_KFUNC "bpf_kfunc" #define TYPE_ATTR_ARENA "address_space(1)" #ifndef KF_FASTCALL @@ -176,6 +178,28 @@ static int resolve_symbols(struct btf *btf) return 0; } +static bool btf_has_decl_tag(struct btf *btf, const char *tag_name, s32 target_id) +{ + const struct btf_type *t; + const char *name; + int nr, id; + + nr = btf__type_cnt(btf); + for (id = 1; id < nr; id++) { + t = btf__type_by_id(btf, id); + if (!btf_is_decl_tag(t)) + continue; + if (t->type != (__u32)target_id) + continue; + if (btf_decl_tag(t)->component_idx != -1) + continue; + name = btf__name_by_offset(btf, t->name_off); + if (strcmp(name, tag_name) == 0) + return true; + } + return false; +} + static void check_kfunc_set(struct btf_id_set8 *set) { unsigned int i, j; @@ -260,6 +284,22 @@ void test_resolve_btfids(void) check_kfunc_set(&test_kfunc_set); check_kfunc_set(&test_kfunc_set_rev); + /* Check resolve_btfids emitted a bpf_kfunc decl_tag for each kfunc */ + for (i = 0; i < ARRAY_SIZE(kfunc_symbols); i++) { + ASSERT_TRUE(btf_has_decl_tag(btf, DECL_TAG_KFUNC, + kfunc_symbols[i].id), + kfunc_symbols[i].name); + } + + /* Check resolve_btfids emitted bpf_fastcall for KF_FASTCALL kfuncs */ + for (i = 0; i < ARRAY_SIZE(kfunc_symbols); i++) { + if (kfunc_symbols[i].flags & KF_FASTCALL) { + ASSERT_TRUE(btf_has_decl_tag(btf, DECL_TAG_FASTCALL, + kfunc_symbols[i].id), + kfunc_symbols[i].name); + } + } + /* * Check resolve_btfids wrapped exactly the arena-flagged return/args * with the address_space(1) type attribute, and left other From fd5425b67355da4972c76b4ca266f05515172a9b Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Thu, 6 Aug 2026 20:20:29 -0700 Subject: [PATCH 233/373] docs, resolve_btfids: Document kfunc BTF annotation emission resolve_btfids now emits the bpf_kfunc and bpf_fastcall BTF decl tags and the arena address_space(1) type attribute for kfuncs, which were previously produced by pahole. Reflect this in the in-tree comments and documentation. Signed-off-by: Ihor Solodrai Reviewed-by: Emil Tsalapatis Link: https://patch.msgid.link/20260807032029.78092-7-ihor.solodrai@linux.dev Signed-off-by: Eduard Zingerman --- Documentation/bpf/kfuncs.rst | 7 +++++++ Documentation/process/changes.rst | 5 ----- tools/bpf/resolve_btfids/main.c | 11 +++++++++++ 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index cbde86d082cc..021be6d93dfb 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -472,6 +472,13 @@ type. An example is shown below:: } late_initcall(init_subsystem); +At kernel build time the ``resolve_btfids`` tool finds all kfuncs declared with +``BTF_KFUNCS_START()`` and emits their BTF annotations into the kernel's BTF. +For each kfunc it emits a ``bpf_kfunc`` BTF decl tag, a ``bpf_fastcall`` decl +tag when the kfunc is flagged ``KF_FASTCALL``, and the ``address_space(1)`` type +attribute on the return value and/or arguments flagged ``KF_ARENA_RET``, +``KF_ARENA_ARG1`` or ``KF_ARENA_ARG2`` (see section 2.8). + 2.7 Specifying no-cast aliases with ___init -------------------------------------------- diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst index 1ca8c5f73ad0..0aa232b117b5 100644 --- a/Documentation/process/changes.rst +++ b/Documentation/process/changes.rst @@ -147,11 +147,6 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel modules as well. This requires pahole v1.22 or later. -Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26 -or later. Without it, such kfuncs will have incorrect BTF prototypes in -vmlinux, causing BPF programs to fail to load with a "func_proto incompatible -with vmlinux" error. Many sched_ext kfuncs are affected. - It is found in the 'dwarves' or 'pahole' distro packages or from https://fedorapeople.org/~acme/dwarves/. diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index a1b921e86ef0..d2e4176339da 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -58,6 +58,17 @@ * __BTF_ID__func__vfs_fallocate__5: * .zero 4 * .word (1 << 3) | (1 << 1) | (1 << 2) + * + * In addition to resolving BTF IDs, resolve_btfids performs kernel-specific + * BTF-to-BTF transformations for kfuncs found in BTF_SET8_KFUNCS sets. For + * each such kfunc it: + * + * - emits a "bpf_kfunc" decl tag, and "bpf_fastcall" when KF_FASTCALL is set; + * - wraps the return value and/or arguments flagged KF_ARENA_RET, + * KF_ARENA_ARG1 or KF_ARENA_ARG2 with the "address_space(1)" type attribute; + * - rewrites the prototype of KF_IMPLICIT_ARGS kfuncs. + * + * These kfunc annotations were historically produced by pahole. */ #define _GNU_SOURCE From 7db0a00445f1a40bacfe9b747405c11cb5f10fc9 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:42 +0200 Subject: [PATCH 234/373] bpf: Reject load-acquire from pointers requiring fault protection A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier, unlike a regular BPF_LDX, so the JIT emits a plain load with no exception table entry and a fault panics the kernel instead of being handled. Reject the source pointer types that a BPF_LDX would have had that fault protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED, PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED. This is reachable e.g. by loading ->mm out of a trusted task_struct yields an untrusted pointer to mm_struct, and it is NULL for a kernel thread: [...] SEC("tp_btf/sched_switch") int BPF_PROG(demo, bool preempt, struct task_struct *prev, struct task_struct *next) { struct mm_struct *mm = next->mm; /* untrusted */ out_ldx = (__u64)mm->pgd; /* BPF_LDX */ out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */ return 0; } [...] Both dereference the same pointer, but only the BPF_LDX is protected (x86-64 JIT, jump targets shown prog-relative): [...] ; out_ldx = (__u64)mm->pgd; 17: movq $-10485760, %r10 1e: movq %rsi, %r11 21: addq $184, %r11 28: subq %r10, %r11 2b: movabsq $140737498841088, %r10 35: cmpq %r10, %r11 38: ja 0x3e <-- kernel addr? 3a: xorl %edi, %edi <-- no: dst = 0, skip the load 3c: jmp 0x45 3e: movq 184(%rsi), %rdi <-- yes: load + extable entry [...] ; load_acquire(&mm->pgd) 53: movq %rsi, %rdi 56: movq 184(%rdi), %rax <-- no check, no extable entry [...] Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as bpf_insn_prepare_dump() rewrites it back to BPF_MEM. A PTR_TRUSTED pointer is deliberately not on the list. Such a load is not converted either, but it does not need to be, since the pointer is guaranteed live, so load-acquire from it stays allowed. The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release error messages are unchanged; writes (RMW / store-release) to such pointers are already rejected elsewhere, so only load-acquire needs this. Fixes: 880442305a39 ("bpf: Introduce load-acquire and store-release instructions") Reported-by: STAR Labs SG Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e6233c0081d1..648c5784178e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4923,6 +4923,30 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno) return reg->type == PTR_TO_ARENA; } +static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno, + struct bpf_insn *insn) +{ + const struct bpf_reg_state *reg = reg_state(env, regno); + + /* + * A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the + * verifier, unlike a regular BPF_LDX. The JIT would emit a plain load + * with no exception table entry, so a fault (e.g. NULL deref) crashes + * the kernel instead of being handled. + * + * Reject the source pointer types that a BPF_LDX would have had that + * fault protection applied to, i.e. the ones bpf_convert_ctx_accesses() + * turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID and any PTR_UNTRUSTED + * pointer (untrusted btf ids, untrusted MEM_ALLOC, rdonly untrusted + * memory). A PTR_TRUSTED pointer is not among them, is not converted, + * and stays allowed. Same for the other flagged PTR_TO_BTF_ID variants + * (MEM_ALLOC, MEM_RCU, ...), hence the exact match on the base type. + */ + return insn->imm == BPF_LOAD_ACQ && + (reg->type == PTR_TO_BTF_ID || + (type_flag(reg->type) & PTR_UNTRUSTED)); +} + /* Return false if @regno contains a pointer whose type isn't supported for * atomic instruction @insn. */ @@ -4939,7 +4963,8 @@ static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno, return false; if (is_arena_reg(env, regno)) return bpf_jit_supports_insn(insn, true); - + if (is_load_acq_unsafe(env, regno, insn)) + return false; return true; } From e2577cd62060be91a3d7d11a56e5a61faae4b7f7 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:43 +0200 Subject: [PATCH 235/373] bpf, riscv: Add and use bpf_atomic_is_load_acq() helper A load-acquire is the only BPF_STX class instruction that reads from src_reg into dst_reg, that is, it has the operand roles of a BPF_LDX. JIT code which tells loads from stores apart by instruction class alone has to special case it, for example when deciding which register holds the faulting address and which one to clear from an exception handler. riscv64 already does so, open coded as a bare insn->imm test. Add a bpf_atomic_is_load_acq() helper and convert riscv64 over to it, so that the x86-64 and arm64 JITs can use the same helper in subsequent patches. Unlike bpf_atomic_is_load_store(), which presumes that its argument is already known to be a BPF_ATOMIC instruction, the new helper is called from code which still sees all instruction classes, so it checks class and mode itself. Also, move bpf_atomic_is_load_store() to filter.h next to BPF_ATOMIC_OP, so that both helpers stay together. No functional change intended. Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 2 +- include/linux/bpf.h | 15 --------------- include/linux/filter.h | 31 +++++++++++++++++++++++++++++++ 3 files changed, 32 insertions(+), 16 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 8fe8969fb8a0..6b9972b07c1b 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1994,7 +1994,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* ret can be 1 (skip-zext); extable entry still needs to be added */ if (ret >= 0) ret = add_exception_handler(insn, - insn->imm == BPF_LOAD_ACQ ? rd : REG_DONT_CLEAR_MARKER, + bpf_atomic_is_load_acq(insn) ? rd : REG_DONT_CLEAR_MARKER, ctx) ?: ret; if (ret) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 73bacfc6444d..d79bf7557ef6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1132,21 +1132,6 @@ static inline bool bpf_pseudo_func(const struct bpf_insn *insn) return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC; } -/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an - * atomic load or store, and false if it is a read-modify-write instruction. - */ -static inline bool -bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn) -{ - switch (atomic_insn->imm) { - case BPF_LOAD_ACQ: - case BPF_STORE_REL: - return true; - default: - return false; - } -} - struct bpf_prog_ops { int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); diff --git a/include/linux/filter.h b/include/linux/filter.h index 32d5297c557e..41b02d53e222 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -383,6 +383,37 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) /* Legacy alias */ #define BPF_STX_XADD(SIZE, DST, SRC, OFF) BPF_ATOMIC_OP(SIZE, BPF_ADD, DST, SRC, OFF) +/* + * Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an + * atomic load or store, and false if it is a read-modify-write instruction. + */ +static inline bool +bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn) +{ + switch (atomic_insn->imm) { + case BPF_LOAD_ACQ: + case BPF_STORE_REL: + return true; + default: + return false; + } +} + +/* + * A load-acquire is the only BPF_STX class instruction that reads into + * dst_reg from src_reg + off16, i.e. it has the operand roles of a BPF_LDX. + * Unlike bpf_atomic_is_load_store(), @insn is not assumed to be a BPF_ATOMIC + * instruction here, so that callers which walk all instruction classes can + * use this directly. + */ +static inline bool bpf_atomic_is_load_acq(const struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_STX && + (BPF_MODE(insn->code) == BPF_ATOMIC || + BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) && + insn->imm == BPF_LOAD_ACQ; +} + /* Memory store, *(uint *) (dst_reg + off16) = imm32 */ #define BPF_ST_MEM(SIZE, DST, OFF, IMM) \ From 4cf8def58b779ad2827f81760837b7a844d6c7d6 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:44 +0200 Subject: [PATCH 236/373] bpf, x86: Fix exception table metadata for arena load-acquire A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but the entry is filled in as if it were a store, since populate_extable() decides based on instruction class alone and a load-acquire is of BPF_STX class: if (BPF_CLASS(insn->code) == BPF_LDX) { arena_reg = reg2pt_regs[src_reg]; fixup_reg = reg2pt_regs[dst_reg]; } else { arena_reg = reg2pt_regs[dst_reg]; fixup_reg = DONT_CLEAR; } For a load-acquire dst_reg holds the loaded value and src_reg holds the address, so both assignments in the else branch are wrong. On a fault over an unmapped arena page ex_handler_bpf() then: - computes the reported address from the value register instead of the address register - reports the access as a WRITE, since it derives the direction from fixup_reg == DONT_CLEAR - leaves dst_reg untouched, so the program continues with a stale value instead of the 0 that BPF_PROBE_* loads deliver The access itself is emitted correctly, emit_atomic_ld_st_index() uses src_reg as the address, so this is a broken probe contract and a wrong diagnostic rather than a memory safety issue. Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path. Fixes: 5341c9a4d833 ("bpf, x86: Support load-acquire and store-release instructions") Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-3-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 01e7ce569c1e..88ed95b2eaa7 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2331,8 +2331,13 @@ st: insn_off = insn->off; * BPF_PROBE_ATOMIC) before being used for the memory access. Pass * the reg holding the unmodified 32-bit address to * ex_handler_bpf(). + * + * A load-acquire is of BPF_STX class, but reads from src_reg + * into dst_reg like a BPF_LDX does, hence it must not be + * treated as a store here. */ - if (BPF_CLASS(insn->code) == BPF_LDX) { + if (BPF_CLASS(insn->code) == BPF_LDX || + bpf_atomic_is_load_acq(insn)) { arena_reg = reg2pt_regs[src_reg]; fixup_reg = reg2pt_regs[dst_reg]; } else { From af22d273aa1f61fb86ec712b3ed785da73c3296e Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:45 +0200 Subject: [PATCH 237/373] bpf, arm64: Fix exception table metadata for arena load-acquire Same problem as on x86-64: add_exception_handler() decides whether an instruction is a load by its class, and a load-acquire is of BPF_STX class even though it reads from src_reg into dst_reg. As a result ... if (BPF_CLASS(insn->code) != BPF_LDX) dst_reg = DONT_CLEAR; ... drops the register to clear, and ... if (BPF_CLASS(insn->code) == BPF_LDX) arena_reg = bpf2a64[insn->src_reg]; else arena_reg = bpf2a64[insn->dst_reg]; ... hands ex_handler_bpf() the value register instead of the address register. A load-acquire from an arena pointer that faults on an unmapped page is therefore reported as a WRITE at a bogus address, and dst_reg keeps its previous value instead of being cleared to 0. Note that emit_atomic_ld_st() already picks src_reg as the address for BPF_LOAD_ACQ, so only the exception table metadata was out of sync with the emitted access. Same as on x86-64, use bpf_atomic_is_load_acq() so a load-acquire takes the load path. Fixes: 9bb12368d539 ("bpf, arm64: Support load-acquire and store-release instructions") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Link: https://lore.kernel.org/bpf/20260806201047.333389-4-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 4cdc7dfb05ba..d14d297ebb96 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1178,7 +1178,12 @@ static int add_exception_handler(const struct bpf_insn *insn, ex->insn = ins_offset; - if (BPF_CLASS(insn->code) != BPF_LDX) + /* + * A load-acquire is of BPF_STX class, but reads from src_reg into + * dst_reg like a BPF_LDX does, hence it must not be treated as a store + * here. + */ + if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn)) dst_reg = DONT_CLEAR; ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); @@ -1193,7 +1198,7 @@ static int add_exception_handler(const struct bpf_insn *insn, * memory access. Pass the reg holding the unmodified 32-bit address to * ex_handler_bpf. */ - if (BPF_CLASS(insn->code) == BPF_LDX) + if (BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn)) arena_reg = bpf2a64[insn->src_reg]; else arena_reg = bpf2a64[insn->dst_reg]; From 007466d9e49738222054f1df87c111a945633ddd Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:46 +0200 Subject: [PATCH 238/373] selftests/bpf: Add arena fault test for load-acquire Add stream_arena_load_acquire_fault, which performs a load-acquire from an unmapped arena address, next to the existing read and write fault tests. The test covers both halves of the JIT bug that treated a load-acquire as a store when populating its exception table entry: - the fault has to be reported as a READ, and at the address held by the source register, which __stderr() and test_address() check, and - the destination register has to be cleared by the fault handler, which the program checks by poisoning it before the load-acquire and returning it, so __retval(0) fails if it is left untouched Note, load-acquire is open coded since linux/filter.h cannot be included alongside vmlinux.h. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream_arena_fault_address [...] #462/1 stream_arena_fault_address/read_fault:OK #462/2 stream_arena_fault_address/write_fault:OK #462/3 stream_arena_fault_address/load_acquire_fault:OK #462 stream_arena_fault_address:OK Summary: 1/3 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-5-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/prog_tests/stream.c | 2 + tools/testing/selftests/bpf/progs/stream.c | 44 +++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/stream.c b/tools/testing/selftests/bpf/prog_tests/stream.c index c3cce5c292bd..15dd3ae2a84b 100644 --- a/tools/testing/selftests/bpf/prog_tests/stream.c +++ b/tools/testing/selftests/bpf/prog_tests/stream.c @@ -103,6 +103,8 @@ void test_stream_arena_fault_address(void) test_address(skel->progs.stream_arena_read_fault, &skel->bss->fault_addr); if (test__start_subtest("write_fault")) test_address(skel->progs.stream_arena_write_fault, &skel->bss->fault_addr); + if (test__start_subtest("load_acquire_fault")) + test_address(skel->progs.stream_arena_load_acquire_fault, &skel->bss->fault_addr); stream__destroy(skel); } diff --git a/tools/testing/selftests/bpf/progs/stream.c b/tools/testing/selftests/bpf/progs/stream.c index 8d8e53d37266..cf5533e11f39 100644 --- a/tools/testing/selftests/bpf/progs/stream.c +++ b/tools/testing/selftests/bpf/progs/stream.c @@ -185,6 +185,50 @@ int stream_arena_read_fault(void *ctx) return 0; } +SEC("syscall") +__arch_x86_64 +__arch_arm64 +__success __retval(0) +__stderr("ERROR: Arena READ access at unmapped address 0x{{.*}}") +__stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}") +__stderr("Call trace:\n" +"{{([a-zA-Z_][a-zA-Z0-9_]*\\+0x[0-9a-fA-F]+/0x[0-9a-fA-F]+\n" +"|[ \t]+[^\n]+\n)*}}") +int stream_arena_load_acquire_fault(void *ctx) +{ + static const struct bpf_insn load_acquire_insn = { + .code = 0xc3, /* BPF_STX | BPF_ATOMIC | BPF_W */ + .dst_reg = 0, /* BPF_REG_0 */ + .src_reg = 1, /* BPF_REG_1 */ + .off = 0x7fff, + .imm = 0x100, /* BPF_LOAD_ACQ */ + }; + struct bpf_arena *ptr = (void *)&arena; + u64 user_vm_start, val; + + /* + * Prevent GCC bounds warning: casting &arena to struct bpf_arena * + * triggers bounds checking since the map definition is smaller than + * struct bpf_arena. barrier_var() makes the pointer opaque to GCC, + * preventing the bounds analysis. + */ + barrier_var(ptr); + user_vm_start = ptr->user_vm_start; + fault_addr = user_vm_start + 0x7fff; + bpf_addr_space_cast(user_vm_start, 0, 1); + asm volatile ( + "r1 = %[user_vm_start];" + "r0 = 1;" + ".8byte %[load_acquire_insn];" /* r0 = load_acquire((u32 *)(r1 + 0x7fff)) */ + "%[val] = r0;" + : [val] "=r" (val) + : [user_vm_start] "r" (user_vm_start), + __imm_insn(load_acquire_insn, load_acquire_insn) + : "r0", "r1" + ); + return val; +} + static __noinline void subprog(void) { int __arena *addr = (int __arena *)0xdeadbeef; From 2b1f9f69ae25cb46d0a84ff14398596eb2bfce44 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Thu, 6 Aug 2026 22:10:47 +0200 Subject: [PATCH 239/373] selftests/bpf: Add load-acquire test for probe-memory pointer types Add a verifier test that a BPF_LOAD_ACQ from a rdonly_untrusted_mem pointer (PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED, obtained via bpf_rdonly_cast()) is rejected. Such a source requires BPF_PROBE_MEM fault protection which is not applied to atomic loads; without the verifier fix the load is accepted and would crash the kernel on a fault. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_load_acquire [...] #621/1 verifier_load_acquire/load-acquire, 8-bit:OK #621/2 verifier_load_acquire/load-acquire, 8-bit @unpriv:OK #621/3 verifier_load_acquire/load-acquire, 16-bit:OK #621/4 verifier_load_acquire/load-acquire, 16-bit @unpriv:OK #621/5 verifier_load_acquire/load-acquire, 32-bit:OK #621/6 verifier_load_acquire/load-acquire, 32-bit @unpriv:OK #621/7 verifier_load_acquire/load-acquire, 64-bit:OK #621/8 verifier_load_acquire/load-acquire, 64-bit @unpriv:OK [...] #621/19 verifier_load_acquire/load-acquire from rdonly_untrusted_mem pointer:OK #621/20 verifier_load_acquire/load-acquire with invalid register R15:OK #621/21 verifier_load_acquire/load-acquire with invalid register R15 @unpriv:OK #621/22 verifier_load_acquire/load-acquire from pkt pointer:OK #621/23 verifier_load_acquire/load-acquire from flow_keys pointer:OK #621/24 verifier_load_acquire/load-acquire from sock pointer:OK #621 verifier_load_acquire:OK Summary: 1/24 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260806201047.333389-6-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/progs/verifier_load_acquire.c | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c index ae1dab1b0cbb..d17026d7480d 100644 --- a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c +++ b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c @@ -3,6 +3,7 @@ #include #include +#include #include "../../../include/linux/filter.h" #include "bpf_misc.h" @@ -221,6 +222,33 @@ __naked void load_acquire_from_sock_pointer(void) : __clobber_all); } +SEC("socket") +__description("load-acquire from rdonly_untrusted_mem pointer") +__failure __msg("BPF_ATOMIC loads from R{{[0-9]+}} rdonly_untrusted_mem is not allowed") +int load_acquire_from_rdonly_untrusted_mem(void *ctx) +{ + __u64 val = 0; + void *p; + + /* + * bpf_rdonly_cast(x, 0) yields PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED. + * A regular BPF_LDX from it is rewritten to BPF_PROBE_MEM, but a + * load-acquire is not, so it must be rejected, otherwise the JIT emits + * a plain load with no exception table entry and a fault would crash + * the kernel. + */ + p = bpf_rdonly_cast(&val, 0); + asm volatile ( + "r1 = %[p];" + ".8byte %[load_acquire_insn];" // r0 = load_acquire((u64 *)(r1 + 0)); + : + : [p] "r" (p), + __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_1, 0)) + : "r0", "r1"); + return 0; +} + SEC("socket") __description("load-acquire with invalid register R15") __failure __failure_unpriv __msg("R15 is invalid") From 3f562c537e9ecf4bc5e206cfffc2cc047f1b7e94 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Fri, 7 Aug 2026 10:44:03 +0000 Subject: [PATCH 240/373] bpf, cgroup: Fix storage null-ptr-deref after replacing prog Syzkaller reported a storage null-ptr-deref issue after replacing prog. This occurs in the following scenario: 1. prog A, an empty prog, is attached to a cgrp. 2. prog B uses BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE and calls the bpf_get_local_storage helper. 3. link_update is called to replace prog A with prog B. The reason is that __cgroup_bpf_replace fails to alloc and assign the required cgrp storage for the incoming replacement prog. Consequently, the new prog inherits an uninit storage, leading to null-ptr-deref panic when kick the new prog. Fix this by rejecting a link update if new_prog's cgroup storage is incompatible with link->prog. Fixes: 0c991ebc8c69 ("bpf: Implement bpf_prog replacement for an active bpf_cgroup_link") Signed-off-by: Pu Lehui Signed-off-by: Andrii Nakryiko Reviewed-by: Amery Hung Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260728132336.2857800-1-pulehui@huaweicloud.com [0] Link: https://lore.kernel.org/bpf/f87b53c0-8f00-45a6-82db-8242fa9b143f@huaweicloud.com [1] Link: https://lore.kernel.org/bpf/20260807104403.1013064-1-pulehui@huaweicloud.com --- kernel/bpf/cgroup.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index d2da5063d8f8..8fbc942a1cc3 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -1026,6 +1026,20 @@ static void replace_effective_prog(struct cgroup *cgrp, } } +static bool cgroup_bpf_storages_compatible(struct bpf_prog *old_prog, + struct bpf_prog *new_prog) +{ + enum bpf_cgroup_storage_type stype; + + for_each_cgroup_storage_type(stype) { + if (old_prog->aux->cgroup_storage[stype] != + new_prog->aux->cgroup_storage[stype]) + return false; + } + + return true; +} + /** * __cgroup_bpf_replace() - Replace link's program and propagate the change * to descendants @@ -1064,6 +1078,9 @@ static int __cgroup_bpf_replace(struct cgroup *cgrp, if (!found) return -ENOENT; + if (!cgroup_bpf_storages_compatible(link->link.prog, new_prog)) + return -EINVAL; + cgrp->bpf.revisions[atype] += 1; old_prog = xchg(&link->link.prog, new_prog); replace_effective_prog(cgrp, atype, pl); From fa9dcacdcdf487f0ffef64bf67622f1caed509f1 Mon Sep 17 00:00:00 2001 From: Sanghyun Park Date: Wed, 5 Aug 2026 12:14:25 +0900 Subject: [PATCH 241/373] bpf: Fix mmap_lock leak in irq_work path stack_map_get_build_id_offset() introduced a per-CPU irq_work to defer mmap_read_unlock() from NMI context, and bpf_find_vma() later reused the same mmap_unlock_work. Both callers only check whether the work is busy before taking mmap_lock, so a nested caller can reuse the slot before the first caller queues it. Two read locks may then be acquired while only one deferred unlock runs, leaking a read lock and blocking exit_mmap(). Reserve the per-CPU slot before mmap_read_trylock(). Use the same wrapper in stackmap and bpf_find_vma() so both callers release the reservation on trylock failure. Keep rejecting the slot while the irq_work remains busy. Release it after the irq_work callback unlocks the mm. Fixes: eac9153f2b58 ("bpf/stackmap: Fix deadlock with rq_lock in bpf_get_stack()") Reported-by: syzbot+cdd6c0925e12b0af60cc@syzkaller.appspotmail.com Reported-by: sashiko-bot@kernel.org Signed-off-by: Sanghyun Park Signed-off-by: Andrii Nakryiko Signed-off-by: Daniel Borkmann Closes: https://syzkaller.appspot.com/bug?extid=cdd6c0925e12b0af60cc Closes: https://lore.kernel.org/r/20260630033745.B80201F000E9@smtp.kernel.org Link: https://lore.kernel.org/bpf/20260805031425.2157475-2-sanghyun.park.cnu@gmail.com --- kernel/bpf/mmap_unlock_work.h | 51 ++++++++++++++++++++--------------- kernel/bpf/stackmap.c | 28 +++++++++++-------- kernel/bpf/task_iter.c | 14 +++++++--- 3 files changed, 56 insertions(+), 37 deletions(-) diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h index 5d18d7d85bef..1834db20b861 100644 --- a/kernel/bpf/mmap_unlock_work.h +++ b/kernel/bpf/mmap_unlock_work.h @@ -4,12 +4,15 @@ #ifndef __MMAP_UNLOCK_WORK_H__ #define __MMAP_UNLOCK_WORK_H__ +#include +#include #include /* irq_work to run mmap_read_unlock() in irq_work */ struct mmap_unlock_irq_work { struct irq_work irq_work; struct mm_struct *mm; + atomic_t active; }; DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); @@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); * We cannot do mmap_read_unlock() when the irq is disabled, because of * risk to deadlock with rq_lock. To look up vma when the irqs are * disabled, we need to run mmap_read_unlock() in irq_work. We use a - * percpu variable to do the irq_work. If the irq_work is already used - * by another lookup, we fall over. + * percpu variable to do the irq_work. The active flag reserves the slot + * before mmap_read_trylock() and until the irq_work callback consumes mm. */ -static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr) +static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void) { - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = false; + struct mmap_unlock_irq_work *work; - if (irqs_disabled()) { - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { - work = this_cpu_ptr(&mmap_unlock_work); - if (irq_work_is_busy(&work->irq_work)) { - /* cannot queue more up_read, fallback */ - irq_work_busy = true; - } - } else { - /* - * PREEMPT_RT does not allow to trylock mmap sem in - * interrupt disabled context. Force the fallback code. - */ - irq_work_busy = true; - } - } + if (!irqs_disabled()) + return NULL; - *work_ptr = work; - return irq_work_busy; + /* + * PREEMPT_RT does not allow to trylock mmap sem in interrupt + * disabled context. Force the fallback code. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + return ERR_PTR(-EBUSY); + + work = this_cpu_ptr(&mmap_unlock_work); + if (irq_work_is_busy(&work->irq_work) || + atomic_cmpxchg_acquire(&work->active, 0, 1)) + return ERR_PTR(-EBUSY); + + return work; +} + +static inline void +bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work) +{ + if (work) + atomic_set_release(&work->active, 0); } static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 8f0f3ff1a869..a839041e0d00 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -414,8 +414,7 @@ static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *i static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, u32 trace_nr, bool user, bool may_fault) { - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + struct mmap_unlock_irq_work *work; bool has_user_ctx = user && current && current->mm; struct stack_map_build_id_cache cache = {}; struct vm_area_struct *vma; @@ -426,15 +425,16 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, return; } - /* If the irq_work is in use, fall back to report ips. Same - * fallback is used for kernel stack (!user) on a stackmap with - * build_id. - */ - if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) { - /* cannot access current->mm, fall back to ips */ - for (i = 0; i < trace_nr; i++) - stack_map_build_id_set_ip(&id_offs[i]); - return; + if (!has_user_ctx) + goto fallback; + + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) + goto fallback; + + if (!mmap_read_trylock(current->mm)) { + bpf_mmap_unlock_guard_put(work); + goto fallback; } for (i = 0; i < trace_nr; i++) { @@ -465,6 +465,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, vma->vm_pgoff); } bpf_mmap_unlock_mm(work, current->mm); + return; + +fallback: + /* cannot access current->mm, fall back to ips */ + for (i = 0; i < trace_nr; i++) + stack_map_build_id_set_ip(&id_offs[i]); } static struct perf_callchain_entry * diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index b256fb9c1214..13e1aabe6f88 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -753,9 +753,8 @@ static struct bpf_iter_reg task_vma_reg_info = { BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags) { - struct mmap_unlock_irq_work *work = NULL; + struct mmap_unlock_irq_work *work; struct vm_area_struct *vma; - bool irq_work_busy = false; bool __maybe_unused mmput_needed = false; struct mm_struct *mm; int ret = -ENOENT; @@ -792,9 +791,14 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, if (!mm) return -ENOENT; - irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) { + ret = PTR_ERR(work); + goto out; + } - if (irq_work_busy || !mmap_read_trylock(mm)) { + if (!mmap_read_trylock(mm)) { + bpf_mmap_unlock_guard_put(work); ret = -EBUSY; goto out; } @@ -1191,6 +1195,8 @@ static void do_mmap_read_unlock(struct irq_work *entry) work = container_of(entry, struct mmap_unlock_irq_work, irq_work); mmap_read_unlock_non_owner(work->mm); + work->mm = NULL; + bpf_mmap_unlock_guard_put(work); } static int __init task_iter_init(void) From 483a1bb0b6cf816fabaf99702a6e4a7938c98b07 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:30 -0700 Subject: [PATCH 242/373] bpf: Do not print a newline after disassembly in bpf_verbose_insn() At the moment there are more callsites that want bpf_verbose_insn() to not print a newline after the instruction, than callsites that want a newline. Drop '\n' from disasm.c. Non-functional change. The changes in bpftool are verified by writing a bpf program using a variety of instructions and comparing `prog dump xlated` output in the following modes: plain, opcodes, visual, visual opcodes. The output before and after the changes is identical. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Reviewed-by: Quentin Monnet Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-1-b6c270013c77@gmail.com --- kernel/bpf/backtrack.c | 1 + kernel/bpf/disasm.c | 68 ++++++++++---------- kernel/bpf/liveness.c | 6 +- kernel/bpf/verifier.c | 1 + tools/bpf/bpftool/xlated_dumper.c | 19 ++---- tools/testing/selftests/bpf/disasm_helpers.c | 3 +- 6 files changed, 43 insertions(+), 55 deletions(-) diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 2f473ad4fd7c..40bd04421a99 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -285,6 +285,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, verbose(env, "stack=%s before ", env->tmp_str_buf); verbose(env, "%d: ", idx); bpf_verbose_insn(env, insn); + verbose(env, "\n"); } /* If there is a history record that some registers gained range at this insn, diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 0391b3bc0073..50b3ca5149a0 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -139,7 +139,7 @@ static void print_bpf_end_insn(bpf_insn_print_t verbose, void *private_data, const struct bpf_insn *insn) { - verbose(private_data, "(%02x) r%d = %s%d r%d\n", + verbose(private_data, "(%02x) r%d = %s%d r%d", insn->code, insn->dst_reg, BPF_SRC(insn->code) == BPF_TO_BE ? "be" : "le", insn->imm, insn->dst_reg); @@ -149,7 +149,7 @@ static void print_bpf_bswap_insn(bpf_insn_print_t verbose, void *private_data, const struct bpf_insn *insn) { - verbose(private_data, "(%02x) r%d = bswap%d r%d\n", + verbose(private_data, "(%02x) r%d = bswap%d r%d", insn->code, insn->dst_reg, insn->imm, insn->dst_reg); } @@ -197,19 +197,19 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, else print_bpf_end_insn(verbose, cbs->private_data, insn); } else if (BPF_OP(insn->code) == BPF_NEG) { - verbose(cbs->private_data, "(%02x) %c%d = -%c%d\n", + verbose(cbs->private_data, "(%02x) %c%d = -%c%d", insn->code, class == BPF_ALU ? 'w' : 'r', insn->dst_reg, class == BPF_ALU ? 'w' : 'r', insn->dst_reg); } else if (is_addr_space_cast(insn)) { - verbose(cbs->private_data, "(%02x) r%d = addr_space_cast(r%d, %u, %u)\n", + verbose(cbs->private_data, "(%02x) r%d = addr_space_cast(r%d, %u, %u)", insn->code, insn->dst_reg, insn->src_reg, ((u32)insn->imm) >> 16, (u16)insn->imm); } else if (is_mov_percpu_addr(insn)) { - verbose(cbs->private_data, "(%02x) r%d = &(void __percpu *)(r%d)\n", + verbose(cbs->private_data, "(%02x) r%d = &(void __percpu *)(r%d)", insn->code, insn->dst_reg, insn->src_reg); } else if (BPF_SRC(insn->code) == BPF_X) { - verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d\n", + verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d", insn->code, class == BPF_ALU ? 'w' : 'r', insn->dst_reg, is_sdiv_smod(insn) ? bpf_alu_sign_string[BPF_OP(insn->code) >> 4] @@ -218,7 +218,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, class == BPF_ALU ? 'w' : 'r', insn->src_reg); } else { - verbose(cbs->private_data, "(%02x) %c%d %s %d\n", + verbose(cbs->private_data, "(%02x) %c%d %s %d", insn->code, class == BPF_ALU ? 'w' : 'r', insn->dst_reg, is_sdiv_smod(insn) ? bpf_alu_sign_string[BPF_OP(insn->code) >> 4] @@ -227,7 +227,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, } } else if (class == BPF_STX) { if (BPF_MODE(insn->code) == BPF_MEM) - verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = r%d\n", + verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = r%d", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, @@ -235,7 +235,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, else if (BPF_MODE(insn->code) == BPF_ATOMIC && (insn->imm == BPF_ADD || insn->imm == BPF_AND || insn->imm == BPF_OR || insn->imm == BPF_XOR)) { - verbose(cbs->private_data, "(%02x) lock *(%s *)(r%d %+d) %s r%d\n", + verbose(cbs->private_data, "(%02x) lock *(%s *)(r%d %+d) %s r%d", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, @@ -246,7 +246,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->imm == (BPF_AND | BPF_FETCH) || insn->imm == (BPF_OR | BPF_FETCH) || insn->imm == (BPF_XOR | BPF_FETCH))) { - verbose(cbs->private_data, "(%02x) r%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)\n", + verbose(cbs->private_data, "(%02x) r%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)", insn->code, insn->src_reg, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_atomic_alu_string[BPF_OP(insn->imm) >> 4], @@ -254,7 +254,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->dst_reg, insn->off, insn->src_reg); } else if (BPF_MODE(insn->code) == BPF_ATOMIC && insn->imm == BPF_CMPXCHG) { - verbose(cbs->private_data, "(%02x) r0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)\n", + verbose(cbs->private_data, "(%02x) r0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)", insn->code, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], @@ -262,44 +262,44 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->src_reg); } else if (BPF_MODE(insn->code) == BPF_ATOMIC && insn->imm == BPF_XCHG) { - verbose(cbs->private_data, "(%02x) r%d = atomic%s_xchg((%s *)(r%d %+d), r%d)\n", + verbose(cbs->private_data, "(%02x) r%d = atomic%s_xchg((%s *)(r%d %+d), r%d)", insn->code, insn->src_reg, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); } else if (BPF_MODE(insn->code) == BPF_ATOMIC && insn->imm == BPF_LOAD_ACQ) { - verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))\n", + verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))", insn->code, insn->dst_reg, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->src_reg, insn->off); } else if (BPF_MODE(insn->code) == BPF_ATOMIC && insn->imm == BPF_STORE_REL) { - verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)\n", + verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); } else { - verbose(cbs->private_data, "BUG_%02x\n", insn->code); + verbose(cbs->private_data, "BUG_%02x", insn->code); } } else if (class == BPF_ST) { if (BPF_MODE(insn->code) == BPF_MEM) { - verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d\n", + verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->imm); } else if (BPF_MODE(insn->code) == 0xc0 /* BPF_NOSPEC, no UAPI */) { - verbose(cbs->private_data, "(%02x) nospec\n", insn->code); + verbose(cbs->private_data, "(%02x) nospec", insn->code); } else { - verbose(cbs->private_data, "BUG_st_%02x\n", insn->code); + verbose(cbs->private_data, "BUG_st_%02x", insn->code); } } else if (class == BPF_LDX) { if (BPF_MODE(insn->code) != BPF_MEM && BPF_MODE(insn->code) != BPF_MEMSX) { - verbose(cbs->private_data, "BUG_ldx_%02x\n", insn->code); + verbose(cbs->private_data, "BUG_ldx_%02x", insn->code); return; } - verbose(cbs->private_data, "(%02x) r%d = *(%s *)(r%d %+d)\n", + verbose(cbs->private_data, "(%02x) r%d = *(%s *)(r%d %+d)", insn->code, insn->dst_reg, BPF_MODE(insn->code) == BPF_MEM ? bpf_ldst_string[BPF_SIZE(insn->code) >> 3] : @@ -307,12 +307,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->src_reg, insn->off); } else if (class == BPF_LD) { if (BPF_MODE(insn->code) == BPF_ABS) { - verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[%d]\n", + verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[%d]", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->imm); } else if (BPF_MODE(insn->code) == BPF_IND) { - verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[r%d + %d]\n", + verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[r%d + %d]", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->src_reg, insn->imm); @@ -332,12 +332,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, if (is_ptr && !allow_ptr_leaks) imm = 0; - verbose(cbs->private_data, "(%02x) r%d = %s\n", + verbose(cbs->private_data, "(%02x) r%d = %s", insn->code, insn->dst_reg, __func_imm_name(cbs, insn, imm, tmp, sizeof(tmp))); } else { - verbose(cbs->private_data, "BUG_ld_%02x\n", insn->code); + verbose(cbs->private_data, "BUG_ld_%02x", insn->code); return; } } else if (class == BPF_JMP32 || class == BPF_JMP) { @@ -347,35 +347,35 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, char tmp[64]; if (insn->src_reg == BPF_PSEUDO_CALL) { - verbose(cbs->private_data, "(%02x) call pc%s\n", + verbose(cbs->private_data, "(%02x) call pc%s", insn->code, __func_get_name(cbs, insn, tmp, sizeof(tmp))); } else { strcpy(tmp, "unknown"); - verbose(cbs->private_data, "(%02x) call %s#%d\n", insn->code, + verbose(cbs->private_data, "(%02x) call %s#%d", insn->code, __func_get_name(cbs, insn, tmp, sizeof(tmp)), insn->imm); } } else if (insn->code == (BPF_JMP | BPF_JA)) { - verbose(cbs->private_data, "(%02x) goto pc%+d\n", + verbose(cbs->private_data, "(%02x) goto pc%+d", insn->code, insn->off); } else if (insn->code == (BPF_JMP | BPF_JA | BPF_X)) { - verbose(cbs->private_data, "(%02x) gotox r%d\n", + verbose(cbs->private_data, "(%02x) gotox r%d", insn->code, insn->dst_reg); } else if (insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO) { - verbose(cbs->private_data, "(%02x) may_goto pc%+d\n", + verbose(cbs->private_data, "(%02x) may_goto pc%+d", insn->code, insn->off); } else if (insn->code == (BPF_JMP32 | BPF_JA)) { - verbose(cbs->private_data, "(%02x) gotol pc%+d\n", + verbose(cbs->private_data, "(%02x) gotol pc%+d", insn->code, insn->imm); } else if (insn->code == (BPF_JMP | BPF_EXIT)) { - verbose(cbs->private_data, "(%02x) exit\n", insn->code); + verbose(cbs->private_data, "(%02x) exit", insn->code); } else if (BPF_SRC(insn->code) == BPF_X) { verbose(cbs->private_data, - "(%02x) if %c%d %s %c%d goto pc%+d\n", + "(%02x) if %c%d %s %c%d goto pc%+d", insn->code, class == BPF_JMP32 ? 'w' : 'r', insn->dst_reg, bpf_jmp_string[BPF_OP(insn->code) >> 4], @@ -383,14 +383,14 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->src_reg, insn->off); } else { verbose(cbs->private_data, - "(%02x) if %c%d %s 0x%x goto pc%+d\n", + "(%02x) if %c%d %s 0x%x goto pc%+d", insn->code, class == BPF_JMP32 ? 'w' : 'r', insn->dst_reg, bpf_jmp_string[BPF_OP(insn->code) >> 4], (u32)insn->imm, insn->off); } } else { - verbose(cbs->private_data, "(%02x) %s\n", + verbose(cbs->private_data, "(%02x) %s", insn->code, bpf_class_string[class]); } } diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 0aadfbae0acc..ff1e68cc4bd1 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -497,7 +497,6 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i pos = env->log.end_pos; verbose(env, "%3d: ", insn_idx); bpf_verbose_insn(env, &insns[insn_idx]); - bpf_vlog_reset(&env->log, env->log.end_pos - 1); /* remove \n */ insn_pos = env->log.end_pos; verbose(env, "%*c;", bpf_vlog_alignment(insn_pos - pos), ' '); pos = env->log.end_pos; @@ -1043,7 +1042,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i if (!printed) { verbose(env, "%3d: ", idx); bpf_verbose_insn(env, insn); - bpf_vlog_reset(&env->log, env->log.end_pos - 1); printed = true; } verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]); @@ -1058,7 +1056,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i if (!printed) { verbose(env, "%3d: ", idx); bpf_verbose_insn(env, insn); - bpf_vlog_reset(&env->log, env->log.end_pos - 1); printed = true; } verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]); @@ -1070,7 +1067,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i if (!printed) { verbose(env, "%3d: ", idx); bpf_verbose_insn(env, insn); - bpf_vlog_reset(&env->log, env->log.end_pos - 1); printed = true; } verbose(env, "\tfp%+d: ", -(i + 1) * 8); verbose_arg_track(env, &at_stack_in[i]); @@ -1545,6 +1541,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, verbose(env, "%3d: ", idx); bpf_verbose_insn(env, &insns[idx]); + verbose(env, "\n"); /* Collect what needs printing */ if (is_ldx_stx_call && @@ -2285,6 +2282,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) verbose(env, "."); verbose(env, " "); bpf_verbose_insn(env, &insns[i]); + verbose(env, "\n"); if (bpf_is_ldimm64(&insns[i])) i++; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index c9533ea700ba..be8818b9e640 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17562,6 +17562,7 @@ static int do_check(struct bpf_verifier_env *env) env->prev_log_pos = env->log.end_pos; verbose(env, "%d: ", env->insn_idx); bpf_verbose_insn(env, insn); + verbose(env, "\n"); env->prev_insn_print_pos = env->log.end_pos - env->prev_log_pos; env->prev_log_pos = env->log.end_pos; } diff --git a/tools/bpf/bpftool/xlated_dumper.c b/tools/bpf/bpftool/xlated_dumper.c index 5e7cb8b36fef..5579173a61e3 100644 --- a/tools/bpf/bpftool/xlated_dumper.c +++ b/tools/bpf/bpftool/xlated_dumper.c @@ -107,14 +107,7 @@ print_insn_for_graph(void *private_data, const char *fmt, ...) p = buf; while (*p != '\0') { - if (*p == '\n') { - memmove(p + 3, p, strlen(buf) + 1 - (p - buf)); - /* Align each instruction dump row left. */ - *p++ = '\\'; - *p++ = 'l'; - /* Output multiline concatenation. */ - *p++ = '\\'; - } else if (*p == '<' || *p == '>' || *p == '|' || *p == '&') { + if (*p == '<' || *p == '>' || *p == '|' || *p == '&') { memmove(p + 1, p, strlen(buf) + 1 - (p - buf)); /* Escape special character. */ *p++ = '\\'; @@ -129,16 +122,10 @@ print_insn_for_graph(void *private_data, const char *fmt, ...) static void __printf(2, 3) print_insn_json(void *private_data, const char *fmt, ...) { - unsigned int l = strlen(fmt); - char chomped_fmt[l]; va_list args; va_start(args, fmt); - if (l > 0) { - strncpy(chomped_fmt, fmt, l - 1); - chomped_fmt[l - 1] = '\0'; - } - jsonw_vprintf_enquote(json_wtr, chomped_fmt, args); + jsonw_vprintf_enquote(json_wtr, fmt, args); va_end(args); } @@ -351,6 +338,7 @@ void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len, printf("%4u: ", i); print_bpf_insn(&cbs, insn + i, true); + printf("\n"); if (opcodes) { printf(" "); @@ -417,6 +405,7 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end, printf("%u: ", insn_off); print_bpf_insn(&cbs, cur, true); + printf("\\l\\\n"); if (opcodes) { printf("\\ \\ \\ \\ "); diff --git a/tools/testing/selftests/bpf/disasm_helpers.c b/tools/testing/selftests/bpf/disasm_helpers.c index f529f1c8c171..30221352568d 100644 --- a/tools/testing/selftests/bpf/disasm_helpers.c +++ b/tools/testing/selftests/bpf/disasm_helpers.c @@ -55,10 +55,9 @@ struct bpf_insn *disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz) * for each instruction (FF stands for instruction `code` byte). * Remove the prefix inplace, and also simplify call instructions. * E.g.: "(85) call foo#10" -> "call foo". - * Also remove newline in the end (the 'max(strlen(buf) - 1, 0)' thing). */ pfx_end = buf + 5; - sfx_start = buf + max((int)strlen(buf) - 1, 0); + sfx_start = buf + (int)strlen(buf); if (strncmp(pfx_end, "call ", 5) == 0 && (tmp = strrchr(buf, '#'))) sfx_start = tmp; len = sfx_start - pfx_end; From d977dca7d0736dc7d68d9ad1434961da3b42de35 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:31 -0700 Subject: [PATCH 243/373] bpf: Extract is_addr_space_cast32() utility function bpf_do_misc_fixups() converts the following address space cast instructions to 32-bit moves: - cast from address space 1 (user) to address space 0 (kernel) - cast from address space 0 (kernel) to address space 1 (user) iff associated arena map has a BPF_F_NO_USER_CONV flag. Extract a predicate detecting such instructions for use in the following patches. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-2-b6c270013c77@gmail.com --- kernel/bpf/fixups.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index a0bddada7964..5f7843648189 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -20,6 +20,26 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn) insn->imm == BPF_CMPXCHG; } +/* Returns true if 'insn' is an address space cast instruction translated as BPF_ALU op */ +static bool is_addr_space_cast32(struct bpf_prog *prog, const struct bpf_insn *insn) +{ + struct bpf_map *arena = (struct bpf_map *)prog->aux->arena; + + if (insn->code != (BPF_ALU64 | BPF_MOV | BPF_X) || insn->off != BPF_ADDR_SPACE_CAST) + return false; + + /* cast from as(1) to as(0) */ + if (insn->imm == 1) + return true; + + /* cast from as(0) to as(1) */ + if (insn->imm == 1 << 16) + return arena && arena->map_flags & BPF_F_NO_USER_CONV; + + /* non-BPF_F_NO_USER_CONV cast from as(0) to as(1) should be handled by JIT */ + return false; +} + /* Return the regno defined by the insn, or -1. */ static int insn_def_regno(const struct bpf_insn *insn) { @@ -1513,15 +1533,12 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) } for (i = 0; i < insn_cnt;) { - if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && insn->imm) { - if ((insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1) || - (((struct bpf_map *)env->prog->aux->arena)->map_flags & BPF_F_NO_USER_CONV)) { - /* convert to 32-bit mov that clears upper 32-bit */ - insn->code = BPF_ALU | BPF_MOV | BPF_X; - /* clear off and imm, so it's a normal 'wX = wY' from JIT pov */ - insn->off = 0; - insn->imm = 0; - } /* cast from as(0) to as(1) should be handled by JIT */ + if (is_addr_space_cast32(env->prog, insn)) { + /* convert to 32-bit mov that clears upper 32-bit */ + insn->code = BPF_ALU | BPF_MOV | BPF_X; + /* clear off and imm, so it's a normal 'wX = wY' from JIT pov */ + insn->off = 0; + insn->imm = 0; goto next_insn; } From 05b71078f30555bab7a40bf2cf12e6da1ddfcaf6 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:32 -0700 Subject: [PATCH 244/373] bpf: Move bpf_is_reg64() to fixups.c The following patches are going to remove bpf_is_reg64() users from everywhere except fixups.c, and also make it dependent on functions local to fixups.c. Move the function before hand to simplify the review. Non functional change. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-3-b6c270013c77@gmail.com --- kernel/bpf/fixups.c | 90 +++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 90 ------------------------------------------- 2 files changed, 90 insertions(+), 90 deletions(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 5f7843648189..d2ff416d7ad6 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -64,6 +64,96 @@ static int insn_def_regno(const struct bpf_insn *insn) } } +/* This function is supposed to be used by the zero extension optimization + * code only. It returns TRUE if the source or destination register operates + * on 64-bit, otherwise return FALSE. + */ +bool bpf_is_reg64(struct bpf_insn *insn, + u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t) +{ + u8 code, class, op; + + code = insn->code; + class = BPF_CLASS(code); + op = BPF_OP(code); + if (class == BPF_JMP) { + /* BPF_EXIT for "main" will reach here. Return TRUE + * conservatively. + */ + if (op == BPF_EXIT) + return true; + if (op == BPF_CALL) { + /* BPF to BPF call will reach here because of marking + * caller saved clobber with DST_OP_NO_MARK for which we + * don't care the register def because they are anyway + * marked as NOT_INIT already. + */ + if (insn->src_reg == BPF_PSEUDO_CALL) + return false; + /* Helper call will reach here because of arg type + * check, conservatively return TRUE. + */ + if (t == SRC_OP) + return true; + + return false; + } + } + + if (class == BPF_ALU64 && op == BPF_END && (insn->imm == 16 || insn->imm == 32)) + return false; + + if (class == BPF_ALU64 || class == BPF_JMP || + (class == BPF_ALU && op == BPF_END && insn->imm == 64)) + return true; + + if (class == BPF_ALU || class == BPF_JMP32) + return false; + + if (class == BPF_LDX) { + if (t != SRC_OP) + return BPF_SIZE(code) == BPF_DW || BPF_MODE(code) == BPF_MEMSX; + /* LDX source must be ptr. */ + return true; + } + + if (class == BPF_STX) { + /* BPF_STX (including atomic variants) has one or more source + * operands, one of which is a ptr. Check whether the caller is + * asking about it. + */ + if (t == SRC_OP && reg->type != SCALAR_VALUE) + return true; + return BPF_SIZE(code) == BPF_DW; + } + + if (class == BPF_LD) { + u8 mode = BPF_MODE(code); + + /* LD_IMM64 */ + if (mode == BPF_IMM) + return true; + + /* Both LD_IND and LD_ABS return 32-bit data. */ + if (t != SRC_OP) + return false; + + /* Implicit ctx ptr. */ + if (regno == BPF_REG_6) + return true; + + /* Explicit source could be any width. */ + return true; + } + + if (class == BPF_ST) + /* The only source register for BPF_ST is a ptr. */ + return true; + + /* Conservatively return true at default. */ + return true; +} + /* Return TRUE if INSN has defined any 32-bit value explicitly. */ static bool insn_has_def32(struct bpf_insn *insn) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index be8818b9e640..abb325194168 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3058,96 +3058,6 @@ static void mark_stack_slots_scratched(struct bpf_verifier_env *env, mark_stack_slot_scratched(env, spi - i); } -/* This function is supposed to be used by the following 32-bit optimization - * code only. It returns TRUE if the source or destination register operates - * on 64-bit, otherwise return FALSE. - */ -bool bpf_is_reg64(struct bpf_insn *insn, - u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t) -{ - u8 code, class, op; - - code = insn->code; - class = BPF_CLASS(code); - op = BPF_OP(code); - if (class == BPF_JMP) { - /* BPF_EXIT for "main" will reach here. Return TRUE - * conservatively. - */ - if (op == BPF_EXIT) - return true; - if (op == BPF_CALL) { - /* BPF to BPF call will reach here because of marking - * caller saved clobber with DST_OP_NO_MARK for which we - * don't care the register def because they are anyway - * marked as NOT_INIT already. - */ - if (insn->src_reg == BPF_PSEUDO_CALL) - return false; - /* Helper call will reach here because of arg type - * check, conservatively return TRUE. - */ - if (t == SRC_OP) - return true; - - return false; - } - } - - if (class == BPF_ALU64 && op == BPF_END && (insn->imm == 16 || insn->imm == 32)) - return false; - - if (class == BPF_ALU64 || class == BPF_JMP || - (class == BPF_ALU && op == BPF_END && insn->imm == 64)) - return true; - - if (class == BPF_ALU || class == BPF_JMP32) - return false; - - if (class == BPF_LDX) { - if (t != SRC_OP) - return BPF_SIZE(code) == BPF_DW || BPF_MODE(code) == BPF_MEMSX; - /* LDX source must be ptr. */ - return true; - } - - if (class == BPF_STX) { - /* BPF_STX (including atomic variants) has one or more source - * operands, one of which is a ptr. Check whether the caller is - * asking about it. - */ - if (t == SRC_OP && reg->type != SCALAR_VALUE) - return true; - return BPF_SIZE(code) == BPF_DW; - } - - if (class == BPF_LD) { - u8 mode = BPF_MODE(code); - - /* LD_IMM64 */ - if (mode == BPF_IMM) - return true; - - /* Both LD_IND and LD_ABS return 32-bit data. */ - if (t != SRC_OP) - return false; - - /* Implicit ctx ptr. */ - if (regno == BPF_REG_6) - return true; - - /* Explicit source could be any width. */ - return true; - } - - if (class == BPF_ST) - /* The only source register for BPF_ST is a ptr. */ - return true; - - /* Conservatively return true at default. */ - return true; -} - static void mark_insn_zext(struct bpf_verifier_env *env, struct bpf_reg_state *reg) { From ef1ddbfcfaef3194453c66255781e38eecd1f7de Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:33 -0700 Subject: [PATCH 245/373] bpf: Track upper 32-bit register halves' liveness in compute_live_registers() Extend compute_live_registers() to track upper and lower register halves' liveness separately. This is mostly straightforward: - use/def masks are extended to track 2 bits per register; - compute_insn_live_regs() is updated to properly track these 2 bits according to the instruction semantics. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-4-b6c270013c77@gmail.com --- kernel/bpf/liveness.c | 89 ++++++++++++++++++++++++++++--------------- kernel/bpf/verifier.c | 1 - 2 files changed, 59 insertions(+), 31 deletions(-) diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index ff1e68cc4bd1..edfc2480b0f0 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2047,29 +2047,38 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env) /* Each field is a register bitmask */ struct insn_live_regs { - u16 use; /* registers read by instruction */ - u16 def; /* registers written by instruction */ - u16 in; /* registers that may be alive before instruction */ - u16 out; /* registers that may be alive after instruction */ + u32 use; /* registers read by instruction */ + u32 def; /* registers written by instruction */ + u32 in; /* registers that may be alive before instruction */ + u32 out; /* registers that may be alive after instruction */ }; /* Bitmask with 1s for all caller saved registers */ #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) +static inline u32 reg32_mask(u32 n) { return BIT(n); } +static inline u32 reg64_mask(u32 n) { return BIT(n) | BIT(n + 16); } +static inline u32 mask_widen(u32 m) { return m | (m << 16); } +static inline u16 mask_lo(u32 m) { return (u16)m; } +static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); } + /* Compute info->{use,def} fields for the instruction */ static void compute_insn_live_regs(struct bpf_verifier_env *env, struct bpf_insn *insn, struct insn_live_regs *info) { struct bpf_call_summary cs; - u8 class = BPF_CLASS(insn->code); - u8 code = BPF_OP(insn->code); - u8 mode = BPF_MODE(insn->code); - u16 src = BIT(insn->src_reg); - u16 dst = BIT(insn->dst_reg); - u16 r0 = BIT(0); - u16 def = 0; - u16 use = 0xffff; + const u8 class = BPF_CLASS(insn->code); + const u8 code = BPF_OP(insn->code); + const u8 mode = BPF_MODE(insn->code); + const u8 size = BPF_SIZE(insn->code); + const u32 src = reg64_mask(insn->src_reg); + const u32 dst = reg64_mask(insn->dst_reg); + const u32 src32 = mask_lo(src); + const u32 dst32 = mask_lo(dst); + const u32 r0 = reg64_mask(0); + u32 def = 0; + u32 use = U32_MAX; switch (class) { case BPF_LD: @@ -2080,8 +2089,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, use = 0; } break; - case BPF_LD | BPF_ABS: - case BPF_LD | BPF_IND: + case BPF_ABS: + case BPF_IND: /* stick with defaults */ break; } @@ -2089,7 +2098,15 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, case BPF_LDX: switch (mode) { case BPF_MEM: + /* a narrow load still redefines the whole register */ + def = dst; + use = src; + break; case BPF_MEMSX: + /* + * sign extension defines the whole register; + * src holds a pointer, hence is used as 64-bit. + */ def = dst; use = src; break; @@ -2107,12 +2124,19 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, switch (mode) { case BPF_MEM: def = 0; - use = dst | src; + use = dst | (size == BPF_DW ? src : src32); break; - case BPF_ATOMIC: + case BPF_ATOMIC: { + /* + * dst holds a pointer and is always used as 64-bit; + * the value operand and r0 are read as 32-bit for BPF_W atomics. + */ + u32 srcv = size == BPF_DW ? src : src32; + u32 r0v = size == BPF_DW ? r0 : mask_lo(r0); + switch (insn->imm) { case BPF_CMPXCHG: - use = r0 | dst | src; + use = r0v | dst | srcv; def = r0; break; case BPF_LOAD_ACQ: @@ -2121,10 +2145,10 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, break; case BPF_STORE_REL: def = 0; - use = dst | src; + use = dst | srcv; break; default: - use = dst | src; + use = dst | srcv; if (insn->imm & BPF_FETCH) def = src; else @@ -2132,6 +2156,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, } break; } + } break; case BPF_ALU: case BPF_ALU64: @@ -2145,14 +2170,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, if (BPF_SRC(insn->code) == BPF_K) use = 0; else - use = src; + use = class == BPF_ALU64 ? src : src32; break; default: def = dst; if (BPF_SRC(insn->code) == BPF_K) - use = dst; + use = class == BPF_ALU64 ? dst : dst32; else - use = dst | src; + use = class == BPF_ALU64 ? (dst | src) : (dst32 | src32); } break; case BPF_JMP: @@ -2178,13 +2203,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, use = def & ~BIT(BPF_REG_0); if (bpf_get_call_summary(env, insn, &cs)) use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1); + def = mask_widen(def); + use = mask_widen(use); break; default: def = 0; - if (BPF_SRC(insn->code) == BPF_K) - use = dst; - else - use = dst | src; + use = class == BPF_JMP ? dst : dst32; + if (BPF_SRC(insn->code) == BPF_X) + use |= class == BPF_JMP ? src : src32; } break; } @@ -2249,8 +2275,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) int insn_idx = env->cfg.insn_postorder[i]; struct insn_live_regs *live = &state[insn_idx]; struct bpf_iarray *succ; - u16 new_out = 0; - u16 new_in = 0; + u32 new_out = 0; + u32 new_in = 0; succ = bpf_insn_successors(env, insn_idx); for (int s = 0; s < succ->cnt; ++s) @@ -2264,8 +2290,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) } } - for (i = 0; i < insn_cnt; ++i) - insn_aux[i].live_regs_before = state[i].in; + for (i = 0; i < insn_cnt; ++i) { + u32 in = state[i].in; + + insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in); + } if (env->log.level & BPF_LOG_LEVEL2) { verbose(env, "Live regs before insn:\n"); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index abb325194168..dc1acbe0172e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16727,7 +16727,6 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, int i; if (bpf_helper_call(call)) { - if (bpf_get_helper_proto(env, call->imm, &fn) < 0) /* error would be reported later */ return false; From 7ce090afbf725e25fff29caacce1eaf8459b1117 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:34 -0700 Subject: [PATCH 246/373] bpf: Infer zext_dst based on static register liveness analysis As reported in the thread [1], the verifier's 32-bit operations zero extension logic is broken. This logic is responsible for correct semantics of 32-bit operations on s390 architecture. According to BPF semantics, operation `w1 += 1` is supposed to zero extend the upper half of the register `r1`. On s390 the JIT relies on the verifier emitting explicit zero extension before such operations. The verifier attempts to minimize the amount of zero extensions inserted by tracking whether upper halves of the 64-bit registers are ever used. Previously such tracking worked as follows: - bpf_reg_state->subreg_def field was set by do_check_insn() for each operation defining lower but not the upper halves of the register. - Whenever an operation reading the whole register was verified, the verifier checked register's subreg_def and set bpf_insn_aux_data->zext_dst flag as true via a call to mark_insn_zext() function. - After the verification was complete, a special pass bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations with bpf_insn_aux_data->zext_dst set as true by adding explicit zero extension. Note that the logic above relies on bpf_reg_state->subreg_def, which is a property of a current verifier state. Before the commit [2] two additional steps happened: - The verifier tracked upper and lower register halves' liveness as flags REG_LIVE_READ{32,64} in bpf_reg_state->live. - The function propagate_liveness() called mark_insn_zext() in order to transfer the knowledge about which registers have their upper halves alive (and thus might require zero extension). The commit [2] removed the two steps described above, hence making possible a situation like below: - The register's upper half is set and is used on some verification path P1 and the register happens not to be marked as precise. - The checkpoint C is created while processing some instruction between register initialization and usage. - On some other verification path P2 the register's upper half is not initialized and that path ends hitting the checkpoint C. - In such a case the register's initialization on path P2 would lack zext_dst mark, making it possible for the program to inject an arbitrary value in the register's upper half. This commit replaces subreg_def based logic with computing zext_dst statically, as a part of the bpf_compute_live_registers() analysis: - The analysis now tracks usage of upper and lower halves of the registers separately. - If some instruction defines a 32-bit subregister, but not the whole register, *and* the upper half of the register is alive after that instruction, the instruction is marked as zext_dst. There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict a noticeable performance penalty. [1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/ [2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness") Fixes: 107e16979905 ("bpf: disable and remove registers chain based liveness") Reported-by: Min-gyu Kim Reported-by: STAR Labs SG Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/ Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-5-b6c270013c77@gmail.com --- include/linux/bpf_verifier.h | 7 +-- kernel/bpf/fixups.c | 27 +++++---- kernel/bpf/liveness.c | 14 +++++ kernel/bpf/verifier.c | 112 ++--------------------------------- 4 files changed, 38 insertions(+), 122 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index a2a40caca0a0..2c74d676ede9 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -162,11 +162,6 @@ struct bpf_reg_state { * pointing to bpf_func_state. */ u32 frameno; - /* Tracks subreg definition. The stored value is the insn_idx of the - * writing insn. This is safe because subreg_def is used before any insn - * patching which only happens after main verification finished. - */ - s32 subreg_def; /* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */ bool precise; }; @@ -1637,7 +1632,6 @@ struct bpf_kfunc_desc_tab { }; /* Functions exported from verifier.c, used by fixups.c */ -bool bpf_is_reg64(struct bpf_insn *insn, u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t); void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len); void bpf_mark_subprog_exc_cb(struct bpf_verifier_env *env, int subprog); bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env); @@ -1661,5 +1655,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env); int bpf_jit_subprogs(struct bpf_verifier_env *env); int bpf_fixup_call_args(struct bpf_verifier_env *env); int bpf_do_misc_fixups(struct bpf_verifier_env *env); +int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn); #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index d2ff416d7ad6..447c54828cb9 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -68,8 +68,8 @@ static int insn_def_regno(const struct bpf_insn *insn) * code only. It returns TRUE if the source or destination register operates * on 64-bit, otherwise return FALSE. */ -bool bpf_is_reg64(struct bpf_insn *insn, - u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t) +static bool bpf_is_reg64(struct bpf_prog *prog, struct bpf_insn *insn, + u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t) { u8 code, class, op; @@ -103,6 +103,10 @@ bool bpf_is_reg64(struct bpf_insn *insn, if (class == BPF_ALU64 && op == BPF_END && (insn->imm == 16 || insn->imm == 32)) return false; + /* address space casts converted to BPF_ALU, see bpf_do_misc_fixups() */ + if (is_addr_space_cast32(prog, insn)) + return false; + if (class == BPF_ALU64 || class == BPF_JMP || (class == BPF_ALU && op == BPF_END && insn->imm == 64)) return true; @@ -154,15 +158,18 @@ bool bpf_is_reg64(struct bpf_insn *insn, return true; } -/* Return TRUE if INSN has defined any 32-bit value explicitly. */ -static bool insn_has_def32(struct bpf_insn *insn) +/* + * Return the 32-bit subregister defined by INSN, or -1 if INSN does not + * explicitly define a 32-bit value. + */ +int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn) { int dst_reg = insn_def_regno(insn); - if (dst_reg == -1) - return false; + if (dst_reg < 0 || bpf_is_reg64(prog, insn, dst_reg, NULL, DST_OP)) + return -1; - return !bpf_is_reg64(insn, dst_reg, NULL, DST_OP); + return dst_reg; } static int kfunc_desc_cmp_by_imm_off(const void *a, const void *b) @@ -279,7 +286,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, * (cnt == 1) is taken or not. There is no guarantee INSN at OFF is the * original insn at old prog. */ - data[off].zext_dst = insn_has_def32(insn + off + cnt - 1); + data[off].zext_dst = bpf_insn_def32(new_prog, insn + off + cnt - 1) >= 0; if (cnt == 1) return; @@ -291,7 +298,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, for (i = off; i < off + cnt - 1; i++) { /* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; - data[i].zext_dst = insn_has_def32(insn + i); + data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0; } /* @@ -730,7 +737,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, * BPF_STX + SRC_OP, so it is safe to pass NULL * here. */ - if (bpf_is_reg64(&insn, load_reg, NULL, DST_OP)) { + if (bpf_is_reg64(env->prog, &insn, load_reg, NULL, DST_OP)) { if (class == BPF_LD && BPF_MODE(code) == BPF_IMM) i++; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index edfc2480b0f0..ef9a5a922887 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2232,6 +2232,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) struct bpf_insn *insns = env->prog->insnsi; struct insn_live_regs *state; int insn_cnt = env->prog->len; + u64 pos, insn_pos; int err = 0, i, j; bool changed; @@ -2291,9 +2292,18 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) } for (i = 0; i < insn_cnt; ++i) { + int def32 = bpf_insn_def32(env->prog, &insns[i]); + u32 out = state[i].out; u32 in = state[i].in; insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in); + /* + * On architectures where 32-bit operations do not reset upper halves + * of the registers, the verifier needs to zero extend a destination + * register if an instruction defines a 32-bit subregister and the + * upper half of that register is alive after the instruction. + */ + insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32)); } if (env->log.level & BPF_LOG_LEVEL2) { @@ -2310,7 +2320,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) else verbose(env, "."); verbose(env, " "); + pos = env->log.end_pos; bpf_verbose_insn(env, &insns[i]); + insn_pos = env->log.end_pos; + if (insn_aux[i].zext_dst) + verbose(env, "%*c; zext", bpf_vlog_alignment(insn_pos - pos), ' '); verbose(env, "\n"); if (bpf_is_ldimm64(&insns[i])) i++; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index dc1acbe0172e..9eabc5123e5a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2132,12 +2132,9 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env, /* Mark a register as having a completely unknown (scalar) value. */ void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg) { - s32 subreg_def = reg->subreg_def; - memset(reg, 0, sizeof(*reg)); reg->type = SCALAR_VALUE; reg->var_off = tnum_unknown; - reg->subreg_def = subreg_def; __mark_reg_unbounded(reg); } @@ -2213,7 +2210,6 @@ static int mark_btf_ld_reg(struct bpf_verifier_env *env, } } -#define DEF_NOT_SUBREG (0) static void init_reg_state(struct bpf_verifier_env *env, struct bpf_func_state *state) { @@ -2222,7 +2218,6 @@ static void init_reg_state(struct bpf_verifier_env *env, for (i = 0; i < MAX_BPF_REG; i++) { bpf_mark_reg_not_init(env, ®s[i]); - regs[i].subreg_def = DEF_NOT_SUBREG; } /* frame pointer */ @@ -3058,30 +3053,14 @@ static void mark_stack_slots_scratched(struct bpf_verifier_env *env, mark_stack_slot_scratched(env, spi - i); } -static void mark_insn_zext(struct bpf_verifier_env *env, - struct bpf_reg_state *reg) -{ - s32 def_idx = reg->subreg_def; - - if (def_idx == DEF_NOT_SUBREG) - return; - - env->insn_aux_data[def_idx - 1].zext_dst = true; - /* The dst will be zero extended, so won't be sub-register anymore. */ - reg->subreg_def = DEF_NOT_SUBREG; -} - static int __check_reg_arg(struct bpf_verifier_env *env, struct bpf_reg_state *regs, u32 regno, enum bpf_reg_arg_type t) { - struct bpf_insn *insn = env->prog->insnsi + env->insn_idx; struct bpf_reg_state *reg; - bool rw64; mark_reg_scratched(env, regno); reg = ®s[regno]; - rw64 = bpf_is_reg64(insn, regno, reg, t); if (t == SRC_OP) { /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { @@ -3092,9 +3071,6 @@ static int __check_reg_arg(struct bpf_verifier_env *env, struct bpf_reg_state *r if (regno == BPF_REG_FP) return 0; - if (rw64) - mark_insn_zext(env, reg); - return 0; } else { /* check whether register used as dest operand can be written to */ @@ -3102,7 +3078,6 @@ static int __check_reg_arg(struct bpf_verifier_env *env, struct bpf_reg_state *r verbose(env, "frame pointer is read only\n"); return -EACCES; } - reg->subreg_def = rw64 ? DEF_NOT_SUBREG : env->insn_idx + 1; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } @@ -3758,11 +3733,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, if (size <= spill_size && bpf_stack_narrow_access_ok(off, size, spill_size)) { - /* The earlier check_reg_arg() has decided the - * subreg_def for this insn. Save it first. - */ - s32 subreg_def = state->regs[dst_regno].subreg_def; - if (env->bpf_capable && size == 4 && spill_size == 4 && get_reg_width(reg) <= 32) /* Ensure stack slot has an ID to build a relation @@ -3770,7 +3740,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, */ assign_scalar_id_before_mov(env, reg); state->regs[dst_regno] = *reg; - state->regs[dst_regno].subreg_def = subreg_def; /* Break the relation on a narrowing fill. * coerce_reg_to_size will adjust the boundaries. @@ -6246,12 +6215,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b } else { mark_reg_known_zero(env, regs, value_regno); - /* A load of ctx field could have different - * actual load size with the one encoded in the - * insn. When the dst is PTR, it is for sure not - * a sub-register. - */ - regs[value_regno].subreg_def = DEF_NOT_SUBREG; if (base_type(info.reg_type) == PTR_TO_BTF_ID) { regs[value_regno].btf = info.btf; regs[value_regno].btf_id = info.btf_id; @@ -7350,10 +7313,6 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat if (spi < 0) return spi; - /* - * For CONST_PTR_TO_DYNPTR, reg is already scratched by check_reg_arg - * in check_helper_call and mark_btf_func_reg_size in check_kfunc_call. - */ mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS); reg = &state->stack[spi].spilled_ptr; @@ -9457,7 +9416,6 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, /* All non-void global functions return a 64-bit SCALAR_VALUE. */ if (!subprog_returns_void(env, subprog)) { mark_reg_unknown(env, caller->regs, BPF_REG_0); - caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; } if (env->subprog_info[subprog].might_throw) { @@ -10477,9 +10435,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn } invalidate_outgoing_stack_args(env, cur_func(env)); - /* helper call returns 64-bit value. */ - regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; - /* update return register (already marked as written above) */ ret_type = fn->ret_type; ret_flag = type_flag(ret_type); @@ -10719,30 +10674,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn return 0; } -/* mark_btf_func_reg_size() is used when the reg size is determined by - * the BTF func_proto's return value size and argument. - */ -static void __mark_btf_func_reg_size(struct bpf_verifier_env *env, struct bpf_reg_state *regs, - u32 regno, size_t reg_size) -{ - struct bpf_reg_state *reg = ®s[regno]; - - if (regno == BPF_REG_0) { - /* Function return value */ - reg->subreg_def = reg_size == sizeof(u64) ? - DEF_NOT_SUBREG : env->insn_idx + 1; - } else if (reg_size == sizeof(u64)) { - /* Function argument */ - mark_insn_zext(env, reg); - } -} - -static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno, - size_t reg_size) -{ - return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size); -} - static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ACQUIRE; @@ -12961,7 +12892,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_call_arg_meta meta; struct bpf_insn_aux_data *insn_aux; int err, insn_idx = *insn_idx_p; - const struct btf_param *args; u32 i, nargs, ptr_type_id; struct bpf_kfunc_desc *desc; struct btf *desc_btf; @@ -13013,7 +12943,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, verbose(env, "failed to mark s32 range for retval in forked state for lock\n"); return err; } - __mark_btf_func_reg_size(env, regs, BPF_REG_0, sizeof(u32)); } else if (!insn->off && insn->imm == special_kfunc_list[KF___bpf_trap]) { verbose(env, "unexpected __bpf_trap() due to uninitialized variable?\n"); return -EFAULT; @@ -13166,7 +13095,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, u32 regno = caller_saved[i]; bpf_mark_reg_not_init(env, ®s[regno]); - regs[regno].subreg_def = DEF_NOT_SUBREG; } invalidate_outgoing_stack_args(env, cur_func(env)); @@ -13188,7 +13116,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, ®s[BPF_REG_0]); - mark_btf_func_reg_size(env, BPF_REG_0, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf); @@ -13279,7 +13206,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, /* For mark_ptr_or_null_reg, see 93c230e3f5bd6 */ regs[BPF_REG_0].id = ++env->id_gen; } - mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *)); if (is_kfunc_acquire(&meta)) { id = acquire_reference(env, insn_idx, 0); if (id < 0) @@ -13316,18 +13242,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, caller_info->stack_arg_cnt = stack_arg_cnt; } - args = (const struct btf_param *)(meta.func_proto + 1); - for (i = 0; i < min_t(int, nargs, MAX_BPF_FUNC_REG_ARGS); i++) { - u32 regno = i + 1; - - t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL); - if (btf_type_is_ptr(t)) - mark_btf_func_reg_size(env, regno, sizeof(void *)); - else - /* scalar. ensured by check_kfunc_args() */ - mark_btf_func_reg_size(env, regno, t->size); - } - if (bpf_is_iter_next_kfunc(&meta)) { err = process_iter_next_call(env, insn_idx, &meta); if (err) @@ -14820,14 +14734,14 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, if (dst_reg->type != PTR_TO_ARENA) *dst_reg = *src_reg; - dst_reg->subreg_def = env->insn_idx + 1; - - if (BPF_CLASS(insn->code) == BPF_ALU64) + if (BPF_CLASS(insn->code) == BPF_ALU64) { /* * 32-bit operations zero upper bits automatically. * 64-bit operations need to be converted to 32. */ aux->needs_zext = true; + aux->zext_dst = true; + } /* Any arithmetic operations are allowed on arena pointers */ return 0; @@ -15023,18 +14937,14 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) if (insn->imm) { /* off == BPF_ADDR_SPACE_CAST */ mark_reg_unknown(env, regs, insn->dst_reg); - if (insn->imm == 1) { /* cast from as(1) to as(0) */ + if (insn->imm == 1) /* cast from as(1) to as(0) */ dst_reg->type = PTR_TO_ARENA; - /* PTR_TO_ARENA is 32-bit */ - dst_reg->subreg_def = env->insn_idx + 1; - } } else if (insn->off == 0) { /* case: R1 = R2 * copy register state to dest reg */ assign_scalar_id_before_mov(env, src_reg); *dst_reg = *src_reg; - dst_reg->subreg_def = DEF_NOT_SUBREG; } else { /* case: R1 = (s8, s16 s32)R2 */ if (is_pointer_value(env, insn->src_reg)) { @@ -15052,7 +14962,6 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) if (!no_sext) clear_scalar_id(dst_reg); coerce_reg_to_size_sx(dst_reg, insn->off >> 3); - dst_reg->subreg_def = DEF_NOT_SUBREG; } else { mark_reg_unknown(env, regs, insn->dst_reg); } @@ -15077,7 +14986,6 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) */ if (!is_src_reg_u32) clear_scalar_id(dst_reg); - dst_reg->subreg_def = env->insn_idx + 1; } else { /* case: W1 = (s8, s16)W2 */ bool no_sext = reg_umax(src_reg) < (1ULL << (insn->off - 1)); @@ -15087,7 +14995,6 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) *dst_reg = *src_reg; if (!no_sext) clear_scalar_id(dst_reg); - dst_reg->subreg_def = env->insn_idx + 1; coerce_subreg_to_size_sx(dst_reg, insn->off >> 3); } } else { @@ -15956,12 +15863,8 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s continue; if ((!(reg->id & BPF_ADD_CONST) && !(known_reg->id & BPF_ADD_CONST)) || reg->delta == known_reg->delta) { - s32 saved_subreg_def = reg->subreg_def; - *reg = *known_reg; - reg->subreg_def = saved_subreg_def; } else { - s32 saved_subreg_def = reg->subreg_def; s32 saved_off = reg->delta; u32 saved_id = reg->id; @@ -15971,12 +15874,11 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s /* reg = known_reg; reg += delta */ *reg = *known_reg; /* - * Must preserve off, id and subreg_def flag, - * otherwise another sync_linked_regs() will be incorrect. + * Must preserve off and id, otherwise another sync_linked_regs() + * will be incorrect. */ reg->delta = saved_off; reg->id = saved_id; - reg->subreg_def = saved_subreg_def; scalar32_min_max_add(reg, &fake_reg); scalar_min_max_add(reg, &fake_reg); @@ -16411,8 +16313,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * Already marked as written above. */ mark_reg_unknown(env, regs, BPF_REG_0); - /* ld_abs load up to 32-bit skb data. */ - regs[BPF_REG_0].subreg_def = env->insn_idx + 1; /* * See bpf_gen_ld_abs() which emits a hidden BPF_EXIT with r0=0 * which must be explored by the verifier when in a subprog. From be4f8d6f2ff7afe31bd481e004b216bcb3fa6707 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:35 -0700 Subject: [PATCH 247/373] bpf: Simplify the bpf_is_reg64() After the previous commit bpf_is_reg64() is only used in a context where destination register's property is queried, and only for instructions for which insn_def_regno() >= 0. Hence, simplify the function by: - removing unused parameters; - removing code paths considering BPF_JMP{,32} instructions; - streamlining the condition expressions. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-6-b6c270013c77@gmail.com --- kernel/bpf/fixups.c | 108 +++++++++++--------------------------------- 1 file changed, 26 insertions(+), 82 deletions(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 447c54828cb9..661e2d13a604 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -64,95 +64,43 @@ static int insn_def_regno(const struct bpf_insn *insn) } } -/* This function is supposed to be used by the zero extension optimization - * code only. It returns TRUE if the source or destination register operates - * on 64-bit, otherwise return FALSE. +/* + * For use only in combination with insn_def_regno() >= 0. + * Returns TRUE if the destination register operates on 64-bit, + * otherwise return FALSE. */ -static bool bpf_is_reg64(struct bpf_prog *prog, struct bpf_insn *insn, - u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t) +static bool bpf_is_reg64(struct bpf_prog *prog, struct bpf_insn *insn) { - u8 code, class, op; + u8 class = BPF_CLASS(insn->code); + u8 mode = BPF_MODE(insn->code); + u8 size = BPF_SIZE(insn->code); + u8 op = BPF_OP(insn->code); + bool mode_mem; - code = insn->code; - class = BPF_CLASS(code); - op = BPF_OP(code); - if (class == BPF_JMP) { - /* BPF_EXIT for "main" will reach here. Return TRUE - * conservatively. - */ - if (op == BPF_EXIT) - return true; - if (op == BPF_CALL) { - /* BPF to BPF call will reach here because of marking - * caller saved clobber with DST_OP_NO_MARK for which we - * don't care the register def because they are anyway - * marked as NOT_INIT already. - */ - if (insn->src_reg == BPF_PSEUDO_CALL) - return false; - /* Helper call will reach here because of arg type - * check, conservatively return TRUE. - */ - if (t == SRC_OP) - return true; + /* subregister endiness swap */ + if ((class == BPF_ALU || class == BPF_ALU64) && op == BPF_END && insn->imm != 64) + return false; - return false; - } - } - - if (class == BPF_ALU64 && op == BPF_END && (insn->imm == 16 || insn->imm == 32)) + /* w0 += 1 */ + if (class == BPF_ALU && op != BPF_END) return false; /* address space casts converted to BPF_ALU, see bpf_do_misc_fixups() */ if (is_addr_space_cast32(prog, insn)) return false; - if (class == BPF_ALU64 || class == BPF_JMP || - (class == BPF_ALU && op == BPF_END && insn->imm == 64)) - return true; - - if (class == BPF_ALU || class == BPF_JMP32) + /* non 64-bit, non signed extended loads */ + mode_mem = mode == BPF_MEM || mode == BPF_PROBE_MEM || mode == BPF_PROBE_MEM32; + if (class == BPF_LDX && mode_mem && size != BPF_DW) return false; - if (class == BPF_LDX) { - if (t != SRC_OP) - return BPF_SIZE(code) == BPF_DW || BPF_MODE(code) == BPF_MEMSX; - /* LDX source must be ptr. */ - return true; - } + /* atomics, see insn_def_regno() */ + if (class == BPF_STX && size != BPF_DW) + return false; - if (class == BPF_STX) { - /* BPF_STX (including atomic variants) has one or more source - * operands, one of which is a ptr. Check whether the caller is - * asking about it. - */ - if (t == SRC_OP && reg->type != SCALAR_VALUE) - return true; - return BPF_SIZE(code) == BPF_DW; - } - - if (class == BPF_LD) { - u8 mode = BPF_MODE(code); - - /* LD_IMM64 */ - if (mode == BPF_IMM) - return true; - - /* Both LD_IND and LD_ABS return 32-bit data. */ - if (t != SRC_OP) - return false; - - /* Implicit ctx ptr. */ - if (regno == BPF_REG_6) - return true; - - /* Explicit source could be any width. */ - return true; - } - - if (class == BPF_ST) - /* The only source register for BPF_ST is a ptr. */ - return true; + /* both LD_IND and LD_ABS return 32-bit data. */ + if (class == BPF_LD && (mode == BPF_IND || mode == BPF_ABS)) + return false; /* Conservatively return true at default. */ return true; @@ -166,7 +114,7 @@ int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn) { int dst_reg = insn_def_regno(insn); - if (dst_reg < 0 || bpf_is_reg64(prog, insn, dst_reg, NULL, DST_OP)) + if (dst_reg < 0 || bpf_is_reg64(prog, insn)) return -1; return dst_reg; @@ -733,11 +681,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, if (load_reg == -1) continue; - /* NOTE: arg "reg" (the fourth one) is only used for - * BPF_STX + SRC_OP, so it is safe to pass NULL - * here. - */ - if (bpf_is_reg64(env->prog, &insn, load_reg, NULL, DST_OP)) { + if (bpf_is_reg64(env->prog, &insn)) { if (class == BPF_LD && BPF_MODE(code) == BPF_IMM) i++; From 8b365b3c68b474a1053ec0755dacdc751578afb0 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 7 Aug 2026 13:59:36 -0700 Subject: [PATCH 248/373] selftests/bpf: Verify zext_dst annotations for various instructions Includes the following test cases: - a test showing that zero extension flags do not propagate through state pruning in the unpatched kernel. - a 32-bit subregister consumed by MOV32 and ALU32 operations (never zext'ed); - a 64-bit MOV (never zext'ed); - a narrow (32-bit) BPF_LDX load whose result is read as 64-bit; - 32-bit atomic fetch_add and cmpxchg whose result is read as 64-bit; - a CFG case where a 32-bit definition's upper half is used only on one of two branches; - no zext for dead registers; - LD_ABS defines only lower 32 bits, hence needs zext when the result is used as 64-bits; - helper, kfunc and subprogram parameters are considered to use full 64 bits; - a 32-bit subregister consumed by JMP32 (X/K) operations; - a 32-bit subregister consumed by JMP (X/K) operations; - a 64-bit register consumed by both JMP and JMP32 operations (never zext'ed); - ALU64 and address space cast operations on arena pointers; - memory loads using BPF_PROBE_MEM instructions. Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807-static-zext-v4-7-b6c270013c77@gmail.com --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_zext.c | 392 ++++++++++++++++++ 2 files changed, 394 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_zext.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index b79bafca68f7..0baa74618fa0 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -126,6 +126,7 @@ #include "verifier_jit_inline.skel.h" #include "irq.skel.h" #include "verifier_ctx_ptr_param.skel.h" +#include "verifier_zext.skel.h" #define MAX_ENTRIES 11 @@ -281,6 +282,7 @@ void test_irq(void) { RUN(irq); } void test_verifier_mtu(void) { RUN(verifier_mtu); } void test_verifier_jit_inline(void) { RUN(verifier_jit_inline); } void test_verifier_ctx_ptr_param(void) { RUN(verifier_ctx_ptr_param); } +void test_verifier_zext(void) { RUN_TESTS(verifier_zext); } static int init_test_val_map(struct bpf_object *obj, char *map_name) { diff --git a/tools/testing/selftests/bpf/progs/verifier_zext.c b/tools/testing/selftests/bpf/progs/verifier_zext.c new file mode 100644 index 000000000000..8f2362da91d6 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_zext.c @@ -0,0 +1,392 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "../../../include/linux/filter.h" +#include +#include +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE | BPF_F_NO_USER_CONV); + __uint(max_entries, 1); +} arena SEC(".maps"); + +extern long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; + +/* to retain debug info for BTF generation */ +void __kfunc_btf_root(void) +{ + bpf_kfunc_call_test4(0, 0, 0, 0); + bpf_arena_alloc_pages(0, 0, 0, 0, 0); + bpf_rdonly_cast(0, 0); +} + +SEC("socket") +__flag(BPF_F_TEST_STATE_FREQ) +__flag(BPF_F_TEST_RND_HI32) +__success __retval(0) +__naked void zext_lost_across_checkpoint(void) +{ + asm volatile (" \ + call %[bpf_ktime_get_ns]; \ + r8 = r0; \ + r6 = 0xdeadbeefcafebabe ll; /* inject some value for r6's upper half */ \ + if r8 != 0 goto 1f; /* fall-through cached first, branch pruned */ \ + r6 = 32; /* full 64-bit def */ \ + goto 2f; \ +1: w6 = 32; /* 32-bit def, zext mark lost */ \ +2: r0 = r6; /* buggy verifier believed upper 32 bits are 0 */ \ + /* and thus did not zero extended w6 = 32. */ \ + r0 >>= 32; \ + exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +/* 32-bit ALU result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void zext_alu32_hi_used(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w1 = w0; \ + r0 = r1; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* 32-bit ALU result read only as 32-bit -> no zext */ +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void no_zext_alu32_hi_unused(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w1 = w0; /* MOV */ \ + w2 = w1; \ + w2 += w1; /* ALU32, BPF_X */ \ + w2 += 1; /* ALU32, BPF_K */ \ + w2 = w2; /* keep w2 alive for previous instruction */ \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* 64-bit definition is never zero extended */ +SEC("socket") +__success __log_level(2) +__not_msg("r1 = r0{{.*}}; zext") +__naked void no_zext_mov64(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + r0 = r1; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* Narrow load result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r1 = *(u32 *)(r10 -8){{ +}}; zext") +__naked void zext_narrow_load_hi_used(void) +{ + asm volatile (" \ + r0 = 0; \ + *(u64 *)(r10 - 8) = r0; \ + r1 = *(u32 *)(r10 - 8); \ + r0 = r1; \ + exit; \ +" ::: __clobber_all); +} + +/* 32-bit atomic fetch result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r1 = atomic_fetch_add((u32 *)(r10 -8), r1){{ +}}; zext") +__naked void zext_atomic_fetch32_hi_used(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64 *)(r10 - 8) = r1; \ + w1 = 1; \ + .8byte %[fetch_add32]; \ + r0 = r1; \ + exit; \ +" : + : __imm_insn(fetch_add32, + BPF_ATOMIC_OP(BPF_W, BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +/* 32-bit atomic cmpxchg result (r0) read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r0 = atomic_cmpxchg((u32 *)(r10 -8), r0, r1){{ +}}; zext") +__naked void zext_cmpxchg32_hi_used(void) +{ + asm volatile (" \ + r1 = 0; \ + *(u64 *)(r10 - 8) = r1; \ + w0 = 0; \ + w1 = 1; \ + .8byte %[cmpxchg32]; \ + r2 = r0; \ + r0 = r2; \ + exit; \ +" : + : __imm_insn(cmpxchg32, + BPF_ATOMIC_OP(BPF_W, BPF_CMPXCHG, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +/* 32-bit def before a branch, upper half used on one branch -> zext */ +SEC("socket") +__success __log_level(2) +__msg("w6 = 32{{ +}}; zext") +__naked void zext_cfg_hi_used_one_branch(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w6 = 32; \ + if r0 == 0 goto 1f; \ + r0 = r6; \ + exit; \ +1: r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* r1's upper half is dead, so 'w1 = 1' must NOT be marked for zero extension. */ +SEC("socket") +__success __log_level(2) +__not_msg("w1 = 1{{.*}}; zext") +__naked void no_zext_other_reg_hi_used(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r6 = r0; \ + r6 <<= 32; \ + w1 = 1; \ + r0 = r6; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* LD_ABS defines r0; when r0 is read as 64-bit it must be zero extended */ +SEC("socket") +__success __log_level(2) +__msg("r0 = *(u8 *)skb[0]{{.*}}; zext") +__naked void zext_ld_abs_hi_used(void) +{ + asm volatile (" \ + r6 = r1; \ + r0 = *(u8 *)skb[0]; \ + r7 = r0; \ + r0 = r7; \ + exit; \ +" ::: __clobber_all); +} + +/* Helper parameters are read as 64-bit (call_use_mask() fallback) */ +SEC("socket") +__success __log_level(2) +__msg("w2 = 1{{ +}}; zext") +__naked void helper_param_read_as_64bit(void) +{ + asm volatile (" \ + r1 = r10; \ + r1 += -8; \ + w2 = 1; \ + call %[bpf_trace_printk]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_trace_printk) + : __clobber_all); +} + +static __used __naked int subprog_reads_arg_as_64bit(void) +{ + asm volatile (" \ + r0 = r1; \ + exit; \ +" ::: __clobber_all); +} + +/* subprogram parameters are conservatively read as 64-bit */ +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void subprog_param_read_as_64bit(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w1 = w0; \ + call subprog_reads_arg_as_64bit; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* kfunc parameters are zero extended */ +SEC("tc") +__success __log_level(2) +__msg("w1 = 1{{ +}}; zext") +__msg("w2 = 1{{ +}}; zext") +__msg("w3 = 1{{ +}}; zext") +__msg("w4 = 1{{ +}}; zext") +__naked void kfunc_param_read_per_btf(void) +{ + asm volatile (" \ + w1 = 1; \ + w2 = 1; \ + w3 = 1; \ + w4 = 1; \ + call bpf_kfunc_call_test4; \ + r0 = 0; \ + exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void alu32_and_32bit_conditional(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w1 = w0; \ + if w1 > 42 goto 1f; /* BPF_K */ \ + w2 = 28; \ + if w2 > w1 goto 1f; /* BPF_X */ \ + r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void alu32_and_64bit_conditional(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + w1 = w0; \ + if r1 > 42 goto 1f; /* BPF_K */ \ + r2 = 28; \ + if r2 > r1 goto 1f; /* BPF_X */ \ + r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void alu64_and_conditionals(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + r1 = r0; \ + if w1 > 42 goto 1f; /* BPF_K */ \ + if r1 > 42 goto 1f; /* BPF_K */ \ + r2 = 28; \ + if w2 > w1 goto 1f; /* BPF_X */ \ + if r2 > r1 goto 1f; /* BPF_X */ \ + r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +#ifdef __BPF_FEATURE_ADDR_SPACE_CAST + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__arch_s390x +__xlated("7: w1 = w0") +__xlated("8: w1 = w1") +__xlated("9: w1 += 8") +__xlated("10: w1 = w1") +__xlated("11: w2 = w1") +__xlated("12: w2 = w2") +__xlated("13: *(u64 *)(r1 +0) = r2") +__naked void arena_ptr(void) +{ + asm volatile (" \ + r1 = %[arena] ll; \ + r2 = 0; \ + r3 = 1; \ + r4 = 0; \ + r5 = 0; \ + call %[bpf_arena_alloc_pages]; \ + r1 = addr_space_cast(r0, 0, 1); /* needs zext */ \ + r1 += 8; /* needs zext */ \ + r2 = addr_space_cast(r1, 1, 0); /* needs zext because of BPF_F_NO_USER_CONV */ \ + *(u64 *)(r1 +0) = r2; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_arena_alloc_pages), + __imm_addr(arena) + : __clobber_all); +} + +#endif + +/* Check if probe mem loads keep their zero extension. */ +SEC("socket") +__success __log_level(2) +__arch_s390x +__xlated("3: r1 = *(u64 *)(r0 +0)") +__xlated("4: r2 = *(u32 *)(r0 +0)") +__xlated("5: w2 = w2") +__xlated("6: r3 = *(u16 *)(r0 +0)") +__xlated("7: w3 = w3") +__xlated("8: r4 = *(u8 *)(r0 +0)") +__xlated("9: w4 = w4") +__naked void probe_mem(void) +{ + asm volatile (" \ + r1 = 0; \ + r2 = 0; \ + call %[bpf_rdonly_cast]; \ + r1 = *(u64 *)(r0 + 0); /* BPF_PROBE_MEM */ \ + r2 = *(u32 *)(r0 + 0); /* BPF_PROBE_MEM */ \ + r3 = *(u16 *)(r0 + 0); /* BPF_PROBE_MEM */ \ + r4 = *(u8 *)(r0 + 0); /* BPF_PROBE_MEM */ \ + r0 = r1; /* make the registers used */ \ + r0 += r2; \ + r0 += r3; \ + r0 += r4; \ +1: exit; \ +" : + : __imm(bpf_rdonly_cast) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; From 04962afb3cd6a3cbf1a3679c0c95049833db36c1 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 8 Aug 2026 02:39:21 +0200 Subject: [PATCH 249/373] bpf: Rename 'early' BTF checking as a preparation phase BTF processing is split around subprogram discovery. The first phase gets program BTF and imports func_info because a BTF-tagged exception callback may not be referenced by any instruction. Subprogram discovery needs this metadata to find it. The later phase validates func_info and line_info against the complete subprogram table and applies CO-RE relocations. This split breaks a real dependency cycle rather than merely running the same checks early. Rename bpf_check_btf_info_early() and check_btf_func_early() to preparation names that reflect this role. Add short call-site comments to make the two phases and their responsibilities clear. No functional change is intended. Signed-off-by: Kumar Kartikeya Dwivedi Reviewed-by: Amery Hung Link: https://patch.msgid.link/20260808003938.3486067-2-memxor@gmail.com Signed-off-by: Eduard Zingerman --- include/linux/bpf_verifier.h | 4 ++-- kernel/bpf/check_btf.c | 14 +++++++------- kernel/bpf/verifier.c | 4 +++- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 2c74d676ede9..1ccf82d7ff8d 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1172,8 +1172,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id) *btf_id = key & 0x7FFFFFFF; } -int bpf_check_btf_info_early(struct bpf_verifier_env *env, - const union bpf_attr *attr, bpfptr_t uattr); +int bpf_prepare_btf_info(struct bpf_verifier_env *env, + const union bpf_attr *attr, bpfptr_t uattr); int bpf_check_btf_info(struct bpf_verifier_env *env, const union bpf_attr *attr, bpfptr_t uattr); diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c index 93bebe6fe12e..0e8b3ccc7a5b 100644 --- a/kernel/bpf/check_btf.c +++ b/kernel/bpf/check_btf.c @@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env) #define MIN_BPF_FUNCINFO_SIZE 8 #define MAX_FUNCINFO_REC_SIZE 252 -static int check_btf_func_early(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +static int prepare_btf_func(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { u32 krec_size = sizeof(struct bpf_func_info); const struct btf_type *type, *func_proto; @@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env, return err; } -int bpf_check_btf_info_early(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +int bpf_prepare_btf_info(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { struct btf *btf; int err; @@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env, } env->prog->aux->btf = btf; - err = check_btf_func_early(env, attr, uattr); + err = prepare_btf_func(env, attr, uattr); if (err) return err; return 0; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9eabc5123e5a..ce755c8b0ee2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20135,7 +20135,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, INIT_LIST_HEAD(&env->explored_states[i]); INIT_LIST_HEAD(&env->free_list); - ret = bpf_check_btf_info_early(env, attr, uattr); + /* Prepare BTF and func_info needed to discover all subprograms. */ + ret = bpf_prepare_btf_info(env, attr, uattr); if (ret < 0) goto skip_full_check; @@ -20147,6 +20148,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* Validate BTF against the complete subprogram layout and apply CO-RE. */ ret = bpf_check_btf_info(env, attr, uattr); if (ret < 0) goto skip_full_check; From 41f36ffa3a87b354a248be4c24f02f06cd52844d Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 8 Aug 2026 02:39:22 +0200 Subject: [PATCH 250/373] bpf: Split subprogram and kfunc collection add_subprog_and_kfunc() combines two operations with different ordering requirements. Subprogram discovery must precede validation of func_info and line_info, while kfunc descriptors are only needed by the verifier after its initial program setup is complete. Split the helper into add_subprogs() and add_kfuncs() so each operation can be placed according to its actual dependencies. Keep both calls adjacent and in their existing phase for now, and add short comments describing their roles. No functional change is intended for valid programs. Signed-off-by: Kumar Kartikeya Dwivedi Reviewed-by: Amery Hung Link: https://patch.msgid.link/20260808003938.3486067-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ce755c8b0ee2..a54f7b63eaba 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2837,7 +2837,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset) return 0; } -static int add_subprog_and_kfunc(struct bpf_verifier_env *env) +static int add_subprogs(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; int i, ret, insn_cnt = env->prog->len, ex_cb_insn; @@ -2849,8 +2849,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) return ret; for (i = 0; i < insn_cnt; i++, insn++) { - if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn) && - !bpf_pseudo_kfunc_call(insn)) + if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn)) continue; if (!env->bpf_capable) { @@ -2858,11 +2857,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) return -EPERM; } - if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn)) - ret = add_subprog(env, i + insn->imm + 1); - else - ret = bpf_add_kfunc_call(env, insn->imm, insn->off); - + ret = add_subprog(env, i + insn->imm + 1); if (ret < 0) return ret; } @@ -2900,6 +2895,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) return 0; } +static int add_kfuncs(struct bpf_verifier_env *env) +{ + struct bpf_insn *insn = env->prog->insnsi; + int i, ret, insn_cnt = env->prog->len; + + for (i = 0; i < insn_cnt; i++, insn++) { + if (!bpf_pseudo_kfunc_call(insn)) + continue; + + if (!env->bpf_capable) { + verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n"); + return -EPERM; + } + + ret = bpf_add_kfunc_call(env, insn->imm, insn->off); + if (ret < 0) + return ret; + } + + return 0; +} + static int check_subprogs(struct bpf_verifier_env *env) { int i, subprog_start, subprog_end, off, cur_subprog = 0; @@ -20140,7 +20157,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; - ret = add_subprog_and_kfunc(env); + /* Discover all subprograms before validating their layout and BTF. */ + ret = add_subprogs(env); + if (ret < 0) + goto skip_full_check; + + /* Collect the kfunc descriptors used during verification. */ + ret = add_kfuncs(env); if (ret < 0) goto skip_full_check; From d98b2d445fc530aa34bfc7abce7e06d2e761dc01 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 8 Aug 2026 02:39:23 +0200 Subject: [PATCH 251/373] bpf: Collect kfuncs after resolving program resources The kfunc descriptors include argument prototypes generated while calls are collected. Some argument classifications need program auxiliary state derived from referenced maps, such as the arena associated with the program. This avoids a footgun in get_kfunc_arg_type() checks where we do validation on whether program has prog->aux->arena and it hasn't been resolved yet. check_and_resolve_insns() records used maps and populates that state. It must remain after bpf_check_btf_info(), which applies kernel-side CO-RE relocations, so that instruction validation and the program tag observe the relocated instruction stream. Move only add_kfuncs() after instruction and resource resolution. Subprogram discovery and validation remain before the full BTF phase because that phase needs the complete subprogram layout. Add a short comment describing the resource resolution phase at the call site. Signed-off-by: Kumar Kartikeya Dwivedi Reviewed-by: Amery Hung Link: https://patch.msgid.link/20260808003938.3486067-4-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a54f7b63eaba..e17084666041 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20162,11 +20162,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; - /* Collect the kfunc descriptors used during verification. */ - ret = add_kfuncs(env); - if (ret < 0) - goto skip_full_check; - ret = check_subprogs(env); if (ret < 0) goto skip_full_check; @@ -20176,10 +20171,16 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* Validate instructions and resolve the program's referenced resources. */ ret = check_and_resolve_insns(env); if (ret < 0) goto skip_full_check; + /* Build kfunc prototypes after resolving program resources. */ + ret = add_kfuncs(env); + if (ret < 0) + goto skip_full_check; + if (bpf_prog_is_offloaded(env->prog->aux)) { ret = bpf_prog_offload_verifier_prep(env->prog); if (ret) From 252d367163268cb8d3fe321c1fc2b15a60faf9ea Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:24 +0200 Subject: [PATCH 252/373] bpf: Support __arena and __arena__nullable kfunc argument suffixes Passing an arena pointer to a kfunc takes two steps today. There is no arena pointer argument type, so the pointer crosses the boundary as a bare scalar, and the kfunc then offsets it by the arena base and casts it before it can touch the memory. Every such kfunc open-codes the same translation. Add the __arena and __arena__nullable argument suffixes to make this more convenient. The kfunc declares the parameter by its real pointer type and dereferences it directly, with the JIT rebasing the value at the call site, rN = kern_vm_start + (u32)rN. No bounds check is needed: the u32 offset stays within the guard-padded arena kernel mapping, and a fault on an unpopulated page recovers through the per-arena scratch page. A suffixed argument accepts a PTR_TO_ARENA or scalar register, matching global subprog arena arguments. __arena rebases unconditionally, so the kfunc never sees NULL and a value with zero in the low 32 bits arrives as the arena base. __arena__nullable preserves NULL for optional arguments by skipping the rebase when the truncated value, arena offset 0, is zero. Keeping the plain form NULL-free saves the NULL test on every call. The double separator makes the annotations composable: __arena__nullable also ends in __nullable and naturally follows the common nullable argument path. Plain __arena follows that path too for verifier type checking because both forms accept a constant zero; the function-model flag still determines whether the JIT preserves NULL or rebases it to the arena base. This patch adds the verifier side: the suffixes are recognized in check_kfunc_args() and distilled into argument flags in the function model stored in the kfunc descriptor. JITs retrieve the model while emitting the call, avoiding per-call state in insn_aux_data. JITs declare support with bpf_jit_supports_arena_args() and verification fails with -ENOTSUPP elsewhere. Co-developed-by: Kumar Kartikeya Dwivedi Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-5-memxor@gmail.com Signed-off-by: Eduard Zingerman --- Documentation/bpf/kfuncs.rst | 29 +++++++++++++++++++++++ include/linux/bpf.h | 6 +++++ include/linux/filter.h | 1 + kernel/bpf/btf.c | 18 +++++++++++++- kernel/bpf/core.c | 5 ++++ kernel/bpf/verifier.c | 46 ++++++++++++++++++++++++++++++++---- 6 files changed, 100 insertions(+), 5 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 021be6d93dfb..9c205d8b5fff 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -278,6 +278,33 @@ An example is given below:: ... } +2.3.8 __arena and __arena__nullable Annotations +----------------------------------------------- + +Both annotations indicate that the pointer argument points into the +calling program's arena. The JIT rebases the value at the call site so +the kfunc receives a directly dereferenceable kernel address, subject to +the access rules described in :ref:`BPF_kfunc_arena_access` (at most +``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access). + +With ``__arena`` the rebase is unconditional and the argument is never +NULL: a value whose lower 32 bits are zero arrives as the arena base +address (arena offset 0). The kfunc must not check the argument for NULL. +With ``__arena__nullable`` such a value arrives as NULL instead and the +kfunc must check before dereferencing. + +An example is given below:: + + __bpf_kfunc int bpf_process_item(struct item *item__arena) + { + ... + } + +Calling such a kfunc requires the program to use an arena map and a JIT with +arena argument support (currently x86-64); verification fails otherwise. The +program can pass any value without compromising the kernel. A value that does +not point into the arena is a program bug. + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function @@ -522,6 +549,8 @@ In order to accommodate such requirements, the verifier will enforce strict PTR_TO_BTF_ID type matching if two types have the exact same name, with one being suffixed with ``___init``. +.. _BPF_kfunc_arena_access: + 2.8 Accessing arena memory through kfunc arguments -------------------------------------------------- diff --git a/include/linux/bpf.h b/include/linux/bpf.h index d79bf7557ef6..ba1b9d8ac348 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1195,6 +1195,12 @@ struct bpf_prog_offload { /* The argument is signed. */ #define BTF_FMODEL_SIGNED_ARG BIT(1) +/* The argument is an arena pointer. */ +#define BTF_FMODEL_ARENA_ARG BIT(2) + +/* The argument is nullable. */ +#define BTF_FMODEL_NULLABLE_ARG BIT(3) + struct btf_func_model { u8 ret_size; u8 ret_flags; diff --git a/include/linux/filter.h b/include/linux/filter.h index 41b02d53e222..4edba8182db1 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1214,6 +1214,7 @@ bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_stack_args(void); +bool bpf_jit_supports_arena_args(void); bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); bool bpf_jit_supports_ptr_xchg(void); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 42414633cf26..4ff6148ae8e8 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7539,6 +7539,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t) return flags; } +static u8 __get_arg_fmodel_flags(const struct btf *btf, + const struct btf_param *arg, + const struct btf_type *t) +{ + u8 flags = __get_type_fmodel_flags(t); + + if (btf_param_match_suffix(btf, arg, "__arena__nullable")) + flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG; + else if (btf_param_match_suffix(btf, arg, "__arena")) + flags |= BTF_FMODEL_ARENA_ARG; + else if (btf_param_match_suffix(btf, arg, "__nullable")) + flags |= BTF_FMODEL_NULLABLE_ARG; + + return flags; +} + int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, const struct btf_type *func, @@ -7604,7 +7620,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log, return -EINVAL; } m->arg_size[i] = ret; - m->arg_flags[i] = __get_type_fmodel_flags(t); + m->arg_flags[i] = __get_arg_fmodel_flags(btf, &args[i], t); } m->nr_args = nargs; return 0; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index e2076667b245..a3e1fae32eac 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void) return false; } +bool __weak bpf_jit_supports_arena_args(void) +{ + return false; +} + bool __weak bpf_jit_supports_far_kfunc_call(void) { return false; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e17084666041..a1b71ac39b3b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10777,7 +10777,8 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg) { - return btf_param_match_suffix(btf, arg, "__nullable"); + return btf_param_match_suffix(btf, arg, "__nullable") || + btf_param_match_suffix(btf, arg, "__arena"); } static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg) @@ -10795,6 +10796,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param return btf_param_match_suffix(btf, arg, "__irq_flag"); } +static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg) +{ + return btf_param_match_suffix(btf, arg, "__arena__nullable") || + btf_param_match_suffix(btf, arg, "__arena"); +} + static bool is_kfunc_arg_scalar_with_name(const struct btf *btf, const struct btf_param *arg, const char *name) @@ -11015,6 +11022,7 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_IRQ_FLAG, KF_ARG_PTR_TO_RES_SPIN_LOCK, KF_ARG_PTR_TO_TASK_WORK, + KF_ARG_PTR_TO_ARENA, }; enum special_kfunc_type { @@ -11300,7 +11308,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, reg_arg_name(env, argno), btf_type_str(t)); return -EINVAL; } - ref_t = btf_type_skip_modifiers(meta->btf, t->type, NULL); ref_tname = btf_name_by_offset(meta->btf, ref_t->name_off); @@ -11349,7 +11356,30 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, arg_type = KF_ARG_PTR_TO_RES_SPIN_LOCK; else if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) arg_type = KF_ARG_PTR_TO_CALLBACK; - else if (arg + 1 < nargs && + else if (is_kfunc_arg_arena(meta->btf, &args[arg])) { + if (!bpf_jit_supports_arena_args()) { + verbose(env, "JIT does not support kfunc %s() with arena pointer arguments\n", + meta->func_name); + return -ENOTSUPP; + } + if (!env->prog->aux->arena) { + verbose(env, + "%s arena pointer requires a program with an associated arena\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + if (reg_from_argno(argno) < 0) { + verbose(env, "%s arena pointer cannot be a stack argument\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + /* + * Both suffixes accept a constant zero. The function model determines + * whether the JIT rebases it to the arena base or preserves NULL. + * The common nullable path below records that verifier property. + */ + arg_type = KF_ARG_PTR_TO_ARENA; + } else if (arg + 1 < nargs && (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) || is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && @@ -12024,7 +12054,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t = btf_type_skip_modifiers(btf, args[i].type, NULL); if (btf_type_is_ptr(t) && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && - !is_kfunc_arg_nullable(meta->btf, &args[i])) { + !type_may_be_null(kf_arg_type)) { verbose(env, "Possibly NULL pointer passed to trusted %s\n", reg_arg_name(env, argno)); return -EACCES; @@ -12077,6 +12107,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me case KF_ARG_PTR_TO_TASK_WORK: case KF_ARG_PTR_TO_IRQ_FLAG: case KF_ARG_PTR_TO_RES_SPIN_LOCK: + case KF_ARG_PTR_TO_ARENA: break; case KF_ARG_PTR_TO_DYNPTR: arg_type = ARG_PTR_TO_DYNPTR; @@ -12143,6 +12174,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me meta->ret_btf_id = ret; } break; + case KF_ARG_PTR_TO_ARENA: + if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { + verbose(env, "%s is not a pointer to arena or scalar\n", + reg_arg_name(env, argno)); + return -EINVAL; + } + break; case KF_ARG_PTR_TO_ALLOC_BTF_ID: if (reg->type == (PTR_TO_BTF_ID | MEM_ALLOC)) { if (!is_bpf_obj_drop_kfunc(meta->func_id)) { From f6c33c4479a7e4d6bfc0e0e533a86376866f7360 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:25 +0200 Subject: [PATCH 253/373] bpf: Support __arena and __arena__nullable on struct_ops arguments A struct_ops callback cannot receive an arena pointer directly, so passing one takes two steps. The pointer arrives as a bare u64 that the callback casts, and because the two sides address the arena through different bases it also has to be rebased by hand on the way in. Add the __arena and __arena__nullable stub argument suffixes to make this convenient. The callback declares the parameter as an arena pointer, receives it as a PTR_TO_ARENA register, and dereferences it directly, while the kernel caller just passes the natural kernel arena address (kaddr). The trampoline converts the value while saving the arguments into the BPF ctx, ctx[slot] = (u32)(kaddr - kern_vm_start), so the program never sees a kernel address and nothing rewrites the ctx after the fact. The converted value keeps the upper 32 bits clear as the JITs require of arena pointer registers and behaves like any cast_kern'ed arena pointer, so cast_user recovers the full user-visible address. __arena converts unconditionally and the kernel caller must not pass NULL. __arena__nullable preserves NULL, tested on the full 64-bit kernel pointer, and surfaces to the verifier as PTR_TO_ARENA (but not as a PTR_TO_ARENA | PTR_MAYBE_NULL). The reason is that PTR_TO_ARENA in the program's type state already encompasses NULL-ness, so it is not meaningful to force a NULL check for the program. The composite suffix intentionally ends in __nullable. Classify __arena__nullable before the generic suffix so scalar arena pointees do not take the generic nullable BTF pointer path. This patch adds the generic side. prepare_arg_info() records arena and nullable argument flags in the struct_ops function model, and bpf_tramp_arena_base() returns the arena base for a single-program struct_ops indirect trampoline. Only that trampoline converts: its program's arena is fixed at generation time. Generic trampolines can mix programs with different arenas and reject arena context arguments defensively, which is unreachable today as only struct_ops programs carry them. Architectures that do not implement the conversion are gated out at verification time with bpf_jit_supports_arena_args(). Co-developed-by: Kumar Kartikeya Dwivedi Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-6-memxor@gmail.com Signed-off-by: Eduard Zingerman --- Documentation/bpf/kfuncs.rst | 10 +++++++ include/linux/bpf.h | 9 ++++++ include/linux/bpf_verifier.h | 10 +++++++ kernel/bpf/bpf_struct_ops.c | 56 +++++++++++++++++++++++++++--------- kernel/bpf/btf.c | 10 +++++-- kernel/bpf/trampoline.c | 37 ++++++++++++++++++++++++ kernel/bpf/verifier.c | 23 +++++++++++---- 7 files changed, 133 insertions(+), 22 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 9c205d8b5fff..1004eb0bec61 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -305,6 +305,16 @@ arena argument support (currently x86-64); verification fails otherwise. The program can pass any value without compromising the kernel. A value that does not point into the arena is a program bug. +The suffixes have the same meaning on the arguments of struct_ops stub +functions, with the conversion running in the opposite direction. The +kernel caller passes the kernel arena address and the trampoline converts +it while saving the arguments, so the callback receives an arena pointer +it can dereference directly. With ``__arena`` the kernel caller must not +pass NULL. With ``__arena__nullable`` a NULL kernel pointer arrives as NULL. +However, there is no obligation to prove to the verifier that such a pointer is +non-NULL before use, in-line with existing semantics of arena pointers used in +a program (or obtained from any other source). + .. _BPF_kfunc_nodef: 2.4 Using an existing kernel function diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ba1b9d8ac348..b4a10c9878cf 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1274,6 +1274,15 @@ struct bpf_tramp_nodes { int nr_nodes; }; +/* + * The arena base against which a struct_ops trampoline converts the + * arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF + * ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline + * converts nothing. + */ +u64 bpf_tramp_arena_base(const struct btf_func_model *m, + struct bpf_tramp_nodes *tnodes, u32 flags); + struct bpf_tramp_run_ctx; /* Different use cases for BPF trampoline: diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 1ccf82d7ff8d..93f7c2075eea 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1297,6 +1297,16 @@ static inline u32 type_flag(u32 type) return type & ~BPF_BASE_TYPE_MASK; } +static inline bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog) +{ + int i; + + for (i = 0; i < prog->aux->ctx_arg_info_size; i++) + if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA) + return true; + return false; +} + static inline enum bpf_prog_type resolve_prog_type(const struct bpf_prog *prog) { return (prog->type == BPF_PROG_TYPE_EXT && prog->aux->saved_dst_prog_type) ? diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 4e7a48c02be5..d7c3030bc63b 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -147,6 +147,8 @@ void bpf_struct_ops_image_free(void *image) #define MAYBE_NULL_SUFFIX "__nullable" #define REFCOUNTED_SUFFIX "__ref" +#define ARENA_SUFFIX "__arena" +#define ARENA_MAYBE_NULL_SUFFIX "__arena__nullable" /* Prepare argument info for every nullable argument of a member of a * struct_ops type. @@ -159,7 +161,7 @@ void bpf_struct_ops_image_free(void *image) * to provide an array of struct bpf_ctx_arg_aux, which in turn provides * the information that used by the verifier to check the arguments of the * BPF struct_ops program assigned to the member. Here, we only care about - * the arguments that are marked as __nullable. + * the arguments that are marked as __nullable, __ref or __arena. * * The array of struct bpf_ctx_arg_aux is eventually assigned to * prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the @@ -172,10 +174,12 @@ static int prepare_arg_info(struct btf *btf, const char *st_ops_name, const char *member_name, const struct btf_type *func_proto, void *stub_func_addr, + struct btf_func_model *model, struct bpf_struct_ops_arg_info *arg_info) { const struct btf_type *stub_func_proto, *pointed_type; - bool is_nullable = false, is_refcounted = false; + bool is_nullable = false, is_refcounted = false, is_arena = false; + bool is_arena_nullable = false; const struct btf_param *stub_args, *args; struct bpf_ctx_arg_aux *info, *info_buf; u32 nargs, arg_no, info_cnt = 0; @@ -225,27 +229,39 @@ static int prepare_arg_info(struct btf *btf, /* Prepare info for every nullable argument */ info = info_buf; for (arg_no = 0; arg_no < nargs; arg_no++) { - /* Skip arguments that is not suffixed with - * "__nullable or __ref". + bool ptr_to_arena, ptr_to_struct; + + /* + * Skip arguments that are not suffixed with "__arena__nullable", + * "__arena", "__nullable", or "__ref". */ - is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no], - MAYBE_NULL_SUFFIX); + is_arena_nullable = btf_param_match_suffix(btf, &stub_args[arg_no], + ARENA_MAYBE_NULL_SUFFIX); + is_arena = btf_param_match_suffix(btf, &stub_args[arg_no], ARENA_SUFFIX); + is_nullable = !is_arena_nullable && + btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX); is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no], REFCOUNTED_SUFFIX); - if (is_nullable) + if (is_arena_nullable) + suffix = ARENA_MAYBE_NULL_SUFFIX; + else if (is_arena) + suffix = ARENA_SUFFIX; + else if (is_nullable) suffix = MAYBE_NULL_SUFFIX; else if (is_refcounted) suffix = REFCOUNTED_SUFFIX; else continue; - /* Should be a pointer to struct */ - pointed_type = btf_type_resolve_ptr(btf, - args[arg_no].type, - &arg_btf_id); - if (!pointed_type || - !btf_type_is_struct(pointed_type)) { + /* + * Should be a pointer to struct, or any pointer for __arena or + * __arena__nullable. + */ + pointed_type = btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id); + ptr_to_arena = pointed_type && (is_arena || is_arena_nullable); + ptr_to_struct = pointed_type && btf_type_is_struct(pointed_type); + if (!ptr_to_arena && !ptr_to_struct) { pr_warn("stub function %s has %s tagging to an unsupported type\n", stub_fname, suffix); goto err_out; @@ -268,7 +284,18 @@ static int prepare_arg_info(struct btf *btf, info->btf_id = arg_btf_id; info->btf = btf; info->offset = offset; - if (is_nullable) { + if (is_arena || is_arena_nullable) { + /* + * Both types get PTR_TO_ARENA. In verifier state, + * PTR_TO_ARENA encompasses potential NULL values, but + * we do not force the program to check it, or maintain + * precision around it, since it has no safety implication. + */ + info->reg_type = PTR_TO_ARENA; + model->arg_flags[arg_no] |= BTF_FMODEL_ARENA_ARG; + if (is_arena_nullable) + model->arg_flags[arg_no] |= BTF_FMODEL_NULLABLE_ARG; + } else if (is_nullable) { info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID | PTR_MAYBE_NULL; } else if (is_refcounted) { info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID; @@ -460,6 +487,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc, stub_func_addr = *(void **)(st_ops->cfi_stubs + moff); err = prepare_arg_info(btf, st_ops->name, mname, func_proto, stub_func_addr, + &st_ops->func_models[i], arg_info + i); if (err) goto errout; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 4ff6148ae8e8..6606187ed4f4 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6963,15 +6963,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, return false; } - /* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */ + /* + * Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or + * PTR_TO_ARENA (both nullable and non-nullable cases). + */ for (i = 0; i < prog->aux->ctx_arg_info_size; i++) { const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i]; u32 type, flag; type = base_type(ctx_arg_info->reg_type); flag = type_flag(ctx_arg_info->reg_type); - if (ctx_arg_info->offset == off && type == PTR_TO_BUF && - (flag & PTR_MAYBE_NULL)) { + if (ctx_arg_info->offset == off && + (type == PTR_TO_ARENA || + (type == PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) { info->reg_type = ctx_arg_info->reg_type; return true; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index ed7999ad6c66..e07af35ed040 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -529,6 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a return tnodes; } +/* + * The arena base against which save_args() converts the arguments marked + * with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline + * converts: it dispatches to a single prog whose arena is known at + * generation time. Return 0 when there is nothing to convert. + */ +u64 bpf_tramp_arena_base(const struct btf_func_model *m, + struct bpf_tramp_nodes *tnodes, u32 flags) +{ + const struct bpf_prog *prog; + int i; + + if (!(flags & BPF_TRAMP_F_INDIRECT) || + tnodes[BPF_TRAMP_FENTRY].nr_nodes != 1) + return 0; + + for (i = 0; i < m->nr_args; i++) + if (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG) + break; + if (i == m->nr_args) + return 0; + + /* Verification rejects an arena argument without an arena. */ + prog = tnodes[BPF_TRAMP_FENTRY].nodes[0]->link->prog; + if (WARN_ON_ONCE(!prog->aux->arena)) + return 0; + + return bpf_arena_get_kern_vm_start(prog->aux->arena); +} + static void bpf_tramp_image_free(struct bpf_tramp_image *im) { bpf_image_ksym_del(&im->ksym); @@ -920,6 +950,13 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node, int cnt = 0, i; kind = bpf_attach_type_to_tramp(node->link->prog); + /* + * Arena ctx args are converted only by struct_ops indirect + * trampolines. They must never be attached to a generic trampoline. + */ + if (WARN_ON_ONCE(bpf_prog_has_arena_ctx_arg(node->link->prog))) + return -ENOTSUPP; + if (tr->extension_prog) /* cannot attach fentry/fexit if extension prog is attached. * cannot overwrite extension prog either. diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a1b71ac39b3b..7d527f7c899e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18685,6 +18685,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) { const struct btf_type *t, *func_proto; const struct bpf_struct_ops_desc *st_ops_desc; + const struct bpf_struct_ops_arg_info *arg_info; const struct bpf_struct_ops *st_ops; const struct btf_member *member; struct bpf_prog *prog = env->prog; @@ -18763,10 +18764,23 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) return -EACCES; } - for (i = 0; i < st_ops_desc->arg_info[member_idx].cnt; i++) { - if (st_ops_desc->arg_info[member_idx].info[i].refcounted) { + arg_info = &st_ops_desc->arg_info[member_idx]; + for (i = 0; i < arg_info->cnt; i++) { + const struct bpf_ctx_arg_aux *info = &arg_info->info[i]; + + if (info->refcounted) has_refcounted_arg = true; - break; + if (base_type(info->reg_type) == PTR_TO_ARENA) { + if (!bpf_jit_supports_arena_args()) { + verbose(env, "JIT does not support arena arguments\n"); + return -ENOTSUPP; + } + if (!prog->aux->arena) { + verbose(env, + "arena argument of %s requires a program with an associated arena\n", + mname); + return -EINVAL; + } } } @@ -18787,8 +18801,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) prog->aux->attach_func_name = mname; env->ops = st_ops->verifier_ops; - return bpf_prog_ctx_arg_info_init(prog, st_ops_desc->arg_info[member_idx].info, - st_ops_desc->arg_info[member_idx].cnt); + return bpf_prog_ctx_arg_info_init(prog, arg_info->info, arg_info->cnt); } #define SECURITY_PREFIX "security_" From 41b75522304c8237151289d3db5e7f275c593e74 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:26 +0200 Subject: [PATCH 254/373] bpf, x86: JIT __arena kfunc argument rebasing Implement arena argument rebasing for kfunc calls on x86. R12 already holds kern_vm_start whenever the prog has an arena, so each tagged argument costs two instructions emitted right before the call: movl %eN, %eN /* truncate, clear the upper 32 bits */ addq %r12, %rN A nullable argument tests the truncated value and jumps over the add: movl %eN, %eN testl %eN, %eN jz 1f addq %r12, %rN 1: addq carries a REX prefix for every argument register and is always three bytes, so the jz displacement is constant. The sequence is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees the rebase and needs no special handling. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-7-memxor@gmail.com Signed-off-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 50 +++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 88ed95b2eaa7..107b9901fba8 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1678,6 +1678,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip, return 0; } +/* + * Rebase the __arena args of a kfunc call to arena kernel addresses, + * rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable + * arg preserves NULL by skipping the add, tested on the truncated value as + * arena NULL is offset 0. Return the number of emitted bytes. + */ +static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog, + const struct bpf_insn *insn, u8 **pprog) +{ + const struct btf_func_model *fm; + u8 *prog = *pprog; + u8 *start = prog; + int i; + + fm = bpf_jit_find_kfunc_model(bpf_prog, insn); + if (!fm) + return -EINVAL; + + for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { + u8 flags = fm->arg_flags[i]; + u32 reg = BPF_REG_1 + i; + + if (!(flags & BTF_FMODEL_ARENA_ARG)) + continue; + if (WARN_ON_ONCE(!bpf_prog->aux->arena)) + return -EINVAL; + + /* mov eN, eN: truncate and clear the upper 32 bits */ + emit_mov_reg(&prog, false, reg, reg); + if (flags & BTF_FMODEL_NULLABLE_ARG) { + /* test eN, eN; jz over the 3-byte add */ + maybe_emit_mod(&prog, reg, reg, false); + EMIT2(0x85, add_2reg(0xC0, reg, reg)); + EMIT2(X86_JE, 3); + } + /* add rN, r12 */ + maybe_emit_mod(&prog, reg, X86_REG_R12, true); + EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12)); + } + + *pprog = prog; + return prog - start; +} + static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding) { @@ -2588,6 +2632,12 @@ st: insn_off = insn->off; } if (!imm32) return -EINVAL; + if (src_reg == BPF_PSEUDO_KFUNC_CALL) { + err = emit_kfunc_arena_args(bpf_prog, insn, &prog); + if (err < 0) + return err; + ip += err; + } if (priv_frame_ptr) { push_r9(&prog); ip += 2; From 5129e9be211114318fd91a62cae0e3836c6611ee Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:27 +0200 Subject: [PATCH 255/373] bpf, x86: Convert struct_ops arena arguments in the trampoline Implement the struct_ops arena argument conversion on x86. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through RAX: movl %esrc, %eax /* truncate and clear the upper 32 bits */ subl $base_lo, %eax movq %rax, ctx_slot A nullable argument tests the full 64-bit kernel pointer first: movq %rsrc, %rax testq %rax, %rax jz 1f subl $base_lo, %eax 1: movq %rax, ctx_slot The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through RAX, so only the subtraction (and the NULL test) is inserted there. Keep arena and nullable classification in btf_func_model. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline; other trampolines pass zero and perform no conversion. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for x86. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-8-memxor@gmail.com Signed-off-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 57 +++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 107b9901fba8..162fbd2ba1df 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3048,8 +3048,35 @@ static int get_nr_used_regs(const struct btf_func_model *m) return nr_used_regs; } +/* + * Convert an arena kernel address into the arena pointer form on its way + * into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg + * preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit + * subtraction both truncates and clears the upper half, so the stored + * value satisfies the JIT invariant for arena pointer registers. + */ +static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo) +{ + u8 *prog = *pprog; + + if (nullable) { + if (src_reg != BPF_REG_0) + emit_mov_reg(&prog, true, BPF_REG_0, src_reg); + /* test rax, rax; jz over the 5-byte sub */ + EMIT3(0x48, 0x85, 0xC0); + EMIT2(X86_JE, 5); + } else if (src_reg != BPF_REG_0) { + emit_mov_reg(&prog, false, BPF_REG_0, src_reg); + } + /* sub eax, base_lo */ + EMIT1_off32(0x2D, base_lo); + + *pprog = prog; +} + static void save_args(const struct btf_func_model *m, u8 **prog, - int stack_size, bool for_call_origin, u32 flags) + int stack_size, bool for_call_origin, u32 flags, + u64 arena_base) { int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0; bool use_jmp = bpf_trampoline_use_jmp(flags); @@ -3061,6 +3088,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog, * mov QWORD PTR [rbp-0x8],rsi */ for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) { + bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG); + bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG; + arg_regs = (m->arg_size[i] + 7) / 8; /* According to the research of Yonghong, struct members @@ -3094,6 +3124,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog, for (j = 0; j < arg_regs; j++) { emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, nr_stack_slots * 8 + 16 + (!use_jmp) * 8); + if (arena_arg) + emit_arena_arg_conv(prog, BPF_REG_0, nullable, + (u32)arena_base); emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); @@ -3114,9 +3147,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog, /* copy the arguments from regs into stack */ for (j = 0; j < arg_regs; j++) { - emit_stx(prog, BPF_DW, BPF_REG_FP, - nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs, - -stack_size); + u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs; + + if (arena_arg) { + emit_arena_arg_conv(prog, src, nullable, (u32)arena_base); + src = BPF_REG_0; + } + emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size); stack_size -= 8; nr_regs++; } @@ -3412,6 +3449,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im void *orig_call = func_addr; int cookie_off, cookie_cnt; u8 **branches = NULL; + u64 arena_base; u64 func_meta; u8 *prog; bool save_ret; @@ -3424,6 +3462,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); + arena_base = bpf_tramp_arena_base(m, tnodes, flags); + for (i = 0; i < m->nr_args; i++) nr_regs += (m->arg_size[i] + 7) / 8 - 1; @@ -3558,7 +3598,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr); } - save_args(m, &prog, regs_off, false, flags); + save_args(m, &prog, regs_off, false, flags, arena_base); if (flags & BPF_TRAMP_F_CALL_ORIG) { /* arg1: mov rdi, im */ @@ -3600,7 +3640,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, regs_off); - save_args(m, &prog, arg_stack_off, true, flags); + save_args(m, &prog, arg_stack_off, true, flags, 0); if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { /* Before calling the original function, load the @@ -4101,6 +4141,11 @@ bool bpf_jit_supports_stack_args(void) return true; } +bool bpf_jit_supports_arena_args(void) +{ + return true; +} + void *bpf_arch_text_copy(void *dst, void *src, size_t len) { if (text_poke_copy(dst, src, len) == NULL) From 0996e93691f1acbf6660fe51f2a90e0df6799769 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:28 +0200 Subject: [PATCH 256/373] selftests/bpf: Add kfunc __arena and __arena__nullable argument tests Add arena-argument kfuncs to bpf_testmod, which also exercises the argument rebasing on module kfuncs, and tests covering the accepted argument forms (arena pointer, low 32 bits as a scalar, full user address as a scalar), the exact rebase semantics via capture kfuncs returning the raw argument (zero low 32 bits arrive as the arena kernel base under __arena and as NULL under __arena__nullable), five arena arguments in one call, a mixed __arena plus __arena__nullable call exercising both bitmasks on one call site, a kernel-side dereference of an unpopulated page recovering through the scratch page, and the rejections (no arena in the program, incompatible register type). The tests run on x86-64 and skip elsewhere, as programs with arena-tagged kfunc args fail verification where the JIT lacks support. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-9-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 3 + .../testing/selftests/bpf/progs/arena_kfunc.c | 234 ++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 44 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 9 + 4 files changed, 290 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 0baa74618fa0..608050c60309 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -2,6 +2,7 @@ #include +#include "arena_kfunc.skel.h" #include "cap_helpers.h" #include "verifier_align.skel.h" #include "verifier_and.skel.h" @@ -162,6 +163,8 @@ static void run_tests_aux(const char *skel_name, #define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL) +void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); } + void test_verifier_align(void) { RUN(verifier_align); } void test_verifier_and(void) { RUN(verifier_and); } void test_verifier_arena(void) { RUN(verifier_arena); } diff --git a/tools/testing/selftests/bpf/progs/arena_kfunc.c b/tools/testing/selftests/bpf/progs/arena_kfunc.c new file mode 100644 index 000000000000..cdcea889da58 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_kfunc.c @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#define BPF_NO_KFUNC_PROTOTYPES +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" +#include +#include "../test_kmods/bpf_testmod_kfunc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + /* page 0 hosts the arena global, page 1 is for allocations */ + __uint(max_entries, 2); +} arena SEC(".maps"); + +/* + * Occupies page 0 so no allocation lands at arena offset 0, which the + * nullable tests below must be able to tell apart from NULL. + */ +u64 __arena arena_pad; + +/* volatile to force the scalar reloads below */ +volatile u64 stash; + +SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_forms(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + u64 ret; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + /* PTR_TO_ARENA argument */ + *val = 41; + ret = bpf_kfunc_arena_arg_test((u64 *)val); + if (ret != 41 || *val != 42) + return 2; + + /* the low 32 bits as a scalar */ + stash = (u32)(u64)val; + ret = bpf_kfunc_arena_arg_test((u64 *)stash); + if (ret != 42 || *val != 43) + return 3; + + /* the full user address as a scalar */ + stash = (u64)val; + bpf_addr_space_cast(stash, 1, 0); + ret = bpf_kfunc_arena_arg_test((u64 *)stash); + if (ret != 43 || *val != 44) + return 4; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +/* + * Pin the rebase semantics using the capture kfuncs, which return the raw + * argument value: __arena rebases unconditionally, so zero low 32 bits + * arrive as the arena kernel base, while __arena__nullable turns them into + * NULL. + */ +SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_rebase(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + u64 base, off; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + base = bpf_kfunc_arena_cap_test(NULL); + if (!base) + return 2; + + /* only the low 32 bits contribute */ + stash = 0xbadc0ffe00000000; + if (bpf_kfunc_arena_cap_test((u64 *)stash) != base) + return 3; + + off = (u32)(u64)val; + if (bpf_kfunc_arena_cap_test((u64 *)val) != base + off) + return 4; + + if (bpf_kfunc_arena_cap_nullable_test(NULL) != 0) + return 5; + + stash = 0xbadc0ffe00000000; + if (bpf_kfunc_arena_cap_nullable_test((u64 *)stash) != 0) + return 6; + + if (bpf_kfunc_arena_cap_nullable_test((u64 *)val) != base + off) + return 7; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_args5(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + val[0] = 1; + val[1] = 2; + val[2] = 4; + val[3] = 8; + val[4] = 16; + + if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1], + (u64 *)&val[2], (u64 *)&val[3], + (u64 *)&val[4]) != 31) + return 2; + if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1], + (u64 *)&val[2], (u64 *)&val[3], NULL) != 15) + return 3; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_mixed(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + val[0] = 7; + val[1] = 5; + + if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], NULL) != 7) + return 2; + + if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], (u64 *)&val[1]) != 12) + return 3; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +/* kernel-side faults on unpopulated pages recover via the scratch page */ +SEC("syscall") +__arch_x86_64 +__success __retval(0) +int arena_arg_unpopulated(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + stash = (u64)val + PAGE_SIZE; + bpf_kfunc_arena_arg_test((u64 *)stash); + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} + +SEC("syscall") +__arch_x86_64 +__failure __msg("arena pointer requires a program with an associated arena") +int arena_arg_no_arena(void *ctx) +{ + bpf_kfunc_arena_arg_test((u64 *)1); + return 0; +} + +SEC("syscall") +__arch_x86_64 +__failure __msg("is not a pointer to arena or scalar") +int arena_arg_bad_reg(void *ctx) +{ + u64 buf = 0; + + /* use the arena so the program passes the arena presence check */ + bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + bpf_kfunc_arena_arg_test(&buf); + return 0; +} + +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT) +SEC("syscall") +__arch_x86_64 +__failure __msg("arena pointer cannot be a stack argument") +int arena_arg_stack(void *ctx) +{ + bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + bpf_kfunc_arena_stack_arg_test(1, 2, 3, 4, 5, (u64 *)1); + return 0; +} +#else +SEC("syscall") +__arch_x86_64 +__description("arena_arg_stack: not supported, dummy test") +__success +int arena_arg_stack(void *ctx) +{ + return 0; +} +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 0585794606ed..2291bb466517 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -237,6 +237,44 @@ __bpf_kfunc void bpf_kfunc_common_test(void) { } +__bpf_kfunc u64 bpf_kfunc_arena_arg_test(u64 *val__arena) +{ + u64 old; + + old = *val__arena; + *val__arena = old + 1; + return old; +} + +__bpf_kfunc u64 bpf_kfunc_arena_cap_test(u64 *val__arena) +{ + return (u64)val__arena; +} + +__bpf_kfunc u64 bpf_kfunc_arena_cap_nullable_test(u64 *val__arena__nullable) +{ + return (u64)val__arena__nullable; +} + +__bpf_kfunc u64 bpf_kfunc_arena_args5_test(u64 *a__arena, u64 *b__arena, + u64 *c__arena, u64 *d__arena, + u64 *e__arena__nullable) +{ + return *a__arena + *b__arena + *c__arena + *d__arena + + (e__arena__nullable ? *e__arena__nullable : 0); +} + +__bpf_kfunc u64 bpf_kfunc_arena_stack_arg_test(u64 a, u64 b, u64 c, u64 d, u64 e, + u64 *f__arena) +{ + return a + b + c + d + e + *f__arena; +} + +__bpf_kfunc u64 bpf_kfunc_arena_mixed_test(u64 *a__arena, u64 *b__arena__nullable) +{ + return *a__arena + (b__arena__nullable ? *b__arena__nullable : 0); +} + __bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nullable) { @@ -755,6 +793,12 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_iter_testmod_seq_value) BTF_ID_FLAGS(func, bpf_kfunc_common_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_arg_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_nullable_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_args5_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_stack_arg_test) +BTF_ID_FLAGS(func, bpf_kfunc_arena_mixed_test) BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1) BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test) BTF_ID_FLAGS(func, bpf_kfunc_nested_acquire_nonzero_offset_test, KF_ACQUIRE) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index c36bb911defa..453bfd94154f 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -98,6 +98,15 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym; void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym; +__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_cap_test(__u64 *val__arena) __ksym; +__u64 bpf_kfunc_arena_cap_nullable_test(__u64 *val__arena__nullable) __ksym; +__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena, + __u64 *c__arena, __u64 *d__arena, + __u64 *e__arena__nullable) __ksym; +__u64 bpf_kfunc_arena_stack_arg_test(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, + __u64 *f__arena) __ksym; +__u64 bpf_kfunc_arena_mixed_test(__u64 *a__arena, __u64 *b__arena__nullable) __ksym; int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) __ksym; int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym; int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym; From 25818556edcf6cb3c2ef2007f7d137e06877f698 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:29 +0200 Subject: [PATCH 257/373] selftests/bpf: Add JIT-sequence tests for __arena kfunc arguments Pin the exact rebase sequences the JITs emit for __arena and __arena__nullable kfunc arguments with __jited assertions on x86-64: the unconditional truncate-and-add, the nullable test-and-skip variant, and all five argument registers in one call, which also covers the REX-prefixed encoding of r8 on x86. The capture kfuncs take the argument without dereferencing, so only the emitted code is under test. The tests skip without LLVM disassembler support. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-10-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 3 + .../selftests/bpf/progs/arena_kfunc_jit.c | 98 +++++++++++++++++++ 2 files changed, 101 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/arena_kfunc_jit.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 608050c60309..5b265af3b1d5 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -3,6 +3,7 @@ #include #include "arena_kfunc.skel.h" +#include "arena_kfunc_jit.skel.h" #include "cap_helpers.h" #include "verifier_align.skel.h" #include "verifier_and.skel.h" @@ -165,6 +166,8 @@ static void run_tests_aux(const char *skel_name, void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); } +void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); } + void test_verifier_align(void) { RUN(verifier_align); } void test_verifier_and(void) { RUN(verifier_and); } void test_verifier_arena(void) { RUN(verifier_arena); } diff --git a/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c b/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c new file mode 100644 index 000000000000..c9b918662616 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +/* + * Verify the JIT-emitted rebase sequences for __arena and __arena__nullable + * kfunc arguments. The capture kfuncs take the argument without + * dereferencing it, so these tests pin only the emitted code. + */ +#define BPF_NO_KFUNC_PROTOTYPES +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" +#include +#include "../test_kmods/bpf_testmod_kfunc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1); +} arena SEC(".maps"); + +/* volatile to force the scalar reloads below */ +volatile u64 stash; + +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + +SEC("syscall") +__arch_x86_64 +__jited("...") +__jited(" movl %edi, %edi") +__jited(" addq %r12, %rdi") +__jited("...") +__jited(" callq {{.*}}") +__success +int arena_arg_jit_rebase(void *ctx) +{ + stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + bpf_kfunc_arena_cap_test((u64 *)stash); + return 0; +} + +SEC("syscall") +__arch_x86_64 +__jited("...") +__jited(" movl %edi, %edi") +__jited(" testl %edi, %edi") +__jited(" je L0") +__jited(" addq %r12, %rdi") +__jited("L0: callq {{.*}}") +__success +int arena_arg_jit_nullable(void *ctx) +{ + stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + bpf_kfunc_arena_cap_nullable_test((u64 *)stash); + return 0; +} + +SEC("syscall") +__arch_x86_64 +__jited("...") +__jited(" movl %edi, %edi") +__jited(" addq %r12, %rdi") +__jited(" movl %esi, %esi") +__jited(" addq %r12, %rsi") +__jited(" movl %edx, %edx") +__jited(" addq %r12, %rdx") +__jited(" movl %ecx, %ecx") +__jited(" addq %r12, %rcx") +__jited(" movl %r8d, %r8d") +__jited(" testl %r8d, %r8d") +__jited(" je L0") +__jited(" addq %r12, %r8") +__jited("L0: callq {{.*}}") +__success +int arena_arg_jit_args5(void *ctx) +{ + u64 __arena *val; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + val[0] = 1; + val[1] = 2; + val[2] = 4; + val[3] = 8; + val[4] = 16; + + bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1], + (u64 *)&val[2], (u64 *)&val[3], + (u64 *)&val[4]); + return 0; +} + +#endif /* __BPF_FEATURE_ADDR_SPACE_CAST */ + +char _license[] SEC("license") = "GPL"; From ba0484197163bb1de490965d779211be281429f8 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:30 +0200 Subject: [PATCH 258/373] selftests/bpf: Add struct_ops __arena and __arena__nullable argument tests Add test_arena and test_arena_nullable members to bpf_testmod_ops3 with arena-tagged stub arguments and kfuncs that forward a caller-provided pointer to them. The kfuncs take arena-tagged arguments, so each round trip exercises both conversion directions end to end: the kfunc receives a kernel arena address and the trampoline converts it back to an arena pointer for the callback. The non-nullable callback dereferences its argument with no NULL branch and captures the raw ctx value, which the trigger program compares against the arena offset of the passed object, pinning the exact (u32)(kaddr - kern_vm_start) conversion. The nullable callback verifies that only a true kernel NULL arrives as NULL. Failure coverage: a program with no arena is rejected when it loads. The tests run on x86-64 and skip elsewhere, as the programs fail verification where the JIT lacks arena argument support. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-11-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../bpf/prog_tests/test_struct_ops_arena.c | 74 +++++++++++++++ .../selftests/bpf/progs/struct_ops_arena.c | 94 +++++++++++++++++++ .../bpf/progs/struct_ops_arena_fail.c | 20 ++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 24 +++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 3 + .../bpf/test_kmods/bpf_testmod_kfunc.h | 2 + 6 files changed, 217 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c new file mode 100644 index 000000000000..323d707c543f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include + +#include "struct_ops_arena.skel.h" +#include "struct_ops_arena_fail.skel.h" + +#if defined(__x86_64__) +/* + * Attach callbacks with __arena and __arena__nullable arguments and drive + * them through the bpf_testmod_ops3_call_test_arena*() kfuncs. + */ +static void arena_arg(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct struct_ops_arena *skel; + struct bpf_link *link = NULL; + int err; + + skel = struct_ops_arena__open_and_load(); + if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load")) + return; + + link = bpf_map__attach_struct_ops(skel->maps.testmod_arena); + if (!ASSERT_OK_PTR(link, "attach_struct_ops")) + goto out; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger), + &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "trigger_retval"); + +out: + bpf_link__destroy(link); + struct_ops_arena__destroy(skel); +} + +/* + * A program with no arena cannot attach to a member with an __arena + * argument. + */ +static void arena_arg_fail(void) +{ + struct struct_ops_arena_fail *skel; + + skel = struct_ops_arena_fail__open_and_load(); + if (ASSERT_ERR_PTR(skel, "struct_ops_arena_fail__open_and_load")) + return; + + struct_ops_arena_fail__destroy(skel); +} +#endif + +/* + * Serialized because it attaches the singleton bpf_testmod_ops3, which + * test_struct_ops_private_stack also attaches; registering it twice fails + * with -EEXIST. + */ +void serial_test_struct_ops_arena(void) +{ + /* + * Arena struct_ops arguments need JIT support, currently x86-64 only. + * Elsewhere verification fails with "JIT does not support arena + * arguments", so the programs cannot even load. + */ +#if defined(__x86_64__) + if (test__start_subtest("arena_arg")) + arena_arg(); + if (test__start_subtest("arena_arg_fail")) + arena_arg_fail(); +#else + test__skip(); +#endif +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena.c b/tools/testing/selftests/bpf/progs/struct_ops_arena.c new file mode 100644 index 000000000000..40c856a748d2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#define BPF_NO_KFUNC_PROTOTYPES +#include +#include +#include "bpf_experimental.h" +#include +#include "../test_kmods/bpf_testmod.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + /* page 0 hosts the arena globals, page 1 is for allocations */ + __uint(max_entries, 2); +} arena SEC(".maps"); + +/* also associates the callbacks with the arena */ +u64 __arena arena_touch; +/* raw value of the last __arena ctx argument, captured by test_arena_cb */ +u64 __arena cb_ptr_val; + +SEC("struct_ops/test_arena") +int test_arena_cb(unsigned long long *ctx) +{ + u64 __arena *ptr = (u64 __arena *)ctx[0]; + + arena_touch++; + cb_ptr_val = ctx[0]; + *ptr += 1; + return 0; +} + +SEC("struct_ops/test_arena_nullable") +int test_arena_nullable_cb(unsigned long long *ctx) +{ + u64 __arena *ptr = (u64 __arena *)ctx[0]; + + arena_touch++; + if (!ptr) + return 0xbee; + *ptr += 1; + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena = { + .test_arena = (void *)test_arena_cb, + .test_arena_nullable = (void *)test_arena_nullable_cb, +}; + +SEC("syscall") +int trigger(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + u64 __arena *val; + int ret; + + val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!val) + return 1; + + *val = 41; + ret = bpf_testmod_ops3_call_test_arena((u64 *)val); + if (ret) + return 2; + if (*val != 42) + return 3; + + /* + * The callback must have seen exactly (u32)(kaddr - kern_vm_start), + * which is the arena offset of val with the upper 32 bits clear. + */ + if (cb_ptr_val != (u32)(u64)val) + return 4; + + ret = bpf_testmod_ops3_call_test_arena_nullable((u64 *)val); + if (ret) + return 5; + if (*val != 43) + return 6; + + /* NULL survives the nullable kfunc and the trampoline as NULL */ + ret = bpf_testmod_ops3_call_test_arena_nullable(NULL); + if (ret != 0xbee) + return 7; + + bpf_arena_free_pages(&arena, (void __arena *)val, 1); +#endif + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c b/tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c new file mode 100644 index 000000000000..1c0ec727d637 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include +#include +#include "../test_kmods/bpf_testmod.h" + +char _license[] SEC("license") = "GPL"; + +/* No arena in the program: attaching to test_arena must be rejected. */ +SEC("struct_ops/test_arena") +int test_arena_no_arena(unsigned long long *ctx) +{ + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops3 testmod_arena_fail = { + .test_arena = (void *)test_arena_no_arena, +}; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 2291bb466517..1e6d632c6f83 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -385,9 +385,21 @@ static int bpf_testmod_test_4(void) return 0; } +static int bpf_testmod_ops3__test_arena(u64 *ptr__arena) +{ + return 0; +} + +static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable) +{ + return 0; +} + static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, + .test_arena = bpf_testmod_ops3__test_arena, + .test_arena_nullable = bpf_testmod_ops3__test_arena_nullable, }; static void bpf_testmod_test_struct_ops3(void) @@ -406,6 +418,16 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void) st_ops3->test_2(); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena) +{ + return st_ops3->test_arena(ptr__arena); +} + +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nullable) +{ + return st_ops3->test_arena_nullable(ptr__arena__nullable); +} + struct bpf_testmod_btf_type_tag_1 { int a; }; @@ -814,6 +836,8 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable) BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test); BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test); BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h index 863fd10f1619..c367ec856776 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h @@ -106,6 +106,9 @@ struct bpf_testmod_ops2 { struct bpf_testmod_ops3 { int (*test_1)(void); int (*test_2)(void); + /* Used to test arena pointer arguments. */ + int (*test_arena)(u64 *ptr); + int (*test_arena_nullable)(u64 *ptr); }; struct st_ops_args { diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index 453bfd94154f..ea1747e2ad1f 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -120,6 +120,8 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym; #endif void bpf_testmod_test_mod_kfunc(int i) __ksym; +int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; +int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym; From 596824de8c10458653f0951d4a345858a1329767 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:31 +0200 Subject: [PATCH 259/373] bpf, x86: Fix stack-passed arguments for indirect trampolines save_args() reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of a struct_ops callback with more than six argument slots is read one slot off. This has gone unnoticed because no in-tree struct_ops member passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treat BPF_TRAMP_F_INDIRECT the same way. Fixes: 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING") Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Tested-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-12-memxor@gmail.com Signed-off-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 162fbd2ba1df..8dddb5d7af21 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3080,6 +3080,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog, { int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0; bool use_jmp = bpf_trampoline_use_jmp(flags); + int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24; int i, j; /* Store function arguments to stack. @@ -3114,16 +3115,16 @@ static void save_args(const struct btf_func_model *m, u8 **prog, /* copy function arguments from origin stack frame * into current stack frame. * - * The starting address of the arguments on-stack - * is: - * rbp + 8(push rbp) + - * 8(return addr of origin call) + - * 8(return addr of the caller) - * which means: rbp + 24 + * The arguments on-stack start above the saved rbp + * and the return addresses: two return addresses + * (origin call and caller) when the trampoline is + * entered through the fentry call, so rbp + 24, and + * a single one when it is entered with a jmp or + * called indirectly, so rbp + 16. */ for (j = 0; j < arg_regs; j++) { emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, - nr_stack_slots * 8 + 16 + (!use_jmp) * 8); + nr_stack_slots * 8 + stack_args_off); if (arena_arg) emit_arena_arg_conv(prog, BPF_REG_0, nullable, (u32)arena_base); From 2d4de9a493a01e977914517bcc43b7b9a63ee50b Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Sat, 8 Aug 2026 02:39:32 +0200 Subject: [PATCH 260/373] selftests/bpf: Test stack-passed struct_ops arena arguments Add a test_arena_stack member with eight leading scalar arguments so the arena pointer is passed on the stack. The callback validates the first and last scalar ctx slots before dereferencing the pointer in ctx[8]. This exercises the indirect trampoline stack layout and arena conversion together, and prevents a regression where stack arguments are read one slot late. Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi Tested-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-13-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/progs/struct_ops_arena.c | 21 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 14 +++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 3 +++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 1 + 4 files changed, 39 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena.c b/tools/testing/selftests/bpf/progs/struct_ops_arena.c index 40c856a748d2..ba04c73d8d96 100644 --- a/tools/testing/selftests/bpf/progs/struct_ops_arena.c +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena.c @@ -46,10 +46,24 @@ int test_arena_nullable_cb(unsigned long long *ctx) return 0; } +SEC("struct_ops/test_arena_stack") +int test_arena_stack_cb(unsigned long long *ctx) +{ + u64 __arena *ptr = (u64 __arena *)ctx[8]; + + arena_touch++; + /* pin the slot layout: the leading args fill ctx[0]..ctx[7] */ + if (ctx[0] != 1 || ctx[7] != 8) + return 0xbad; + *ptr += 1; + return 0; +} + SEC(".struct_ops.link") struct bpf_testmod_ops3 testmod_arena = { .test_arena = (void *)test_arena_cb, .test_arena_nullable = (void *)test_arena_nullable_cb, + .test_arena_stack = (void *)test_arena_stack_cb, }; SEC("syscall") @@ -88,6 +102,13 @@ int trigger(void *ctx) if (ret != 0xbee) return 7; + /* the arena pointer is stack-passed into the trampoline here */ + ret = bpf_testmod_ops3_call_test_arena_stack((u64 *)val); + if (ret) + return 8; + if (*val != 44) + return 9; + bpf_arena_free_pages(&arena, (void __arena *)val, 1); #endif return 0; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 1e6d632c6f83..a6133f7521f3 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -395,11 +395,19 @@ static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable) return 0; } +static int bpf_testmod_ops3__test_arena_stack(u64 a, u64 b, u64 c, u64 d, + u64 e, u64 f, u64 g, u64 h, + u64 *ptr__arena) +{ + return 0; +} + static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, .test_arena = bpf_testmod_ops3__test_arena, .test_arena_nullable = bpf_testmod_ops3__test_arena_nullable, + .test_arena_stack = bpf_testmod_ops3__test_arena_stack, }; static void bpf_testmod_test_struct_ops3(void) @@ -428,6 +436,11 @@ __bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nulla return st_ops3->test_arena_nullable(ptr__arena__nullable); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena) +{ + return st_ops3->test_arena_stack(1, 2, 3, 4, 5, 6, 7, 8, ptr__arena); +} + struct bpf_testmod_btf_type_tag_1 { int a; }; @@ -838,6 +851,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack) BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test); BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test); BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h index c367ec856776..33f2af5b7085 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h @@ -109,6 +109,9 @@ struct bpf_testmod_ops3 { /* Used to test arena pointer arguments. */ int (*test_arena)(u64 *ptr); int (*test_arena_nullable)(u64 *ptr); + /* enough leading args to force @ptr onto the stack on x86 and arm64 */ + int (*test_arena_stack)(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, + u64 g, u64 h, u64 *ptr); }; struct st_ops_args { diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index ea1747e2ad1f..c4383acb53c1 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -122,6 +122,7 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym; void bpf_testmod_test_mod_kfunc(int i) __ksym; int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym; +int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym; From fd6094ac877cf5efe6702c0cfc86802dd9a4f322 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 8 Aug 2026 02:39:33 +0200 Subject: [PATCH 261/373] bpf: Reject tracing/freplace progs for struct_ops with arena args Reject tracing and freplace attachments to a target program with arena context arguments. The struct_ops indirect trampoline converts those arguments before entering the target, so a generic tracing trampoline would otherwise expose arena offsets using the target BTF pointer type. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-14-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7d527f7c899e..add3affc5703 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19068,6 +19068,16 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "Subprog %s doesn't exist\n", tname); return -EINVAL; } + /* + * A struct_ops indirect trampoline converts arena arguments + * before invoking its program. A tracing or extension program + * attached to the main program would see the converted offset as a + * regular BTF pointer. + */ + if (subprog == 0 && bpf_prog_has_arena_ctx_arg(tgt_prog)) { + bpf_log(log, "Cannot attach to a target with arena context arguments\n"); + return -EOPNOTSUPP; + } if (aux->func && aux->func[subprog]->aux->exception_cb) { bpf_log(log, "%s programs cannot attach to exception callback\n", From 4976cce08baa12e4b7ea6dbd1a820d66b09014db Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 8 Aug 2026 02:39:34 +0200 Subject: [PATCH 262/373] selftests/bpf: Test attach rejection for struct_ops arena programs Exercise fentry, fexit, and freplace programs that target a struct_ops callback with an arena context argument. Verify each load is rejected with -EOPNOTSUPP and the arena-specific verifier diagnostic. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260808003938.3486067-15-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../bpf/prog_tests/test_struct_ops_arena.c | 54 +++++++++++++++++++ .../bpf/progs/struct_ops_arena_attach.c | 25 +++++++++ 2 files changed, 79 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c index 323d707c543f..940ec2cda0d5 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c @@ -3,6 +3,7 @@ #include #include "struct_ops_arena.skel.h" +#include "struct_ops_arena_attach.skel.h" #include "struct_ops_arena_fail.skel.h" #if defined(__x86_64__) @@ -49,6 +50,57 @@ static void arena_arg_fail(void) struct_ops_arena_fail__destroy(skel); } + +static void arena_arg_attach_one(int target_fd, const char *prog_name) +{ + struct struct_ops_arena_attach *skel; + struct bpf_program *prog, *pos; + char log_buf[64 * 1024]; + int err; + + skel = struct_ops_arena_attach__open(); + if (!ASSERT_OK_PTR(skel, "struct_ops_arena_attach__open")) + return; + + prog = bpf_object__find_program_by_name(skel->obj, prog_name); + if (!ASSERT_OK_PTR(prog, prog_name)) + goto out; + + bpf_object__for_each_program(pos, skel->obj) + bpf_program__set_autoload(pos, pos == prog); + + err = bpf_program__set_attach_target(prog, target_fd, "test_arena_cb"); + if (!ASSERT_OK(err, "set_attach_target")) + goto out; + + log_buf[0] = '\0'; + bpf_program__set_log_buf(prog, log_buf, sizeof(log_buf)); + err = struct_ops_arena_attach__load(skel); + + ASSERT_EQ(err, -EOPNOTSUPP, prog_name); + ASSERT_HAS_SUBSTR(log_buf, "Cannot attach to a target with arena context arguments", + "verifier_log"); + +out: + struct_ops_arena_attach__destroy(skel); +} + +static void arena_arg_attach(void) +{ + struct struct_ops_arena *skel; + int target_fd; + + skel = struct_ops_arena__open_and_load(); + if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load")) + return; + + target_fd = bpf_program__fd(skel->progs.test_arena_cb); + arena_arg_attach_one(target_fd, "fentry_test_arena"); + arena_arg_attach_one(target_fd, "fexit_test_arena"); + arena_arg_attach_one(target_fd, "freplace_test_arena"); + + struct_ops_arena__destroy(skel); +} #endif /* @@ -68,6 +120,8 @@ void serial_test_struct_ops_arena(void) arena_arg(); if (test__start_subtest("arena_arg_fail")) arena_arg_fail(); + if (test__start_subtest("arena_arg_attach")) + arena_arg_attach(); #else test__skip(); #endif diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c b/tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c new file mode 100644 index 000000000000..081a770307e5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +SEC("fentry") +int BPF_PROG(fentry_test_arena, unsigned long long *st_ops_ctx) +{ + return 0; +} + +SEC("fexit") +int BPF_PROG(fexit_test_arena, unsigned long long *st_ops_ctx, int ret) +{ + return 0; +} + +SEC("freplace") +int freplace_test_arena(unsigned long long *st_ops_ctx) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; From 83608e303b95d07afba1c15da0b5d9e513c2f15a Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Mon, 10 Aug 2026 20:59:54 -0700 Subject: [PATCH 263/373] bpf: Compare iterator types during state pruning An iterator stack slot can be MEM_RCU or PTR_UNTRUSTED. These states must not be equal, or the verifier can prune an unsafe path. Compare the pointer type for STACK_ITER slots. Fixes: dfab99df147b ("bpf: teach the verifier to enforce css_iter and task_iter in RCU CS") Signed-off-by: Ning Ding Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260811035955.132989-2-dingning04@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/states.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c index ea2153cf28d0..4e6aafad33bd 100644 --- a/kernel/bpf/states.c +++ b/kernel/bpf/states.c @@ -812,7 +812,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, * infinite loop check triggering, see * iter_active_depths_differ() */ - if (old_reg->iter.btf != cur_reg->iter.btf || + if (old_reg->type != cur_reg->type || + old_reg->iter.btf != cur_reg->iter.btf || old_reg->iter.btf_id != cur_reg->iter.btf_id || old_reg->iter.state != cur_reg->iter.state || /* ignore {old_reg,cur_reg}->iter.depth, see above */ From 81f209d5f7435646df047400a88bc81e0a16b9eb Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Mon, 10 Aug 2026 20:59:55 -0700 Subject: [PATCH 264/373] selftests/bpf: Test RCU iterator state pruning Add a path where RCU protection reaches zero and then starts again. The iterator is untrusted after this gap and must be rejected. Signed-off-by: Ning Ding Link: https://patch.msgid.link/20260811035955.132989-3-dingning04@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/progs/iters_task_failure.c | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/iters_task_failure.c b/tools/testing/selftests/bpf/progs/iters_task_failure.c index fe3663dedbe1..566a1d3dffea 100644 --- a/tools/testing/selftests/bpf/progs/iters_task_failure.c +++ b/tools/testing/selftests/bpf/progs/iters_task_failure.c @@ -61,6 +61,52 @@ int BPF_PROG(iter_tasks_lock_and_unlock) return 0; } +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__failure __msg("expected an RCU CS when using bpf_iter_task_next") +__flag(BPF_F_TEST_STATE_FREQ) +int BPF_PROG(iter_tasks_rcu_state_pruning) +{ + struct bpf_iter_task it; + + asm volatile ( + "call %[bpf_rcu_read_lock];" + "r1 = %[it];" + "r2 = 0;" + "r3 = 0;" /* BPF_TASK_ITER_ALL_PROCS */ + "call %[bpf_iter_task_new];" + + "call %[bpf_get_prandom_u32];" + "if w0 == 0 goto unprotected_%=;" + + /* Keep the outer RCU lock active on the straight-line path. */ + "call %[bpf_rcu_read_lock];" + "call %[bpf_rcu_read_unlock];" + "goto merge_%=;" + + "unprotected_%=:" + /* Create an unprotected gap on the taken path. */ + "call %[bpf_rcu_read_unlock];" + "call %[bpf_rcu_read_lock];" + + "merge_%=: r1 = %[it];" + "call %[bpf_iter_task_next];" + "r1 = %[it];" + "call %[bpf_iter_task_destroy];" + "call %[bpf_rcu_read_unlock];" + : + : __imm_ptr(it), + __imm(bpf_get_prandom_u32), + __imm(bpf_iter_task_new), + __imm(bpf_iter_task_next), + __imm(bpf_iter_task_destroy), + __imm(bpf_rcu_read_lock), + __imm(bpf_rcu_read_unlock) + : __clobber_common + ); + + return 0; +} + SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") __failure __msg("expected an RCU CS when using bpf_iter_css_next") int BPF_PROG(iter_css_lock_and_unlock) From 41c5dbb4be3c1ef4a5e2ce4c28de60b2be3cdccf Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:15:55 +0200 Subject: [PATCH 265/373] bpf: Derive the atomic load register in one place check_atomic_rmw() open codes the mapping from a BPF_ATOMIC to the register it reads the old value into, the BPF_STX case of insn_def_regno() open codes the very same mapping a second time, the const folding and the liveness transfer functions a third and a fourth time, and BPF JITs need it as well to know which register a faulting BPF_PROBE_ATOMIC has to clear. Add a small helper so that all of them can share it. No functional change. The BPF_LOAD_ACQ case is there for the JITs, which do walk all instruction classes. const_reg_xfer() loses its explicit BPF_ATOMIC mode test since the helper checks class and mode itself; the BPF_PROBE_ATOMIC it additionally accepts cannot be seen there as it is only set from bpf_do_misc_fixups(), that is, after const folding has run. arg_track_xfer() keeps its mode test since that also guards the stack clearing next to it. Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260811131600.506721-1-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- include/linux/filter.h | 24 ++++++++++++++++++++++++ kernel/bpf/const_fold.c | 11 +++-------- kernel/bpf/fixups.c | 11 +---------- kernel/bpf/liveness.c | 9 +++------ kernel/bpf/verifier.c | 13 ++----------- 5 files changed, 33 insertions(+), 35 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 4edba8182db1..15d83684c6e9 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -414,6 +414,30 @@ static inline bool bpf_atomic_is_load_acq(const struct bpf_insn *insn) insn->imm == BPF_LOAD_ACQ; } +/* + * Given an instruction @insn, return the number of the BPF register that a + * BPF_ATOMIC reads the value at its memory operand into, or -1 if there is + * no such register. That is the register a BPF_PROBE_ATOMIC has to clear when + * the access faults. Like bpf_atomic_is_load_acq(), @insn is not assumed to + * be a BPF_ATOMIC here. + */ +static inline int bpf_atomic_load_reg(const struct bpf_insn *insn) +{ + if (BPF_CLASS(insn->code) != BPF_STX || + (BPF_MODE(insn->code) != BPF_ATOMIC && + BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)) + return -1; + + switch (insn->imm) { + case BPF_LOAD_ACQ: + return insn->dst_reg; + case BPF_CMPXCHG: + return BPF_REG_0; + default: + return (insn->imm & BPF_FETCH) ? insn->src_reg : -1; + } +} + /* Memory store, *(uint *) (dst_reg + off16) = imm32 */ #define BPF_ST_MEM(SIZE, DST, OFF, IMM) \ diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c index b2a19acadb91..4cf120c7b2cb 100644 --- a/kernel/bpf/const_fold.c +++ b/kernel/bpf/const_fold.c @@ -199,14 +199,9 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * ci_out[r] = unknown; break; case BPF_STX: - if (mode != BPF_ATOMIC) - break; - if (insn->imm == BPF_CMPXCHG) - ci_out[BPF_REG_0] = unknown; - else if (insn->imm == BPF_LOAD_ACQ) - *dst = unknown; - else if (insn->imm & BPF_FETCH) - *src = unknown; + r = bpf_atomic_load_reg(insn); + if (r >= 0) + ci_out[r] = unknown; break; } } diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 661e2d13a604..c4bd70befbb5 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -49,16 +49,7 @@ static int insn_def_regno(const struct bpf_insn *insn) case BPF_ST: return -1; case BPF_STX: - if (BPF_MODE(insn->code) == BPF_ATOMIC || - BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) { - if (insn->imm == BPF_CMPXCHG) - return BPF_REG_0; - else if (insn->imm == BPF_LOAD_ACQ) - return insn->dst_reg; - else if (insn->imm & BPF_FETCH) - return insn->src_reg; - } - return -1; + return bpf_atomic_load_reg(insn); default: return insn->dst_reg; } diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index ef9a5a922887..1c997aeba6fa 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -1209,12 +1209,9 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, clear_stack_for_all_offs(insn, at_out, insn->dst_reg, at_stack_out, sz); - if (insn->imm == BPF_CMPXCHG) - at_out[BPF_REG_0] = none; - else if (insn->imm == BPF_LOAD_ACQ) - *dst = none; - else if (insn->imm & BPF_FETCH) - *src = none; + r = bpf_atomic_load_reg(insn); + if (r >= 0) + at_out[r] = none; } } else if (class == BPF_ST && BPF_MODE(insn->code) == BPF_MEM) { u32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code)); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index add3affc5703..61ef43325c6f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6485,21 +6485,12 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, return -EACCES; } - if (insn->imm & BPF_FETCH) { - if (insn->imm == BPF_CMPXCHG) - load_reg = BPF_REG_0; - else - load_reg = insn->src_reg; - + load_reg = bpf_atomic_load_reg(insn); + if (load_reg >= 0) { /* check and record load of old value */ err = check_reg_arg(env, load_reg, DST_OP); if (err) return err; - } else { - /* This instruction accesses a memory location but doesn't - * actually load it into a register. - */ - load_reg = -1; } dst_reg = cur_regs(env) + insn->dst_reg; From 1519f488e8ce430c68ade11d24b8459631f5dca9 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:15:56 +0200 Subject: [PATCH 266/373] bpf, riscv: Clear fetch destination on faulting arena atomic A RMW atomic on an arena pointer is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but that entry records no destination register to clear unless the instruction is a load-acquire today. That is right for a plain BPF_{ADD,AND,OR,XOR}, which only writes memory, but an RMW carrying BPF_FETCH also reads the old value into a register: src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and r0 for BPF_CMPXCHG. emit_atomic_rmw() emits it that way, e.g.: [...] case BPF_XCHG: ctx->ex_insn_off = ctx->ninsns; emit(is64 ? rv_amoswap_d(rs, rs, rd, 1, 1) : rv_amoswap_w(rs, rs, rd, 1, 1), ctx); [...] Thus, a fault over an unmapped arena page ex_handler_bpf() jumps over the access but leaves rs untouched, and the program resumes with whatever it held before the atomic instead of the 0 that every other BPF_PROBE_* access delivers. Fill the exception table entry in from bpf_atomic_load_reg(), which returns the BPF register an atomic reads the memory operand into or -1 when it has none. A load-acquire ends up with the same register it gets today, it just goes through the helper. Unlike x86-64 and arm64, riscv64 does not report arena violations from its exception handler, so there is no access direction to correct here, only the missing register clear. Fixes: fb7cefabae81 ("riscv, bpf: Add support arena atomics for RV64") Signed-off-by: Daniel Borkmann Reviewed-by: Pu Lehui Link: https://patch.msgid.link/20260811131600.506721-2-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- arch/riscv/net/bpf_jit_comp64.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 6b9972b07c1b..2504df1fa111 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1992,10 +1992,19 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, ret = emit_atomic_rmw(rd, rs, insn, ctx); /* ret can be 1 (skip-zext); extable entry still needs to be added */ - if (ret >= 0) - ret = add_exception_handler(insn, - bpf_atomic_is_load_acq(insn) ? rd : REG_DONT_CLEAR_MARKER, - ctx) ?: ret; + if (ret >= 0) { + /* + * A load-acquire reads into dst_reg, and a read-modify-write + * carrying BPF_FETCH reads the old value into src_reg, or into + * r0 for a BPF_CMPXCHG. Clear that register on fault, the + * remaining atomics have no destination register. + */ + int load_reg = bpf_atomic_load_reg(insn); + + ret = add_exception_handler(insn, load_reg < 0 ? + REG_DONT_CLEAR_MARKER : regmap[load_reg], + ctx) ?: ret; + } if (ret) return ret; From cf92a108601a3f2465b550e44a98c8a78cabf8d1 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:15:57 +0200 Subject: [PATCH 267/373] bpf, x86: Clear fetch destination on faulting arena atomic populate_extable() encodes "there is no destination register to clear" as DONT_CLEAR in the DST_REG field of the exception table metadata, and later ex_handler_bpf() then reuses that very value to derive the direction it reports the fault with is_write = (reg == DONT_CLEAR). The two coincide for a plain load or store, but not for a RMW carrying BPF_FETCH. Such an atomic writes memory, so it has to be reported as a WRITE, and it also reads the old value into a register, src_reg for BPF_ADD | BPF_FETCH and BPF_XCHG, r0 for BPF_CMPXCHG, so that register has to be cleared on fault. A single DONT_CLEAR cannot say both, and the store branch picks it unconditionally: [...] } else { arena_reg = reg2pt_regs[dst_reg]; fixup_reg = DONT_CLEAR; } [...] The reported direction is therefore right, but on a fault over an unmapped arena page the fetch destination keeps whatever it held before the atomic, where every other BPF_PROBE_* access delivers 0. Give the metadata its own ARENA_WRITE bit so that the reported direction no longer depends on whether there is a register to clear, and fill DST_REG in from bpf_atomic_load_reg(). BPF_{AND,OR,XOR} | BPF_FETCH need no handling here, bpf_jit_supports_insn() already rejects those in the arena. Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260811131600.506721-3-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 8dddb5d7af21..d920772af7d5 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1473,17 +1473,20 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size, * * Bit layout of `fixup` (32-bit): * - * +-----------+--------+-----------+---------+----------+ - * | 31 | 30-24 | 23-16 | 15-8 | 7-0 | - * | | | | | | - * | ARENA_ACC | Unused | ARENA_REG | DST_REG | INSN_LEN | - * +-----------+--------+-----------+---------+----------+ + * +-----------+-------------+--------+-----------+---------+----------+ + * | 31 | 30 | 29-24 | 23-16 | 15-8 | 7-0 | + * | | | | | | | + * | ARENA_ACC | ARENA_WRITE | Unused | ARENA_REG | DST_REG | INSN_LEN | + * +-----------+-------------+--------+-----------+---------+----------+ * * - INSN_LEN (8 bits): Length of faulting insn (max x86 insn = 15 bytes (fits in 8 bits)). * - DST_REG (8 bits): Offset of dst_reg from reg2pt_regs[] (max offset = 112 (fits in 8 bits)). - * This is set to DONT_CLEAR if the insn is a store. + * This is set to DONT_CLEAR if the insn does not read into a register. * - ARENA_REG (8 bits): Offset of the register that is used to calculate the * address for load/store when accessing the arena region. + * - ARENA_WRITE (1 bit): This bit is set when the faulting instruction wrote to the arena region. + * It is independent of DST_REG, since a read-modify-write both writes to + * memory and reads the old value into a register. * - ARENA_ACCESS (1 bit): This bit is set when the faulting instruction accessed the arena region. * * Bit layout of `data` (32-bit): @@ -1502,6 +1505,7 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size, #define FIXUP_INSN_LEN_MASK GENMASK(7, 0) #define FIXUP_REG_MASK GENMASK(15, 8) #define FIXUP_ARENA_REG_MASK GENMASK(23, 16) +#define FIXUP_ARENA_WRITE BIT(30) #define FIXUP_ARENA_ACCESS BIT(31) #define DATA_ARENA_OFFSET_MASK GENMASK(31, 16) @@ -1510,7 +1514,7 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs) u32 reg = FIELD_GET(FIXUP_REG_MASK, x->fixup); u32 insn_len = FIELD_GET(FIXUP_INSN_LEN_MASK, x->fixup); bool is_arena = !!(x->fixup & FIXUP_ARENA_ACCESS); - bool is_write = (reg == DONT_CLEAR); + bool is_write = !!(x->fixup & FIXUP_ARENA_WRITE); unsigned long addr; s16 off; u32 arena_reg; @@ -2348,6 +2352,7 @@ st: insn_off = insn->off; struct exception_table_entry *ex; u8 *_insn = image + proglen + (start_of_ldx - temp); u32 arena_reg, fixup_reg; + bool is_write; s64 delta; if (!bpf_prog->aux->extable) @@ -2384,15 +2389,29 @@ st: insn_off = insn->off; bpf_atomic_is_load_acq(insn)) { arena_reg = reg2pt_regs[src_reg]; fixup_reg = reg2pt_regs[dst_reg]; + is_write = false; } else { + /* + * A store has no destination register to clear, + * except for a read-modify-write with BPF_FETCH, + * which also reads the old value into src_reg, or + * into r0 for a BPF_CMPXCHG. Either way the access + * is still reported as a write. + */ + int load_reg = bpf_atomic_load_reg(insn); + arena_reg = reg2pt_regs[dst_reg]; - fixup_reg = DONT_CLEAR; + fixup_reg = load_reg < 0 ? DONT_CLEAR : + reg2pt_regs[load_reg]; + is_write = true; } ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) | FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) | FIELD_PREP(FIXUP_REG_MASK, fixup_reg); ex->fixup |= FIXUP_ARENA_ACCESS; + if (is_write) + ex->fixup |= FIXUP_ARENA_WRITE; ex->data |= FIELD_PREP(DATA_ARENA_OFFSET_MASK, insn->off); } From ea3f20cb5918fea8e3786899b78fc7bb867c6a5e Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:15:58 +0200 Subject: [PATCH 268/373] bpf, arm64: Clear fetch destination on faulting arena atomic Same problem as on x86-64: add_exception_handler() folds "there is no destination register to clear" and "this is a store" into one DONT_CLEAR value ... if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn)) dst_reg = DONT_CLEAR; ... which ex_handler_bpf() then reads back as the access direction: bool is_write = (dst_reg == DONT_CLEAR); A RMW carrying BPF_FETCH is both. emit_lse_atomic() reads the old value into src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and into r0 for BPF_CMPXCHG, so a fault over an unmapped arena page is correctly reported as a WRITE but leaves that register holding a stale value instead of the 0 that every other BPF_PROBE_* access delivers. Same as on x86-64, add a separate ARENA_WRITE bit for the direction. FIXUP_REG is now filled in by the callers of add_exception_handler(), the BPF_PROBE_ATOMIC one deriving it from bpf_atomic_load_reg(), so that the helper only has to determine the direction. This is how the riscv64 JIT already does it, and it stops the two store callers from handing in a dst_reg that was only going to be overwritten with DONT_CLEAR anyway. Fixes: e612b5c1d3ee ("bpf, arm64: Add support for lse atomics in bpf_arena") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Link: https://patch.msgid.link/20260811131600.506721-4-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- arch/arm64/net/bpf_jit_comp.c | 44 ++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index d14d297ebb96..74b4083791da 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1082,23 +1082,27 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) * * Bit layout of `fixup` (32-bit): * - * +-----------+--------+-----------+-----------+----------+ - * | 31-27 | 26-22 | 21 | 20-16 | 15-0 | - * | | | | | | - * | FIXUP_REG | Unused | ARENA_ACC | ARENA_REG | OFFSET | - * +-----------+--------+-----------+-----------+----------+ + * +-----------+--------+-------------+-----------+-----------+----------+ + * | 31-27 | 26-23 | 22 | 21 | 20-16 | 15-0 | + * | | | | | | | + * | FIXUP_REG | Unused | ARENA_WRITE | ARENA_ACC | ARENA_REG | OFFSET | + * +-----------+--------+-------------+-----------+-----------+----------+ * * - OFFSET (16 bits): Offset used to compute address for Load/Store instruction. * - ARENA_REG (5 bits): Register that is used to calculate the address for load/store when * accessing the arena region. * - ARENA_ACCESS (1 bit): This bit is set when the faulting instruction accessed the arena region. + * - ARENA_WRITE (1 bit): This bit is set when the faulting instruction wrote to the arena region. + * It is independent of FIXUP_REG, since a read-modify-write both writes to + * memory and reads the old value into a register. * - FIXUP_REG (5 bits): Destination register for the load instruction (cleared on fault) or set to - * DONT_CLEAR if it is a store instruction. + * DONT_CLEAR if the instruction does not read into a register. */ #define BPF_FIXUP_OFFSET_MASK GENMASK(15, 0) #define BPF_FIXUP_ARENA_REG_MASK GENMASK(20, 16) #define BPF_ARENA_ACCESS BIT(21) +#define BPF_ARENA_WRITE BIT(22) #define BPF_FIXUP_REG_MASK GENMASK(31, 27) #define DONT_CLEAR 5 /* Unused ARM64 register from BPF's POV */ @@ -1109,7 +1113,7 @@ bool ex_handler_bpf(const struct exception_table_entry *ex, s16 off = FIELD_GET(BPF_FIXUP_OFFSET_MASK, ex->fixup); int arena_reg = FIELD_GET(BPF_FIXUP_ARENA_REG_MASK, ex->fixup); bool is_arena = !!(ex->fixup & BPF_ARENA_ACCESS); - bool is_write = (dst_reg == DONT_CLEAR); + bool is_write = !!(ex->fixup & BPF_ARENA_WRITE); unsigned long addr; if (is_arena) { @@ -1132,7 +1136,7 @@ static int add_exception_handler(const struct bpf_insn *insn, { off_t ins_offset; s16 off = insn->off; - bool is_arena; + bool is_arena, is_write; int arena_reg; unsigned long pc; struct exception_table_entry *ex; @@ -1181,15 +1185,18 @@ static int add_exception_handler(const struct bpf_insn *insn, /* * A load-acquire is of BPF_STX class, but reads from src_reg into * dst_reg like a BPF_LDX does, hence it must not be treated as a store - * here. + * here. A read-modify-write carrying BPF_FETCH is reported as a write + * even though it does have a register to clear, see the callers. */ - if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn)) - dst_reg = DONT_CLEAR; + is_write = BPF_CLASS(insn->code) != BPF_LDX && + !bpf_atomic_is_load_acq(insn); ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); if (is_arena) { ex->fixup |= BPF_ARENA_ACCESS; + if (is_write) + ex->fixup |= BPF_ARENA_WRITE; /* * insn->src_reg/dst_reg holds the address in the arena region with upper 32-bits * being zero because of a preceding addr_space_cast(r, 0x0, 0x1) instruction. @@ -1889,7 +1896,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn break; } - ret = add_exception_handler(insn, ctx, dst); + ret = add_exception_handler(insn, ctx, DONT_CLEAR); if (ret) return ret; break; @@ -1956,7 +1963,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn break; } - ret = add_exception_handler(insn, ctx, dst); + ret = add_exception_handler(insn, ctx, DONT_CLEAR); if (ret) return ret; break; @@ -1979,7 +1986,16 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn return ret; if (BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) { - ret = add_exception_handler(insn, ctx, dst); + /* + * A load-acquire reads into dst_reg, and a read-modify-write + * carrying BPF_FETCH reads the old value into src_reg, or into + * r0 for a BPF_CMPXCHG. Clear that register on fault, the + * remaining atomics have no destination register. + */ + int load_reg = bpf_atomic_load_reg(insn); + + ret = add_exception_handler(insn, ctx, load_reg < 0 ? + DONT_CLEAR : bpf2a64[load_reg]); if (ret) return ret; } From cc3e12330599f097f0e1f792435686ddc276f26d Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:15:59 +0200 Subject: [PATCH 269/373] bpf, s390: Clear fetch destination on faulting arena atomic Same missing register clear as on riscv64. A RMW atomic on an arena pointer is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but bpf_jit_probe_atomic_pre() only fills in the arena base and the probe offset, leaving probe->reg at the -1 that bpf_jit_probe_init() set, which bpf_jit_probe_post() writes into the entry and ex_handler_bpf() then reads back as "there is nothing to clear". That is right for a plain BPF_{ADD,AND,OR,XOR}, which only writes memory, but an RMW carrying BPF_FETCH also reads the old value into a register: src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and r0 for BPF_CMPXCHG. So on a fault over an unmapped arena page the program resumes at the landing pad with whatever that register held before the atomic instead of the 0 that every other BPF_PROBE_* access delivers. Fill probe->reg in from bpf_atomic_load_reg(). Unlike x86-64 and arm64, s390x does not report arena violations from its exception handler, so there is no access direction to correct here, only the missing register clear. Fixes: 2f9469484a3b ("s390/bpf: Support arena atomics") Signed-off-by: Daniel Borkmann Reviewed-by: Ilya Leoshkevich Link: https://patch.msgid.link/20260811131600.506721-5-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- arch/s390/net/bpf_jit_comp.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index b60877478b45..c46872b071ce 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -774,6 +774,8 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit, struct bpf_insn *insn, struct bpf_jit_probe *probe) { + int load_reg; + if (BPF_MODE(insn->code) != BPF_PROBE_ATOMIC) return; @@ -783,6 +785,14 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit, EMIT4(0xb9080000, REG_W1, insn->dst_reg); probe->arena_reg = REG_W1; probe->prg = jit->prg; + /* + * A read-modify-write carrying BPF_FETCH reads the old value into + * src_reg, or into r0 for a BPF_CMPXCHG. Clear that register on + * fault, the remaining atomics only write memory. + */ + load_reg = bpf_atomic_load_reg(insn); + if (load_reg >= 0) + probe->reg = reg2hex[load_reg]; } static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp, @@ -1684,6 +1694,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, if (load_probe.prg != -1) { probe.prg = jit->prg; probe.arena_reg = load_probe.arena_reg; + probe.reg = load_probe.reg; } loop_start = jit->prg; /* 0: {csy|csg} %w0,%src,off(%arena) */ From 611a9f0d3dcaaddd7ac5ede1c3d98d5f64092159 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Tue, 11 Aug 2026 15:16:00 +0200 Subject: [PATCH 270/373] selftests/bpf: Add arena fault tests for atomics with fetch Add stream_arena_xchg_fault and stream_arena_cmpxchg_fault next to the existing read, write and load-acquire fault tests, covering the two places a read-modify-write can deposit the old value: src_reg for a BPF_XCHG and r0 for a BPF_CMPXCHG. Both cover both halves of the JIT bug that left the fetch destination alone when a RMW on an arena pointer faulted: - the fault has to be reported as a WRITE, and at the address held by the destination register, which __stderr() and test_address() check - the register receiving the fetched value has to be cleared by the fault handler, which the programs check by poisoning it before the atomic and returning it, so __retval(0) fails if it is left untouched The __stderr() annotation can only wildcard the faulting address since the arena base is not known until runtime, hence the two test_address() subtests on top, which pin it to the address held by dst_reg rather than src_reg. Note, the atomics are open coded since linux/filter.h cannot be included alongside vmlinux.h. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream [...] #464/1 stream_arena_fault_address/read_fault:OK #464/2 stream_arena_fault_address/write_fault:OK #464/3 stream_arena_fault_address/load_acquire_fault:OK #464/4 stream_arena_fault_address/xchg_fault:OK #464/5 stream_arena_fault_address/cmpxchg_fault:OK #464 stream_arena_fault_address:OK [...] #466/5 stream_success/stream_arena_write_fault:OK #466/6 stream_success/stream_arena_read_fault:OK #466/7 stream_success/stream_arena_load_acquire_fault:OK #466/8 stream_success/stream_arena_xchg_fault:OK #466/9 stream_success/stream_arena_cmpxchg_fault:OK [...] Summary: 4/22 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Acked-by: Puranjay Mohan Link: https://patch.msgid.link/20260811131600.506721-6-daniel@iogearbox.net Signed-off-by: Eduard Zingerman --- .../testing/selftests/bpf/prog_tests/stream.c | 4 + tools/testing/selftests/bpf/progs/stream.c | 101 ++++++++++++++++++ 2 files changed, 105 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/stream.c b/tools/testing/selftests/bpf/prog_tests/stream.c index 15dd3ae2a84b..e4e9374309e2 100644 --- a/tools/testing/selftests/bpf/prog_tests/stream.c +++ b/tools/testing/selftests/bpf/prog_tests/stream.c @@ -105,6 +105,10 @@ void test_stream_arena_fault_address(void) test_address(skel->progs.stream_arena_write_fault, &skel->bss->fault_addr); if (test__start_subtest("load_acquire_fault")) test_address(skel->progs.stream_arena_load_acquire_fault, &skel->bss->fault_addr); + if (test__start_subtest("xchg_fault")) + test_address(skel->progs.stream_arena_xchg_fault, &skel->bss->fault_addr); + if (test__start_subtest("cmpxchg_fault")) + test_address(skel->progs.stream_arena_cmpxchg_fault, &skel->bss->fault_addr); stream__destroy(skel); } diff --git a/tools/testing/selftests/bpf/progs/stream.c b/tools/testing/selftests/bpf/progs/stream.c index cf5533e11f39..00a37933e411 100644 --- a/tools/testing/selftests/bpf/progs/stream.c +++ b/tools/testing/selftests/bpf/progs/stream.c @@ -229,6 +229,107 @@ int stream_arena_load_acquire_fault(void *ctx) return val; } +SEC("syscall") +__arch_x86_64 +__arch_arm64 +__success __retval(0) +__stderr("ERROR: Arena WRITE access at unmapped address 0x{{.*}}") +__stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}") +__stderr("Call trace:\n" +"{{([a-zA-Z_][a-zA-Z0-9_]*\\+0x[0-9a-fA-F]+/0x[0-9a-fA-F]+\n" +"|[ \t]+[^\n]+\n)*}}") +int stream_arena_xchg_fault(void *ctx) +{ + static const struct bpf_insn xchg_insn = { + .code = 0xc3, /* BPF_STX | BPF_ATOMIC | BPF_W */ + .dst_reg = 1, /* BPF_REG_1 */ + .src_reg = 2, /* BPF_REG_2 */ + .off = 0x7fff, + .imm = 0xe1, /* BPF_XCHG */ + }; + struct bpf_arena *ptr = (void *)&arena; + u64 user_vm_start, val; + + /* + * Prevent GCC bounds warning: casting &arena to struct bpf_arena * + * triggers bounds checking since the map definition is smaller than + * struct bpf_arena. barrier_var() makes the pointer opaque to GCC, + * preventing the bounds analysis. + */ + barrier_var(ptr); + user_vm_start = ptr->user_vm_start; + fault_addr = user_vm_start + 0x7fff; + bpf_addr_space_cast(user_vm_start, 0, 1); + /* + * A read-modify-write carrying BPF_FETCH writes to memory, so the fault + * has to be reported as a WRITE from the dst_reg address, but it also + * reads the old value into src_reg, so the exception handler has to + * clear src_reg. Poison it up front, the returned value must be 0. + */ + asm volatile ( + "r1 = %[user_vm_start];" + "r2 = 1;" + ".8byte %[xchg_insn];" /* r2 = xchg((u32 *)(r1 + 0x7fff), r2) */ + "%[val] = r2;" + : [val] "=r" (val) + : [user_vm_start] "r" (user_vm_start), + __imm_insn(xchg_insn, xchg_insn) + : "r1", "r2" + ); + return val; +} + +SEC("syscall") +__arch_x86_64 +__arch_arm64 +__success __retval(0) +__stderr("ERROR: Arena WRITE access at unmapped address 0x{{.*}}") +__stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}") +__stderr("Call trace:\n" +"{{([a-zA-Z_][a-zA-Z0-9_]*\\+0x[0-9a-fA-F]+/0x[0-9a-fA-F]+\n" +"|[ \t]+[^\n]+\n)*}}") +int stream_arena_cmpxchg_fault(void *ctx) +{ + static const struct bpf_insn cmpxchg_insn = { + .code = 0xc3, /* BPF_STX | BPF_ATOMIC | BPF_W */ + .dst_reg = 1, /* BPF_REG_1 */ + .src_reg = 2, /* BPF_REG_2 */ + .off = 0x7fff, + .imm = 0xf1, /* BPF_CMPXCHG */ + }; + struct bpf_arena *ptr = (void *)&arena; + u64 user_vm_start, val; + + /* + * Prevent GCC bounds warning: casting &arena to struct bpf_arena * + * triggers bounds checking since the map definition is smaller than + * struct bpf_arena. barrier_var() makes the pointer opaque to GCC, + * preventing the bounds analysis. + */ + barrier_var(ptr); + user_vm_start = ptr->user_vm_start; + fault_addr = user_vm_start + 0x7fff; + bpf_addr_space_cast(user_vm_start, 0, 1); + /* + * Same as the exchange above, except that a BPF_CMPXCHG reads the old + * value into r0 rather than into src_reg, so r0 is the register the + * exception handler has to clear. It doubles as the compare value, but + * the comparison never happens since the access faults first. + */ + asm volatile ( + "r1 = %[user_vm_start];" + "r0 = 1;" + "r2 = 2;" + ".8byte %[cmpxchg_insn];" /* r0 = cmpxchg((u32 *)(r1 + 0x7fff), r0, r2) */ + "%[val] = r0;" + : [val] "=r" (val) + : [user_vm_start] "r" (user_vm_start), + __imm_insn(cmpxchg_insn, cmpxchg_insn) + : "r0", "r1", "r2" + ); + return val; +} + static __noinline void subprog(void) { int __arena *addr = (int __arena *)0xdeadbeef; From e28b4922679ed56062ac0dcd77f393cbafa90070 Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 7 Aug 2026 13:44:31 -0700 Subject: [PATCH 271/373] selftests/bpf: map_kptr: Expect BPF_ST reject msg on cpuv4 toolchains reject_scalar_store_to_kptr stores a scalar constant to a kptr field: *(volatile u64 *)&v->unref_ptr = 0xBADC0DE; Compilers generate one of two encodings for that: 1. Materialize the constant into a register and emit BPF_STX: r1 = 0xbadc0de *(u64 *)(r0 + 0x8) = r1 2. Or fold it into a single BPF_ST (store immediate): *(u64 *)(r0 + 0x8) = 0xbadc0de These go through different rejection paths and output different messages. - BPF_STX goes through map_kptr_match_type(), which prints "invalid kptr access, R...". - BPF_ST only gets the immediate check printing "BPF_ST imm must be 0 when storing to kptr" The test only expects the BPF_STX message, so it fails on a toolchain that folds the constant - bpf-gcc, and clang -mcpu=v4: 7: (7a) *(u64 *)(r0 +8) = 195936478 BPF_ST imm must be 0 when storing to kptr at off=8 ... EXPECTED SUBSTR: 'invalid kptr access, R' Pick the expected message with __BPF_FEATURE_ST, which clang and bpf-gcc both define exactly when BPF_ST codegen is available - cpuv4 for clang, and by default for bpf-gcc, whose default cpu is v4. bpf-gcc, before: #229/20 map_kptr/reject_scalar_store_to_kptr:FAIL bpf-gcc, after : #229/20 map_kptr/reject_scalar_store_to_kptr:OK Two caveats worth noting: - On a BPF_ST toolchain the test now only exercises the imm != 0 check and never reaches map_kptr_match_type(), so the scalar-vs-PTR_TO_BTF_ID rejection the test is named for is only covered by the non-ST builds. The imm path itself is already covered compiler-independently by verifier/map_kptr.c ("map_kptr: BPF_ST imm != 0"). - __BPF_FEATURE_ST says the compiler *can* emit BPF_ST, not that it will. The encoding also depends on the optimization level: clang -mcpu=v4 -O0 still emits BPF_STX, which would send the #ifdef down the wrong branch and fail the test. Selftests always build BPF objects at -O2 so this does not bite today, but it is a latent failure mode if that changes. Signed-off-by: Vineet Gupta Signed-off-by: Daniel Borkmann Acked-by: Yonghong Song Link: https://lore.kernel.org/bpf/20260807204434.1036279-2-vineet.gupta@linux.dev --- tools/testing/selftests/bpf/progs/map_kptr_fail.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/map_kptr_fail.c b/tools/testing/selftests/bpf/progs/map_kptr_fail.c index f11848dfa78f..5e25ca806060 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/map_kptr_fail.c @@ -386,7 +386,16 @@ int kptr_xchg_possibly_null(struct __sk_buff *ctx) } SEC("?tc") +/* + * A compiler with BPF_ST folds the constant into a store-immediate, which the + * verifier rejects on a different path (and with a different message) than the + * BPF_STX form. + */ +#ifdef __BPF_FEATURE_ST +__failure __msg("BPF_ST imm must be 0 when storing to kptr at off=8") +#else __failure __msg("invalid kptr access, R") +#endif int reject_scalar_store_to_kptr(struct __sk_buff *ctx) { struct map_value *v; From 2614285f32627ce263d92ce1c07f33fc88ffb3b6 Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 7 Aug 2026 13:44:32 -0700 Subject: [PATCH 272/373] selftests/bpf: Add --no-error-summary to skip end-of-run error log dump By default test_progs re-prints the aggregated error logs of all failed tests at the end of the run (when not in verbose mode), starting with "All error logs:". With bpf-gcc the current failures and a couple runaway 1M fails cause a huge print overhead/delay at the end. Add a subtractive --no-error-summary flag, gated on a new env.error_summary field which defaults to true, so the default behavior is unchanged. Passing --no-error-summary suppresses the final "All error logs:" dump. Only the human readable output is elided. dump_test_log() also emits the per-test and per-subtest entries of the --json-summary "results" array, so it keeps being called (via a new @quiet argument) and the JSON report is bit for bit what it was before. Signed-off-by: Vineet Gupta Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807204434.1036279-3-vineet.gupta@linux.dev --- tools/testing/selftests/bpf/test_progs.c | 43 +++++++++++++++++------- tools/testing/selftests/bpf/test_progs.h | 1 + 2 files changed, 31 insertions(+), 13 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index aa06bab30966..b274d98faac4 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -424,10 +424,12 @@ static void jsonw_write_log_message(json_writer_t *w, char *log_buf, size_t log_ } } +/* @quiet elides the human readable output, the JSON report is unaffected */ static void dump_test_log(const struct prog_test_def *test, const struct test_state *test_state, bool skip_ok_subtests, bool par_exec_result, + bool quiet, json_writer_t *w) { bool test_failed = test_state->error_cnt > 0; @@ -449,7 +451,7 @@ static void dump_test_log(const struct prog_test_def *test, if (verbose() && !par_exec_result) return; - if (test_state->log_cnt && print_test) + if (test_state->log_cnt && print_test && !quiet) print_test_log(test_state->log_buf, test_state->log_cnt); if (w && print_test) { @@ -471,15 +473,16 @@ static void dump_test_log(const struct prog_test_def *test, if ((skip_ok_subtests && !subtest_failed) || subtest_filtered) continue; - if (subtest_state->log_cnt && print_subtest) { + if (subtest_state->log_cnt && print_subtest && !quiet) { print_test_log(subtest_state->log_buf, subtest_state->log_cnt); } - print_subtest_name(test->test_num, i + 1, - test->test_name, subtest_state->name, - test_result(subtest_state->error_cnt, - subtest_state->skipped)); + if (!quiet) + print_subtest_name(test->test_num, i + 1, + test->test_name, subtest_state->name, + test_result(subtest_state->error_cnt, + subtest_state->skipped)); if (w && print_subtest) { jsonw_start_object(w); @@ -496,7 +499,8 @@ static void dump_test_log(const struct prog_test_def *test, jsonw_end_object(w); } - print_test_result(test, test_state); + if (!quiet) + print_test_result(test, test_state); } /* A bunch of tests set custom affinity per-thread and/or per-process. Reset @@ -899,6 +903,7 @@ enum ARG_KEYS { ARG_JSON_SUMMARY = 'J', ARG_TRAFFIC_MONITOR = 'm', ARG_WATCHDOG_TIMEOUT = 'w', + ARG_NO_ERROR_SUMMARY = -2, }; static const struct argp_option opts[] = { @@ -931,6 +936,8 @@ static const struct argp_option opts[] = { #endif { "watchdog-timeout", ARG_WATCHDOG_TIMEOUT, "SECONDS", 0, "Kill the process if tests are not making progress for specified number of seconds." }, + { "no-error-summary", ARG_NO_ERROR_SUMMARY, NULL, 0, + "Do not re-print the aggregated error logs of failed tests at the end of the run." }, {}, }; @@ -1132,6 +1139,9 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) case ARG_DEBUG: env->debug = true; break; + case ARG_NO_ERROR_SUMMARY: + env->error_summary = false; + break; case ARG_JSON_SUMMARY: env->json = fopen(arg, "w"); if (env->json == NULL) { @@ -1304,7 +1314,7 @@ static void dump_crash_log(void) if (env.test) { env.test_state->error_cnt++; - dump_test_log(env.test, env.test_state, true, false, NULL); + dump_test_log(env.test, env.test_state, true, false, false, NULL); } } @@ -1462,7 +1472,7 @@ static void run_one_test(int test_num) free(stop_libbpf_log_capture()); - dump_test_log(test, state, false, false, NULL); + dump_test_log(test, state, false, false, false, NULL); } struct dispatch_data { @@ -1623,7 +1633,7 @@ static void *dispatch_thread(void *ctx) } while (false); pthread_mutex_lock(&stdout_output_lock); - dump_test_log(test, state, false, true, NULL); + dump_test_log(test, state, false, true, false, NULL); pthread_mutex_unlock(&stdout_output_lock); } /* while (true) */ error: @@ -1686,9 +1696,14 @@ static void calculate_summary_and_print_errors(struct test_env *env) * We only print error logs summary when there are failed tests and * verbose mode is not enabled. Otherwise, results may be inconsistent. * + * --no-error-summary elides the human readable dump. The walk still + * happens when a JSON report was requested, so the JSON output keeps + * its per-test results; with no JSON report there is nothing left to + * do and the whole loop is skipped. */ - if (!verbose() && fail_cnt) { - printf("\nAll error logs:\n"); + if (!verbose() && fail_cnt && (env->error_summary || w)) { + if (env->error_summary) + printf("\nAll error logs:\n"); /* print error logs again */ for (i = 0; i < prog_test_cnt; i++) { @@ -1698,7 +1713,8 @@ static void calculate_summary_and_print_errors(struct test_env *env) if (!state->tested || !state->error_cnt) continue; - dump_test_log(test, state, true, true, w); + dump_test_log(test, state, true, true, + !env->error_summary, w); } } @@ -2028,6 +2044,7 @@ int main(int argc, char **argv) env.secs_till_notify = 10; env.secs_till_kill = 120; + env.error_summary = true; err = argp_parse(&argp, argc, argv, 0, NULL, &env); if (err) return err; diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h index 2cf950afcd85..e66d5c457901 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -105,6 +105,7 @@ struct test_env { struct test_selector tmon_selector; bool verifier_stats; bool debug; + bool error_summary; enum verbosity verbosity; bool jit_enabled; From 7bd1dd3fb87d83b9141617bc817501762dd133eb Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 7 Aug 2026 13:44:33 -0700 Subject: [PATCH 273/373] selftests/bpf: Report failed subtest count in test_progs summary The final summary line is asymmetric: the PASSED field reports both the number of top-level tests and the number of subtests within them, while the FAILED field reports only top-level tests: Summary: 640/5750 PASSED, 7760 SKIPPED, 100 FAILED There is no way to tell whether those 100 failing tests amount to 100 broken subtests or 1000. So count subtests with a non-zero error_cnt into a new sub_fail_cnt and print it alongside fail_cnt: Summary: 640/5750 PASSED, 7760 SKIPPED, 100/342 FAILED ^^^^^ This is correct for -j runs, as subtest_states[] is populated both in sequential and parallel modes. A test that fails without declaring any subtests contributes 0 to sub_fail_cnt. That mirrors the existing behaviour of sub_succ_cnt for tests that pass without subtests, keeping the two numerators comparable. Also emit the new count as a "failed_subtest" field in the JSON output, for parity with the existing "success_subtest". Note that this changes the trailing field of the summary line from a bare integer to "A/B", so anything scraping "N FAILED" out of it needs updating. While here, fix the fail_cnt comment in struct test_env, which claims it counts "total failed tests + sub-tests". Signed-off-by: Vineet Gupta Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807204434.1036279-4-vineet.gupta@linux.dev --- tools/testing/selftests/bpf/test_progs.c | 21 +++++++++++++-------- tools/testing/selftests/bpf/test_progs.h | 2 +- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index b274d98faac4..46eb201b96a3 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -1656,8 +1656,8 @@ static void *dispatch_thread(void *ctx) static void calculate_summary_and_print_errors(struct test_env *env) { - int i; - int succ_cnt = 0, fail_cnt = 0, sub_succ_cnt = 0, skip_cnt = 0; + int i, j; + int succ_cnt = 0, fail_cnt = 0, sub_succ_cnt = 0, sub_fail_cnt = 0, skip_cnt = 0; json_writer_t *w = NULL; for (i = 0; i < prog_test_cnt; i++) { @@ -1670,10 +1670,14 @@ static void calculate_summary_and_print_errors(struct test_env *env) sub_succ_cnt += state->sub_succ_cnt; skip_cnt += state->skip_cnt; - if (state->error_cnt) + if (state->error_cnt) { fail_cnt++; - else if (!test->not_built) + for (j = 0; j < state->subtest_num; j++) + if (state->subtest_states[j].error_cnt) + sub_fail_cnt++; + } else if (!test->not_built) { succ_cnt++; + } } if (env->json) { @@ -1688,6 +1692,7 @@ static void calculate_summary_and_print_errors(struct test_env *env) jsonw_uint_field(w, "success_subtest", sub_succ_cnt); jsonw_uint_field(w, "skipped", skip_cnt); jsonw_uint_field(w, "failed", fail_cnt); + jsonw_uint_field(w, "failed_subtest", sub_fail_cnt); jsonw_name(w, "results"); jsonw_start_array(w); } @@ -1728,12 +1733,12 @@ static void calculate_summary_and_print_errors(struct test_env *env) fclose(env->json); if (env->not_built_cnt) - printf("Summary: %d/%d PASSED, %d SKIPPED (%d not built), %d FAILED\n", + printf("Summary: %d/%d PASSED, %d SKIPPED (%d not built), %d/%d FAILED\n", succ_cnt, sub_succ_cnt, skip_cnt, env->not_built_cnt, - fail_cnt); + fail_cnt, sub_fail_cnt); else - printf("Summary: %d/%d PASSED, %d SKIPPED, %d FAILED\n", - succ_cnt, sub_succ_cnt, skip_cnt, fail_cnt); + printf("Summary: %d/%d PASSED, %d SKIPPED, %d/%d FAILED\n", + succ_cnt, sub_succ_cnt, skip_cnt, fail_cnt, sub_fail_cnt); env->succ_cnt = succ_cnt; env->sub_succ_cnt = sub_succ_cnt; diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h index e66d5c457901..ea493c477fbd 100644 --- a/tools/testing/selftests/bpf/test_progs.h +++ b/tools/testing/selftests/bpf/test_progs.h @@ -124,7 +124,7 @@ struct test_env { int succ_cnt; /* successful tests */ int sub_succ_cnt; /* successful sub-tests */ - int fail_cnt; /* total failed tests + sub-tests */ + int fail_cnt; /* failed tests */ int skip_cnt; /* skipped tests */ int not_built_cnt; /* tests not built */ From 3a59f11e0f989bdd637c87151992605a6559a7cb Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 7 Aug 2026 13:44:34 -0700 Subject: [PATCH 274/373] selftests/bpf: vmtest.sh: Preserve command quoting when running in the VM vmtest.sh captures the trailing command with command="$@", which flattens the arguments into a single space-separated string, and then pastes it into the generated guest init script: cd /root/bpf echo ${command} stdbuf -oL -eL ${command} That here-doc is unquoted, so the host expands ${command} and the flattened text lands in the script verbatim. The guest bash then parses those lines as shell source, re-splitting the text on whitespace and glob-expanding it against /root/bpf. As a result any command with a glob or an argument containing spaces is corrupted before it reaches the test binary. For example: vmtest.sh -- ./test_progs -a 'verifier_*' has 'verifier_*' expanded in the guest into the matching object/skeleton files (verifier_align.bpf.o verifier_align.skel.h ...), so test_progs is handed a list of filenames instead of the intended name filter and runs no matching tests. Quote each argument with printf '%q ' so the command is reproduced verbatim inside the VM: the escaped text goes through exactly one round of quote removal when the guest parses the init script, yielding the original argv with globs and special characters intact. The common case (e.g. -t ) is unaffected. Only do this when there is a command to quote. printf '%q ' with no arguments still applies the format once and emits '', which the -s (debug shell) path would take for a real command and try to run. Note this makes the trailing command strictly an argv rather than a shell snippet: passing it pre-quoted as one word, e.g. vmtest.sh -- "./test_progs -t foo" no longer works, and neither does embedding guest-side shell syntax such as ';' or a redirection. 'sh -c ...' still works. The RV64 recipe in README.rst does depend on the old double parse: it wraps the denylist in \" so the literal quotes reach the guest, whose second parse of the init script removes them. Under %q those quotes now survive into argv, and parse_test_list() strtok_r()s on ',' turns them into junk filters: -d ",exceptions," -> ["] [exceptions] ["] That is harmless for DENYLIST.riscv64 only because its first line is a comment, so the leading field is empty. A denylist starting with a real entry would silently lose it - ["*arena*] never matches - so drop the backslashes and let the host consume the quotes instead. Fixes: c9709f52386d ("bpf: Helper script for running BPF presubmit tests") Signed-off-by: Vineet Gupta Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260807204434.1036279-5-vineet.gupta@linux.dev --- tools/testing/selftests/bpf/README.rst | 4 ++-- tools/testing/selftests/bpf/vmtest.sh | 13 +++++++++++-- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst index 37164322a102..07c834433b38 100644 --- a/tools/testing/selftests/bpf/README.rst +++ b/tools/testing/selftests/bpf/README.rst @@ -107,12 +107,12 @@ Docker container and local rootfs image. The overall steps are as follows: tools/testing/selftests/bpf/vmtest.sh \ -l -- \ ./test_progs -d \ - \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \ + "$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \ | cut -d'#' -f1 \ | sed -e 's/^[[:space:]]*//' \ -e 's/[[:space:]]*$//' \ | tr -s '\n' ',' \ - )\" + )" Link: https://github.com/pulehui/riscv-bpf-vmtest.git [0] Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [1] diff --git a/tools/testing/selftests/bpf/vmtest.sh b/tools/testing/selftests/bpf/vmtest.sh index 9ca802285393..6a3d026d76bd 100755 --- a/tools/testing/selftests/bpf/vmtest.sh +++ b/tools/testing/selftests/bpf/vmtest.sh @@ -428,8 +428,17 @@ main() if [[ $# -eq 0 && "${debug_shell}" == "no" ]]; then echo "No command specified, will run ${DEFAULT_COMMAND} in the vm" - else - command="$@" + elif [[ $# -gt 0 ]]; then + # Quote each argument so the command survives into the guest: the + # host expands ${command} into the generated init script, which + # the guest bash then parses as shell source. Without the %q + # escapes an argument with a space or a glob (e.g. -a 'verifier_*') + # is re-split and expanded against /root/bpf there. + # + # Skip this when there is no command: printf '%q ' would still + # apply the format once and emit '', which is not the empty + # command that -s (debug shell) expects. + command=$(printf '%q ' "$@") fi local kconfig_file="${OUTPUT_DIR}/latest.config" From 14c950ac2be8cadb63e1bfe22111ab0fdc829eb8 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:18 +0200 Subject: [PATCH 275/373] bpf: Track verifier instruction stats for each subprogram The verifier currently records one instruction count for the main program and each global subprogram checked independently. Static subprograms are explored within callers, so their verification cost cannot be reported separately. Track both self and inclusive instruction counts for every subprogram. Charge each processed instruction as self work to the current subprogram and to a path-local subtotal in its function frame. When a function returns, add the callee subtotal to its inclusive count and to its parent subtotal. Fold any remaining frames when a path terminates or is pruned. Instruction subtotals are accounting state, not semantic verifier state. Clear them when a verifier state is copied so work before a path fork is charged once, rather than again when a saved branch is explored. If copying a saved state fails before all frames are allocated, skip missing frames while folding the current path. This generic frame accounting also records self and inclusive totals when an asynchronous callback starts as a fresh frame-zero state. It does not yet charge that independently explored callback path back to the main or global exploration root which scheduled it. That will be done in subsequent changes. This does not change the verification statistics output format. It only prepares the counters for per-subprogram reporting. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com Signed-off-by: Eduard Zingerman --- include/linux/bpf_verifier.h | 5 +++- kernel/bpf/verifier.c | 55 ++++++++++++++++++++++++++++++------ 2 files changed, 50 insertions(+), 10 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 93f7c2075eea..7c376451db82 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -380,6 +380,8 @@ struct bpf_func_state { * | number of simulations is tracked in frame N */ u32 callback_depth; + /* Instructions processed in this frame and callees on the current path. */ + u32 insns_subtotal; /* The following fields should be last. See copy_func_state() */ /* The state of the stack. Each element of the array describes BPF_REG_SIZE @@ -798,7 +800,8 @@ struct bpf_subprog_info { u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */ u16 stack_depth; /* max. stack depth used by this function */ u16 stack_extra; - u32 insn_processed; + u32 insns_total; + u32 insns_self; /* offsets in range [stack_depth .. fastcall_stack_off) * are used for bpf_fastcall spills and fills. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 61ef43325c6f..51d754bdef5d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1593,6 +1593,8 @@ static int copy_func_state(struct bpf_func_state *dst, const struct bpf_func_state *src) { memcpy(dst, src, offsetof(struct bpf_func_state, stack)); + /* Instruction accounting is path-local, not part of verifier state. */ + dst->insns_subtotal = 0; return copy_stack_state(dst, src); } @@ -9708,6 +9710,42 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env, static bool is_rbtree_lock_required_kfunc(u32 btf_id); +static void account_processed_insn(struct bpf_verifier_env *env) +{ + struct bpf_func_state *frame = cur_func(env); + + env->insn_processed++; + frame->insns_subtotal++; + env->subprog_info[frame->subprogno].insns_self++; +} + +static void account_processed_insns(struct bpf_verifier_env *env, + struct bpf_func_state *callee, + struct bpf_func_state *caller) +{ + u32 insns; + + if (!callee) + return; + + insns = callee->insns_subtotal; + + env->subprog_info[callee->subprogno].insns_total += insns; + if (caller) + caller->insns_subtotal += insns; + callee->insns_subtotal = 0; +} + +static void account_current_path(struct bpf_verifier_env *env) +{ + struct bpf_verifier_state *state = env->cur_state; + int frame; + + for (frame = state->curframe; frame >= 0; frame--) + account_processed_insns(env, state->frame[frame], + frame ? state->frame[frame - 1] : NULL); +} + /* Are we currently verifying the callback for a rbtree helper that must * be called with lock held? If so, no need to complain about unreleased * lock @@ -9804,6 +9842,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) verbose(env, "to caller at %d:\n", *insn_idx); print_verifier_state(env, state, caller->frameno, true); } + account_processed_insns(env, callee, caller); /* clear everything in the callee. In case of exceptional exits using * bpf_throw, this will be done by copy_verifier_state for extra frames. */ free_func_state(callee); @@ -17359,7 +17398,9 @@ static int do_check(struct bpf_verifier_env *env) insn = &insns[env->insn_idx]; insn_aux = &env->insn_aux_data[env->insn_idx]; - if (++env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) { + account_processed_insn(env); + + if (env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) { verbose(env, "BPF program is too large. Processed %d insn\n", env->insn_processed); @@ -17500,6 +17541,7 @@ static int do_check(struct bpf_verifier_env *env) "speculation barrier after jump instruction may not have the desired effect")) return -EFAULT; process_bpf_exit: + account_current_path(env); mark_verifier_state_scratched(env); err = bpf_update_branch_counts(env, env->cur_state); if (err) @@ -18544,6 +18586,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) ret = do_check(env); out: + account_current_path(env); if (!ret && pop_log) bpf_vlog_reset(&env->log, 0); free_states(env); @@ -18575,7 +18618,6 @@ static int do_check_subprogs(struct bpf_verifier_env *env) struct bpf_prog_aux *aux = env->prog->aux; struct bpf_func_info_aux *sub_aux; int i, ret, new_cnt; - u32 insn_processed; if (!aux->func_info) return 0; @@ -18590,8 +18632,6 @@ static int do_check_subprogs(struct bpf_verifier_env *env) if (!bpf_subprog_is_global(env, i)) continue; - insn_processed = env->insn_processed; - sub_aux = subprog_aux(env, i); if (!sub_aux->called || sub_aux->verified) continue; @@ -18599,7 +18639,6 @@ static int do_check_subprogs(struct bpf_verifier_env *env) env->insn_idx = env->subprog_info[i].start; WARN_ON_ONCE(env->insn_idx == 0); ret = do_check_common(env, i); - env->subprog_info[i].insn_processed = env->insn_processed - insn_processed; if (ret) { return ret; } else if (env->log.level & BPF_LOG_LEVEL) { @@ -18626,12 +18665,10 @@ static int do_check_subprogs(struct bpf_verifier_env *env) static int do_check_main(struct bpf_verifier_env *env) { - u32 insn_processed = env->insn_processed; int ret; env->insn_idx = 0; ret = do_check_common(env, 0); - env->subprog_info[0].insn_processed = env->insn_processed - insn_processed; if (!ret) env->prog->aux->stack_depth = env->subprog_info[0].stack_depth; return ret; @@ -18650,10 +18687,10 @@ static void print_verification_stats(struct bpf_verifier_env *env) for (i = 1; i < subprog_cnt; i++) verbose(env, "+%d", env->subprog_info[i].stack_depth); verbose(env, " max %d\n", env->max_stack_depth); - verbose(env, "insns processed %d", env->subprog_info[0].insn_processed); + verbose(env, "insns processed %d", env->subprog_info[0].insns_total); for (i = 1; i < subprog_cnt; i++) if (bpf_subprog_is_global(env, i)) - verbose(env, "+%d", env->subprog_info[i].insn_processed); + verbose(env, "+%d", env->subprog_info[i].insns_total); verbose(env, "\n"); } verbose(env, "processed %d insns (limit %d) max_states_per_insn %d " From 6137fb7c5f830d07909cbc789f52b1e0fffd6cd2 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:19 +0200 Subject: [PATCH 276/373] bpf: Attribute async callback instructions to verification roots Asynchronous callbacks are explored as fresh frame-zero verifier states, so normal callee-to-caller accounting cannot propagate their instruction budget to the main or global subprogram whose verification scheduled them. The callback exploration still happens within the same do_check_common() invocation as that independent verification root. Record env->insn_processed at do_check_common() entry and override the root's inclusive count with the delta before returning. This includes all directly and transitively scheduled asynchronous callbacks in the root's total without maintaining a separate accounting call stack. Static subprogram and callback totals remain local to their synchronous call paths. Their self counts continue to account for each processed instruction exactly once. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 51d754bdef5d..fced21ec2304 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18448,6 +18448,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) struct bpf_prog_aux *aux = env->prog->aux; struct bpf_verifier_state *state; struct bpf_reg_state *regs; + u32 insn_processed = env->insn_processed; int ret, i; env->prev_linfo = NULL; @@ -18590,6 +18591,15 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) if (!ret && pop_log) bpf_vlog_reset(&env->log, 0); free_states(env); + + /* + * The override is needed to account for async subprograms, which + * are verified with their own set of stack frames and thus are + * not accounted as callees by account_current_path(). + * Accumulate their total counts as total counts of the main or + * global subprog hosting the async call. + */ + env->subprog_info[subprog].insns_total = env->insn_processed - insn_processed; return ret; } From c2e6c7de883034a16132b974d6236f8189d5babe Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:20 +0200 Subject: [PATCH 277/373] bpf: Show more useful info in stack depth stats Stack depth statistics list captured depths in subprogram-number order, while per-verification instruction counts are reported separately. Since libbpf determines subprogram numbers, it is hard to associate either statistic with its subprogram name or see where verifier work is spent. Now that self and inclusive instruction counts are available for every subprogram, keep the combined maximum stack depth on its own line and print one uniform record for each subprogram. Represent the main program as subprog 0, then classify each record as main, global, or static before reporting insns_self, insns_total, and stack depth. The aggregate processed count is the sum of all self counts, while each total shows verifier work rooted at that subprogram. When no subprogram name is available, print . Keep the existing aggregate "processed ... insns" record unchanged for compatibility. Suggested-by: Andrii Nakryiko Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 23 +++++++++------ .../bpf/progs/verifier_basic_stack.c | 6 ++-- .../bpf/progs/verifier_bpf_fastcall.c | 28 +++++++++++++------ .../bpf/progs/verifier_global_subprogs.c | 5 +++- .../bpf/progs/verifier_private_stack.c | 15 ++++++++-- .../selftests/bpf/progs/verifier_var_off.c | 6 ++-- 6 files changed, 58 insertions(+), 25 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index fced21ec2304..73d6cd563cdf 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18693,15 +18693,20 @@ static void print_verification_stats(struct bpf_verifier_env *env) if (env->log.level & BPF_LOG_STATS) { verbose(env, "verification time %lld usec\n", div_u64(env->verification_time, 1000)); - verbose(env, "stack depth %d", env->subprog_info[0].stack_depth); - for (i = 1; i < subprog_cnt; i++) - verbose(env, "+%d", env->subprog_info[i].stack_depth); - verbose(env, " max %d\n", env->max_stack_depth); - verbose(env, "insns processed %d", env->subprog_info[0].insns_total); - for (i = 1; i < subprog_cnt; i++) - if (bpf_subprog_is_global(env, i)) - verbose(env, "+%d", env->subprog_info[i].insns_total); - verbose(env, "\n"); + verbose(env, "stack depth max %d\n", env->max_stack_depth); + for (i = 0; i < subprog_cnt; i++) { + const char *name = env->subprog_info[i].name; + const char *kind; + + if (!name || !name[0]) + name = ""; + kind = i == 0 ? "main" : + bpf_subprog_is_global(env, i) ? "global" : "static"; + verbose(env, "subprog %d (%s) %s insns_self %d insns_total %d stack %d\n", + i, name, kind, env->subprog_info[i].insns_self, + env->subprog_info[i].insns_total, + env->subprog_info[i].stack_depth); + } } verbose(env, "processed %d insns (limit %d) max_states_per_insn %d " "total_states %d peak_states %d mark_read %d\n", diff --git a/tools/testing/selftests/bpf/progs/verifier_basic_stack.c b/tools/testing/selftests/bpf/progs/verifier_basic_stack.c index d3df7a9f1d8c..0eb495ce85c1 100644 --- a/tools/testing/selftests/bpf/progs/verifier_basic_stack.c +++ b/tools/testing/selftests/bpf/progs/verifier_basic_stack.c @@ -27,7 +27,8 @@ __naked void stack_out_of_bounds(void) SEC("socket") __description("uninitialized stack1") -__success __log_level(4) __msg("stack depth 8") +__success __log_level(4) +__msg("subprog 0 (uninitialized_stack1) main {{.*}} stack 8") __failure_unpriv __msg_unpriv("invalid read from stack") __naked void uninitialized_stack1(void) { @@ -45,7 +46,8 @@ __naked void uninitialized_stack1(void) SEC("socket") __description("uninitialized stack2") -__success __log_level(4) __msg("stack depth 8") +__success __log_level(4) +__msg("subprog 0 (uninitialized_stack2) main insns_self {{[0-9]+}} insns_total {{[0-9]+}} stack 8") __failure_unpriv __msg_unpriv("invalid read from stack") __naked void uninitialized_stack2(void) { diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index 83707faea049..4cfaa6b4ab40 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -10,7 +10,8 @@ SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 8") +__log_level(4) +__msg("subprog 0 (simple) main insns_self {{[0-9]+}} insns_total {{[0-9]+}} stack 8") __xlated("4: r5 = 5") __xlated("5: r0 = ") __xlated("6: r0 = &(void __percpu *)(r0)") @@ -96,7 +97,8 @@ __naked void canary_zero_spills(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 16") +__log_level(4) +__msg("subprog 0 (wrong_reg_in_pattern1) main {{.*}} stack 16") __xlated("1: *(u64 *)(r10 -16) = r1") __xlated("...") __xlated("3: r0 = &(void __percpu *)(r0)") @@ -598,7 +600,8 @@ __naked static void subprogs_use_independent_offsets_aux(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 8") +__log_level(4) +__msg("subprog 0 (helper_call_does_not_prevent_bpf_fastcall) main {{.*}} stack 8") __xlated("2: r0 = &(void __percpu *)(r0)") __success __naked void helper_call_does_not_prevent_bpf_fastcall(void) @@ -620,7 +623,8 @@ __naked void helper_call_does_not_prevent_bpf_fastcall(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 24") +__log_level(4) +__msg("subprog 0 (may_goto_interaction_x86_64) main {{.*}} stack 24") /* may_goto counter at -24 */ __xlated("0: *(u64 *)(r10 -24) =") /* may_goto timestamp at -16 */ @@ -661,7 +665,8 @@ __naked void may_goto_interaction_x86_64(void) SEC("raw_tp") __arch_arm64 __arch_riscv64 -__log_level(4) __msg("stack depth 24") +__log_level(4) +__msg("subprog 0 (may_goto_interaction) main {{.*}} stack 24") /* may_goto counter at -24 */ __xlated("0: *(u64 *)(r10 -24) =") /* may_goto timestamp at -16 */ @@ -708,7 +713,9 @@ __naked static void dummy_loop_callback(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 32+0") +__log_level(4) +__msg("subprog 0 (bpf_loop_interaction1) main {{.*}} stack 32") +__msg("subprog 1 (dummy_loop_callback) static {{.*}} stack 0") __xlated("2: r1 = 1") __xlated("3: r0 =") __xlated("4: r0 = &(void __percpu *)(r0)") @@ -756,7 +763,9 @@ __naked int bpf_loop_interaction1(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 40+0") +__log_level(4) +__msg("subprog 0 (bpf_loop_interaction2) main {{.*}} stack 40") +__msg("subprog 1 (dummy_loop_callback) static {{.*}} stack 0") /* call bpf_get_smp_processor_id */ __xlated("2: r1 = 42") __xlated("3: r0 =") @@ -800,7 +809,10 @@ __naked int bpf_loop_interaction2(void) SEC("raw_tp") __arch_x86_64 -__log_level(4) __msg("stack depth 512+0 max 512") +__log_level(4) +__msg("stack depth max 512") +__msg("subprog 0 (cumulative_stack_depth) main {{.*}} stack 512") +__msg("subprog 1 (cumulative_stack_depth_subprog) static {{.*}} stack 0") /* just to print xlated version when debugging */ __xlated("r0 = &(void __percpu *)(r0)") __success diff --git a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c index 67dc352addfd..7b65eea97ebc 100644 --- a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c +++ b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c @@ -52,7 +52,10 @@ __msg("('global_calls_good_only') is global and assumed valid.") /* eventually global_good() is transitively validated as well */ __msg("Validating global_good() func") __msg("('global_good') is safe for any args that match its prototype") -__msg("insns processed {{[0-9]+\\+[0-9]+\\+[0-9]+$}}") +__msg("subprog 0 (chained_global_func_calls_success) main insns_self 7 insns_total 7 stack") +__msg("subprog {{[0-9]+}} (global_calls_good_only) global insns_self 2 insns_total 2 stack") +__msg("subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack") +__msg("processed 14 insns") int chained_global_func_calls_success(void) { int sum = 0; diff --git a/tools/testing/selftests/bpf/progs/verifier_private_stack.c b/tools/testing/selftests/bpf/progs/verifier_private_stack.c index bb8206e10880..ea0a7e73331d 100644 --- a/tools/testing/selftests/bpf/progs/verifier_private_stack.c +++ b/tools/testing/selftests/bpf/progs/verifier_private_stack.c @@ -86,7 +86,9 @@ __naked static void cumulative_stack_depth_subprog(void) SEC("kprobe") __description("Private stack, subtree > MAX_BPF_STACK") __success -__log_level(4) __msg("stack depth 512+32 max 512") +__log_level(4) __msg("stack depth max 512") +__msg("subprog 0 (private_stack_nested_1) main {{.*}} stack 512") +__msg("subprog 1 (cumulative_stack_depth_subprog) static {{.*}} stack 32") __arch_x86_64 /* private stack fp for the main prog */ __jited(" movabsq $0x{{.*}}, %r9") @@ -331,7 +333,11 @@ SEC("fentry/bpf_fentry_test9") __description("Private stack, async callback, potential nesting") __success __retval(0) __load_if_JITed() -__log_level(4) __msg("stack depth 8+0+256+0 max 272") +__log_level(4) __msg("stack depth max 272") +__msg("subprog 0 (private_stack_async_callback_2) main {{.*}} stack 8") +__msg("subprog 1 (timer_cb1) static {{.*}} stack 0") +__msg("subprog 2 (subprog1) static {{.*}} stack 256") +__msg("subprog 3 (subprog2) static {{.*}} stack 0") __arch_x86_64 __jited(" subq $0x100, %rsp") __arch_arm64 @@ -355,7 +361,10 @@ int private_stack_async_callback_2(void) SEC("fentry/bpf_fentry_test9") __description("private stack, max stack depth is private stack") __success -__log_level(4) __msg("stack depth 8+256+0 max 256") +__log_level(4) __msg("stack depth max 256") +__msg("subprog 0 (private_stack_max_depth) main {{.*}} stack 8") +__msg("subprog 1 (subprog1) static insns_self {{[0-9]+}} insns_total {{[0-9]+}} stack 256") +__msg("subprog 2 (subprog2) static insns_self {{[0-9]+}} insns_total {{[0-9]+}} stack 0") int private_stack_max_depth(void) { int x = 0; diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c index 24cd0a763673..a63e33675091 100644 --- a/tools/testing/selftests/bpf/progs/verifier_var_off.c +++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c @@ -198,7 +198,8 @@ __success /* Check that the maximum stack depth is correctly maintained according to the * maximum possible variable offset. */ -__log_level(4) __msg("stack depth 16") +__log_level(4) +__msg("subprog 0 (stack_write_priv_vs_unpriv) main {{.*}} stack 16") __failure_unpriv /* Variable stack access is rejected for unprivileged. */ @@ -238,7 +239,8 @@ __success /* Check that the maximum stack depth is correctly maintained according to the * maximum possible variable offset. */ -__log_level(4) __msg("stack depth 16") +__log_level(4) +__msg("subprog 0 (stack_write_followed_by_read) main {{.*}} stack 16") __failure_unpriv __msg_unpriv("R2 variable stack access prohibited for !root") __retval(0) From 502686233493deca7a516234dd041e22eef97c8b Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:21 +0200 Subject: [PATCH 278/373] selftests/bpf: Adjust veristat stack depth parsing The verifier now reports instruction and stack depth statistics using uniform "subprog () " records. Subprogram 0 is classified as main, while other records are global or static. Each record carries insns_self, insns_total, and stack depth. Teach veristat to parse the new records while retaining support for the legacy one-line stack depth format used by older kernels. Skip both instruction counts and match only through the stack value so fields can still be appended without breaking parsing. Increase the bounded backward scan so it can include all 256 per-subprogram records. Zero-initialize the legacy stack buffer because logs using the new format do not populate it before the trailing tokenizer loop. This makes the loop see an empty string instead of reading uninitialized data. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-5-memxor@gmail.com Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/veristat.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c index c9c257784ee3..5c0ec3edce72 100644 --- a/tools/testing/selftests/bpf/veristat.c +++ b/tools/testing/selftests/bpf/veristat.c @@ -993,13 +993,15 @@ static void free_verif_stats(struct verif_stats *stats, size_t stat_cnt) static char verif_log_buf[64 * 1024]; -#define MAX_PARSED_LOG_LINES 100 +/* Keep room for all 256 subprogram records and trailing statistics. */ +#define MAX_PARSED_LOG_LINES 300 static int parse_verif_log(char * const buf, size_t buf_sz, struct verif_stats *s) { const char *cur; - int pos, lines, sub_stack, cnt = 0; - char *state = NULL, *token, stack[512]; + long sub_stack; + int pos, lines, cnt = 0; + char *state = NULL, *token, stack[512] = {}; buf[buf_sz - 1] = '\0'; @@ -1025,11 +1027,24 @@ static int parse_verif_log(char * const buf, size_t buf_sz, struct verif_stats * &s->stats[MARK_READ_MAX_LEN])) continue; + /* + * New kernels emit one "subprog () " record + * per subprogram with the stack depth at the end, while old + * kernels emit a single "stack depth max " + * line. Match both formats so veristat works against either + * kernel. + */ + if (sscanf(cur, "stack depth max %ld", &s->stats[MAX_STACK]) == 1) + continue; + if (sscanf(cur, "subprog %*d %*s %*s insns_self %*d insns_total %*d stack %ld", &sub_stack) == 1) { + s->stats[STACK] += sub_stack; + continue; + } if (2 == sscanf(cur, "stack depth %511s max %ld", stack, &s->stats[MAX_STACK])) continue; } while ((token = strtok_r(cnt++ ? NULL : stack, "+", &state))) { - if (sscanf(token, "%d", &sub_stack) == 0) + if (sscanf(token, "%ld", &sub_stack) == 0) break; s->stats[STACK] += sub_stack; } From b961cf317100b436a78554597ebd4623b4c4e3ba Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:22 +0200 Subject: [PATCH 279/373] selftests/bpf: Test stack depth stats without BTF subprog names Test the per-program insns_self, insns_total, and stack depth statistics emitted when BTF function info does not provide subprogram names. Check that the subprog 0 main record and static-subprogram records use . Make VERBOSE_ACCEPT request verifier statistics so the raw-insn test can validate the output without a test-specific log level. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-6-memxor@gmail.com Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/test_verifier.c | 2 +- tools/testing/selftests/bpf/verifier/calls.c | 12 +++++++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index a8ae03c57bba..bffb7360434c 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -1560,7 +1560,7 @@ static void do_test_single(struct bpf_test *test, bool unpriv, opts.expected_attach_type = test->expected_attach_type; if (expected_ret == VERBOSE_ACCEPT) - opts.log_level = 2; + opts.log_level = 2 | 4; else if (verbose) opts.log_level = verif_log_level | 4; /* force stats */ else diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c index 8cd626e04551..eb6e3baef412 100644 --- a/tools/testing/selftests/bpf/verifier/calls.c +++ b/tools/testing/selftests/bpf/verifier/calls.c @@ -1091,7 +1091,17 @@ /* stack_main=32, stack_A=256, stack_B=64 * and max(main+A, main+A+B) < 512 */ - .result = ACCEPT, + .result = VERBOSE_ACCEPT, + .errstr = "stack depth max 352\t" + "subprog 0 () main insns_self \t" + " insns_total \t" + " stack 32\t" + "subprog 1 () static insns_self \t" + " insns_total \t" + " stack 256\t" + "subprog 2 () static insns_self \t" + " insns_total \t" + " stack 64", }, { "calls: stack depth check using three frames. test2", From 5cb481e1391a886b13f9a6cdf2853a521f660d3f Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 00:19:23 +0200 Subject: [PATCH 280/373] selftests/bpf: Test subprogram instruction statistics Add small verifier programs with deterministic instruction streams to exercise per-subprogram self and inclusive instruction accounting. Use assembly for normal call chains and straight-line callback bodies containing only moves, calls, and returns or exits, so control-flow pruning does not make the expected counts unstable. Pass callback arguments as explicit assembly operands so the compiler keeps their registers live across the asm block. Cover asynchronous callback attribution separately: main verification-root totals include all callback exploration, while static and callback totals remain local to their synchronous paths. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260812221925.3358041-7-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_subprog_insn_stats.c | 223 ++++++++++++++++++ 2 files changed, 225 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 5b265af3b1d5..8113fea7ba86 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -102,6 +102,7 @@ #include "verifier_stack_arg_order.skel.h" #include "verifier_stack_ptr.skel.h" #include "verifier_store_release.skel.h" +#include "verifier_subprog_insn_stats.skel.h" #include "verifier_subprog_precision.skel.h" #include "verifier_subprog_topo.skel.h" #include "verifier_subreg.skel.h" @@ -262,6 +263,7 @@ void test_verifier_stack_arg(void) { RUN(verifier_stack_arg); } void test_verifier_stack_arg_order(void) { RUN(verifier_stack_arg_order); } void test_verifier_stack_ptr(void) { RUN(verifier_stack_ptr); } void test_verifier_store_release(void) { RUN(verifier_store_release); } +void test_verifier_subprog_insn_stats(void) { RUN(verifier_subprog_insn_stats); } void test_verifier_subprog_precision(void) { RUN(verifier_subprog_precision); } void test_verifier_subprog_topo(void) { RUN(verifier_subprog_topo); } void test_verifier_subreg(void) { RUN(verifier_subreg); } diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c new file mode 100644 index 000000000000..8f6082fdb5c8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +struct timer_value { + struct bpf_timer timer; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, struct timer_value); +} timer_map SEC(".maps"); + +SEC("?raw_tp") +__success __log_level(4) +__msg("subprog 0 (stats_main_only) main insns_self 2 insns_total 2 stack 0") +__msg("processed 2 insns") +__naked int stats_main_only(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked __noinline __used +static int stats_chain_leaf(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked __noinline __used +static int stats_chain_parent(void) +{ + asm volatile ( + "call stats_chain_leaf;" + "exit;" + ); +} + +SEC("?raw_tp") +__success __log_level(4) +/* + * self: 2 + 2 + 2 = 6 + * totals: leaf 2, parent 2 + 2 = 4, main 2 + 4 = 6 + */ +__msg("subprog 0 (stats_static_chain) main insns_self 2 insns_total 6 stack 0") +__msg("subprog {{[0-9]+}} (stats_chain_parent) static insns_self 2 insns_total 4 stack 0") +__msg("subprog {{[0-9]+}} (stats_chain_leaf) static insns_self 2 insns_total 2 stack 0") +__msg("processed 6 insns") +__naked int stats_static_chain(void) +{ + asm volatile ( + "call stats_chain_parent;" + "exit;" + ); +} + +__naked __noinline __used +static int stats_shared_leaf(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked __noinline __used +int stats_global_root(void) +{ + asm volatile ( + "call stats_shared_leaf;" + "exit;" + ); +} + +SEC("?raw_tp") +__success __log_level(4) +/* + * stats_shared_leaf is explored once under each independent root. + * self: main 3 + leaf 4 + global 2 = 9 + * root totals: main 5 + global 4 = 9 + */ +__msg("subprog 0 (stats_shared_roots) main insns_self 3 insns_total 5 stack 0") +__msg("subprog {{[0-9]+}} (stats_shared_leaf) static insns_self 4 insns_total 4 stack 0") +__msg("subprog {{[0-9]+}} (stats_global_root) global insns_self 2 insns_total 4 stack 0") +__msg("processed 9 insns") +__naked int stats_shared_roots(void) +{ + asm volatile ( + "call stats_shared_leaf;" + "call stats_global_root;" + "exit;" + ); +} + +__noinline __used +static int stats_async_leaf(void *map, __u32 *key, struct bpf_timer *timer) +{ + return 0; +} + +__noinline __used +static __u64 stats_async_schedule(struct bpf_map *map, __u32 *key, + struct timer_value *value, void *ctx) +{ + asm volatile ( + "r1 = %[timer];" + "r2 = %[stats_async_leaf];" + "call %[bpf_timer_set_callback];" + : + : [timer] "r" (value), + __imm_ptr(stats_async_leaf), + __imm(bpf_timer_set_callback) + : __clobber_common + ); + return 0; +} + +SEC("?raw_tp") +__success __log_level(4) +/* + * self: 9 + 7 + 2 = 18 + * totals: leaf 2, scheduler 7, main root 18 + */ +__msg("subprog 0 (stats_async_direct) main insns_self 9 insns_total 18 stack 0") +__msg("subprog {{[0-9]+}} (stats_async_schedule) static insns_self 7 insns_total 7 stack 0") +__msg("subprog {{[0-9]+}} (stats_async_leaf) static insns_self 2 insns_total 2 stack 0") +__msg("processed 18 insns") +__naked int stats_async_direct(void) +{ + asm volatile ( + "r1 = %[timer_map] ll;" + "r2 = %[stats_async_schedule];" + "r3 = 0;" + "r4 = 0;" + "call %[bpf_for_each_map_elem];" + "r0 = 0;" + "exit;" + : + : __imm_addr(timer_map), + __imm_ptr(stats_async_schedule), + __imm(bpf_for_each_map_elem) + : __clobber_common + ); +} + +__noinline __used +static int stats_async_nested_leaf(void *map, __u32 *key, struct bpf_timer *timer) +{ + return 0; +} + +__noinline __used +static int stats_async_outer(void *map, __u32 *key, struct bpf_timer *timer) +{ + asm volatile ( + "r1 = %[timer];" + "r2 = %[stats_async_nested_leaf];" + "call %[bpf_timer_set_callback];" + : + : [timer] "r" (timer), + __imm_ptr(stats_async_nested_leaf), + __imm(bpf_timer_set_callback) + : __clobber_common + ); + return 0; +} + +__noinline __used +static __u64 stats_async_nested_schedule(struct bpf_map *map, __u32 *key, + struct timer_value *value, void *ctx) +{ + asm volatile ( + "r1 = %[timer];" + "r2 = %[stats_async_outer];" + "call %[bpf_timer_set_callback];" + : + : [timer] "r" (value), + __imm_ptr(stats_async_outer), + __imm(bpf_timer_set_callback) + : __clobber_common + ); + return 0; +} + +SEC("?raw_tp") +__success __log_level(4) +/* + * self: 9 + 7 + 7 + 2 = 25 + * totals: leaf 2, outer 7, scheduler 7, main root 25 + */ +__msg("subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0") +__msg("subprog {{[0-9]+}} (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0") +__msg("subprog {{[0-9]+}} (stats_async_outer) static insns_self 7 insns_total 7 stack 0") +__msg("subprog {{[0-9]+}} (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0") +__msg("processed 25 insns") +__naked int stats_async_nested(void) +{ + asm volatile ( + "r1 = %[timer_map] ll;" + "r2 = %[stats_async_nested_schedule];" + "r3 = 0;" + "r4 = 0;" + "call %[bpf_for_each_map_elem];" + "r0 = 0;" + "exit;" + : + : __imm_addr(timer_map), + __imm_ptr(stats_async_nested_schedule), + __imm(bpf_for_each_map_elem) + : __clobber_common + ); +} + +char _license[] SEC("license") = "GPL"; From 0253073fb7d79a2dd2eae9581ea16db2aef395a6 Mon Sep 17 00:00:00 2001 From: Hui Zhu Date: Tue, 11 Aug 2026 10:46:19 +0800 Subject: [PATCH 281/373] bpf: Fix UAF in bpf_trampoline_multi_attach_free on update failure When bpf_trampoline_update() fails before modify_fentry_multi()/ unregister_fentry_multi() is called, cur_image is unchanged (cur_image == old_image) and ftrace still calls into it. Freeing old_image in that case causes a UAF. Only free old_image when it differs from cur_image. Fixes: aef4dfa790b2 ("bpf: Add bpf_trampoline_multi_attach/detach functions") Signed-off-by: Hui Zhu Acked-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/aaa3829e11e2e26bcd3bda9ee6df7a0101a718ac.1786412280.git.zhuhui@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/trampoline.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index e07af35ed040..008448bb4a1f 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -1632,7 +1632,17 @@ static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr) static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr) { - if (tr->multi_attach.old_image) + /* + * Only free old_image if it is no longer the active image. + * When bpf_trampoline_update() fails before modify_fentry_multi()/ + * unregister_fentry_multi() is called, cur_image is unchanged + * (cur_image == old_image) and ftrace still points to it. Freeing + * it would cause a UAF when ftrace calls into the freed memory. + * On success, cur_image is either a new image or NULL, so + * old_image != cur_image means the image is stale. + */ + if (tr->multi_attach.old_image && + tr->multi_attach.old_image != tr->cur_image) bpf_tramp_image_put(tr->multi_attach.old_image); tr->multi_attach.old_image = NULL; From 7c6beeb8c88f92866daab4516220667d1234d3c9 Mon Sep 17 00:00:00 2001 From: Hui Zhu Date: Tue, 11 Aug 2026 10:46:20 +0800 Subject: [PATCH 282/373] bpf: Make bpf_trampoline_multi_detach return void bpf_trampoline_multi_detach() always returns 0 and the sole caller ignores the return value. Change it to return void and drop the WARN_ON_ONCE at the call site. Signed-off-by: Hui Zhu Acked-by: Leon Hwang Acked-by: Jiri Olsa Link: https://lore.kernel.org/bpf/12beba657f5c9e86a016a097750209287a2f262a.1786412280.git.zhuhui@kylinos.cn Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 9 ++++----- kernel/bpf/trampoline.c | 4 ++-- kernel/trace/bpf_trace.c | 2 +- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b4a10c9878cf..f4e8d372253a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1518,8 +1518,8 @@ int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_func int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, struct bpf_tracing_multi_link *link); -int bpf_trampoline_multi_detach(struct bpf_prog *prog, - struct bpf_tracing_multi_link *link); +void bpf_trampoline_multi_detach(struct bpf_prog *prog, + struct bpf_tracing_multi_link *link); void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags); /* @@ -1639,10 +1639,9 @@ static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, { return -ENOTSUPP; } -static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog, - struct bpf_tracing_multi_link *link) +static inline void bpf_trampoline_multi_detach(struct bpf_prog *prog, + struct bpf_tracing_multi_link *link) { - return -ENOTSUPP; } static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {} #endif diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 008448bb4a1f..90b70ea0d370 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -1766,7 +1766,8 @@ int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, return err; } -int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link) +void bpf_trampoline_multi_detach(struct bpf_prog *prog, + struct bpf_tracing_multi_link *link) { struct bpf_tracing_multi_data *data = &link->data; struct bpf_tracing_multi_node *mnode; @@ -1796,7 +1797,6 @@ int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_ bpf_trampoline_put(mnode->trampoline); clear_tracing_multi_data(data); - return 0; } #undef for_each_mnode_cnt diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 891897f8a1b3..29260951aa87 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -3687,7 +3687,7 @@ static void bpf_tracing_multi_link_release(struct bpf_link *link) struct bpf_tracing_multi_link *tr_link = container_of(link, struct bpf_tracing_multi_link, link); - WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link)); + bpf_trampoline_multi_detach(link->prog, tr_link); } static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) From 5e31d32843d3ad4b349ea99e534662713687e810 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Wed, 12 Aug 2026 21:38:40 +0200 Subject: [PATCH 283/373] resolve_btfids: Emit arena attributes from kfunc parameter suffixes Kfunc declarations can identify arena arguments through parameter name suffixes without repeating KF_ARENA_ARG flags in their BTF ID sets. resolve_btfids currently misses those arguments when synthesizing the address_space(1) attributes used by generated vmlinux.h files. Teach the arena prototype rewrite to recognize __arena and __arena__nullable directly on each parameter. Keep KF_ARENA_ARG1 and KF_ARENA_ARG2 handling for explicitly flagged kfuncs, while allowing suffixes on any argument without synthesizing kfunc flags. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260812193842.2879226-2-memxor@gmail.com Signed-off-by: Eduard Zingerman --- Documentation/bpf/kfuncs.rst | 4 +- tools/bpf/resolve_btfids/main.c | 72 ++++++++++++++++++++++++--------- 2 files changed, 54 insertions(+), 22 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 1004eb0bec61..10e725cbe64c 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -513,8 +513,8 @@ At kernel build time the ``resolve_btfids`` tool finds all kfuncs declared with ``BTF_KFUNCS_START()`` and emits their BTF annotations into the kernel's BTF. For each kfunc it emits a ``bpf_kfunc`` BTF decl tag, a ``bpf_fastcall`` decl tag when the kfunc is flagged ``KF_FASTCALL``, and the ``address_space(1)`` type -attribute on the return value and/or arguments flagged ``KF_ARENA_RET``, -``KF_ARENA_ARG1`` or ``KF_ARENA_ARG2`` (see section 2.8). +attribute on the return value and/or arguments that use arena pointers (see +sections 2.3.8 and 2.8). 2.7 Specifying no-cast aliases with ___init -------------------------------------------- diff --git a/tools/bpf/resolve_btfids/main.c b/tools/bpf/resolve_btfids/main.c index d2e4176339da..37d7e7224207 100644 --- a/tools/bpf/resolve_btfids/main.c +++ b/tools/bpf/resolve_btfids/main.c @@ -64,8 +64,8 @@ * each such kfunc it: * * - emits a "bpf_kfunc" decl tag, and "bpf_fastcall" when KF_FASTCALL is set; - * - wraps the return value and/or arguments flagged KF_ARENA_RET, - * KF_ARENA_ARG1 or KF_ARENA_ARG2 with the "address_space(1)" type attribute; + * - wraps the return value and/or arguments that use arena pointers + * with the "address_space(1)" type attribute; * - rewrites the prototype of KF_IMPLICIT_ARGS kfuncs. * * These kfunc annotations were historically produced by pahole. @@ -182,6 +182,8 @@ struct object { #define KF_IMPLICIT_ARGS (1 << 16) #define KF_IMPL_SUFFIX "_impl" #define TYPE_ATTR_ARENA "address_space(1)" +#define PARAM_SUFFIX_ARENA "__arena" +#define PARAM_SUFFIX_ARENA_NULLABLE "__arena__nullable" struct kfunc { struct rb_node rb_node; @@ -1067,6 +1069,22 @@ static int collect_decl_tags(struct btf2btf_context *ctx) return 0; } +static bool param_name_has_suffix(const char *name, const char *suffix) +{ + size_t name_len = strlen(name); + size_t suffix_len = strlen(suffix); + + return name_len >= suffix_len && !strcmp(name + name_len - suffix_len, suffix); +} + +static bool is_arena_param(const struct btf *btf, const struct btf_param *param) +{ + const char *name = btf__name_by_offset(btf, param->name_off); + + return param_name_has_suffix(name, PARAM_SUFFIX_ARENA) || + param_name_has_suffix(name, PARAM_SUFFIX_ARENA_NULLABLE); +} + static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx) { Elf_Data *idlist = obj->efile.idlist; @@ -1299,8 +1317,12 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct return 0; } -static bool is_arena_arg(struct kfunc *kfunc, u32 idx) +static bool is_arena_arg(const struct btf *btf, const struct kfunc *kfunc, + const struct btf_param *param, u32 idx) { + if (is_arena_param(btf, param)) + return true; + switch (idx) { case 0: return kfunc->flags & KF_ARENA_ARG1; @@ -1339,23 +1361,36 @@ static s32 arena_tag_ptr(struct btf *btf, u32 ptr_id, struct kfunc *kfunc) } /* - * Add a FUNC_PROTO for @kfunc with each relevant pointer tagged with - * an "address_space(1)" attribute. The original proto may be shared - * with other FUNCs, so it is never modified in place. + * Add a FUNC_PROTO for @kfunc with each arena pointer tagged with an + * "address_space(1)" attribute. The original proto may be shared with + * other FUNCs, so it is never modified in place. Returns the original + * proto id when @kfunc has no arena return value or arguments. */ static s32 add_arena_tagged_proto(struct btf *btf, struct kfunc *kfunc) { const struct btf_type *func = btf__type_by_id(btf, kfunc->btf_id); u32 proto_id = func->type; const struct btf_type *proto = btf__type_by_id(btf, proto_id); + const struct btf_param *params = btf_params(proto); u32 nr_params = btf_vlen(proto); s32 ret_type_id = proto->type; const struct btf_type *t; - struct btf_param *params; + struct btf_param *tag_params; s32 new_proto_id, id; const char *name; + bool has_arena_arg = false; int err, i; + for (i = 0; i < nr_params; i++) { + if (is_arena_arg(btf, kfunc, ¶ms[i], i)) { + has_arena_arg = true; + break; + } + } + + if (!(kfunc->flags & KF_ARENA_RET) && !has_arena_arg) + return proto_id; + if (kfunc->flags & KF_ARENA_RET) { ret_type_id = arena_tag_ptr(btf, ret_type_id, kfunc); if (ret_type_id < 0) @@ -1383,19 +1418,18 @@ static s32 add_arena_tagged_proto(struct btf *btf, struct kfunc *kfunc) } for (i = 0; i < nr_params; i++) { - if (!is_arena_arg(kfunc, i)) + t = btf__type_by_id(btf, new_proto_id); + tag_params = btf_params(t); + if (!is_arena_arg(btf, kfunc, &tag_params[i], i)) continue; - t = btf__type_by_id(btf, new_proto_id); - params = btf_params(t); - - id = arena_tag_ptr(btf, params[i].type, kfunc); + id = arena_tag_ptr(btf, tag_params[i].type, kfunc); if (id < 0) return id; t = btf__type_by_id(btf, new_proto_id); - params = btf_params(t); - params[i].type = id; + tag_params = btf_params(t); + tag_params[i].type = id; } pr_debug("added arena-tagged proto for kfunc %s: %d\n", kfunc->name, new_proto_id); @@ -1403,7 +1437,7 @@ static s32 add_arena_tagged_proto(struct btf *btf, struct kfunc *kfunc) return new_proto_id; } -static int process_kfunc_with_arena_flags(struct btf2btf_context *ctx, +static int process_kfunc_with_arena_attrs(struct btf2btf_context *ctx, struct kfunc *kfunc) { struct btf_type *t; @@ -1463,11 +1497,9 @@ static int btf2btf(struct object *obj) goto out; } - if (kfunc->flags & (KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2)) { - err = process_kfunc_with_arena_flags(&ctx, kfunc); - if (err) - goto out; - } + err = process_kfunc_with_arena_attrs(&ctx, kfunc); + if (err) + goto out; } err = 0; From d7842e98ca172b30e15fb498a63af103ecedefab Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Wed, 12 Aug 2026 21:38:41 +0200 Subject: [PATCH 284/373] selftests/bpf: Test resolve_btfids arena argument suffixes Add a suffix-only kfunc declaration with arena annotations on all five arguments. Verify that resolve_btfids emits address_space(1) type tags for every position without KF_ARENA_ARG flags in the BTF ID sets. Represent expected arena arguments as a per-parameter bitmap so the test covers suffixes beyond the two positions expressible by flags. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260812193842.2879226-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/resolve_btfids.c | 41 +++++++++++-------- tools/testing/selftests/bpf/progs/btf_data.c | 20 +++++++++ 2 files changed, 45 insertions(+), 16 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c index 732cfed35e1c..3f9949e8227d 100644 --- a/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +++ b/tools/testing/selftests/bpf/prog_tests/resolve_btfids.c @@ -15,6 +15,7 @@ #define DECL_TAG_FASTCALL "bpf_fastcall" #define DECL_TAG_KFUNC "bpf_kfunc" #define TYPE_ATTR_ARENA "address_space(1)" +#define ARENA_ARG(n) (1U << (n)) #ifndef KF_FASTCALL #define KF_FASTCALL (1 << 12) @@ -49,13 +50,20 @@ struct kfunc_symbol { const char *name; s32 id; u32 flags; + u32 arena_args; + bool arena_ret; }; static struct kfunc_symbol kfunc_symbols[] = { - { "kfunc_a", -1, 0 }, - { "kfunc_b", -1, KF_FASTCALL }, - { "kfunc_c", -1, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2 }, - { "kfunc_d", -1, KF_ARENA_ARG2 }, + { "kfunc_a", -1, 0, 0, false }, + { "kfunc_b", -1, KF_FASTCALL, 0, false }, + { "kfunc_c", -1, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2, + ARENA_ARG(0) | ARENA_ARG(1), true }, + { "kfunc_d", -1, KF_ARENA_ARG2, ARENA_ARG(1), false }, + { "kfunc_e", -1, 0, ARENA_ARG(0) | ARENA_ARG(1) | ARENA_ARG(2) | + ARENA_ARG(3) | ARENA_ARG(4), false }, + { "kfunc_f", -1, 0, ARENA_ARG(1), false }, + { "kfunc_g", -1, KF_ARENA_RET, ARENA_ARG(0) | ARENA_ARG(1), true }, }; /* Align the .BTF_ids section to 4 bytes */ @@ -105,6 +113,9 @@ BTF_ID_FLAGS(func, kfunc_a) BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) BTF_ID_FLAGS(func, kfunc_c, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2) BTF_ID_FLAGS(func, kfunc_d, KF_ARENA_ARG2) +BTF_ID_FLAGS(func, kfunc_e) +BTF_ID_FLAGS(func, kfunc_f) +BTF_ID_FLAGS(func, kfunc_g, KF_ARENA_RET) BTF_KFUNCS_END(test_kfunc_set) /* @@ -112,6 +123,9 @@ BTF_KFUNCS_END(test_kfunc_set) * actually sort at least one of the two sets. */ BTF_KFUNCS_START(test_kfunc_set_rev) +BTF_ID_FLAGS(func, kfunc_g, KF_ARENA_RET) +BTF_ID_FLAGS(func, kfunc_f) +BTF_ID_FLAGS(func, kfunc_e) BTF_ID_FLAGS(func, kfunc_d, KF_ARENA_ARG2) BTF_ID_FLAGS(func, kfunc_c, KF_ARENA_RET | KF_ARENA_ARG1 | KF_ARENA_ARG2) BTF_ID_FLAGS(func, kfunc_b, KF_FASTCALL) @@ -301,15 +315,15 @@ void test_resolve_btfids(void) } /* - * Check resolve_btfids wrapped exactly the arena-flagged return/args - * with the address_space(1) type attribute, and left other + * Check resolve_btfids wrapped exactly the arena-flagged or suffixed + * return/args with the address_space(1) type attribute, and left other * pointers/returns untouched. */ for (i = 0; i < ARRAY_SIZE(kfunc_symbols); i++) { const struct btf_type *fn, *proto; const struct btf_param *params; const char *name = kfunc_symbols[i].name; - u32 fl = kfunc_symbols[i].flags; + u32 arena_args = kfunc_symbols[i].arena_args; __u32 nr; fn = btf__type_by_id(btf, kfunc_symbols[i].id); @@ -322,15 +336,10 @@ void test_resolve_btfids(void) nr = btf_vlen(proto); ASSERT_EQ(is_arena_tagged_ptr(btf, proto->type), - !!(fl & KF_ARENA_RET), name); - if (nr > 0) { - ASSERT_EQ(is_arena_tagged_ptr(btf, params[0].type), - !!(fl & KF_ARENA_ARG1), name); - } - if (nr > 1) { - ASSERT_EQ(is_arena_tagged_ptr(btf, params[1].type), - !!(fl & KF_ARENA_ARG2), name); - } + kfunc_symbols[i].arena_ret, name); + for (j = 0; j < nr; j++) + ASSERT_EQ(is_arena_tagged_ptr(btf, params[j].type), + !!(arena_args & ARENA_ARG(j)), name); } out: diff --git a/tools/testing/selftests/bpf/progs/btf_data.c b/tools/testing/selftests/bpf/progs/btf_data.c index ec34f7a6e038..8082c13490ab 100644 --- a/tools/testing/selftests/bpf/progs/btf_data.c +++ b/tools/testing/selftests/bpf/progs/btf_data.c @@ -68,3 +68,23 @@ int kfunc_d(struct root_struct *a, struct root_struct *b) { return 0; } + +int kfunc_e(struct root_struct *a__arena, + struct root_struct *b__arena__nullable, + struct root_struct *c__arena, + struct root_struct *d__arena__nullable, + struct root_struct *e__arena) +{ + return 0; +} + +int kfunc_f(struct root_struct *a, struct root_struct *b__arena, int flags) +{ + return 0; +} + +struct root_struct *kfunc_g(struct root_struct *a__arena, + struct root_struct *b__arena__nullable) +{ + return a__arena; +} From 7c3e54cb82fda75a374e2b29b25d2a399911a008 Mon Sep 17 00:00:00 2001 From: Xu Kuohai Date: Tue, 28 Jul 2026 20:25:49 +0000 Subject: [PATCH 285/373] bpf: Eliminate dup/restore of insn_aux_data The dup/restore of insn_aux_data was introduced to resolve the inconsistency between insnsi and insn_aux_data arrays, which occurs on the failure path where insnsi was rolled back to the original state before constants blinding, while insn_aux_data was not. After JIT failure, there is only one user, bpf_clear_insn_aux_data(), that requires insnsi and insn_aux_data to be synchronized. It accesses both insnsi and insn_aux_data using the same array size and index. However, the access to insnsi in bpf_clear_insn_aux_data() is not necessary. It is checked to skip the second slot of an ldimm64 instruction, whose jt is never set and can be absorbed into the jt check itself. So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a specific length field for insn_aux_data to allow it to have a different length from the insnsi array. Then remove dup/restore of insn_aux_data. Signed-off-by: Xu Kuohai Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 1 + include/linux/filter.h | 13 ------------- kernel/bpf/core.c | 16 ---------------- kernel/bpf/fixups.c | 36 ++---------------------------------- kernel/bpf/verifier.c | 4 ++-- 5 files changed, 5 insertions(+), 65 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 7c376451db82..27b43fda9b17 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -946,6 +946,7 @@ struct bpf_verifier_env { bool seen_direct_write; bool seen_exception; bool signature; + u32 insn_aux_data_len; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/include/linux/filter.h b/include/linux/filter.h index 15d83684c6e9..4a9bc6a848f2 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1267,25 +1267,12 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, #ifdef CONFIG_BPF_SYSCALL struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, const struct bpf_insn *patch, u32 len); -struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env); -void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux); #else static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, const struct bpf_insn *patch, u32 len) { return ERR_PTR(-ENOTSUPP); } - -static inline struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env) -{ - return NULL; -} - -static inline void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux) -{ -} #endif /* CONFIG_BPF_SYSCALL */ int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index a3e1fae32eac..6a94370a2448 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2634,22 +2634,10 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc { #ifdef CONFIG_BPF_JIT struct bpf_prog *orig_prog; - struct bpf_insn_aux_data *orig_insn_aux; if (!bpf_prog_need_blind(prog)) return bpf_int_jit_compile(env, prog); - if (env) { - /* - * If env is not NULL, we are called from the end of bpf_check(), at this - * point, only insn_aux_data is used after failure, so it should be restored - * on failure. - */ - orig_insn_aux = bpf_dup_insn_aux_data(env); - if (!orig_insn_aux) - return prog; - } - orig_prog = prog; prog = bpf_jit_blind_constants(env, prog); /* @@ -2662,8 +2650,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc prog = bpf_int_jit_compile(env, prog); if (prog->jited) { bpf_jit_prog_release_other(prog, orig_prog); - if (env) - vfree(orig_insn_aux); return prog; } @@ -2671,8 +2657,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc out_restore: prog = orig_prog; - if (env) - bpf_restore_insn_aux_data(env, orig_insn_aux); #endif return prog; } diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index c4bd70befbb5..2417a3461652 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -230,6 +230,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env, if (cnt == 1) return; prog_len = new_prog->len; + env->insn_aux_data_len = prog_len; memmove(data + off + cnt - 1, data + off, sizeof(struct bpf_insn_aux_data) * (prog_len - off - cnt + 1)); @@ -496,7 +497,6 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off, void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len) { struct bpf_insn_aux_data *aux_data = env->insn_aux_data; - struct bpf_insn *insns = env->prog->insnsi; int end = start + len; int i; @@ -505,9 +505,6 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len) kvfree(aux_data[i].jt); aux_data[i].jt = NULL; } - - if (bpf_is_ldimm64(&insns[i])) - i++; } } @@ -520,7 +517,6 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) if (bpf_prog_is_offloaded(env->prog->aux)) bpf_prog_offload_remove_insns(env, off, cnt); - /* Should be called before bpf_remove_insns, as it uses prog->insnsi */ bpf_clear_insn_aux_data(env, off, cnt); err = bpf_remove_insns(env->prog, off, cnt); @@ -539,6 +535,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) memmove(aux_data + off, aux_data + off + cnt, sizeof(*aux_data) * (orig_prog_len - off - cnt)); + env->insn_aux_data_len -= cnt; return 0; } @@ -1057,26 +1054,6 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s env->subprog_info[env->subprog_cnt].start = env->prog->len; } -struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env) -{ - size_t size; - void *new_aux; - - size = array_size(sizeof(struct bpf_insn_aux_data), env->prog->len); - new_aux = __vmalloc(size, GFP_KERNEL_ACCOUNT); - if (new_aux) - memcpy(new_aux, env->insn_aux_data, size); - return new_aux; -} - -void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, - struct bpf_insn_aux_data *orig_insn_aux) -{ - /* the expanded elements are zero-filled, so no special handling is required */ - vfree(env->insn_aux_data); - env->insn_aux_data = orig_insn_aux; -} - static int jit_subprogs(struct bpf_verifier_env *env) { struct bpf_prog *prog = env->prog, **func, *tmp; @@ -1351,7 +1328,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) bool blinded = false; struct bpf_insn *insn; struct bpf_prog *prog, *orig_prog; - struct bpf_insn_aux_data *orig_insn_aux; u32 *orig_subprog_starts; if (env->subprog_cnt <= 1) @@ -1359,14 +1335,8 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) prog = orig_prog = env->prog; if (bpf_prog_need_blind(prog)) { - orig_insn_aux = bpf_dup_insn_aux_data(env); - if (!orig_insn_aux) { - err = -ENOMEM; - goto out_cleanup; - } orig_subprog_starts = bpf_dup_subprog_starts(env); if (!orig_subprog_starts) { - vfree(orig_insn_aux); err = -ENOMEM; goto out_cleanup; } @@ -1386,7 +1356,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) if (blinded) { bpf_jit_prog_release_other(prog, orig_prog); kvfree(orig_subprog_starts); - vfree(orig_insn_aux); } return 0; @@ -1416,7 +1385,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env) out_restore: bpf_restore_subprog_starts(env, orig_subprog_starts); - bpf_restore_insn_aux_data(env, orig_insn_aux); kvfree(orig_subprog_starts); out_cleanup: /* cleanup main prog to be interpreted */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 73d6cd563cdf..7a4e1b88d73a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20213,7 +20213,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, if (!is_priv) mutex_lock(&bpf_verifier_lock); - len = env->prog->len; + len = env->insn_aux_data_len = env->prog->len; env->insn_aux_data = __vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len), GFP_KERNEL_ACCOUNT | __GFP_ZERO); @@ -20468,7 +20468,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, release_btfs(env); err_free_env: if (env->insn_aux_data) - bpf_clear_insn_aux_data(env, 0, env->prog->len); + bpf_clear_insn_aux_data(env, 0, env->insn_aux_data_len); vfree(env->insn_aux_data); kvfree(env->fd_array); bpf_stack_liveness_free(env); From 6f033615ef8fb2374daa7e50a8ff68616bc850d2 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 00:48:43 +0800 Subject: [PATCH 286/373] bpf: Trim special_kfunc_list in verifier The commit 7619a0ee9340 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE") dropped some helpers in verifier, which also eliminated the use of the following kfuncs from the special_kfunc_list: * bpf_arena_reserve_pages * bpf_stream_vprintk * bpf_stream_print_stack So, drop them from the special_kfunc_list. Signed-off-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7a4e1b88d73a..164d16c243ca 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11122,10 +11122,7 @@ enum special_kfunc_type { KF_bpf_task_work_schedule_resume, KF_bpf_arena_alloc_pages, KF_bpf_arena_free_pages, - KF_bpf_arena_reserve_pages, KF_bpf_session_is_return, - KF_bpf_stream_vprintk, - KF_bpf_stream_print_stack, }; BTF_ID_LIST(special_kfunc_list) @@ -11215,14 +11212,11 @@ BTF_ID(func, bpf_task_work_schedule_signal) BTF_ID(func, bpf_task_work_schedule_resume) BTF_ID(func, bpf_arena_alloc_pages) BTF_ID(func, bpf_arena_free_pages) -BTF_ID(func, bpf_arena_reserve_pages) #ifdef CONFIG_BPF_EVENTS BTF_ID(func, bpf_session_is_return) #else BTF_ID_UNUSED #endif -BTF_ID(func, bpf_stream_vprintk) -BTF_ID(func, bpf_stream_print_stack) static bool is_bpf_obj_new_kfunc(u32 func_id) { From 9786d424a36b3c600250290e15168e2d401b995b Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Thu, 13 Aug 2026 08:06:41 -0700 Subject: [PATCH 287/373] selftests/bpf: Fix chained_global_func_calls_success() for cpu v4 The chained_global_func_calls_success() test hardcodes the instruction counts reported by the verifier's per-subprog stats: subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack processed 14 insns global_good() does 'return arr[0]', where arr[] is an int array and the return type is long. Without cpu v4 this is a zero-extending load followed by a <<32/s>>32 sign-extension pair. With -mcpu=v4 llvm emits a single sign-extending load instead: 18: (18) r1 = 0xffa00000008eb000 20: (81) r0 = *(s32 *)(r1 +0) 21: (95) exit so the subprog is 3 insns rather than 5, and the whole program is 12 processed insns rather than 14. test_progs-cpuv4 fails with: EXPECTED REGEX: 'subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack' #606/1 verifier_global_subprogs/chained_global_func_calls_success:FAIL Select the expected counts based on __BPF_CPU_VERSION__. Fixes: c2e6c7de8830 ("bpf: Show more useful info in stack depth stats") Signed-off-by: Yonghong Song Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260813150641.3347662-1-yonghong.song@linux.dev --- tools/testing/selftests/bpf/progs/verifier_global_subprogs.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c index 7b65eea97ebc..966f49348787 100644 --- a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c +++ b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c @@ -54,8 +54,13 @@ __msg("Validating global_good() func") __msg("('global_good') is safe for any args that match its prototype") __msg("subprog 0 (chained_global_func_calls_success) main insns_self 7 insns_total 7 stack") __msg("subprog {{[0-9]+}} (global_calls_good_only) global insns_self 2 insns_total 2 stack") +#if defined(__BPF_CPU_VERSION__) && __BPF_CPU_VERSION__ >= 4 +__msg("subprog {{[0-9]+}} (global_good) global insns_self 3 insns_total 3 stack") +__msg("processed 12 insns") +#else __msg("subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack") __msg("processed 14 insns") +#endif int chained_global_func_calls_success(void) { int sum = 0; From 0a07e75b16b6788948198ca8576a3a72324c7ca3 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:14 +0800 Subject: [PATCH 288/373] bpf: Drop duplicate blank lines in kernel/bpf/ There are many adjacent blank lines in kernel/bpf/ that have accumulated over time. Drop them for cleanup. No functional changes intended. Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260813152324.97937-2-leon.hwang@linux.dev --- kernel/bpf/backtrack.c | 2 -- kernel/bpf/btf.c | 1 - kernel/bpf/cfg.c | 1 - kernel/bpf/fixups.c | 1 - kernel/bpf/hashtab.c | 2 -- kernel/bpf/helpers.c | 1 - kernel/bpf/liveness.c | 2 -- kernel/bpf/queue_stack_maps.c | 1 - kernel/bpf/syscall.c | 5 ----- kernel/bpf/verifier.c | 14 -------------- 10 files changed, 30 deletions(-) diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 40bd04421a99..a2b18a9f1694 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg) return bt->reg_masks[bt->frame] & (1 << reg); } - /* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask) { @@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask) } } - /* For given verifier state backtrack_insn() is called from the last insn to * the first insn. Its purpose is to compute a bitmask of registers and * stack slots that needs precision in the parent verifier state. diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6606187ed4f4..87ffde865a50 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void *data, u8 bits_offset, btf_int128_print(show, print_num); } - static void btf_int_bits_show(const struct btf *btf, const struct btf_type *t, void *data, u8 bits_offset, diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index db3416a7c904..818f7afac83a 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -47,7 +47,6 @@ enum { BRANCH = 2, }; - static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off) { struct bpf_subprog_info *subprog; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 2417a3461652..0caf1bbd9494 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) return err; } - /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) { diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 9f394e1aa2e8..d40cb5dd446c 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab *htab) atomic_dec(&htab->count); } - static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) { htab_put_fd_value(htab, l); @@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v return 0; } - static long rhtab_map_delete_elem(struct bpf_map *map, void *key) { struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 6388b6b23e49..45e2f19387b2 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = { .set = &generic_btf_ids, }; - BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) BTF_ID(func, bpf_task_release_dtor) diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 1c997aeba6fa..74fc4b3f80d6 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx) __diag_pop(); - static inline bool update_insn(struct bpf_verifier_env *env, struct func_instance *instance, u32 frame, u32 insn_idx) { @@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env *env, if (need_resched()) cond_resched(); - /* * When an instance is reused (must_write_initialized == true), * record into a fresh instance and merge afterward. This avoids diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index c1c9dee4dcdd..6e8b18c32a10 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete) return err; } - static long __stack_map_get(struct bpf_map *map, void *value, bool delete) { struct bpf_queue_stack *qs = bpf_queue_stack(map); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8d111da88655..7d8c3e8e6d62 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid, return ret; } - static int btf_field_cmp(const void *a, const void *b) { const struct btf_field *f1 = a, *f2 = b; @@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr *attr) return err; } - #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) @@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer) if (fd < 0) return fd; - id = bpf_link_alloc_id(link); if (id < 0) { put_unused_fd(fd); @@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file *file, return 0; } - static int token_get_info_by_fd(struct file *file, struct bpf_token *token, const union bpf_attr *attr, @@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0); } - /* To shut up -Wmissing-prototypes. * This function is used by the kernel light skeleton * to load bpf programs when modules are loaded or during kernel boot. diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 164d16c243ca..cdb79a66b156 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type, bool first_slot, int id, int parent_id); - static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, struct bpf_reg_state *sreg1, struct bpf_reg_state *sreg2, @@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta return true; } - void bpf_free_backedges(struct bpf_scc_visit *visit) { struct bpf_scc_backedge *backedge, *next; @@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, return &elem->st; } - static int cmp_subprogs(const void *a, const void *b) { return ((struct bpf_subprog_info *)a)->start - @@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env *env, return err; } - /* check_stack_write dispatches to check_stack_write_fixed_off or * check_stack_write_var_off. * @@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, valid = false; } - if (valid) { env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size; @@ -6635,7 +6630,6 @@ static int check_stack_range_initialized( if (err) return err; - if (tnum_is_const(reg->var_off)) { min_off = max_off = reg->var_off.value + off; } else { @@ -7347,7 +7341,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta) return meta->kfunc_flags & KF_ITER_NEW; } - static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta) { return meta->kfunc_flags & KF_ITER_DESTROY; @@ -11607,7 +11600,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state * return 0; } - static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg) { struct btf_record *rec = reg_btf_record(reg); @@ -16412,7 +16404,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return 0; } - static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range) { enum bpf_prog_type prog_type = resolve_prog_type(env->prog); @@ -18361,8 +18352,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env) bpf_insn_array_release(env->insn_array_maps[i]); } - - /* The verifier does more data flow analysis than llvm and will not * explore branches that are dead at run time. Malicious programs can * have dead code too. Therefore replace all dead at-run-time code @@ -18390,8 +18379,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env) } } - - static void free_states(struct bpf_verifier_env *env) { struct bpf_verifier_state_list *sl; @@ -18678,7 +18665,6 @@ static int do_check_main(struct bpf_verifier_env *env) return ret; } - static void print_verification_stats(struct bpf_verifier_env *env) { /* Skip over hidden subprogs which are not verified. */ From bed7d65ff499e58c8504a2a3387ee95b551f6951 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:15 +0800 Subject: [PATCH 289/373] bpf: Factor out check_map_mem_read helper in verifier In the next commit, percpu_array map will add map_direct_value_addr support. IOW, it will add a map_type check in the iff condition of the bpf_map_direct_read() code block, which will reduce the code block readability. Hence, factor out check_map_mem_read helper to improve the readability, and the maintainability for the percpu_array map case. Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev --- kernel/bpf/verifier.c | 75 +++++++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 32 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index cdb79a66b156..4fac230122d9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6078,6 +6078,48 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val) reg_bounds_sync(dst_reg); } +static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off, + int bpf_size, int value_regno, bool is_ldsx) +{ + struct bpf_reg_state *regs = cur_regs(env); + int size = bpf_size_to_bytes(bpf_size); + struct bpf_map *map = reg->map_ptr; + + switch (map->map_type) { + case BPF_MAP_TYPE_INSN_ARRAY: + if (bpf_size != BPF_DW) { + verbose(env, "Invalid read of %d bytes from insn_array\n", size); + return -EACCES; + } + regs[value_regno] = *reg; + add_scalar_to_reg(®s[value_regno], off); + regs[value_regno].type = PTR_TO_INSN; + return 0; + default: + break; + } + + /* If map is read-only, track its contents as scalars. */ + if (tnum_is_const(reg->var_off) && + bpf_map_is_rdonly(map) && + map->ops->map_direct_value_addr) { + int map_off = off + reg->var_off.value; + u64 val = 0; + int err; + + err = bpf_map_direct_read(map, map_off, size, &val, is_ldsx); + if (err) + return err; + + regs[value_regno].type = SCALAR_VALUE; + __mark_reg_known(®s[value_regno], val); + return 0; + } + + mark_reg_unknown(env, regs, value_regno); + return 0; +} + /* check whether memory at (regno + off) is accessible for t = (read | write) * if t==write, value_regno is a register which value is stored into memory * if t==read, value_regno is a register which will receive the value from memory @@ -6132,38 +6174,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b if (kptr_field) { err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field); } else if (t == BPF_READ && value_regno >= 0) { - struct bpf_map *map = reg->map_ptr; - - /* - * If map is read-only, track its contents as scalars, - * unless it is an insn array (see the special case below) - */ - if (tnum_is_const(reg->var_off) && - bpf_map_is_rdonly(map) && - map->ops->map_direct_value_addr && - map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { - int map_off = off + reg->var_off.value; - u64 val = 0; - - err = bpf_map_direct_read(map, map_off, size, - &val, is_ldsx); - if (err) - return err; - - regs[value_regno].type = SCALAR_VALUE; - __mark_reg_known(®s[value_regno], val); - } else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { - if (bpf_size != BPF_DW) { - verbose(env, "Invalid read of %d bytes from insn_array\n", - size); - return -EACCES; - } - regs[value_regno] = *reg; - add_scalar_to_reg(®s[value_regno], off); - regs[value_regno].type = PTR_TO_INSN; - } else { - mark_reg_unknown(env, regs, value_regno); - } + err = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx); } } else if (base_type(reg->type) == PTR_TO_MEM) { bool rdonly_mem = type_is_rdonly_mem(reg->type); From 6e61f4f8b04362d03136a4ad8f68adf72127a3af Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:16 +0800 Subject: [PATCH 290/373] bpf: Introduce global percpu data Introduce global percpu data, inspired by the commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the include/linux/percpu-defs.h::DEFINE_PER_CPU() macro. For example, in BPF, it is able to define a global percpu variable like: int data SEC(".percpu"); With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify their BPF code for handling LBRs. The code can be updated from static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs"); to static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs"); This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper. Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps. Links: [1] https://github.com/anakryiko/retsnoop [2] https://github.com/bpfsnoop/bpfsnoop Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev --- kernel/bpf/arraymap.c | 38 ++++++++++++++++++++++++++++++++++++-- kernel/bpf/const_fold.c | 1 - kernel/bpf/fixups.c | 37 +++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 11 +++++++++++ 4 files changed, 84 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 248b4818178c..34865701f7f7 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) return this_cpu_ptr(array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (off >= map->value_size) + return -EINVAL; + + *imm = (u64)(__force unsigned long) array->pptrs[0]; + return 0; +} + +static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + u64 base = (u64)(__force unsigned long) array->pptrs[0]; + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (imm < base || imm >= base + array->elem_size) + return -ENOENT; + + *off = imm - base; + return 0; +} + /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) { @@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map, const struct btf_type *key_type, const struct btf_type *value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map */ + /* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */ if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY || + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) || map->max_entries != 1) return -EINVAL; @@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem, diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c index 4cf120c7b2cb..7f1b30059cc8 100644 --- a/kernel/bpf/const_fold.c +++ b/kernel/bpf/const_fold.c @@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * u64 val = 0; if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr || - map->map_type == BPF_MAP_TYPE_INSN_ARRAY || off < 0 || off + size > map->value_size || bpf_map_direct_read(map, off, size, &val, is_ldsx)) { *dst = unknown; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 0caf1bbd9494..177a3fcbb63a 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1834,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } + if (bpf_jit_supports_percpu_insn() && + insn->code == (BPF_LD | BPF_IMM | BPF_DW) && + (insn->src_reg == BPF_PSEUDO_MAP_VALUE || + insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) { + struct bpf_map *map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; + if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) + goto next_insn; + + prog->jit_required = true; + + /* + * We are *skipping* first half of ld_imm64 insn + * with 'i++;', patching over second half of it + * with that same half + mov64_percpu_reg insn. + * All because bpf_patch_insn_data() can only + * replace one 8-byte insn, which does not work + * well for ld_imm64 insn. + */ + + insn_buf[0] = insn[1]; + insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg); + cnt = 2; + + i++; + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } + if (insn->code != (BPF_JMP | BPF_CALL)) goto next_insn; if (insn->src_reg == BPF_PSEUDO_CALL) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4fac230122d9..6ac1afced20b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5582,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, u64 addr; int err; + if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err) return err; @@ -6095,6 +6097,8 @@ static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state add_scalar_to_reg(®s[value_regno], off); regs[value_regno].type = PTR_TO_INSN; return 0; + case BPF_MAP_TYPE_PERCPU_ARRAY: + goto reg_unknown; default: break; } @@ -6116,6 +6120,7 @@ static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state return 0; } +reg_unknown: mark_reg_unknown(env, regs, value_regno); return 0; } @@ -8129,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env, return -EACCES; } + if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { + verbose(env, "%s points to percpu_array map which cannot be used as const string\n", + reg_arg_name(env, argno)); + return -EACCES; + } + if (!bpf_map_is_rdonly(map)) { verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno)); return -EACCES; From 3634e4f2dc59559c4225ffdf2b2348a7d21f19a1 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:17 +0800 Subject: [PATCH 291/373] libbpf: Probe percpu data feature libbpf needs a reliable way to distinguish kernels that can support global percpu data from those that cannot. Add a dedicated feature probe, so libbpf can make capability decisions early and fail predictably when global percpu data is unavailable. Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Acked-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260813152324.97937-5-leon.hwang@linux.dev --- tools/lib/bpf/features.c | 35 +++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf_internal.h | 2 ++ 2 files changed, 37 insertions(+) diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c index b7e388f99d0b..ef9581c11303 100644 --- a/tools/lib/bpf/features.c +++ b/tools/lib/bpf/features.c @@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd) return probe_sys_bpf_ext(); } +static int probe_kern_percpu_data(int token_fd) +{ + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + int ret, map, insn_cnt = ARRAY_SIZE(insns); + + map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1, + &map_opts); + if (map < 0) { + pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n", + __func__, errstr(map)); + return map; + } + + insns[0].imm = map; + + ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts); + close(map); + return probe_fd(ret); +} + typedef int (*feature_probe_fn)(int /* token_fd */); static struct kern_feature_cache feature_cache; @@ -707,6 +739,9 @@ static struct kern_feature_desc { [FEAT_BPF_SYSCALL_COMMON_ATTRS] = { "BPF syscall common attributes support", probe_bpf_syscall_common_attrs, }, + [FEAT_PERCPU_DATA] = { + "kernel supports percpu data", probe_kern_percpu_data, + }, }; bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id) diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 7a74abb904f8..4c46d34fc055 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -401,6 +401,8 @@ enum kern_feature_id { FEAT_BTF_LAYOUT, /* Kernel supports BPF syscall common attributes */ FEAT_BPF_SYSCALL_COMMON_ATTRS, + /* Kernel supports percpu data */ + FEAT_PERCPU_DATA, __FEAT_CNT, }; From f4e64cb65f5adfc88f2155bfb3cc82b7d20a1d8f Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:18 +0800 Subject: [PATCH 292/373] libbpf: Add support for global percpu data Add support for global percpu data in libbpf by adding a new ".percpu" section, similar to ".data". It enables efficient handling of percpu global variables in bpf programs. When generating loader for lightweight skeleton, update the percpu_array map used for global percpu data using BPF_F_ALL_CPUS, in order to update values across all CPUs using one value slot. Unlike global data, the mmaped data for global percpu data will be marked as read-only after populating the percpu_array map. Thereafter, users can read those initialized percpu data after loading prog. If they want to update the percpu data after loading prog, they have to update the percpu_array map using key=0 instead. Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev --- tools/lib/bpf/bpf_gen_internal.h | 3 +- tools/lib/bpf/gen_loader.c | 3 +- tools/lib/bpf/libbpf.c | 81 +++++++++++++++++++++++++++----- 3 files changed, 72 insertions(+), 15 deletions(-) diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h index 042569187752..6c5ad6c55e8a 100644 --- a/tools/lib/bpf/bpf_gen_internal.h +++ b/tools/lib/bpf/bpf_gen_internal.h @@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen, enum bpf_prog_type prog_type, const char *prog_name, const char *license, struct bpf_insn *insns, size_t insn_cnt, struct bpf_prog_load_opts *load_attr, int prog_idx); -void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size); +void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size, + __u64 flags); void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx); void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type); void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak, diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 6e3dd5242761..af3a04f161ac 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, } void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, - __u32 value_size) + __u32 value_size, __u64 flags) { int attr_size = offsetofend(union bpf_attr, flags); int map_update_attr, value, key; @@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, int zero = 0; memset(&attr, 0, attr_size); + attr.flags = tgt_endian(flags); value = add_data(gen, pvalue, value_size); key = add_data(gen, &zero, sizeof(zero)); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 514e4e9daa82..e574870fb716 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -541,6 +541,7 @@ struct bpf_struct_ops { }; #define DATA_SEC ".data" +#define PERCPU_SEC ".percpu" #define BSS_SEC ".bss" #define RODATA_SEC ".rodata" #define KCONFIG_SEC ".kconfig" @@ -555,6 +556,7 @@ enum libbpf_map_type { LIBBPF_MAP_BSS, LIBBPF_MAP_RODATA, LIBBPF_MAP_KCONFIG, + LIBBPF_MAP_PERCPU, }; struct bpf_map_def { @@ -666,6 +668,7 @@ enum sec_type { SEC_DATA, SEC_RODATA, SEC_ST_OPS, + SEC_PERCPU, }; struct elf_sec_desc { @@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map) switch (map->def.type) { case BPF_MAP_TYPE_ARRAY: return array_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_PERCPU_ARRAY: + return map->def.value_size; case BPF_MAP_TYPE_ARENA: return page_sz * map->def.max_entries; default: @@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz return 0; } -static char *internal_map_name(struct bpf_object *obj, const char *real_name) +static char *internal_map_name(struct bpf_object *obj, const char *real_name, + enum libbpf_map_type type) { char map_name[BPF_OBJ_NAME_LEN], *p; int pfx_len, sfx_len = max((size_t)7, strlen(real_name)); @@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name) if (sfx_len >= BPF_OBJ_NAME_LEN) sfx_len = BPF_OBJ_NAME_LEN - 1; - /* if there are two or more dots in map name, it's a custom dot map */ - if (strchr(real_name + 1, '.') != NULL) + /* + * Don't prefix the bpf_object name if this is a custom dot map + * (containing two or more dots) or a percpu data map. + */ + if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU) pfx_len = 0; else pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name)); @@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) if (!map->btf_value_type_id) return false; + /* + * The internal PERCPU maps are not mmapble because the underlying + * percpu_array maps do not have mmap support. + */ + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return false; + t = btf__type_by_id(obj->btf, map->btf_value_type_id); if (!btf_is_datasec(t)) return false; @@ -1962,6 +1978,7 @@ static int bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, const char *real_name, int sec_idx, void *data, size_t data_sz) { + bool is_percpu = type == LIBBPF_MAP_PERCPU; struct bpf_map_def *def; struct bpf_map *map; size_t mmap_sz; @@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, map->sec_idx = sec_idx; map->sec_offset = 0; map->real_name = strdup(real_name); - map->name = internal_map_name(obj, real_name); + map->name = internal_map_name(obj, real_name, type); if (!map->real_name || !map->name) { zfree(&map->real_name); zfree(&map->name); @@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, } def = &map->def; - def->type = BPF_MAP_TYPE_ARRAY; + def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY; def->key_size = sizeof(int); def->value_size = data_sz; def->max_entries = 1; @@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, if (map_is_mmapable(obj, map)) def->map_flags |= BPF_F_MMAPABLE; - pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", - map->name, map->sec_idx, map->sec_offset, def->map_flags); + pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n", + map->name, is_percpu ? "percpu " : "", map->sec_idx, + map->sec_offset, def->map_flags); mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, @@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj) NULL, sec_desc->data->d_size); break; + case SEC_PERCPU: + sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); + err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU, + sec_name, sec_idx, + sec_desc->data->d_buf, + sec_desc->data->d_size); + break; default: /* skip */ break; @@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj) sec_desc->sec_type = SEC_RODATA; sec_desc->shdr = sh; sec_desc->data = data; + } else if (strcmp(name, PERCPU_SEC) == 0 || + str_has_pfx(name, PERCPU_SEC ".")) { + sec_desc->sec_type = SEC_PERCPU; + sec_desc->shdr = sh; + sec_desc->data = data; } else if (strcmp(name, STRUCT_OPS_SEC) == 0 || strcmp(name, STRUCT_OPS_LINK_SEC) == 0 || strcmp(name, "?" STRUCT_OPS_SEC) == 0 || @@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj, case SEC_BSS: case SEC_DATA: case SEC_RODATA: + case SEC_PERCPU: return true; default: return false; @@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) return LIBBPF_MAP_DATA; case SEC_RODATA: return LIBBPF_MAP_RODATA; + case SEC_PERCPU: + return LIBBPF_MAP_PERCPU; default: return LIBBPF_MAP_UNSPEC; } @@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map) /* * LLVM annotates global data differently in BTF, that is, - * only as '.data', '.bss' or '.rodata'. + * only as '.data', '.bss', '.percpu' or '.rodata'. */ if (!bpf_map__is_internal(map)) return -ENOENT; @@ -5293,18 +5326,20 @@ static int bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) { enum libbpf_map_type map_type = map->libbpf_type; + bool is_percpu = map_type == LIBBPF_MAP_PERCPU; + const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0; int err, zero = 0; size_t mmap_sz; if (obj->gen_loader) { bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps, - map->mmaped, map->def.value_size); + map->mmaped, map->def.value_size, update_flags); if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG) bpf_gen__map_freeze(obj->gen_loader, map - obj->maps); return 0; } - err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0); + err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags); if (err) { err = -errno; pr_warn("map '%s': failed to set initial contents: %s\n", @@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) return err; } map->mmaped = mmaped; + } else if (is_percpu) { + if (mprotect(map->mmaped, mmap_sz, PROT_READ)) { + err = -errno; + pr_warn("map '%s': failed to mprotect() contents: %s\n", + bpf_map__name(map), errstr(err)); + return err; + } } else if (map->mmaped) { munmap(map->mmaped, mmap_sz); map->mmaped = NULL; @@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object *obj) * runtime due to bpf_program__set_autoload(prog, false), * bpf_object loading will succeed just fine even on old * kernels. + * Same skipping applies to percpu data. */ - if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA)) - map->autocreate = false; + if (bpf_map__is_internal(map)) { + bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU; + enum kern_feature_id feat_id; + + feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA; + if (!kernel_supports(obj, feat_id)) + map->autocreate = false; + } if (!map->autocreate) { pr_debug("map '%s': skipped auto-creating...\n", map->name); @@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map *map) * such map's corresponding ELF section name as a map name. * This check distinguishes .data/.rodata from .data.* and .rodata.* * maps to know which name has to be returned to the user. + * Map name of the custom .percpu.* maps might be truncated to + * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data + * maps must use real name for their user-visible name. */ if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0) return true; if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0) return true; + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return true; return false; } @@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size) size_t mmap_old_sz, mmap_new_sz; int err; - if (map->def.type != BPF_MAP_TYPE_ARRAY) + if (map->def.type != BPF_MAP_TYPE_ARRAY && + map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY) return libbpf_err(-EOPNOTSUPP); mmap_old_sz = bpf_map_mmap_sz(map); From 68d4fde73cf78a7ab858ed040f54e1851cc3815c Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:19 +0800 Subject: [PATCH 293/373] bpftool: Generate skeleton for global percpu data Enhance bpftool to generate skeletons that properly handle global percpu variables. The generated skeleton now includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables more efficiently. For global percpu variables, the skeleton now includes a nested structure, e.g.: struct test_global_percpu_data { struct bpf_object_skeleton *skeleton; struct bpf_object *obj; struct { struct bpf_map *percpu; } maps; // ... struct test_global_percpu_data__percpu { int data; char run; struct { char set; int i; int nums[7]; } struct_data; int nums[7]; } *percpu; // ... }; * The "struct test_global_percpu_data__percpu *percpu" points to initialized data, which is actually "maps.percpu->mmaped". * Before loading the skeleton, updating the "struct test_global_percpu_data__percpu *percpu" modifies the initial value of the corresponding global percpu variables. * After loading the skeleton, "maps.percpu->mmaped" has been marked as read-only in libbpf. If users want to update the global percpu variables, they have to update the "maps.percpu" map instead. * For lightweight skeleton, "lskel->percpu" will be protected by "mprotect(p, sz, PROT_READ)". * For subskeleton, those variables of global percpu data will be skipped. Assisted-by: Codex:gpt-5.5-xhigh Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Acked-by: Quentin Monnet Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev --- tools/bpf/bpftool/gen.c | 53 +++++++++++++++++++++++++++-------- tools/lib/bpf/skel_internal.h | 24 ++++++++++++++-- 2 files changed, 63 insertions(+), 14 deletions(-) diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index a01d06d22d1a..a50540ef6521 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz) return true; } + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + snprintf(buf, buf_sz, "%s", name + 1); + sanitize_identifier(buf); + return true; + } + for (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) { const char *sfx = sfxs[i], *p; @@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz) static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz) { - static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; + static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" }; int i, n; /* recognize hard coded LLVM section name */ @@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map return NULL; } -static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) +static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz) { size_t tmp_sz; @@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) return true; } - if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (!bpf_map__is_internal(map)) return false; if (!get_map_ident(map, buf, sz)) return false; - return true; + if (bpf_map__map_flags(map) & BPF_F_MMAPABLE) + return true; + + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) + return bpf_map__btf_value_type_id(map) != 0; + + return false; +} + +static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) +{ + return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY; } static int codegen_datasecs(struct bpf_object *obj, const char *obj_name) @@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name) bpf_object__for_each_map(map, obj) { /* only generate definitions for memory-mapped internal maps */ - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name) ", obj_name); bpf_object__for_each_map(map, obj) { - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name) bpf_object__for_each_map(map, obj) { if (!get_map_ident(map, ident, sizeof(ident))) continue; - if (bpf_map__is_internal(map) && - (bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (is_skel_data(map, ident, sizeof(ident))) printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n", ident, bpf_map_mmap_sz(map)); codegen("\ @@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h const void *mmap_data = NULL; size_t mmap_size = 0; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; codegen("\ @@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h bpf_object__for_each_map(map, obj) { const char *mmap_flags; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + codegen("\ + \n\ + err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\ + if (err) \n\ + return err; \n\ + #ifdef __KERNEL__ \n\ + skel->%1$s = NULL; \n\ + #endif \n\ + ", + ident, bpf_map_mmap_sz(map)); + continue; + } + if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG) mmap_flags = "PROT_READ"; else @@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool map->map = &obj->maps.%s; \n\ ", i, bpf_map__name(map), ident); - /* memory-mapped internal maps */ - if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) { + if (mmaped && is_skel_data(map, ident, sizeof(ident))) { printf("\tmap->mmaped = (void **)&obj->%s;\n", ident); } diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index 53fee53d36d5..1f3f332dffbe 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz) { if (addr != ~0ULL) kvfree(p); - /* When addr == ~0ULL the 'p' points to - * ((struct bpf_array *)map)->value. See skel_finalize_map_data. + /* + * When addr == ~0ULL the init buffer has already been released. + * For skel_finalize_map_data(), 'p' points to + * ((struct bpf_array *)map)->value. */ } @@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int return addr; } +static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz) +{ + (void)sz; + + kvfree(p); + *init_val = ~0ULL; + return 0; +} + #else static inline void *skel_alloc(size_t size) @@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int return NULL; return addr; } + +static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz) +{ + (void)init_val; + + if (mprotect(p, sz, PROT_READ)) + return -errno; + return 0; +} #endif static inline int skel_closenz(int fd) From 4c9241bd731a0205cee2c4a4fa0abe866c767e2f Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:20 +0800 Subject: [PATCH 294/373] selftests/bpf: Add tests to verify global percpu data If the arch, like s390x, does not support percpu insn, these cases won't test global percpu data by checking FEAT_PERCPU_DATA support. The following APIs have been tested for global percpu data: 1. bpf_map__set_initial_value() 2. bpf_map__initial_value() 3. bpf_map__set_value_size() 4. generated percpu struct pointer pointing to internal map's mmaped data 5. bpf_map__lookup_elem() for global percpu data map 6. bpf_map_lookup_elem_flags() for global percpu data map At the same time, the case is also tested with 'bpftool gen skeleton -L'. Assisted-by: Codex:gpt-5.5-xhigh Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260813152324.97937-8-leon.hwang@linux.dev --- tools/testing/selftests/bpf/Makefile | 2 +- .../bpf/prog_tests/global_data_init.c | 188 ++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 41 ++++ 3 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index d3655a706482..560ce4016fbf 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c # Generate both light skeleton and libbpf skeleton for these LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \ - kfunc_call_test_subprog.c + kfunc_call_test_subprog.c test_global_percpu_data.c SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED) test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 8466332d7406..06d163a022dc 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -1,5 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include "bpf/libbpf_internal.h" +#include "test_global_percpu_data.skel.h" +#include "test_global_percpu_data.lskel.h" void test_global_data_init(void) { @@ -60,3 +63,188 @@ void test_global_data_init(void) free(newval); bpf_object__close(obj); } + +static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_fd, int *runp) +{ + struct test_global_percpu_data__percpu *data = NULL; + int i, err, key = 0, num_online, run = 0; + __u64 args[2] = {0x1234ULL, 0x5678ULL}; + size_t data_sz; + bool *online; + LIBBPF_OPTS(bpf_test_run_opts, topts, + .ctx_in = args, + .ctx_size_in = sizeof(args), + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + + err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online); + if (!ASSERT_OK(err, "parse_cpu_mask_file")) + return; + + data_sz = map ? bpf_map__value_size(map) : sizeof(*data); + data = calloc(1, data_sz); + if (!ASSERT_OK_PTR(data, "calloc percpu data")) + goto out; + + /* run on every online-CPU */ + for (i = 0; i < num_online; i++) { + __u64 flags; + + if (!online[i]) + continue; + + topts.cpu = i; + topts.retval = -1; + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval"); + + memset(data, 0, data_sz); + flags = ((__u64) i << 32) | BPF_F_CPU; + if (map) + err = bpf_map__lookup_elem(map, &key, sizeof(key), data, data_sz, flags); + else + err = bpf_map_lookup_elem_flags(map_fd, &key, data, flags); + if (!ASSERT_OK(err, "lookup_elem on cpu")) + break; + + ASSERT_EQ(*runp, ++run, "run"); + ASSERT_EQ(data->cpu_id[0], i, "cpu_id"); + ASSERT_EQ(data->data, 1, "data"); + ASSERT_TRUE(data->set, "set"); + ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]"); + ASSERT_EQ(data->struct_data.i, 1, "struct_data.i"); + ASSERT_TRUE(data->struct_data.set, "struct_data.set"); + ASSERT_EQ(data->struct_data.nums[6], 0xc0de, "struct_data.nums[6]"); + } + +out: + free(data); + free(online); +} + +static void test_global_percpu_data_init(void) +{ + struct test_global_percpu_data__percpu init_value = {}; + struct test_global_percpu_data__percpu *init_data; + const __u32 desired_sz = sysconf(_SC_PAGE_SIZE); + struct test_global_percpu_data *skel = NULL; + size_t init_data_sz; + struct bpf_map *map; + int prog_fd, err; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->data_percpu, "skel->data_percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong")) + goto out; + + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data", + ".percpu.data map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.data_percpu), ".data.percpu", + ".data.percpu map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong", + "long map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name"); + ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data"); + ASSERT_FALSE(skel->percpu->set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]"); + ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i"); + ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]"); + + map = skel->maps.percpu; + if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type")) + goto out; + + init_value.data = 2; + init_value.nums[6] = -1; + init_value.struct_data.i = 2; + init_value.struct_data.nums[6] = -1; + err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value)); + if (!ASSERT_OK(err, "bpf_map__set_initial_value")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value")) + goto out; + + ASSERT_EQ(init_data->data, init_value.data, "init_value data"); + ASSERT_EQ(init_data->set, init_value.set, "init_value set"); + ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i"); + ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6], + "init_value struct_data.nums[6]"); + ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size"); + ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu eq init_data"); + ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data"); + ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i, + "skel->percpu->struct_data.i"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6], + "skel->percpu->struct_data.nums[6]"); + + ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz"); + err = bpf_map__set_value_size(map, desired_sz); + if (!ASSERT_OK(err, "bpf_map__set_value_size")) + goto out; + if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size")) + goto out; + if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value")) + goto out; + if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size")) + goto out; + if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data")) + goto out; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + ASSERT_OK_PTR(skel->percpu, "skel->percpu"); + + prog_fd = bpf_program__fd(skel->progs.update_percpu_data); + test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run); + +out: + test_global_percpu_data__destroy(skel); +} + +static void test_global_percpu_data_lskel(void) +{ + struct test_global_percpu_data_lskel *lskel = NULL; + int prog_fd, map_fd; + + lskel = test_global_percpu_data_lskel__open_and_load(); + if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load")) + goto out; + + map_fd = lskel->maps.percpu.map_fd; + prog_fd = lskel->progs.update_percpu_data.prog_fd; + test_percpu_data_on_cpus(NULL, map_fd, prog_fd, &lskel->bss->run); + +out: + test_global_percpu_data_lskel__destroy(lskel); +} + +void test_global_percpu_data(void) +{ + if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { + test__skip(); + return; + } + + if (test__start_subtest("init")) + test_global_percpu_data_init(); + if (test__start_subtest("lskel")) + test_global_percpu_data_lskel(); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file mode 100644 index 000000000000..416841cd3569 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "bpf_misc.h" + +/* Used for testing map name. */ +int loong SEC(".percpu.looooooooong"); +int data3 SEC(".data.percpu"); +int data2 SEC(".percpu.data"); + +int run; +/* cpu_id as array to verify map value resizing. */ +int cpu_id[1] SEC(".percpu"); +int data SEC(".percpu") = -1; +int nums[7] SEC(".percpu"); +bool set SEC(".percpu") = false; +struct { + char set; + int i; + int nums[7]; +} struct_data SEC(".percpu") = { + .set = 0, + .i = -1, +}; + +SEC("raw_tp/task_rename") +__auxiliary +int update_percpu_data(void *ctx) +{ + struct_data.nums[6] = 0xc0de; + struct_data.set = 1; + struct_data.i = 1; + nums[6] = 0xc0de; + data = 1; + run++; + set = true; + cpu_id[0] = bpf_get_smp_processor_id(); + return 0; +} + +char _license[] SEC("license") = "GPL"; From 3993d5beef61ca9234c5aa11adff552b8b9f133f Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:21 +0800 Subject: [PATCH 295/373] selftests/bpf: Test direct reading/writing read-only percpu_array map Verify these two cases: 1. Direct reading the data of read-only percpu data's percpu_array map is allowed. 2. Direct writing the data of read-only percpu data's percpu_array map is disallowed. Assisted-by: Codex:gpt-5.5-xhigh Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260813152324.97937-9-leon.hwang@linux.dev --- .../bpf/prog_tests/global_data_init.c | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 06d163a022dc..9688b417a90b 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -236,6 +236,92 @@ static void test_global_percpu_data_lskel(void) test_global_percpu_data_lskel__destroy(lskel); } +static int create_rdonly_percpu_array(void) +{ + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .map_flags = BPF_F_RDONLY_PROG, + ); + int key = 0, map_fd, err; + __u64 value = 0; + + map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int), + sizeof(__u64), 1, &map_opts); + if (!ASSERT_GE(map_fd, 0, "bpf_map_create")) + return -1; + + err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS); + if (!ASSERT_OK(err, "bpf_map_update_elem")) + goto out; + + err = bpf_map_freeze(map_fd); + if (!ASSERT_OK(err, "bpf_map_freeze")) + goto out; + + return map_fd; + +out: + close(map_fd); + return -1; +} + +static void test_global_percpu_data_rdonly_direct_read(void) +{ + /* + * Raw instructions with manually prepared rdonly percpu_array map + * for testing direct-read global percpu data, because libbpf + * doesn't have rdonly internal percpu_array map support for + * global percpu data. + */ + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + int map_fd, prog_fd; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), NULL); + if (ASSERT_GE(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + close(map_fd); +} + +static void test_global_percpu_data_rdonly_direct_write(void) +{ + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts); + /* See the comment in test_global_percpu_data_rdonly_direct_read() */ + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0), + BPF_EXIT_INSN(), + }; + char log_buf[256] = {}; + int map_fd, prog_fd; + + prog_opts.log_buf = log_buf; + prog_opts.log_size = sizeof(log_buf); + prog_opts.log_level = 1; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), &prog_opts); + if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + else + ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log"); + close(map_fd); +} + void test_global_percpu_data(void) { if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { @@ -247,4 +333,8 @@ void test_global_percpu_data(void) test_global_percpu_data_init(); if (test__start_subtest("lskel")) test_global_percpu_data_lskel(); + if (test__start_subtest("rdonly_direct_read")) + test_global_percpu_data_rdonly_direct_read(); + if (test__start_subtest("rdonly_direct_write")) + test_global_percpu_data_rdonly_direct_write(); } From 1ed2294b31fc9c403ee2bfbd5920d1b03de969e6 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:22 +0800 Subject: [PATCH 296/373] selftests/bpf: Test verifier log for global percpu data Add two tests to verify the verifier log "R%d points to percpu_array map which cannot be used as const string\n". Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260813152324.97937-10-leon.hwang@linux.dev --- .../bpf/prog_tests/global_data_init.c | 6 +++++ .../bpf/progs/test_global_percpu_data.c | 23 +++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 9688b417a90b..d308ca3b3045 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -322,6 +322,11 @@ static void test_global_percpu_data_rdonly_direct_write(void) close(map_fd); } +static void test_global_percpu_data_verifier_log(void) +{ + RUN_TESTS(test_global_percpu_data); +} + void test_global_percpu_data(void) { if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { @@ -337,4 +342,5 @@ void test_global_percpu_data(void) test_global_percpu_data_rdonly_direct_read(); if (test__start_subtest("rdonly_direct_write")) test_global_percpu_data_rdonly_direct_write(); + test_global_percpu_data_verifier_log(); } diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c index 416841cd3569..d086e9417f9f 100644 --- a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -38,4 +38,27 @@ int update_percpu_data(void *ctx) return 0; } +static const char fmt[] SEC(".percpu.fmt") = "data %d\n"; + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_strncmp(void *ctx) +{ + return bpf_strncmp("test", 5, fmt); +} + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_snprintf(void *ctx) +{ + u64 args[] = { data }; + char buf[128]; + int len; + + len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args)); + if (len > 0) + bpf_printk("snprintf: %s\n", buf); + return 0; +} + char _license[] SEC("license") = "GPL"; From 15945d02c6e1b54106b3b2fa4275dbfd73167973 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Thu, 13 Aug 2026 23:23:23 +0800 Subject: [PATCH 297/373] selftests/bpf: Verify bpf_iter for global percpu data Add a test to verify that it is OK to iter the percpu_array map used for global percpu data. Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260813152324.97937-11-leon.hwang@linux.dev --- .../bpf/prog_tests/global_data_init.c | 52 +++++++++++++++++++ .../bpf/progs/test_global_percpu_data.c | 25 +++++++++ 2 files changed, 77 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index d308ca3b3045..7d6bda909295 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -327,6 +327,56 @@ static void test_global_percpu_data_verifier_log(void) RUN_TESTS(test_global_percpu_data); } +static void test_global_percpu_data_iter(void) +{ + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + struct test_global_percpu_data *skel; + union bpf_iter_link_info linfo = {}; + struct bpf_link *link = NULL; + int fd, num_cpus, len, err; + char buf[16]; + + num_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus")) + return; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + return; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data); + skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int); + skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8); + skel->percpu->struct_data.nums[6] = 0xc0de; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + linfo.map.map_fd = bpf_map__fd(skel->maps.percpu); + opts.link_info = &linfo; + opts.link_info_len = sizeof(linfo); + link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto out; + + fd = bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_GE(fd, 0, "bpf_iter_create")) + goto out; + + while ((len = read(fd, buf, sizeof(buf))) > 0) + do { } while (0); + ASSERT_EQ(len, 0, "read iter"); + ASSERT_TRUE(skel->bss->run_iter, "run_iter"); + ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum"); + + close(fd); +out: + bpf_link__destroy(link); + test_global_percpu_data__destroy(skel); +} + void test_global_percpu_data(void) { if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { @@ -343,4 +393,6 @@ void test_global_percpu_data(void) if (test__start_subtest("rdonly_direct_write")) test_global_percpu_data_rdonly_direct_write(); test_global_percpu_data_verifier_log(); + if (test__start_subtest("iter")) + test_global_percpu_data_iter(); } diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c index d086e9417f9f..71ff8d1bf49e 100644 --- a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -61,4 +61,29 @@ int verifier_snprintf(void *ctx) return 0; } +volatile const __u32 num_cpus = 0; +volatile const int offsetof_num; +volatile const int elem_sz; +__u32 percpu_data_sum = 0; +bool run_iter = false; + +SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx) +{ + void *pptr = ctx->value; + int i; + + if (!pptr) + return 0; + + run_iter = true; + + for (i = 0; i < num_cpus; i++) { + percpu_data_sum += *(int *) (pptr + offsetof_num); + pptr += elem_sz; + } + return 0; +} + char _license[] SEC("license") = "GPL"; From aacd13e1eb68f2c9049fc0cf7aed89694c3e0713 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 13 Aug 2026 01:15:05 +0200 Subject: [PATCH 298/373] bpf: Fix func_info_aux desync after dead code elimination The verifier keeps per-subprogram metadata in three parallel arrays: subprog_info, func_info, and func_info_aux. Dead code elimination can remove whole subprograms, and adjust_subprog_starts_after_remove() shifts subprog_info and func_info to close the gap, but leaves func_info_aux in place. From that point on, func_info_aux[i] no longer describes subprogram i. Shift func_info_aux together with func_info so the three arrays stay aligned after subprogram removal. Reported-by: Sashiko Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260808064523.DE3E71F000E9@smtp.kernel.org Link: https://lore.kernel.org/bpf/20260812231506.3558128-1-memxor@gmail.com --- kernel/bpf/fixups.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 177a3fcbb63a..70f22eb63ed5 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -402,13 +402,17 @@ static int adjust_subprog_starts_after_remove(struct bpf_verifier_env *env, sizeof(*env->subprog_info) * move); env->subprog_cnt -= j - i; - /* remove func_info */ + /* remove func_info and its aux */ if (aux->func_info) { move = aux->func_info_cnt - j; memmove(aux->func_info + i, aux->func_info + j, sizeof(*aux->func_info) * move); + if (aux->func_info_aux) + memmove(aux->func_info_aux + i, + aux->func_info_aux + j, + sizeof(*aux->func_info_aux) * move); aux->func_info_cnt -= j - i; /* func_info->insn_off is set after all code rewrites, * in adjust_btf_func() - no need to adjust From 259d60f5bfa41056fe01cbf2ba3f6f0331865a16 Mon Sep 17 00:00:00 2001 From: Yuan Chen Date: Mon, 10 Aug 2026 22:22:22 +0800 Subject: [PATCH 299/373] bpftool: Fix double close in map dump map_dump() closes the map fd in its error path, and do_dump() then closes the same fd again after a successful dump. Closing an already closed fd leaves errno set to EBADF, which poisons later errno checks such as the batch file read check in do_batch(). Let do_dump() own the fd and remove the close from map_dump(). The same double-close pattern exists in do_show_subset(): both show_map_close_json() and show_map_close_plain() already close the fd, so drop the extra close() there as well. Also propagate the error when bpf_map_get_info_by_fd() fails on a subsequent map in do_dump(): set err = -1 before breaking out of the loop, so a later failure is not silently hidden after an earlier iteration succeeded. Fixes: 99f9863a0c45f ("bpftool: Match maps by name") Signed-off-by: Yuan Chen Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260810142224.2907373-2-chenyuan_fl@163.com --- tools/bpf/bpftool/map.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index 6b9649294ca1..684a8fb72414 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -659,8 +659,6 @@ static int do_show_subset(int argc, char **argv) show_map_close_json(fds[i], &info); else show_map_close_plain(fds[i], &info); - - close(fds[i]); } if (json_output && nb_fds > 1) jsonw_end_array(json_wtr); /* root array */ @@ -895,7 +893,6 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr, exit_free: free(key); free(value); - close(fd); free_map_kv_btf(btf); return err; @@ -944,6 +941,7 @@ static int do_dump(int argc, char **argv) for (i = 0; i < nb_fds; i++) { if (bpf_map_get_info_by_fd(fds[i], &info, &len)) { p_err("can't get map info: %s", strerror(errno)); + err = -1; break; } err = map_dump(fds[i], &info, wtr, nb_fds > 1); From 98d309ec8189fd91698d1a72946c3d888270c57e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Tue, 11 Aug 2026 23:05:42 -0700 Subject: [PATCH 300/373] selftests/bpf: Fix for veristat file/prog filters processing At the moment veristat filtering behaves unexpectedly for the following filter expression: -f !file/prog The expression rejects all programs with name 'prog', and all programs in a file with name 'file'. This commit fixes the expression to exclude only a program 'prog' from a file 'file'. Additionally, the commit makes empty filters like '-f ""' or '-f "/"' and error. Here is the filtering behaviour compared old versus new: | filter | file | prog | old verdict | new verdict | |----------+------+------+-------------+-------------| | !foo | foo | bar | skipped | skipped | | !foo | bar | foo | skipped | skipped | | !foo | bar | bar | processed | processed | | !foo/bar | foo | bar | skipped | skipped | | !foo/bar | foo | buz | skipped | processed | (!) | !foo/bar | bar | bar | skipped | processed | (!) | !foo/ | foo | bar | skipped | skipped | | !foo/ | bar | bar | processed | processed | | !/bar | foo | bar | skipped | skipped | | !/bar | foo | foo | processed | processed | | !/ | foo | bar | processed | error | (!) | ! | foo | bar | processed | error | (!) |----------+------+------+-------------+-------------| | foo | foo | bar | processed | processed | | foo | bar | foo | processed | processed | | foo | bar | bar | skipped | skipped | | foo/bar | foo | bar | processed | processed | | foo/bar | foo | buz | skipped | skipped | | foo/bar | bar | bar | skipped | skipped | | foo/ | foo | bar | processed | processed | | foo/ | bar | bar | skipped | skipped | | /bar | foo | bar | processed | processed | | /bar | foo | foo | skipped | skipped | | / | foo | bar | processed | error | (!) | | foo | bar | skipped | error | (!) Fixes: 10b1b3f3e56a ("selftests/bpf: consolidate and improve file/prog filtering in veristat") Signed-off-by: Eduard Zingerman Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v2-1-6c234c4cd6ef@gmail.com --- tools/testing/selftests/bpf/veristat.c | 76 +++++++++++++++++--------- 1 file changed, 49 insertions(+), 27 deletions(-) diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c index 5c0ec3edce72..e70741c6b9b7 100644 --- a/tools/testing/selftests/bpf/veristat.c +++ b/tools/testing/selftests/bpf/veristat.c @@ -514,6 +514,40 @@ static bool is_bpf_obj_file(const char *path) { return err == 0; } +/* Exact filter match */ +static bool name_filter_matches(struct filter *f, const char *filename, const char *prog_name) +{ + if (f->any_glob) + return glob_matches(filename, f->any_glob) || + (prog_name && glob_matches(prog_name, f->any_glob)); + if (f->file_glob && f->prog_glob) + return prog_name && + glob_matches(filename, f->file_glob) && + glob_matches(prog_name, f->prog_glob); + if (f->file_glob) + return glob_matches(filename, f->file_glob); + if (f->prog_glob) + return prog_name && glob_matches(prog_name, f->prog_glob); + return false; +} + +/* Check if the filter does not outright reject the file name */ +static bool name_filter_may_match(struct filter *f, const char *filename) +{ + if (f->file_glob) + return glob_matches(filename, f->file_glob); + /* + * If we don't know program name yet, any_glob filter + * has to assume that current BPF object file might be + * relevant; we'll check again later on after opening + * BPF object file, at which point program name will + * be known finally. + */ + if (f->any_glob || f->prog_glob) + return true; + return false; +} + static bool should_process_file_prog(const char *filename, const char *prog_name) { struct filter *f; @@ -521,16 +555,7 @@ static bool should_process_file_prog(const char *filename, const char *prog_name for (i = 0; i < env.deny_filter_cnt; i++) { f = &env.deny_filters[i]; - if (f->kind != FILTER_NAME) - continue; - - if (f->any_glob && glob_matches(filename, f->any_glob)) - return false; - if (f->any_glob && prog_name && glob_matches(prog_name, f->any_glob)) - return false; - if (f->file_glob && glob_matches(filename, f->file_glob)) - return false; - if (f->prog_glob && prog_name && glob_matches(prog_name, f->prog_glob)) + if (f->kind == FILTER_NAME && name_filter_matches(f, filename, prog_name)) return false; } @@ -540,24 +565,15 @@ static bool should_process_file_prog(const char *filename, const char *prog_name continue; allow_cnt++; - if (f->any_glob) { - if (glob_matches(filename, f->any_glob)) - return true; - /* If we don't know program name yet, any_glob filter - * has to assume that current BPF object file might be - * relevant; we'll check again later on after opening - * BPF object file, at which point program name will - * be known finally. - */ - if (!prog_name || glob_matches(prog_name, f->any_glob)) - return true; - } else { - if (f->file_glob && !glob_matches(filename, f->file_glob)) - continue; - if (f->prog_glob && prog_name && !glob_matches(prog_name, f->prog_glob)) - continue; + if (prog_name && name_filter_matches(f, filename, prog_name)) + return true; + /* + * If there is no prog_name and the file name is not blocked by + * the filter, allow to open the file. Afterwards there would be + * a second refining query with prog_name set. + */ + if (!prog_name && name_filter_may_match(f, filename)) return true; - } } /* if there are no file/prog name allow filters, allow all progs, @@ -703,6 +719,12 @@ static int append_filter(struct filter **filters, int *cnt, const char *str) } } + if ((!f->any_glob && !f->file_glob && !f->prog_glob) || + (f->any_glob && strcmp(f->any_glob, "") == 0)) { + fprintf(stderr, "Invalid filter: '%s'\n", str); + return -EINVAL; + } + *cnt += 1; return 0; } From 6c034d962bdf8e33e21f117a9f8d514ba773f744 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Tue, 11 Aug 2026 23:05:43 -0700 Subject: [PATCH 301/373] selftests/bpf: Exercise veristat filtering logic in a selftest Test cases for veristat file/prog name filtering logic. Check various formulations for any (*foo*), file (*foo*/), prog (/bar) and file/prog (*foo*/bar) filters, alongside erroneous filters and mixed allow/deny filter expressions. Signed-off-by: Eduard Zingerman Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v2-2-6c234c4cd6ef@gmail.com --- .../selftests/bpf/prog_tests/test_veristat.c | 101 ++++++++++++++++++ .../selftests/bpf/progs/veristat_bar.c | 3 + .../selftests/bpf/progs/veristat_foo.c | 31 ++++++ 3 files changed, 135 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/veristat_bar.c create mode 100644 tools/testing/selftests/bpf/progs/veristat_foo.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_veristat.c b/tools/testing/selftests/bpf/prog_tests/test_veristat.c index 9aff08ac55c0..4cd41080eed3 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_veristat.c +++ b/tools/testing/selftests/bpf/prog_tests/test_veristat.c @@ -37,6 +37,14 @@ static struct fixture *init_fixture(void) return fix; } +static void read_output(struct fixture *fix) +{ + ssize_t len = pread(fix->fd, fix->output, fix->sz - 1, 0); + + fix->output[len < 0 ? 0 : len] = 0; + ASSERT_GE(len, 0, "pread"); +} + static void teardown_fixture(struct fixture *fix) { free(fix->output); @@ -230,6 +238,97 @@ static void test_no_array_index_for_array(void) teardown_fixture(fix); } +/* + * Name filter tests below run veristat on veristat_foo.bpf.o and + * veristat_bar.bpf.o, both defining programs 'foo', 'bar' and 'buz'. + * Every entry describes a single (filters, file, prog) combination and + * tells whether that program is expected in the veristat output: + * 'true' if it is, 'false' if it is not and -1 if veristat is expected + * to reject the filter. + */ +#define FILTER_OBJS "veristat_foo.bpf.o veristat_bar.bpf.o" + +static const struct name_filter_case { + const char *filters; + const char *file; + const char *prog; + int included; +} name_filter_cases[] = { + /* no filters, every program is processed */ + { "", "foo", "foo", true }, + { "", "foo", "bar", true }, + { "", "foo", "buz", true }, + { "", "bar", "foo", true }, + { "", "bar", "bar", true }, + { "", "bar", "buz", true }, + /* deny filters */ + { "-f '!*foo*'", "foo", "bar", false }, + { "-f '!*foo*'", "bar", "foo", false }, + { "-f '!*foo*'", "bar", "bar", true }, + { "-f '!*foo*/bar'", "foo", "bar", false }, + { "-f '!*foo*/bar'", "foo", "buz", true }, + { "-f '!*foo*/bar'", "bar", "bar", true }, + { "-f '!*foo*/'", "foo", "bar", false }, + { "-f '!*foo*/'", "bar", "bar", true }, + { "-f '!/bar'", "foo", "bar", false }, + { "-f '!/bar'", "foo", "foo", true }, + { "-f '!/'", "foo", "bar", -1 }, + { "-f '!'", "foo", "bar", -1 }, + /* allow filters */ + { "-f '*foo*'", "foo", "bar", true }, + { "-f '*foo*'", "bar", "foo", true }, + { "-f '*foo*'", "bar", "bar", false }, + { "-f '*foo*/bar'", "foo", "bar", true }, + { "-f '*foo*/bar'", "foo", "buz", false }, + { "-f '*foo*/bar'", "bar", "bar", false }, + { "-f '*foo*/'", "foo", "bar", true }, + { "-f '*foo*/'", "bar", "bar", false }, + { "-f '/bar'", "foo", "bar", true }, + { "-f '/bar'", "foo", "foo", false }, + { "-f '/'", "foo", "bar", -1 }, + { "-f ''", "foo", "bar", -1 }, + /* allow and deny filters combined */ + { "-f '*foo*/' -f '!/bar'", "foo", "foo", true }, + { "-f '*foo*/' -f '!/bar'", "foo", "bar", false }, + { "-f '*foo*/' -f '!/bar'", "bar", "foo", false }, +}; + +static void test_name_filters(void) +{ + struct fixture *fix = init_fixture(); + const struct name_filter_case *t; + char cmd[512], row[64], name[128]; + int i, err; + + for (i = 0; i < ARRAY_SIZE(name_filter_cases); i++) { + t = &name_filter_cases[i]; + /* stderr is merged with stdout in order to catch error messages */ + snprintf(cmd, sizeof(cmd), "%s " FILTER_OBJS " -q -o csv -e file,prog %s > %s 2>&1", + fix->veristat, t->filters, fix->tmpfile); + err = system(cmd); + read_output(fix); + + snprintf(row, sizeof(row), "veristat_%s.bpf.o,%s", t->file, t->prog); + snprintf(name, sizeof(name), "veristat %s: %s", t->filters, row); + switch (t->included) { + case true: + ASSERT_OK(err, name); + ASSERT_HAS_SUBSTR(fix->output, row, name); + break; + case false: + ASSERT_OK(err, name); + ASSERT_FALSE(!!strstr(fix->output, row), name); + break; + case -1: + ASSERT_NEQ(err, 0, name); + ASSERT_HAS_SUBSTR(fix->output, "Invalid filter", name); + break; + } + } + + teardown_fixture(fix); +} + void test_veristat(void) { if (test__start_subtest("set_global_vars_succeeds")) @@ -256,6 +355,8 @@ void test_veristat(void) if (test__start_subtest("test_no_array_index_for_array")) test_no_array_index_for_array(); + if (test__start_subtest("name_filters")) + test_name_filters(); } #undef __CHECK_STR diff --git a/tools/testing/selftests/bpf/progs/veristat_bar.c b/tools/testing/selftests/bpf/progs/veristat_bar.c new file mode 100644 index 000000000000..83d2a2a1dfc9 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/veristat_bar.c @@ -0,0 +1,3 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include "veristat_foo.c" diff --git a/tools/testing/selftests/bpf/progs/veristat_foo.c b/tools/testing/selftests/bpf/progs/veristat_foo.c new file mode 100644 index 000000000000..bd24b97664b4 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/veristat_foo.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ +#include +#include + +/* + * Programs below exist only to exercise veristat's -f name filters, + * their bodies are irrelevant, only the names matter. + * This file is also included by veristat_bar.c, so that the same set of + * program names is available in two differently named object files. + */ + +SEC("socket") +int foo(void *ctx) +{ + return 0; +} + +SEC("socket") +int bar(void *ctx) +{ + return 0; +} + +SEC("socket") +int buz(void *ctx) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; From 033506fc06674ecd57f219de5c29f00c06367a36 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Tue, 11 Aug 2026 23:05:44 -0700 Subject: [PATCH 302/373] selftests/bpf: Guarantee zero termination for veristat test buffers In veristat tests replace direct read() calls with calls to read_output() utility function, which: - guarantees that the input buffer is zero terminated; - asserts that read operation succeeded. Signed-off-by: Eduard Zingerman Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v2-3-6c234c4cd6ef@gmail.com --- .../testing/selftests/bpf/prog_tests/test_veristat.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/test_veristat.c b/tools/testing/selftests/bpf/prog_tests/test_veristat.c index 4cd41080eed3..11f3de2b66ad 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_veristat.c +++ b/tools/testing/selftests/bpf/prog_tests/test_veristat.c @@ -82,7 +82,7 @@ static void test_set_global_vars_succeeds(void) " -G \"struct11 [ 7 ] [ 5 ] .struct2[0][1].u.mat[3][0] = 175\" " \ " -vl2 > %s", fix->veristat, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("=0xf000000000000001 ", "var_s64 = 0xf000000000000001"); __CHECK_STR("=0xfedcba9876543210 ", "var_u64 = 0xfedcba9876543210"); __CHECK_STR("=0x80000000 ", "var_s32 = -0x80000000"); @@ -124,7 +124,7 @@ static void test_set_global_vars_from_file_succeeds(void) syncfs(fd); SYS(out, "%s set_global_vars.bpf.o -G \"@%s\" -vl2 > %s", fix->veristat, input_file, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("=0x8000 ", "var_s16 = -32768"); __CHECK_STR("=0xecec ", "var_u16 = 60652"); @@ -142,7 +142,7 @@ static void test_set_global_vars_out_of_range(void) "%s set_global_vars.bpf.o -G \"var_s32 = 2147483648\" -vl2 2> %s", fix->veristat, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("is out of range [-2147483648; 2147483647]", "out of range"); out: @@ -157,7 +157,7 @@ static void test_unsupported_ptr_array_type(void) "%s set_global_vars.bpf.o -G \"ptr_arr[0] = 0\" -vl2 2> %s", fix->veristat, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("Can't set ptr_arr[0]. Only ints and enums are supported", "ptr_arr"); out: @@ -172,7 +172,7 @@ static void test_array_out_of_bounds(void) "%s set_global_vars.bpf.o -G \"arr[99] = 0\" -vl2 2> %s", fix->veristat, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("Array index 99 is out of bounds", "arr[99]"); out: @@ -187,7 +187,7 @@ static void test_array_index_not_found(void) "%s set_global_vars.bpf.o -G \"arr[EG2] = 0\" -vl2 2> %s", fix->veristat, fix->tmpfile); - read(fix->fd, fix->output, fix->sz); + read_output(fix); __CHECK_STR("Can't resolve enum value EG2", "arr[EG2]"); out: From 50de1c47a41d4031f6002969e71dc6954dedb6b2 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:48 -0700 Subject: [PATCH 303/373] bpf, arm64: Fix stack-passed arguments for indirect trampolines save_args() reads stack-passed arguments relative to FP assuming the trampoline is entered through the fentry call from a traced function, in which case both the parent frame (FP/x9) and the traced function frame (FP/LR) are saved before FP is set, so the arguments start at FP + 32. An indirect trampoline for a struct_ops callback is entered through a function pointer (blr), so only the FP/LR frame is pushed and the arguments start at FP + 16, not FP + 32. Every stack-passed argument of a struct_ops callback with more than eight argument slots is read two slots off. This went unnoticed because no struct_ops member passed arguments on the stack until bpf_testmod_ops3::test_arena_stack, added by commit 2d4de9a493a0 ("selftests/bpf: Test stack-passed struct_ops arena arguments"). That member covers this on arm64 once the JIT gains arena argument support later in this series. Pass is_struct_ops into save_args() and pick the offset accordingly, mirroring the x86 fix. Fixes: 9014cf56f13d ("bpf, arm64: Support up to 12 function arguments") Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-2-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 74b4083791da..7938b3422d3c 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2525,9 +2525,8 @@ static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes) } static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, - const struct btf_func_model *m, - const struct arg_aux *a, - bool for_call_origin) + const struct btf_func_model *m, const struct arg_aux *a, + bool for_call_origin, bool is_struct_ops) { int i; int reg; @@ -2547,7 +2546,15 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, bargs_off += 8; } - soff = 32; /* on stack arguments start from FP + 32 */ + /* + * On-stack arguments start above the frame(s) pushed by the trampoline + * prologue. Entered through the fentry call from a traced function, the + * prologue saves both the parent (FP/x9) and the traced function + * (FP/LR) frames, so the arguments start at FP + 32. A struct_ops + * callback is called indirectly and only the FP/LR frame is saved, so + * they start at FP + 16. + */ + soff = is_struct_ops ? 16 : 32; doff = (for_call_origin ? oargs_off : bargs_off); /* save on stack arguments */ @@ -2737,7 +2744,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, store_func_meta(ctx, func_meta, func_meta_off); /* save args for bpf */ - save_args(ctx, bargs_off, oargs_off, m, a, false); + save_args(ctx, bargs_off, oargs_off, m, a, false, is_struct_ops); /* save callee saved registers */ emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); @@ -2786,7 +2793,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_CALL_ORIG) { /* save args for original func */ - save_args(ctx, bargs_off, oargs_off, m, a, true); + save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); From f4adb983ef3d7a23d7f67834bc12cec528312899 Mon Sep 17 00:00:00 2001 From: Tejun Heo Date: Thu, 13 Aug 2026 12:03:49 -0700 Subject: [PATCH 304/373] arm64: insn: Add encoder for ADD/SUB (extended register) The insn library encodes the immediate and shifted-register forms of ADD/SUB but not the extended-register form. The BPF JIT wants it to rebase a 32-bit arena offset onto the arena kernel base in a single instruction, add xN, xBASE, wN, uxtw, instead of a separate zero-extend followed by a plain add. Add aarch64_insn_gen_add_sub_extended_reg(), modeled on the shifted-register generator. The option and imm3 fields occupy the same bits as the shifted form's shift amount, so they are encoded through the existing IMM_6 field type. The opt field in bits 23:22 is part of the opcode here rather than a shift type, and any value other than 00 is unallocated, so the decode masks cover it. Note that register 31 does not mean the same thing in the two forms: in the extended-register encoding it is SP for Rn, and for Rd unless the instruction sets the flags, while it stays XZR for Rm. Callers porting a shifted-register site that passes A64_ZR need to be aware of that, so say so above the function. Signed-off-by: Tejun Heo Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-3-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/include/asm/insn.h | 23 ++++++++++++++ arch/arm64/lib/insn.c | 60 +++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index cc0702fa64a7..1ce75a442638 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -205,6 +205,18 @@ enum aarch64_insn_adsb_type { AARCH64_INSN_ADSB_SUB_SETFLAGS }; +/* option field of add/sub (extended register) */ +enum aarch64_insn_extend_type { + AARCH64_INSN_EXTEND_UXTB, + AARCH64_INSN_EXTEND_UXTH, + AARCH64_INSN_EXTEND_UXTW, + AARCH64_INSN_EXTEND_UXTX, + AARCH64_INSN_EXTEND_SXTB, + AARCH64_INSN_EXTEND_SXTH, + AARCH64_INSN_EXTEND_SXTW, + AARCH64_INSN_EXTEND_SXTX, +}; + enum aarch64_insn_movewide_type { AARCH64_INSN_MOVEWIDE_ZERO, AARCH64_INSN_MOVEWIDE_KEEP, @@ -378,6 +390,10 @@ __AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000) __AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000) __AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000) __AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000) +__AARCH64_INSN_FUNCS(add_ext, 0x7FE00000, 0x0B200000) +__AARCH64_INSN_FUNCS(adds_ext, 0x7FE00000, 0x2B200000) +__AARCH64_INSN_FUNCS(sub_ext, 0x7FE00000, 0x4B200000) +__AARCH64_INSN_FUNCS(subs_ext, 0x7FE00000, 0x6B200000) __AARCH64_INSN_FUNCS(madd, 0x7FE08000, 0x1B000000) __AARCH64_INSN_FUNCS(msub, 0x7FE08000, 0x1B008000) __AARCH64_INSN_FUNCS(udiv, 0x7FE0FC00, 0x1AC00800) @@ -637,6 +653,13 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst, int shift, enum aarch64_insn_variant variant, enum aarch64_insn_adsb_type type); +u32 aarch64_insn_gen_add_sub_extended_reg(enum aarch64_insn_register dst, + enum aarch64_insn_register src, + enum aarch64_insn_register reg, + enum aarch64_insn_extend_type extend, + int shift, + enum aarch64_insn_variant variant, + enum aarch64_insn_adsb_type type); u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst, enum aarch64_insn_register src, enum aarch64_insn_variant variant, diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c index 37ce75f7f1f0..e70ac0238515 100644 --- a/arch/arm64/lib/insn.c +++ b/arch/arm64/lib/insn.c @@ -986,6 +986,66 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst, return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift); } +/* + * Unlike the shifted-register form, register 31 is not XZR everywhere here: + * it encodes SP for @src, and for @dst too unless @type sets the flags. Only + * @reg keeps the XZR meaning. + */ +u32 aarch64_insn_gen_add_sub_extended_reg(enum aarch64_insn_register dst, + enum aarch64_insn_register src, + enum aarch64_insn_register reg, + enum aarch64_insn_extend_type extend, + int shift, + enum aarch64_insn_variant variant, + enum aarch64_insn_adsb_type type) +{ + u32 insn; + + switch (type) { + case AARCH64_INSN_ADSB_ADD: + insn = aarch64_insn_get_add_ext_value(); + break; + case AARCH64_INSN_ADSB_SUB: + insn = aarch64_insn_get_sub_ext_value(); + break; + case AARCH64_INSN_ADSB_ADD_SETFLAGS: + insn = aarch64_insn_get_adds_ext_value(); + break; + case AARCH64_INSN_ADSB_SUB_SETFLAGS: + insn = aarch64_insn_get_subs_ext_value(); + break; + default: + pr_err("%s: unknown add/sub encoding %d\n", __func__, type); + return AARCH64_BREAK_FAULT; + } + + switch (variant) { + case AARCH64_INSN_VARIANT_32BIT: + break; + case AARCH64_INSN_VARIANT_64BIT: + insn |= AARCH64_INSN_SF_BIT; + break; + default: + pr_err("%s: unknown variant encoding %d\n", __func__, variant); + return AARCH64_BREAK_FAULT; + } + + if (shift < 0 || shift > 4) { + pr_err("%s: invalid shift encoding %d\n", __func__, shift); + return AARCH64_BREAK_FAULT; + } + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg); + + /* option in bits [15:13] and imm3 in [12:10] together fill IMM_6 */ + return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, + (extend << 3) | shift); +} + u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst, enum aarch64_insn_register src, enum aarch64_insn_variant variant, From 760cb40cfd6a4e5781e4a32b46b1ef49950fd271 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:50 -0700 Subject: [PATCH 305/373] bpf, arm64: JIT __arena kfunc argument rebasing Implement arena argument rebasing for kfunc calls on arm64. x28 already holds kern_vm_start whenever the prog has an arena, and the newly added extended-register add zero-extends the 32-bit arena offset in place, so an unconditional argument costs a single instruction emitted right before the call: add xN, x28, wN, uxtw A nullable argument first truncates into wN so that a zero offset leaves xN holding a real NULL, then tests it and jumps over the add: mov wN, wN cbz wN, 1f add xN, x28, wN, uxtw 1: The rebase is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees it and needs no special handling. The emitted count depends only on the kfunc model, so it is identical across JIT passes. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-4-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit.h | 11 +++++++++ arch/arm64/net/bpf_jit_comp.c | 42 +++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index d13de4222cfb..b2fe6e6dcf44 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -243,6 +243,17 @@ /* Rn - Rm; set condition flags */ #define A64_CMP(sf, Rn, Rm) A64_SUBS(sf, A64_ZR, Rn, Rm) +/* Add/subtract (extended register) */ +#define A64_ADDSUB_EREG(sf, Rd, Rn, Rm, ext, shift, type) \ + aarch64_insn_gen_add_sub_extended_reg(Rd, Rn, Rm, \ + AARCH64_INSN_EXTEND_##ext, shift, A64_VARIANT(sf), \ + AARCH64_INSN_ADSB_##type) +/* Rd = Rn + (EXT(Rm) << shift) */ +#define A64_ADD_EXT(sf, Rd, Rn, Rm, ext, shift) \ + A64_ADDSUB_EREG(sf, Rd, Rn, Rm, ext, shift, ADD) +/* Rd = Rn + (u32)Rm */ +#define A64_ADD_UXTW(Rd, Rn, Rm) A64_ADD_EXT(1, Rd, Rn, Rm, UXTW, 0) + /* Data-processing (1 source) */ #define A64_DATA1(sf, Rd, Rn, type) aarch64_insn_gen_data1(Rd, Rn, \ A64_VARIANT(sf), AARCH64_INSN_DATA1_##type) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 7938b3422d3c..e31490c0e331 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1256,6 +1256,43 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct } } +/* + * Rebase the __arena args of a kfunc call to arena kernel addresses, + * xN = kern_vm_start + (u32)xN, with the arena base register holding + * kern_vm_start. A nullable arg preserves NULL by skipping the add, tested + * on the truncated value as arena NULL is offset 0. + */ +static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn) +{ + const u8 arena_vm_base = bpf2a64[ARENA_VM_START]; + const struct btf_func_model *fm; + int i; + + fm = bpf_jit_find_kfunc_model(ctx->prog, insn); + if (!fm) + return -EINVAL; + + for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { + const u8 reg = bpf2a64[BPF_REG_1 + i]; + u8 flags = fm->arg_flags[i]; + + if (!(flags & BTF_FMODEL_ARENA_ARG)) + continue; + if (WARN_ON_ONCE(!ctx->arena_vm_start)) + return -EINVAL; + + if (flags & BTF_FMODEL_NULLABLE_ARG) { + /* 32-bit mov clears the upper 32 bits */ + emit(A64_MOV(0, reg, reg), ctx); + /* skip the add so that NULL stays NULL */ + emit(A64_CBZ(0, reg, 2), ctx); + } + emit(A64_ADD_UXTW(reg, arena_vm_base, reg), ctx); + } + + return 0; +} + /* JITs an eBPF instruction. * Returns: * 0 - successfully JITed an 8-byte eBPF instruction. @@ -1678,6 +1715,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn &func_addr, &func_addr_fixed); if (ret < 0) return ret; + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { + ret = emit_kfunc_arena_args(ctx, insn); + if (ret < 0) + return ret; + } emit_call(func_addr, ctx); /* * Call to arch_bpf_timed_may_goto() is emitted by the From bb5bad6a78b6334ab9f8ab99cdb92bae11d9acea Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:51 -0700 Subject: [PATCH 306/373] bpf, arm64: Convert struct_ops arena arguments in the trampoline Implement the struct_ops arena argument conversion on arm64. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through x10 with the low half of the base materialized once into x11: sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */ str x10, [sp, #slot] A nullable argument tests the full 64-bit kernel pointer first: mov x10, xsrc cbz x10, 1f sub w10, w10, w11 1: str x10, [sp, #slot] The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through x10, so only the subtraction (and the NULL test) is inserted there. The register loop now walks arguments rather than registers so that the per-argument flags line up with the slots a multi-slot argument occupies; the sequence of stores is otherwise unchanged. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline, so a tracing trampoline emits exactly what it did before and never touches x11. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. Conversion must never reach the original function, which takes kernel addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin save_args() and assert the flag combination the same way x86 does, rather than leaving the invariant to a comment. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only qualifier from the kfunc documentation. Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-5-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- Documentation/bpf/kfuncs.rst | 6 +-- arch/arm64/net/bpf_jit_comp.c | 92 +++++++++++++++++++++++++++++------ 2 files changed, 80 insertions(+), 18 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 10e725cbe64c..85f73e0bbd0f 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -301,9 +301,9 @@ An example is given below:: } Calling such a kfunc requires the program to use an arena map and a JIT with -arena argument support (currently x86-64); verification fails otherwise. The -program can pass any value without compromising the kernel. A value that does -not point into the arena is a program bug. +arena argument support (currently x86-64 and arm64); verification fails +otherwise. The program can pass any value without compromising the kernel. A +value that does not point into the arena is a program bug. The suffixes have the same meaning on the arguments of struct_ops stub functions, with the conversion running in the opposite direction. The diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index e31490c0e331..c18e005a41db 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2393,6 +2393,11 @@ bool bpf_jit_supports_stack_args(void) return true; } +bool bpf_jit_supports_arena_args(void) +{ + return true; +} + void *bpf_arch_text_copy(void *dst, void *src, size_t len) { if (!aarch64_insn_copy(dst, src, len)) @@ -2566,26 +2571,58 @@ static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes) } } +/* + * Convert an arena kernel address into the arena pointer form on its way into + * the BPF ctx, dst = (u32)(src - kern_vm_start), with @base_lo holding the low + * 32 bits of kern_vm_start. A nullable arg preserves NULL, tested on the full + * 64-bit kernel pointer. The 32-bit subtraction both truncates and clears the + * upper half, so the stored value satisfies the JIT invariant for arena + * pointer registers. + */ +static void emit_arena_arg_conv(struct jit_ctx *ctx, u8 dst, u8 src, bool nullable, u8 base_lo) +{ + if (nullable) { + if (dst != src) + emit(A64_MOV(1, dst, src), ctx); + /* skip the subtraction so that NULL stays NULL */ + emit(A64_CBZ(1, dst, 2), ctx); + src = dst; + } + emit(A64_SUB(0, dst, src, base_lo), ctx); +} + static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, const struct btf_func_model *m, const struct arg_aux *a, - bool for_call_origin, bool is_struct_ops) + bool for_call_origin, bool is_struct_ops, u64 arena_base) { - int i; - int reg; - int doff; - int soff; - int slots; u8 tmp = bpf2a64[TMP_REG_1]; + u8 base_lo = bpf2a64[TMP_REG_2]; + int i, reg, doff, soff, slots; + + /* only the low 32 bits of the base take part in the subtraction */ + if (arena_base) + emit_a64_mov_i(0, base_lo, (s32)(u32)arena_base, ctx); /* store arguments to the stack for the bpf program, or restore * arguments from stack for the original function */ - for (reg = 0; reg < a->regs_for_args; reg++) { - emit(for_call_origin ? - A64_LDR64I(reg, A64_SP, bargs_off) : - A64_STR64I(reg, A64_SP, bargs_off), - ctx); - bargs_off += 8; + for (i = 0, reg = 0; i < a->args_in_regs; i++) { + bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG); + bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG; + + slots = (m->arg_size[i] + 7) / 8; + while (slots-- > 0) { + if (for_call_origin) { + emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx); + } else if (arena_arg) { + emit_arena_arg_conv(ctx, tmp, reg, nullable, base_lo); + emit(A64_STR64I(tmp, A64_SP, bargs_off), ctx); + } else { + emit(A64_STR64I(reg, A64_SP, bargs_off), ctx); + } + reg++; + bargs_off += 8; + } } /* @@ -2601,6 +2638,9 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, /* save on stack arguments */ for (i = a->args_in_regs; i < m->nr_args; i++) { + bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG); + bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG; + slots = (m->arg_size[i] + 7) / 8; /* verifier ensures arg_size <= 16, so slots equals 1 or 2 */ while (slots-- > 0) { @@ -2610,6 +2650,15 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, */ if (slots == 0 && !for_call_origin) clear_garbage(ctx, tmp, m->arg_size[i] % 8); + /* + * No guard on for_call_origin here: only the indirect + * trampoline is given a base, and it never calls the + * original function, so arguments are never converted + * on their way back out to it. See the WARN_ON_ONCE() + * in prepare_trampoline(). + */ + if (arena_arg) + emit_arena_arg_conv(ctx, tmp, tmp, nullable, base_lo); emit(A64_STR64I(tmp, A64_SP, doff), ctx); soff += 8; doff += 8; @@ -2669,8 +2718,21 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, bool is_struct_ops = is_struct_ops_tramp(fentry); int cookie_off, cookie_cnt, cookie_bargs_off; int fsession_cnt = bpf_fsession_cnt(tnodes); + u64 arena_base; u64 func_meta; + /* + * F_INDIRECT is only compatible with F_RET_FENTRY_RET, it is explicitly + * incompatible with F_CALL_ORIG | F_SKIP_FRAME | F_IP_ARG because + * @func_addr. Arena conversion relies on this: bpf_tramp_arena_base() + * only returns a base for the indirect trampoline, which therefore + * never calls the original function with converted arguments. + */ + WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && + (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); + + arena_base = bpf_tramp_arena_base(m, tnodes, flags); + /* trampoline stack layout: * [ parent ip ] * [ FP ] @@ -2786,7 +2848,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, store_func_meta(ctx, func_meta, func_meta_off); /* save args for bpf */ - save_args(ctx, bargs_off, oargs_off, m, a, false, is_struct_ops); + save_args(ctx, bargs_off, oargs_off, m, a, false, is_struct_ops, arena_base); /* save callee saved registers */ emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); @@ -2834,8 +2896,8 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, } if (flags & BPF_TRAMP_F_CALL_ORIG) { - /* save args for original func */ - save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops); + /* the original func takes kernel addresses, never converted ones */ + save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); From 1c5bc60f957d853ec7183d79419360979dddd421 Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:52 -0700 Subject: [PATCH 307/373] selftests/bpf: Add arm64 JIT-sequence tests for __arena kfunc arguments Pin the arm64 counterparts of the x86-64 rebase sequences: the single extended-register add for an unconditional argument, the nullable truncate-test-and-skip variant, and all five argument registers in one call. The nullable cases use a local label so the branch is pinned to the instruction right after the add, and the label line does not spell out the call because arm64 emits either a direct bl or a materialize- and-blr pair depending on the distance to the kfunc. Note that on arm64 an unconditional argument is one instruction with nothing to anchor it against, so arena_arg_jit_rebase alone cannot tell the two forms apart; it only requires that nothing is emitted between the rebase and the call. The args5 test is what pins the distinction, since its four consecutive adds leave no room for a nullable truncate-and-branch pair between them. Signed-off-by: Puranjay Mohan Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-6-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/arena_kfunc_jit.c | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c b/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c index c9b918662616..b5a01cbc33a7 100644 --- a/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c +++ b/tools/testing/selftests/bpf/progs/arena_kfunc_jit.c @@ -32,6 +32,10 @@ __jited(" movl %edi, %edi") __jited(" addq %r12, %rdi") __jited("...") __jited(" callq {{.*}}") +__arch_arm64 +__jited("...") +__jited(" add x0, x28, w0, uxtw") +__jited(" {{(bl|mov) .*}}") __success int arena_arg_jit_rebase(void *ctx) { @@ -48,6 +52,12 @@ __jited(" testl %edi, %edi") __jited(" je L0") __jited(" addq %r12, %rdi") __jited("L0: callq {{.*}}") +__arch_arm64 +__jited("...") +__jited(" mov w0, w0") +__jited(" cbz w0, L0") +__jited(" add x0, x28, w0, uxtw") +__jited("L0: {{.*}}") __success int arena_arg_jit_nullable(void *ctx) { @@ -72,6 +82,16 @@ __jited(" testl %r8d, %r8d") __jited(" je L0") __jited(" addq %r12, %r8") __jited("L0: callq {{.*}}") +__arch_arm64 +__jited("...") +__jited(" add x0, x28, w0, uxtw") +__jited(" add x1, x28, w1, uxtw") +__jited(" add x2, x28, w2, uxtw") +__jited(" add x3, x28, w3, uxtw") +__jited(" mov w4, w4") +__jited(" cbz w4, L0") +__jited(" add x4, x28, w4, uxtw") +__jited("L0: {{.*}}") __success int arena_arg_jit_args5(void *ctx) { From 05a3575f225c81712b925c28b1caa1c24ac6ed3b Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:53 -0700 Subject: [PATCH 308/373] selftests/bpf: Enable __arena argument tests on arm64 The arena kfunc and struct_ops argument tests were restricted to x86-64 because it was the only JIT that implemented the conversions. arm64 does now, so let them run there too: tag every program in arena_kfunc.c with __arch_arm64 in addition to __arch_x86_64, and widen the __x86_64__ guards in the struct_ops arena test. Without this the tests report SKIP on arm64 rather than exercising the newly added JIT support. Signed-off-by: Puranjay Mohan Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-7-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/test_struct_ops_arena.c | 10 +++++----- tools/testing/selftests/bpf/progs/arena_kfunc.c | 9 +++++++++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c index 940ec2cda0d5..7f9f54ba3fbe 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c @@ -6,7 +6,7 @@ #include "struct_ops_arena_attach.skel.h" #include "struct_ops_arena_fail.skel.h" -#if defined(__x86_64__) +#if defined(__x86_64__) || defined(__aarch64__) /* * Attach callbacks with __arena and __arena__nullable arguments and drive * them through the bpf_testmod_ops3_call_test_arena*() kfuncs. @@ -111,11 +111,11 @@ static void arena_arg_attach(void) void serial_test_struct_ops_arena(void) { /* - * Arena struct_ops arguments need JIT support, currently x86-64 only. - * Elsewhere verification fails with "JIT does not support arena - * arguments", so the programs cannot even load. + * Arena struct_ops arguments need JIT support, currently x86-64 and + * arm64 only. Elsewhere verification fails with "JIT does not support + * arena arguments", so the programs cannot even load. */ -#if defined(__x86_64__) +#if defined(__x86_64__) || defined(__aarch64__) if (test__start_subtest("arena_arg")) arena_arg(); if (test__start_subtest("arena_arg_fail")) diff --git a/tools/testing/selftests/bpf/progs/arena_kfunc.c b/tools/testing/selftests/bpf/progs/arena_kfunc.c index cdcea889da58..bf0d304e0e59 100644 --- a/tools/testing/selftests/bpf/progs/arena_kfunc.c +++ b/tools/testing/selftests/bpf/progs/arena_kfunc.c @@ -27,6 +27,7 @@ volatile u64 stash; SEC("syscall") __arch_x86_64 +__arch_arm64 __success __retval(0) int arena_arg_forms(void *ctx) { @@ -70,6 +71,7 @@ int arena_arg_forms(void *ctx) */ SEC("syscall") __arch_x86_64 +__arch_arm64 __success __retval(0) int arena_arg_rebase(void *ctx) { @@ -111,6 +113,7 @@ int arena_arg_rebase(void *ctx) SEC("syscall") __arch_x86_64 +__arch_arm64 __success __retval(0) int arena_args5(void *ctx) { @@ -142,6 +145,7 @@ int arena_args5(void *ctx) SEC("syscall") __arch_x86_64 +__arch_arm64 __success __retval(0) int arena_arg_mixed(void *ctx) { @@ -169,6 +173,7 @@ int arena_arg_mixed(void *ctx) /* kernel-side faults on unpopulated pages recover via the scratch page */ SEC("syscall") __arch_x86_64 +__arch_arm64 __success __retval(0) int arena_arg_unpopulated(void *ctx) { @@ -189,6 +194,7 @@ int arena_arg_unpopulated(void *ctx) SEC("syscall") __arch_x86_64 +__arch_arm64 __failure __msg("arena pointer requires a program with an associated arena") int arena_arg_no_arena(void *ctx) { @@ -198,6 +204,7 @@ int arena_arg_no_arena(void *ctx) SEC("syscall") __arch_x86_64 +__arch_arm64 __failure __msg("is not a pointer to arena or scalar") int arena_arg_bad_reg(void *ctx) { @@ -213,6 +220,7 @@ int arena_arg_bad_reg(void *ctx) defined(__BPF_FEATURE_STACK_ARGUMENT) SEC("syscall") __arch_x86_64 +__arch_arm64 __failure __msg("arena pointer cannot be a stack argument") int arena_arg_stack(void *ctx) { @@ -223,6 +231,7 @@ int arena_arg_stack(void *ctx) #else SEC("syscall") __arch_x86_64 +__arch_arm64 __description("arena_arg_stack: not supported, dummy test") __success int arena_arg_stack(void *ctx) From 197d34b169435a447830ddd915d596431b9f311a Mon Sep 17 00:00:00 2001 From: Puranjay Mohan Date: Thu, 13 Aug 2026 12:03:54 -0700 Subject: [PATCH 309/373] selftests/bpf: Test a multi-slot argument before a struct_ops arena argument The trampoline reads the __arena flag from the btf_func_model per argument but stores the ctx one register slot at a time, so the two only line up if every preceding argument occupies exactly one slot. Every arena-bearing member of bpf_testmod_ops3 takes single-slot arguments, so nothing exercises the mapping and a mis-indexed arg_flags lookup would go unnoticed on any architecture. Add test_arena_multislot(), whose first argument is a 16-byte struct passed by value. It fills ctx[0] and ctx[1], putting the arena pointer at argument index one but slot two. The callback checks both halves of the struct before dereferencing ctx[2], so a JIT that walks registers instead of arguments converts the wrong slot and fails the test. Signed-off-by: Puranjay Mohan Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-8-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/progs/struct_ops_arena.c | 24 +++++++++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 15 ++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.h | 8 +++++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 1 + 4 files changed, 48 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/struct_ops_arena.c b/tools/testing/selftests/bpf/progs/struct_ops_arena.c index ba04c73d8d96..8aa8639df91f 100644 --- a/tools/testing/selftests/bpf/progs/struct_ops_arena.c +++ b/tools/testing/selftests/bpf/progs/struct_ops_arena.c @@ -59,11 +59,28 @@ int test_arena_stack_cb(unsigned long long *ctx) return 0; } +SEC("struct_ops/test_arena_multislot") +int test_arena_multislot_cb(unsigned long long *ctx) +{ + u64 __arena *ptr = (u64 __arena *)ctx[2]; + + arena_touch++; + /* + * The 16-byte struct occupies ctx[0] and ctx[1], so @ptr is argument + * one but slot two. Getting that wrong hands the callback a scalar. + */ + if (ctx[0] != 11 || ctx[1] != 22) + return 0xbad; + *ptr += 1; + return 0; +} + SEC(".struct_ops.link") struct bpf_testmod_ops3 testmod_arena = { .test_arena = (void *)test_arena_cb, .test_arena_nullable = (void *)test_arena_nullable_cb, .test_arena_stack = (void *)test_arena_stack_cb, + .test_arena_multislot = (void *)test_arena_multislot_cb, }; SEC("syscall") @@ -109,6 +126,13 @@ int trigger(void *ctx) if (*val != 44) return 9; + /* a multi-slot arg precedes the arena pointer here */ + ret = bpf_testmod_ops3_call_test_arena_multislot((u64 *)val); + if (ret) + return 10; + if (*val != 45) + return 11; + bpf_arena_free_pages(&arena, (void __arena *)val, 1); #endif return 0; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index a6133f7521f3..9366a3c578f1 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -402,12 +402,19 @@ static int bpf_testmod_ops3__test_arena_stack(u64 a, u64 b, u64 c, u64 d, return 0; } +static int bpf_testmod_ops3__test_arena_multislot(struct bpf_testmod_arena_pair p, + u64 *ptr__arena) +{ + return 0; +} + static struct bpf_testmod_ops3 __bpf_testmod_ops3 = { .test_1 = bpf_testmod_test_3, .test_2 = bpf_testmod_test_4, .test_arena = bpf_testmod_ops3__test_arena, .test_arena_nullable = bpf_testmod_ops3__test_arena_nullable, .test_arena_stack = bpf_testmod_ops3__test_arena_stack, + .test_arena_multislot = bpf_testmod_ops3__test_arena_multislot, }; static void bpf_testmod_test_struct_ops3(void) @@ -441,6 +448,13 @@ __bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena) return st_ops3->test_arena_stack(1, 2, 3, 4, 5, 6, 7, 8, ptr__arena); } +__bpf_kfunc int bpf_testmod_ops3_call_test_arena_multislot(u64 *ptr__arena) +{ + struct bpf_testmod_arena_pair p = { .a = 11, .b = 22 }; + + return st_ops3->test_arena_multislot(p, ptr__arena); +} + struct bpf_testmod_btf_type_tag_1 { int a; }; @@ -852,6 +866,7 @@ BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable) BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack) +BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_multislot) BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test); BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test); BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h index 33f2af5b7085..210b919290cc 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h @@ -103,6 +103,12 @@ struct bpf_testmod_ops2 { int (*test_1)(void); }; +/* 16 bytes, so it takes two argument slots when passed by value */ +struct bpf_testmod_arena_pair { + u64 a; + u64 b; +}; + struct bpf_testmod_ops3 { int (*test_1)(void); int (*test_2)(void); @@ -112,6 +118,8 @@ struct bpf_testmod_ops3 { /* enough leading args to force @ptr onto the stack on x86 and arm64 */ int (*test_arena_stack)(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, u64 h, u64 *ptr); + /* a multi-slot leading arg, so @ptr is not at the slot its arg index suggests */ + int (*test_arena_multislot)(struct bpf_testmod_arena_pair p, u64 *ptr); }; struct st_ops_args { diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index c4383acb53c1..7d81070eefe7 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -123,6 +123,7 @@ void bpf_testmod_test_mod_kfunc(int i) __ksym; int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym; int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym; int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym; +int bpf_testmod_ops3_call_test_arena_multislot(__u64 *ptr__arena) __ksym; __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b, __u32 c, __u64 d) __ksym; From 073574da7a8e6055b1687c256bea070ab4b4ebea Mon Sep 17 00:00:00 2001 From: Andrii Nakryiko Date: Thu, 13 Aug 2026 14:35:58 -0700 Subject: [PATCH 310/373] selftests/bpf: Use ping_command() for IPv6 pings in lwt_ip_encap lwt_ip_encap hardcodes the ping6 binary for its IPv6 pings. iputils merged ping6 into ping long ago and distros have started dropping the compat symlink -- Arch's iputils 20250605 ships only arping, clockdiff, ping and tracepath. There, every lwt_ip_encap subtest fails: check_ping_ok:FAIL:ip netns exec ns-lwt-ip-encap-1-0101330 ping6 -c 1 \ -W1 -I veth1 fb04::1 > /dev/null unexpected error: 256 (errno 2) #217/1 lwt_ip_encap_ipv4/egress:FAIL The IPv4 subtests fail too, because check_ping_ok() pings both families. SYS() runs the command through system(), so a missing binary is indistinguishable from an unreachable peer. network_helpers.c has had ping_command() for exactly this since commit 372642ea83ff ("selftests/bpf: Move netcnt test under test_progs"): it falls back to "ping -6" when ping6 is not present. lwt_ip_encap.c is the last hardcoded ping6 user. Fix that. Fixes: f5e288943e2c ("selftests/bpf: Move test_lwt_ip_encap to test_progs") Signed-off-by: Andrii Nakryiko Signed-off-by: Daniel Borkmann Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260813213558.3103179-1-andrii@kernel.org --- tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c b/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c index 6606f0ed9a9a..39e8a3b8b6af 100644 --- a/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c +++ b/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c @@ -410,7 +410,8 @@ static int test_gso_fix(const char *ns1, const char *ns3, int family) static int check_ping_ok(const char *ns1) { SYS(fail, "ip netns exec %s ping -c 1 -W1 -I veth1 %s > /dev/null", ns1, IP4_ADDR_DST); - SYS(fail, "ip netns exec %s ping6 -c 1 -W1 -I veth1 %s > /dev/null", ns1, IP6_ADDR_DST); + SYS(fail, "ip netns exec %s %s -c 1 -W1 -I veth1 %s > /dev/null", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); return 0; fail: return -1; @@ -424,7 +425,8 @@ static int check_ping_fails(const char *ns1) if (!ret) return -1; - ret = SYS_NOFAIL("ip netns exec %s ping6 -c 1 -W1 -I veth1 %s", ns1, IP6_ADDR_DST); + ret = SYS_NOFAIL("ip netns exec %s %s -c 1 -W1 -I veth1 %s", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); if (!ret) return -1; @@ -657,9 +659,10 @@ static void lwt_ip_encap_vxlan(bool ipv4_encap) skel->bss->fexit_triggered = false; if (ipv4_encap) - SYS(out, "ip netns exec %s ping -c 1 -W1 %s", ns1, IP4_ADDR_DST); + SYS(out, "ip netns exec %s ping -c 1 -W1 %s", ns1, IP4_ADDR_DST); else - SYS(out, "ip netns exec %s ping6 -c 1 -W1 %s", ns1, IP6_ADDR_DST); + SYS(out, "ip netns exec %s %s -c 1 -W1 %s", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); if (!ASSERT_TRUE(skel->bss->fexit_triggered, "fexit_triggered")) goto out; From c7e617552938dc5e28d9ad87ae485caa35389be7 Mon Sep 17 00:00:00 2001 From: Andrii Nakryiko Date: Thu, 13 Aug 2026 16:29:43 -0700 Subject: [PATCH 311/373] selftests/bpf: Make pyperf600 a success again pyperf600 has been running into 8K BPF_COMPLEXITY_LIMIT_JMP_SEQ limitations for a long while now, after some internal compiler changes. Until BPF verifier is bestowed with scalar evolution logic, make that test actually work by doing what would anyone should do in such situations: by moving repeatable per-iteration work into independently verified global functions. `void *` argument is a problem for global funcs, but a static function wrapper doing necessary casts and a bit of __arg_nonnull magic dust is all it takes. Signed-off-by: Andrii Nakryiko Signed-off-by: Daniel Borkmann Acked-by: Yonghong Song Tested-by: Pu Lehui # riscv Reviewed-by: Pu Lehui Link: https://lore.kernel.org/bpf/20260813232943.581283-1-andrii@kernel.org --- tools/testing/selftests/bpf/progs/pyperf.h | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/pyperf.h b/tools/testing/selftests/bpf/progs/pyperf.h index 86484f07e1d1..fd93a96e5901 100644 --- a/tools/testing/selftests/bpf/progs/pyperf.h +++ b/tools/testing/selftests/bpf/progs/pyperf.h @@ -85,9 +85,11 @@ static void *get_thread_state(void *tls_base, PidData *pidData) return thread_state; } -static __always_inline bool get_frame_data(void *frame_ptr, PidData *pidData, - FrameData *frame, Symbol *symbol) +__weak bool __get_frame_data(long frame_ptr_, PidData *pidData __arg_nonnull, + FrameData *frame __arg_nonnull, Symbol *symbol __arg_nonnull) { + void *frame_ptr = (void *)frame_ptr_; + // read data from PyFrameObject bpf_probe_read_user(&frame->f_back, sizeof(frame->f_back), @@ -119,6 +121,12 @@ static __always_inline bool get_frame_data(void *frame_ptr, PidData *pidData, return true; } +static __always_inline bool get_frame_data(void *frame_ptr, PidData *pidData, + FrameData *frame, Symbol *symbol) +{ + return __get_frame_data((long)frame_ptr, pidData, frame, symbol); +} + struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 1); From b0e872a31e157d48479507c2821ba8bf6323b7ad Mon Sep 17 00:00:00 2001 From: Mykyta Yatsenko Date: Wed, 12 Aug 2026 07:47:22 -0700 Subject: [PATCH 312/373] bpf: Fix arm64 KASAN false positive after bpf_throw arm64 passes zero as the stack pointer while walking BPF frames, so bpf_throw() leaves stale KASAN stack poison after jumping to the exception callback. Use the frame pointer as the fallback stack watermark. Fixes: e74cb1b42213 ("arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT") Signed-off-by: Mykyta Yatsenko Signed-off-by: Daniel Borkmann Tested-by: Ihor Solodrai Link: https://lore.kernel.org/bpf/20260812-hello_world-v1-1-c3c2ddcb362d@meta.com --- kernel/bpf/helpers.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 45e2f19387b2..b3cc5c8fc875 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3395,11 +3395,13 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN_ON_ONCE(!ctx.aux->exception_boundary); WARN_ON_ONCE(!ctx.bp); WARN_ON_ONCE(!ctx.cnt); - /* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning + /* + * Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning * deeper stack depths than ctx.sp as we do not return from bpf_throw, - * which skips compiler generated instrumentation to do the same. + * which skips compiler generated instrumentation to do the same. Some + * architectures cannot recover sp while unwinding, so fall back to bp. */ - kasan_unpoison_task_stack_below((void *)(long)ctx.sp); + kasan_unpoison_task_stack_below((void *)(long)(ctx.sp ?: ctx.bp)); ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0); WARN(1, "A call to BPF exception callback should never return\n"); } From f2aaa621591093cfe8224a25ef2f04a3b1e304b0 Mon Sep 17 00:00:00 2001 From: Pu Lehui Date: Fri, 14 Aug 2026 06:47:26 +0000 Subject: [PATCH 313/373] riscv, bpf: Fix missing sign-ext for signed 1-byte and 2-byte kfunc args On RV64, the ABI requires sign-extension for signed 1-byte and 2-byte kfunc args. However, the RV64 JIT currently does not perform sign-extension for such kfunc args. Before commit 7ce090afbf72 ("bpf: Infer zext_dst based on static register liveness analysis"), state pruning could potentially omit zero-extension of 32-bit subregisters, which inadvertently masked the above issue by making the args appear as if they had been properly sign-extended. After that commit, the problem is exposed, causing the kfunc_call/kfunc_call_test4 selftest to fail. Fix this by extending the existing sign-extension logic to handle signed 1-byte and 2-byte kfunc args as well. Fixes: 443574b03387 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi") Signed-off-by: Pu Lehui Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260814064726.3607615-1-pulehui@huaweicloud.com --- arch/riscv/net/bpf_jit_comp64.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 2504df1fa111..74efe4b138d2 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -1823,9 +1823,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, for (idx = 0; idx < fm->nr_args; idx++) { u8 reg = bpf_to_rv_reg(BPF_REG_1 + idx, ctx); + bool sign = fm->arg_flags[idx] & BTF_FMODEL_SIGNED_ARG; - if (fm->arg_size[idx] == sizeof(int)) - emit_sextw(reg, reg, ctx); + if (sign_extend(reg, reg, fm->arg_size[idx], sign, ctx)) + return -EINVAL; } } From 3808171428f563633d532a801197ed9b410735bf Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Sat, 15 Aug 2026 01:32:03 +0800 Subject: [PATCH 314/373] libbpf: Avoid unnecessary mmap resize for percpu data maps Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz(). This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap() when the old and new value sizes occupy the same number of pages. Fix some typos btw: * mmapble -> mmapable * satisified -> satisfied * relocatin -> relocation * atach_btf_obj_fd -> attach_btf_obj_fd * len_secnd -> len_second * precendence -> precedence Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev --- tools/lib/bpf/libbpf.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index e574870fb716..b749c01742ee 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -1841,9 +1841,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map) switch (map->def.type) { case BPF_MAP_TYPE_ARRAY: - return array_map_mmap_sz(map->def.value_size, map->def.max_entries); case BPF_MAP_TYPE_PERCPU_ARRAY: - return map->def.value_size; + return array_map_mmap_sz(map->def.value_size, map->def.max_entries); case BPF_MAP_TYPE_ARENA: return page_sz * map->def.max_entries; default: @@ -1951,7 +1950,7 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) return false; /* - * The internal PERCPU maps are not mmapble because the underlying + * The internal PERCPU maps are not mmapable because the underlying * percpu_array maps do not have mmap support. */ if (map->libbpf_type == LIBBPF_MAP_PERCPU) @@ -2831,7 +2830,7 @@ static size_t adjust_ringbuf_sz(size_t sz) return 0; /* Kernel expects BPF_MAP_TYPE_RINGBUF's max_entries to be * a power-of-2 multiple of kernel's page size. If user diligently - * satisified these conditions, pass the size through. + * satisfied these conditions, pass the size through. */ if ((sz % page_sz) == 0 && is_pow_of_2(sz / page_sz)) return sz; @@ -6989,7 +6988,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, * +-----------+------+------+ * * At this point, we relocate subA calls, then go one level up and finish with - * relocatin mainA calls. mainA is done. + * relocation mainA calls. mainA is done. * * For mainB process is similar but results in different order. We start with * mainB and skip subA and subB, as mainB never calls them (at least @@ -7936,7 +7935,7 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, prog->attach_btf_id = btf_type_id; /* but by now libbpf common logic is not utilizing - * prog->atach_btf_obj_fd/prog->attach_btf_id anymore because + * prog->attach_btf_obj_fd/prog->attach_btf_id anymore because * this callback is called after opts were populated by * libbpf, so this callback has to update opts explicitly here */ @@ -14219,7 +14218,7 @@ perf_event_read_simple(void *mmap_mem, size_t mmap_size, size_t page_size, if (((void *)ehdr) + ehdr_size > base + mmap_size) { void *copy_start = ehdr; size_t len_first = base + mmap_size - copy_start; - size_t len_secnd = ehdr_size - len_first; + size_t len_second = ehdr_size - len_first; if (*copy_size < ehdr_size) { free(*copy_mem); @@ -14233,7 +14232,7 @@ perf_event_read_simple(void *mmap_mem, size_t mmap_size, size_t page_size, } memcpy(*copy_mem, copy_start, len_first); - memcpy(*copy_mem + len_first, base, len_secnd); + memcpy(*copy_mem + len_first, base, len_second); ehdr = *copy_mem; } @@ -14251,7 +14250,7 @@ struct perf_buffer; struct perf_buffer_params { struct perf_event_attr *attr; - /* if event_cb is specified, it takes precendence */ + /* if event_cb is specified, it takes precedence */ perf_buffer_event_fn event_cb; /* sample_cb and lost_cb are higher-level common-case callbacks */ perf_buffer_sample_fn sample_cb; From 90bd0329abd4a1f9fbf6c46c3aa64f638443c368 Mon Sep 17 00:00:00 2001 From: Leon Hwang Date: Sat, 15 Aug 2026 01:32:06 +0800 Subject: [PATCH 315/373] selftests/bpf: Improve readability in iter test for percpu data The original 'offsetof()' + offset is equal to the new 'offsetof()'. Use the new 'offsetof()' instead. Rename two variables btw: * offsetof_num -> num_off * percpu_data_sum -> sum Signed-off-by: Leon Hwang Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814173206.93082-6-leon.hwang@linux.dev --- tools/testing/selftests/bpf/prog_tests/global_data_init.c | 6 +++--- tools/testing/selftests/bpf/progs/test_global_percpu_data.c | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 7d6bda909295..5671c31085cd 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -345,8 +345,8 @@ static void test_global_percpu_data_iter(void) return; skel->rodata->num_cpus = num_cpus; - skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data); - skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int); + skel->rodata->num_off = offsetof(struct test_global_percpu_data__percpu, + struct_data.nums[6]); skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8); skel->percpu->struct_data.nums[6] = 0xc0de; @@ -369,7 +369,7 @@ static void test_global_percpu_data_iter(void) do { } while (0); ASSERT_EQ(len, 0, "read iter"); ASSERT_TRUE(skel->bss->run_iter, "run_iter"); - ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum"); + ASSERT_EQ(skel->bss->sum, 0xc0de * num_cpus, "sum"); close(fd); out: diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c index 71ff8d1bf49e..5dc21b3b4cb5 100644 --- a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -62,9 +62,9 @@ int verifier_snprintf(void *ctx) } volatile const __u32 num_cpus = 0; -volatile const int offsetof_num; +volatile const int num_off; volatile const int elem_sz; -__u32 percpu_data_sum = 0; +__u32 sum = 0; bool run_iter = false; SEC("iter/bpf_map_elem") @@ -80,7 +80,7 @@ int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx) run_iter = true; for (i = 0; i < num_cpus; i++) { - percpu_data_sum += *(int *) (pptr + offsetof_num); + sum += *(int *) (pptr + num_off); pptr += elem_sz; } return 0; From a2b83a8c8430b5a4cb43a1671c62535afcfce78d Mon Sep 17 00:00:00 2001 From: Ihor Solodrai Date: Fri, 14 Aug 2026 10:35:21 -0700 Subject: [PATCH 316/373] selftests/bpf: Fix selftest build after filter.h update Upstream commit 7a1f400ff5e5 ("tools: Ensure tools copy of linux/filter.h exports the UAPI") caused selftests/bpf build to fail [1] with: In file included from progs/arena_atomics.c:9: /codebuild/output/src2365462129/src/actions-runner/_work/bpf/bpf/tools/testing/selftests/bpf/../../../include/linux/filter.h:9:10: fatal error: 'uapi/linux/filter.h' file not found 9 | #include | ^~~~~~~~~~~~~~~~~~~~~ 1 error generated. CLNG-BPF [test_progs] bind_perm.bpf.o make: *** [Makefile:888: /codebuild/output/src2365462129/src/actions-runner/_work/bpf/bpf/tools/testing/selftests/bpf/arena_atomics.bpf.o] Error 1 make: *** Waiting for unfinished jobs.... GEN-OBJ [libarena] libarena.bpf.o GEN-SKEL [libarena] libarena.skel.h make: Leaving directory '/codebuild/output/src2365462129/src/actions-runner/_work/bpf/bpf/tools/testing/selftests/bpf' Process completed with exit code 2. BPF selftest programs include the tools header directly, but BPF_CFLAGS only exposes tools/include/uapi. Compiler therefore cannot resolve the nested UAPI include. Add tools/include after tools/include/uapi in BPF_CFLAGS. This preserves the existing UAPI header precedence while allowing tools headers to include uapi headers. [1] https://github.com/kernel-patches/bpf/actions/runs/31806678733/job/94787271162 Fixes: 7a1f400ff5e5 ("tools: Ensure tools copy of linux/filter.h exports the UAPI") Signed-off-by: Ihor Solodrai Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814173522.2783625-1-ihor.solodrai@linux.dev --- tools/testing/selftests/bpf/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 560ce4016fbf..2b2f93dec474 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -464,7 +464,7 @@ endif CLANG_SYS_INCLUDES = $(call get_sys_includes,$(CLANG),$(CLANG_TARGET_ARCH)) BPF_CFLAGS = -g -Wall -Werror -D__TARGET_ARCH_$(SRCARCH) $(MENDIAN) \ -I$(INCLUDE_DIR) -I$(CURDIR) -I$(APIDIR) \ - -I$(CURDIR)/libarena/include \ + -I$(TOOLSINCDIR) -I$(CURDIR)/libarena/include \ -I$(abspath $(OUTPUT)/../usr/include) \ -std=gnu11 \ -fno-strict-aliasing \ From 6ff5b56a50c5351aeeb180e34327736576c038fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Israel=20T=C3=A9llez=20Garc=C3=ADa?= Date: Fri, 14 Aug 2026 14:48:40 +0200 Subject: [PATCH 317/373] bpf: Fix pending_pos walk on 32-bit ring position wrap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reservation path caches the position of the oldest not-yet-committed record in rb->pending_pos and advances it past already committed records on every reservation: while (pend_pos < prod_pos) { consumer_pos, producer_pos and pending_pos are unsigned long, i.e. 32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states that these counters may wrap around there. Every other comparison in the file is written as a difference, so modular arithmetic keeps them correct across the wrap. This one is an ordering comparison, and it is not wrap-safe. Once producer_pos wraps past 2^32, prod_pos is small while pend_pos still holds its pre-wrap value, so the loop condition is false and pending_pos is never advanced again. Reservations keep succeeding for a while, because bpf_ringbuf_has_space() uses differences, but new_prod_pos - pend_pos grows as the producer advances, and once it exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails: the kernel believes a pending record spans the whole buffer. The ring never recovers, bpf_ringbuf_output() drops every event from then on, and nothing is logged. Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a tracepoint-based collector with a 512 KiB ring and 160-byte records. Every one of them stopped delivering after exactly 26846821 records and 4295491360 bytes had passed through the ring, at event rates between 441 and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is the byte count, not time or load. That figure is 2^32 plus 524064 bytes, and the excess is one ring's worth of grace period, as expected while new_prod_pos - pend_pos is still below rb->mask. The last reservation that fits is the largest record boundary X with X + 160 <= 524287, and since 2^32 mod 160 = 96 the boundaries after the wrap sit at X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally until the producer stopped, then read zero records for good. With this patch applied, one of the four devices took 10 GiB through the same ring with no stall, while the three unpatched ones kept wedging at the same byte count. 64-bit hosts are unaffected in practice: their counters would need 16 EiB to wrap. Compare the two positions as a difference instead. pending_pos never runs ahead of producer_pos, so the unsigned difference is the real distance between them and stays correct across the wrap. Fixes: cfa1a2329a69 ("bpf: Fix overrunning reservations in ringbuf") Signed-off-by: Israel Téllez García Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814124843.22041-2-i.tellez@btesa.com --- kernel/bpf/ringbuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index c1bf7a197a96..99487019a8a8 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -482,7 +482,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) prod_pos = rb->producer_pos; new_prod_pos = prod_pos + len; - while (pend_pos < prod_pos) { + while (prod_pos - pend_pos > 0) { hdr = (void *)rb->data + (pend_pos & rb->mask); hdr_len = READ_ONCE(hdr->len); if (hdr_len & BPF_RINGBUF_BUSY_BIT) From 3f611e9b820ee0d01af89bb0643ccfac76cc569d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Israel=20T=C3=A9llez=20Garc=C3=ADa?= Date: Fri, 14 Aug 2026 14:48:41 +0200 Subject: [PATCH 318/373] bpf: Fix available-data accounting on 32-bit wrap in overwrite mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer and overwrite positions before measuring how much data is available: return prod_pos - max(cons_pos, over_pos); max() is an ordering comparison, and consumer_pos, producer_pos and overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures, where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the two positions has wrapped and the other has not, max() returns the older one: the result is then a modular difference close to 2^32, so the function reports far more available data than the ring can hold. Pollers using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be woken with nothing to read. Compare distances rather than positions. prod_pos - X is the amount of data produced since X for either position, wrap or no wrap, so the newer position is simply the one with the smaller distance, which is also the value the function wants to return. 64-bit hosts are unaffected in practice: their counters would need 16 EiB to wrap. Found by review of the same class of bug fixed in "bpf: Fix pending_pos walk on 32-bit ring position wrap". Signed-off-by: Israel Téllez García Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com --- kernel/bpf/ringbuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 99487019a8a8..3f1013d80544 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -321,7 +321,7 @@ static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb) if (unlikely(rb->overwrite_mode)) { over_pos = smp_load_acquire(&rb->overwrite_pos); prod_pos = smp_load_acquire(&rb->producer_pos); - return prod_pos - max(cons_pos, over_pos); + return min(prod_pos - cons_pos, prod_pos - over_pos); } else { prod_pos = smp_load_acquire(&rb->producer_pos); return prod_pos - cons_pos; From fdd4fad0bbbd08501465c5f9b556963093c5d58a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Israel=20T=C3=A9llez=20Garc=C3=ADa?= Date: Fri, 14 Aug 2026 14:48:43 +0200 Subject: [PATCH 319/373] libbpf: Fix ring buffer consumer loop on 32-bit position wrap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ringbuf_process_ring() walks the records between the consumer and the producer with an ordering comparison: while (cons_pos < prod_pos) { cons_pos and prod_pos mirror the kernel's ring positions and are unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic. When producer_pos has wrapped and consumer_pos has not, prod_pos is the smaller of the two, the loop body never runs and no record is consumed. Since consumer_pos only advances inside that loop, it never wraps either and the consumer stops delivering samples for good, with no error returned to the caller: ring_buffer__poll() keeps reporting zero records while the kernel side fills up and starts dropping. Compare the distance instead. The consumer never runs ahead of the producer, so prod_pos - cons_pos is the amount of unconsumed data and stays correct across the wrap. 64-bit hosts are unaffected in practice: the counters would need 16 EiB to wrap. This is the userspace counterpart of the kernel-side walk fixed in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit consumer hits whichever of the two comes first. Signed-off-by: Israel Téllez García Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com --- tools/lib/bpf/ringbuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c index 00ec4837a06d..c8fe57401a8b 100644 --- a/tools/lib/bpf/ringbuf.c +++ b/tools/lib/bpf/ringbuf.c @@ -244,7 +244,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) do { got_new_data = false; prod_pos = smp_load_acquire(r->producer_pos); - while (cons_pos < prod_pos) { + while (prod_pos - cons_pos > 0) { len_ptr = r->data + (cons_pos & r->mask); len = smp_load_acquire(len_ptr); From f5b57e9e9cfd9736246eb9a5f385da451d199039 Mon Sep 17 00:00:00 2001 From: Song Liu Date: Fri, 14 Aug 2026 08:56:23 -0700 Subject: [PATCH 320/373] bpf: Populate mmap-able array map memory lazily An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all). Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path. This makes mmap() O(1), and munmap() proportional to the number of pages that were actually faulted in rather than to the size of the map. The handler is reached through a new optional ->map_mmap_fault callback. Maps that provide it get a vm_operations_struct with a .fault handler; maps that populate their mapping eagerly keep the one they had. Both share the same open/close callbacks, so the existing VMA accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized rather than each map installing its own vm_operations_struct. Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE. Kernel-side access to the map (via the vmalloc address) is unaffected. Time for one mmap()+munmap() of an 8MiB mmap-able array map: before after no MAP_POPULATE, no access 226us 1.1us no MAP_POPULATE, access all pages 236us 1341us MAP_POPULATE, no access 312us 493us MAP_POPULATE, access all pages 318us 519us Mapping without touching the data, which is what this change targets, gets ~160x cheaper. Faulting in the whole mapping one page at a time is more expensive than the eager remap_vmalloc_range() loop, so users that do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE is not free before this change either: it adds ~85us (226us => 312us) for no benefit, as the mapping is already fully populated. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Song Liu Signed-off-by: Andrii Nakryiko Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org --- include/linux/bpf.h | 1 + kernel/bpf/arraymap.c | 34 ++++++++++++++++++++++++++++++---- kernel/bpf/syscall.c | 15 ++++++++++++++- 3 files changed, 45 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f4e8d372253a..04cadd987169 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -145,6 +145,7 @@ struct bpf_map_ops { int (*map_direct_value_meta)(const struct bpf_map *map, u64 imm, u32 *off); int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma); + vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf); __poll_t (*map_poll)(struct bpf_map *map, struct file *filp, struct poll_table_struct *pts); unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr, diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 34865701f7f7..ef315b168b29 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -608,17 +608,42 @@ static int array_map_check_btf(struct bpf_map *map, static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) { struct bpf_array *array = container_of(map, struct bpf_array, map); - pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT; if (!(map->map_flags & BPF_F_MMAPABLE)) return -EINVAL; - if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > + /* use u64 math so the offset cannot overflow on 32-bit archs */ + if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) > PAGE_ALIGN((u64)array->map.max_entries * array->elem_size)) return -EINVAL; - return remap_vmalloc_range(vma, array_map_vmalloc_addr(array), - vma->vm_pgoff + pgoff); + /* + * Pages are faulted in on demand by array_map_mmap_fault(). Set the + * same flags that the eager remap_vmalloc_range() path used to set + * via vm_insert_page(), so that e.g. NUMA balancing keeps skipping + * these VMAs. + */ + vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP); + + return 0; +} + +static vm_fault_t array_map_mmap_fault(struct bpf_map *map, + struct vm_fault *vmf) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + struct page *page; + + page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT)); + if (!page) + return VM_FAULT_SIGBUS; + + /* the eager remap_vmalloc_range() flushed via vm_insert_page() */ + flush_dcache_folio(page_folio(page)); + get_page(page); + vmf->page = page; + + return 0; } static bool array_map_meta_equal(const struct bpf_map *meta0, @@ -844,6 +869,7 @@ const struct bpf_map_ops array_map_ops = { .map_direct_value_addr = array_map_direct_value_addr, .map_direct_value_meta = array_map_direct_value_meta, .map_mmap = array_map_mmap, + .map_mmap_fault = array_map_mmap_fault, .map_seq_show_elem = array_map_seq_show_elem, .map_check_btf = array_map_check_btf, .map_lookup_batch = generic_map_lookup_batch, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7d8c3e8e6d62..6874ba1424af 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1076,11 +1076,24 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma) bpf_map_write_active_dec(map); } +static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf) +{ + struct bpf_map *map = vmf->vma->vm_private_data; + + return map->ops->map_mmap_fault(map, vmf); +} + static const struct vm_operations_struct bpf_map_default_vmops = { .open = bpf_map_mmap_open, .close = bpf_map_mmap_close, }; +static const struct vm_operations_struct bpf_map_lazy_vmops = { + .open = bpf_map_mmap_open, + .close = bpf_map_mmap_close, + .fault = bpf_map_mmap_fault, +}; + static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma) { struct bpf_map *map = filp->private_data; @@ -1116,7 +1129,7 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma) return err; /* set default open/close callbacks */ - vma->vm_ops = &bpf_map_default_vmops; + vma->vm_ops = map->ops->map_mmap_fault ? &bpf_map_lazy_vmops : &bpf_map_default_vmops; vma->vm_private_data = map; vm_flags_clear(vma, VM_MAYEXEC); /* If mapping is read-only, then disallow potentially re-mapping with From 5bbbce02e500d47d8e259a45be5a7be9741d0533 Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 14 Aug 2026 15:02:53 -0700 Subject: [PATCH 321/373] bpf, x86: Fix per-CPU address resolution into an extended register The destination of the per-CPU address MOV is encoded in ModRM.reg, which is extended by REX.R, but the REX prefix is built with add_1mod(), which sets REX.B. REX.B extends ModRM.rm and SIB.base, and this instruction addresses memory as disp32 with no base, so the bit has no effect at all and the high register bit is simply lost. Every is_ereg() destination therefore resolves to the wrong register, picking whichever one shares the low three bits: R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI With BPF_REG_5, whose reg2hex is 0, the emitted 65 49 03 04 25 add %gs:,%rax adds the per-CPU offset to RAX rather than R8. The destination keeps the unadjusted address and RAX is clobbered, so the program goes on to dereference a pointer that was never made per-CPU: BUG: unable to handle page fault for address: 0000607e386a8894 RIP: bpf_prog_707837aafd2aa9ae_update_percpu_data+0x93/0xc9 Call Trace: __bpf_prog_test_run_raw_tp+0x2dc/0x7d0 __flush_smp_call_function_queue+0x1e9/0xc80 Kernel panic - not syncing: Fatal exception in interrupt R5 is the mildest of the four, aliasing a scratch register and faulting at the store. R7 aliases RBP and would corrupt the frame pointer, R8 and R9 alias the argument registers. Use add_2mod() so the register goes through REX.R, matching how add_2reg() places it in ModRM.reg and how emit_priv_frame_ptr() hardcodes 0x4c for the same instruction with R9. Encodings for the non-extended registers are unchanged. Problem showed up when trying to resurrect BPF_GCC CI (selftests built with BPF_GCC). This has gone unnoticed because clang reloads the address into R1 before each per-CPU access, so the destination is never an extended register. GCC keeps several per-CPU addresses live at once, and test_progs-bpf_gcc panics the kernel in global_percpu_data/init, where the address of a .percpu variable ends up in R5. Fixes: 7bdbf7446305 ("bpf: add special internal-only MOV instruction to resolve per-CPU addrs") Signed-off-by: Vineet Gupta Reviewed-by: Eduard Zingerman Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260814220254.3797467-2-vineet.gupta@linux.dev Signed-off-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d920772af7d5..1a9fb530adc3 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1935,7 +1935,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * EMIT_mov(dst_reg, src_reg); #ifdef CONFIG_SMP /* add , gs:[] */ - EMIT2(0x65, add_1mod(0x48, dst_reg)); + EMIT2(0x65, add_2mod(0x48, 0, dst_reg)); EMIT3(0x03, add_2reg(0x04, 0, dst_reg), 0x25); EMIT((u32)(unsigned long)&this_cpu_off, 4); #endif From f61306e8c98ce63021efb3091d422260b6be25bb Mon Sep 17 00:00:00 2001 From: Vineet Gupta Date: Fri, 14 Aug 2026 15:02:54 -0700 Subject: [PATCH 322/373] selftests/bpf: Check per-CPU address resolution per register An ld_imm64 of a per-CPU map value is followed by a mov_percpu_addr that reuses the same register, so which register the address lands in decides how the JIT encodes the add. Getting the REX prefix wrong there is invisible to a functional test unless the address happens to land in an extended register, which is why this went unnoticed. Load a .percpu variable into every register in one program and match the JITed add against the register each one must resolve into. Signed-off-by: Vineet Gupta Link: https://patch.msgid.link/20260814220254.3797467-3-vineet.gupta@linux.dev Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_percpu_addr.c | 72 +++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_percpu_addr.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 8113fea7ba86..64ac49ad67e6 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -79,6 +79,7 @@ #include "verifier_netfilter_retcode.skel.h" #include "verifier_bpf_fastcall.skel.h" #include "verifier_or_jmp32_k.skel.h" +#include "verifier_percpu_addr.skel.h" #include "verifier_precision.skel.h" #include "verifier_prevent_map_lookup.skel.h" #include "verifier_private_stack.skel.h" @@ -240,6 +241,7 @@ void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); } void test_verifier_netfilter_retcode(void) { RUN(verifier_netfilter_retcode); } void test_verifier_bpf_fastcall(void) { RUN(verifier_bpf_fastcall); } void test_verifier_or_jmp32_k(void) { RUN(verifier_or_jmp32_k); } +void test_verifier_percpu_addr(void) { RUN(verifier_percpu_addr); } void test_verifier_precision(void) { RUN(verifier_precision); } void test_verifier_prevent_map_lookup(void) { RUN(verifier_prevent_map_lookup); } void test_verifier_private_stack(void) { RUN(verifier_private_stack); } diff --git a/tools/testing/selftests/bpf/progs/verifier_percpu_addr.c b/tools/testing/selftests/bpf/progs/verifier_percpu_addr.c new file mode 100644 index 000000000000..967f4e6e3a49 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_percpu_addr.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "bpf_misc.h" + +#if defined(__TARGET_ARCH_x86) + +int percpu_data SEC(".percpu"); + +/* + * An ld_imm64 of a per-CPU map value is followed by a mov_percpu_addr that + * reuses the same register, so check that the add resolves into the register + * the address was loaded into, for every register. + */ +SEC("raw_tp") +__description("per-CPU address resolution") +__success +__arch_x86_64 +__jited(" movabsq $0x{{.*}}, %rax") +__jited(" addq %gs:{{.*}}, %rax") +__jited(" movabsq $0x{{.*}}, %rdi") +__jited(" addq %gs:{{.*}}, %rdi") +__jited(" movabsq $0x{{.*}}, %rsi") +__jited(" addq %gs:{{.*}}, %rsi") +__jited(" movabsq $0x{{.*}}, %rdx") +__jited(" addq %gs:{{.*}}, %rdx") +__jited(" movabsq $0x{{.*}}, %rcx") +__jited(" addq %gs:{{.*}}, %rcx") +__jited(" movabsq $0x{{.*}}, %r8") +__jited(" addq %gs:{{.*}}, %r8") +__jited(" movabsq $0x{{.*}}, %rbx") +__jited(" addq %gs:{{.*}}, %rbx") +__jited(" movabsq $0x{{.*}}, %r13") +__jited(" addq %gs:{{.*}}, %r13") +__jited(" movabsq $0x{{.*}}, %r14") +__jited(" addq %gs:{{.*}}, %r14") +__jited(" movabsq $0x{{.*}}, %r15") +__jited(" addq %gs:{{.*}}, %r15") +__naked void percpu_addr(void) +{ + asm volatile (" \ + r0 = %[percpu_data] ll; \ + r1 = %[percpu_data] ll; \ + r2 = %[percpu_data] ll; \ + r3 = %[percpu_data] ll; \ + r4 = %[percpu_data] ll; \ + r5 = %[percpu_data] ll; \ + r6 = %[percpu_data] ll; \ + r7 = %[percpu_data] ll; \ + r8 = %[percpu_data] ll; \ + r9 = %[percpu_data] ll; \ + r0 = 0; \ + exit; \ +" : + : __imm_addr(percpu_data) + : __clobber_all); +} + +#else + +SEC("raw_tp") +__description("percpu addr dummy") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; From 5ad746166341e3c07250ee09518d7e4ab5cfb966 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:45:56 +0200 Subject: [PATCH 323/373] bpf: Add verifier diagnostics report helpers Add the initial diagnostics renderer for verifier reports and wire it into the BPF build. The helper emits the common failure header through the verifier log. Later patches add prose wrapping, reusable report sections, and source and instruction context for category-specific diagnostics. Gate the helpers on normal verifier log output from the start, so BPF_LOG_STATS-only loads do not collect or render diagnostics. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-2-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/Makefile | 2 +- kernel/bpf/diagnostics.c | 47 ++++++++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 14 ++++++++++++ 3 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/diagnostics.c create mode 100644 kernel/bpf/diagnostics.h diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 4dc41bf5780c..90255d80e5be 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse endif CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) -obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c new file mode 100644 index 000000000000..e75753552a4d --- /dev/null +++ b/kernel/bpf/diagnostics.c @@ -0,0 +1,47 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright (c) 2026 Meta Platforms, Inc. and affiliates. + +#include +#include +#include + +#include "diagnostics.h" + +bool bpf_diag_enabled(const struct bpf_verifier_env *env) +{ + return env->log.level & BPF_LOG_LEVEL; +} + +static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); + +static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) +{ + va_list args; + + if (!bpf_diag_enabled(env)) + return; + + va_start(args, fmt); + bpf_verifier_vlog(&env->log, fmt, args); + va_end(args); +} + +static void bpf_diag_header(struct bpf_verifier_env *env, const char *category, + const char *problem) +{ + char first; + + if (!bpf_diag_enabled(env)) + return; + + category = category ?: "Verifier Error"; + problem = problem ?: ""; + + if (!problem[0]) { + diag_write(env, "\nVerification failed: %s\n", category); + return; + } + + first = toupper(problem[0]); + diag_write(env, "\nVerification failed: %s: %c%s\n", category, first, problem + 1); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h new file mode 100644 index 000000000000..f51aa39f0909 --- /dev/null +++ b/kernel/bpf/diagnostics.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#ifndef __BPF_DIAGNOSTICS_H +#define __BPF_DIAGNOSTICS_H + +#include +#include + +struct bpf_verifier_env; + +bool bpf_diag_enabled(const struct bpf_verifier_env *env); + +#endif /* __BPF_DIAGNOSTICS_H */ From b9c5d822f677e065971481c4bafe5d84b5451082 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:45:57 +0200 Subject: [PATCH 324/373] bpf: Add source and instruction diagnostic context Teach verifier diagnostics to annotate an instruction with BTF source line information and nearby BPF instructions. The renderer keeps source text in a fixed-width lane and prints instructions in a stable right-hand gutter. Wrap annotation text under the source line so long error labels remain readable while the source and instruction lanes keep their fixed layout. Keeping source and instruction context in one commit preserves the visual layout contract that later diagnostic reports rely on. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- include/linux/bpf.h | 12 +- include/linux/bpf_verifier.h | 4 + include/linux/btf.h | 1 + kernel/bpf/btf.c | 10 + kernel/bpf/core.c | 35 ++- kernel/bpf/diagnostics.c | 490 +++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 7 + kernel/bpf/verifier.c | 47 +++- 8 files changed, 576 insertions(+), 30 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 04cadd987169..ffa5626411ac 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -4147,8 +4147,16 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) } const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off); -void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo, - const char **filep, const char **linep, int *nump); +struct bpf_linfo_source { + const char *file; + const char *line; + u32 file_name_off; + int line_num; + int line_col; +}; + +void bpf_get_linfo_source(struct btf *btf, const struct bpf_line_info *linfo, + struct bpf_linfo_source *src); int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep, const char **linep, int *nump); struct bpf_prog *bpf_prog_find_from_stack(void); diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 27b43fda9b17..579a288bc8de 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -833,6 +833,7 @@ static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub) return 0; } +struct bpf_diag; struct bpf_verifier_env; struct backtrack_state { @@ -950,6 +951,7 @@ struct bpf_verifier_env { struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; + struct bpf_diag *diag; struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */ /* subprog indices sorted in topological order: leaves first, callers last */ int subprog_topo_order[BPF_MAX_SUBPROGS + 2]; @@ -1433,8 +1435,10 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate, u32 frameno); u32 bpf_vlog_alignment(u32 pos); +const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn); struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off); +const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog); int bpf_jmp_offset(struct bpf_insn *insn); struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx); void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask); diff --git a/include/linux/btf.h b/include/linux/btf.h index 3f5255d095a2..7ea13768c979 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -214,6 +214,7 @@ int btf_type_seq_show_flags(const struct btf *btf, u32 type_id, void *obj, */ int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj, char *buf, int len, u64 flags); +int btf_type_name_to_buf(const struct btf *btf, u32 type_id, char *buf, int len); int btf_get_fd_by_id(u32 id); u32 btf_obj_id(const struct btf *btf); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 87ffde865a50..5b9d767895c9 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -8316,6 +8316,16 @@ int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj, return ssnprintf.len; } +int btf_type_name_to_buf(const struct btf *btf, u32 type_id, char *buf, int len) +{ + struct btf_show show = { + .btf = btf, + .state.type_id = type_id, + }; + + return snprintf(buf, len, "%s", btf_show_name(&show)); +} + #ifdef CONFIG_PROC_FS static void bpf_btf_show_fdinfo(struct seq_file *m, struct file *filp) { diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6a94370a2448..d55e737ed75a 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3461,24 +3461,14 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(xdp_bulk_tx); #ifdef CONFIG_BPF_SYSCALL -void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo, - const char **filep, const char **linep, int *nump) +void bpf_get_linfo_source(struct btf *btf, const struct bpf_line_info *linfo, + struct bpf_linfo_source *src) { - /* Get base component of the file path. */ - if (filep) { - *filep = btf_name_by_offset(btf, linfo->file_name_off); - *filep = kbasename(*filep); - } - - /* Obtain the source line, and strip whitespace in prefix. */ - if (linep) { - *linep = btf_name_by_offset(btf, linfo->line_off); - while (isspace(**linep)) - *linep += 1; - } - - if (nump) - *nump = BPF_LINE_INFO_LINE_NUM(linfo->line_col); + src->file = kbasename(btf_name_by_offset(btf, linfo->file_name_off)); + src->line = btf_name_by_offset(btf, linfo->line_off); + src->file_name_off = linfo->file_name_off; + src->line_num = BPF_LINE_INFO_LINE_NUM(linfo->line_col); + src->line_col = BPF_LINE_INFO_LINE_COL(linfo->line_col); } const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off) @@ -3521,6 +3511,7 @@ const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep, const char **linep, int *nump) { + struct bpf_linfo_source src; int idx = -1, insn_start, insn_end, len; struct bpf_line_info *linfo; void **jited_linfo; @@ -3552,7 +3543,15 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char * if (idx == -1) return -ENOENT; - bpf_get_linfo_file_line(btf, &linfo[idx], filep, linep, nump); + bpf_get_linfo_source(btf, &linfo[idx], &src); + while (isspace(*src.line)) + src.line++; + if (filep) + *filep = src.file; + if (linep) + *linep = src.line; + if (nump) + *nump = src.line_num; return 0; } diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index e75753552a4d..815aa7938b50 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -1,12 +1,61 @@ // SPDX-License-Identifier: GPL-2.0-only // Copyright (c) 2026 Meta Platforms, Inc. and affiliates. +#include #include +#include #include +#include +#include +#include +#include #include +#include +#include "disasm.h" #include "diagnostics.h" +#define BPF_DIAG_TEXT_WIDTH 100 +#define BPF_DIAG_CONTEXT 2 +#define BPF_DIAG_CONTEXT_CNT (1 + BPF_DIAG_CONTEXT * 2) +#define BPF_DIAG_SOURCE_LANE_WIDTH 88 +#define BPF_DIAG_TAB_WIDTH 8 +#define BPF_DIAG_FMT_CHUNK_SIZE (PAGE_SIZE - sizeof(struct diag_fmt_chunk)) +#define BPF_DIAG_FMT_BUF_SIZE 256 +#define DISASM_LINE_LEN 160 + +struct disasm_line { + char text[DISASM_LINE_LEN]; + int idx; + bool valid; +}; + +struct disasm_ctx { + struct bpf_verifier_env *env; + struct seq_buf seq; +}; + +struct diag_fmt_chunk { + struct list_head node; + struct seq_buf seq; + char data[]; +}; + +struct diag_fmt_mark { + struct diag_fmt_chunk *chunk; + size_t len; +}; + +struct bpf_diag_scratch { + struct bpf_linfo_source source_lines[BPF_DIAG_CONTEXT_CNT]; + struct disasm_line disasm_lines[BPF_DIAG_CONTEXT_CNT]; +}; + +struct bpf_diag { + struct bpf_diag_scratch scratch; + struct list_head fmt_chunks; +}; + bool bpf_diag_enabled(const struct bpf_verifier_env *env) { return env->log.level & BPF_LOG_LEVEL; @@ -14,6 +63,138 @@ bool bpf_diag_enabled(const struct bpf_verifier_env *env) static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); +int bpf_diag_init(struct bpf_verifier_env *env) +{ + if (!bpf_diag_enabled(env)) + return 0; + + env->diag = kzalloc_obj(struct bpf_diag, GFP_KERNEL_ACCOUNT); + if (!env->diag) + return -ENOMEM; + + INIT_LIST_HEAD(&env->diag->fmt_chunks); + return 0; +} + +static char *diag_fmt_alloc(struct bpf_verifier_env *env, size_t size) +{ + struct bpf_diag *diag = env->diag; + struct diag_fmt_chunk *chunk; + size_t capacity, available; + char *buf; + + if (!diag || !size || size > INT_MAX) + return NULL; + + if (!list_empty(&diag->fmt_chunks)) { + chunk = list_last_entry(&diag->fmt_chunks, struct diag_fmt_chunk, node); + available = seq_buf_get_buf(&chunk->seq, &buf); + if (available >= size) + goto commit; + } + + capacity = max_t(size_t, BPF_DIAG_FMT_CHUNK_SIZE, size); + chunk = kmalloc(struct_size(chunk, data, capacity), GFP_KERNEL_ACCOUNT); + if (!chunk) + return NULL; + + seq_buf_init(&chunk->seq, chunk->data, capacity); + list_add_tail(&chunk->node, &diag->fmt_chunks); + available = seq_buf_get_buf(&chunk->seq, &buf); + if (WARN_ON_ONCE(available < size)) + return NULL; + +commit: + seq_buf_commit(&chunk->seq, size); + return buf; +} + +char *bpf_diag_fmt_buf(struct bpf_verifier_env *env, size_t size) +{ + char *buf; + + buf = diag_fmt_alloc(env, size); + if (buf) + buf[0] = '\0'; + return buf; +} + +const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list args) +{ + va_list copy; + char *buf; + int len; + + va_copy(copy, args); + len = vsnprintf(NULL, 0, fmt, copy); + va_end(copy); + if (len < 0 || len == INT_MAX) + return ""; + + buf = diag_fmt_alloc(env, len + 1); + if (buf) + vsnprintf(buf, len + 1, fmt, args); + return buf ?: ""; +} + +const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) +{ + const char *buf; + va_list args; + + va_start(args, fmt); + buf = bpf_diag_vfmt(env, fmt, args); + va_end(args); + return buf; +} + +static struct diag_fmt_mark diag_fmt_save(struct bpf_verifier_env *env) +{ + struct bpf_diag *diag = env->diag; + struct diag_fmt_mark mark = {}; + + if (!diag || list_empty(&diag->fmt_chunks)) + return mark; + + mark.chunk = list_last_entry(&diag->fmt_chunks, struct diag_fmt_chunk, node); + mark.len = mark.chunk->seq.len; + return mark; +} + +static void diag_fmt_restore(struct bpf_verifier_env *env, struct diag_fmt_mark mark) +{ + struct bpf_diag *diag = env->diag; + struct diag_fmt_chunk *chunk; + + if (!diag) + return; + + while (!list_empty(&diag->fmt_chunks)) { + chunk = list_last_entry(&diag->fmt_chunks, struct diag_fmt_chunk, node); + if (chunk == mark.chunk) + break; + list_del(&chunk->node); + kfree(chunk); + } + + if (mark.chunk) { + mark.chunk->seq.len = mark.len; + seq_buf_str(&mark.chunk->seq); + } +} + +void bpf_diag_free(struct bpf_verifier_env *env) +{ + struct bpf_diag *diag = env->diag; + + if (!diag) + return; + + diag_fmt_restore(env, (struct diag_fmt_mark){}); + kfree(diag); + env->diag = NULL; +} + static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) { va_list args; @@ -26,6 +207,179 @@ static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) va_end(args); } +static void diag_print_wrapped_prefixed(struct bpf_verifier_env *env, const char *first_prefix, + const char *next_prefix, const char *text) +{ + const char *prefix = first_prefix; + + while (*text) { + const char *line = text; + int prefix_len = strlen(prefix); + int text_width = BPF_DIAG_TEXT_WIDTH - prefix_len; + int len = 0, last_space = -1; + + if (text_width < 1) + text_width = 1; + + while (line[len] && line[len] != '\n' && len < text_width) { + if (line[len] == ' ') + last_space = len; + len++; + } + + if (line[len] && line[len] != '\n' && line[len] != ' ' && last_space > 0) + len = last_space; + + diag_write(env, "%s%.*s\n", prefix, len, line); + + text = line + len; + while (*text == ' ') + text++; + if (*text == '\n') + text++; + + prefix = next_prefix; + } +} + +static int diag_line_width(unsigned int line) +{ + int width = 1; + + while (line >= 10) { + line /= 10; + width++; + } + + return width; +} + +static int diag_line_indent(const char *line) +{ + int indent = 0; + + while (*line == ' ' || *line == '\t') { + if (*line == '\t') + indent = round_up(indent + 1, BPF_DIAG_TAB_WIDTH); + else + indent++; + line++; + } + + return indent; +} + +static void disasm_print(void *private_data, const char *fmt, ...) __printf(2, 3); + +static void disasm_print(void *private_data, const char *fmt, ...) +{ + struct disasm_ctx *ctx = private_data; + va_list args; + + va_start(args, fmt); + seq_buf_vprintf(&ctx->seq, fmt, args); + va_end(args); +} + +static const char *disasm_kfunc_name(void *private_data, const struct bpf_insn *insn) +{ + struct disasm_ctx *ctx = private_data; + + return bpf_disasm_kfunc_name(ctx->env, insn); +} + +static void format_disasm_line(struct bpf_verifier_env *env, int insn_idx, + struct disasm_line *line) +{ + struct disasm_ctx ctx = { .env = env }; + struct bpf_insn *insn; + const struct bpf_insn_cbs cbs = { + .cb_call = disasm_kfunc_name, + .cb_print = disasm_print, + .private_data = &ctx, + }; + + line->idx = insn_idx; + line->valid = false; + seq_buf_init(&ctx.seq, line->text, sizeof(line->text)); + + if (insn_idx < 0 || insn_idx >= env->prog->len) + return; + + if (insn_idx > 0 && bpf_is_ldimm64(&env->prog->insnsi[insn_idx - 1])) + return; + + insn = &env->prog->insnsi[insn_idx]; + if (bpf_is_ldimm64(insn) && insn_idx + 1 >= env->prog->len) + return; + + print_bpf_insn(&cbs, insn, env->allow_ptr_leaks); + seq_buf_str(&ctx.seq); + ctx.seq.len = strnlen(line->text, sizeof(line->text)); + while (ctx.seq.len && line->text[ctx.seq.len - 1] == '\n') + seq_buf_pop(&ctx.seq); + seq_buf_str(&ctx.seq); + + line->valid = true; +} + +static void diag_format_source_text(char *buf, size_t size, const char *line, int width) +{ + int col = 0, len = 0; + + if (!size) + return; + if (width <= 0) { + buf[0] = '\0'; + return; + } + + line = line ?: "..."; + while (*line && col < width && len + 1 < size) { + if (*line == '\t') { + int next = round_up(col + 1, BPF_DIAG_TAB_WIDTH); + + while (col < next && col < width && len + 1 < size) { + buf[len++] = ' '; + col++; + } + line++; + continue; + } + + buf[len++] = *line++; + col++; + } + + if (*line) { + int ellipsis_len = min(3, width); + + while (len > 0 && col > width - ellipsis_len) { + len--; + col--; + } + while (ellipsis_len-- && len + 1 < size) + buf[len++] = '.'; + } + + buf[len] = '\0'; +} + +static void diag_format_source_lane(char *buf, size_t size, const char *source_prefix, + int source_line_width, int line_num, const char *line) +{ + int len, text_width; + + if (line_num <= 0) { + buf[0] = '\0'; + return; + } + + len = scnprintf(buf, size, "%s%*d | ", source_prefix, source_line_width, line_num); + text_width = BPF_DIAG_SOURCE_LANE_WIDTH - len; + diag_format_source_text(buf + len, size - len, line, text_width); +} + static void bpf_diag_header(struct bpf_verifier_env *env, const char *category, const char *problem) { @@ -45,3 +399,139 @@ static void bpf_diag_header(struct bpf_verifier_env *env, const char *category, first = toupper(problem[0]); diag_write(env, "\nVerification failed: %s: %c%s\n", category, first, problem + 1); } + +static void diag_print_source_annotation(struct bpf_verifier_env *env, int line_width, int indent, + const char *label, const char *msg) +{ + const char *first_prefix, *next_prefix, *text; + + indent = min_t(int, indent, max_t(int, 0, BPF_DIAG_SOURCE_LANE_WIDTH - line_width - 8)); + text = bpf_diag_fmt(env, "%s: %s", label, msg); + first_prefix = bpf_diag_fmt(env, " %*s | %*s^-- ", line_width + 4, "", indent, ""); + next_prefix = bpf_diag_fmt(env, " %*s | %*s ", line_width + 4, "", indent, ""); + + diag_print_wrapped_prefixed(env, first_prefix, next_prefix, text); +} + +static void diag_print_insn_context(struct bpf_verifier_env *env, u32 insn_idx, + struct disasm_line *disasm_lines) +{ + int insn_width = diag_line_width(env->prog->len ? env->prog->len - 1 : 0); + int i; + + for (i = 0; i < BPF_DIAG_CONTEXT_CNT; i++) { + int row = i - BPF_DIAG_CONTEXT; + + format_disasm_line(env, insn_idx + row, &disasm_lines[i]); + } + + diag_write(env, " Instruction context:\n"); + for (i = 0; i < BPF_DIAG_CONTEXT_CNT; i++) { + struct disasm_line *line = &disasm_lines[i]; + + if (line->valid) + diag_write(env, " %s%*d | %s\n", + line->idx == insn_idx ? ">>> " : " ", + insn_width, line->idx, line->text); + } +} + +static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const char *label, + const char *fmt, ...) +{ + struct bpf_diag_scratch *scratch; + struct bpf_linfo_source *source_lines; + struct disasm_line *disasm_lines; + struct bpf_linfo_source src = {}; + struct diag_fmt_mark mark; + const struct bpf_line_info *linfo; + const struct bpf_subprog_info *subprog; + struct btf *btf = env->prog->aux->btf; + char *source_lane; + const char *msg; + const char *func; + int start_line, end_line, width, indent, subprogno, linfo_start, linfo_end, i; + va_list args; + + if (!bpf_diag_enabled(env)) + return; + if (!env->diag) + return; + + mark = diag_fmt_save(env); + label = label ?: "note"; + scratch = &env->diag->scratch; + source_lines = scratch->source_lines; + disasm_lines = scratch->disasm_lines; + memset(source_lines, 0, sizeof(scratch->source_lines)); + memset(disasm_lines, 0, sizeof(scratch->disasm_lines)); + + va_start(args, fmt); + msg = bpf_diag_vfmt(env, fmt, args); + va_end(args); + if (!*msg) + msg = ""; + + linfo = bpf_find_linfo(env->prog, insn_idx); + if (btf && linfo) + bpf_get_linfo_source(btf, linfo, &src); + if (!src.file || !*src.file || !src.line || !*src.line) { + diag_write(env, " insn %u\n", insn_idx); + diag_print_source_annotation(env, 0, 0, label, msg); + diag_print_insn_context(env, insn_idx, disasm_lines); + goto out_restore; + } + + subprog = bpf_find_containing_subprog(env, insn_idx); + subprogno = subprog ? subprog - env->subprog_info : -ENOENT; + func = subprogno >= 0 ? bpf_subprog_name(env, subprogno) : NULL; + if (func && *func) + diag_write(env, " %s @ %s:%d:%d\n", func, src.file, src.line_num, src.line_col); + else + diag_write(env, " %s:%d:%d\n", src.file, src.line_num, src.line_col); + + start_line = src.line_num - BPF_DIAG_CONTEXT; + end_line = src.line_num + BPF_DIAG_CONTEXT; + width = diag_line_width(end_line); + indent = diag_line_indent(src.line); + for (i = 0; i < BPF_DIAG_CONTEXT_CNT; i++) + source_lines[i].line_num = start_line + i; + + linfo = env->prog->aux->linfo; + linfo_start = subprog ? subprog->linfo_idx : 0; + linfo_end = subprogno >= 0 && subprogno + 1 < env->subprog_cnt ? + env->subprog_info[subprogno + 1].linfo_idx : env->prog->aux->nr_linfo; + for (i = linfo_start; i < linfo_end; i++) { + struct bpf_linfo_source line_src; + int idx; + + bpf_get_linfo_source(btf, &linfo[i], &line_src); + if (line_src.file_name_off != src.file_name_off || + line_src.line_num < start_line || line_src.line_num > end_line || + !line_src.line || !*line_src.line) + continue; + + idx = line_src.line_num - start_line; + if (!source_lines[idx].line) + source_lines[idx] = line_src; + } + + diag_write(env, " Source context:\n"); + source_lane = bpf_diag_fmt_buf(env, BPF_DIAG_FMT_BUF_SIZE); + if (!source_lane) + goto out_restore; + for (i = 0; i < BPF_DIAG_CONTEXT_CNT; i++) { + const char *source_prefix; + + source_prefix = source_lines[i].line_num == src.line_num ? ">>> " : " "; + diag_format_source_lane(source_lane, BPF_DIAG_FMT_BUF_SIZE, source_prefix, width, + source_lines[i].line_num, source_lines[i].line); + diag_write(env, " %s\n", source_lane); + if (source_lines[i].line_num == src.line_num) + diag_print_source_annotation(env, width, indent, label, msg); + } + diag_print_insn_context(env, insn_idx, disasm_lines); + +out_restore: + diag_fmt_restore(env, mark); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index f51aa39f0909..ba268b589ac9 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -5,10 +5,17 @@ #define __BPF_DIAGNOSTICS_H #include +#include #include struct bpf_verifier_env; bool bpf_diag_enabled(const struct bpf_verifier_env *env); +int bpf_diag_init(struct bpf_verifier_env *env); +char *bpf_diag_fmt_buf(struct bpf_verifier_env *env, size_t size); +const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list args) + __printf(2, 0); +const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); +void bpf_diag_free(struct bpf_verifier_env *env); #endif /* __BPF_DIAGNOSTICS_H */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6ac1afced20b..2f330230f8d5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -34,6 +34,7 @@ #include #include +#include "diagnostics.h" #include "disasm.h" static const struct bpf_verifier_ops * const bpf_verifier_ops[] = { @@ -405,7 +406,7 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog) return btf_type_is_void(type); } -static const char *subprog_name(const struct bpf_verifier_env *env, int subprog) +const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog) { struct bpf_func_info *info; @@ -2624,6 +2625,26 @@ static struct btf *find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset) return btf_vmlinux ?: ERR_PTR(-ENOENT); } +static struct btf *find_kfunc_desc_btf_cached(struct bpf_verifier_env *env, s16 offset) +{ + struct bpf_kfunc_btf kf_btf = { .offset = offset }; + struct bpf_kfunc_btf_tab *tab; + struct bpf_kfunc_btf *b; + + if (!offset) + return btf_vmlinux ?: ERR_PTR(-ENOENT); + if (offset < 0) + return ERR_PTR(-EINVAL); + + tab = env->prog->aux->kfunc_btf_tab; + if (!tab) + return ERR_PTR(-ENOENT); + + b = bsearch(&kf_btf, tab->descs, tab->nr_descs, + sizeof(tab->descs[0]), kfunc_btf_cmp_by_off); + return b ? b->btf : ERR_PTR(-ENOENT); +} + #define KF_IMPL_SUFFIX "_impl" static const struct btf_type *find_kfunc_impl_proto(struct bpf_verifier_log *log, @@ -3031,8 +3052,8 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env) if (bpf_pseudo_func(&insn[idx])) continue; verbose(env, "recursive call from %s() to %s()\n", - subprog_name(env, cur), - subprog_name(env, callee)); + bpf_subprog_name(env, cur), + bpf_subprog_name(env, callee)); ret = -EINVAL; goto out; } @@ -3053,7 +3074,7 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env) if (env->log.level & BPF_LOG_LEVEL2) for (i = 0; i < cnt; i++) verbose(env, "topo_order[%d] = %s\n", - i, subprog_name(env, env->subprog_topo_order[i])); + i, bpf_subprog_name(env, env->subprog_topo_order[i])); out: kvfree(dfs_stack); kvfree(color); @@ -3197,7 +3218,7 @@ static void linked_regs_unpack(u64 val, struct linked_regs *s) } } -static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) +const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn) { const struct btf_type *func; struct btf *desc_btf; @@ -3205,18 +3226,20 @@ static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) if (insn->src_reg != BPF_PSEUDO_KFUNC_CALL) return NULL; - desc_btf = find_kfunc_desc_btf(data, insn->off); + desc_btf = find_kfunc_desc_btf_cached(data, insn->off); if (IS_ERR(desc_btf)) return ""; func = btf_type_by_id(desc_btf, insn->imm); + if (!func || !btf_type_is_func(func)) + return ""; return btf_name_by_offset(desc_btf, func->name_off); } void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn) { const struct bpf_insn_cbs cbs = { - .cb_call = disasm_kfunc_name, + .cb_call = bpf_disasm_kfunc_name, .cb_print = verbose, .private_data = env, }; @@ -9408,7 +9431,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (err == -EFAULT) return err; if (bpf_subprog_is_global(env, subprog)) { - const char *sub_name = subprog_name(env, subprog); + const char *sub_name = bpf_subprog_name(env, subprog); if (env->cur_state->active_locks) { verbose(env, "global function calls are not allowed while holding a lock,\n" @@ -18479,7 +18502,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) regs = state->frame[state->curframe]->regs; if (subprog || env->prog->type == BPF_PROG_TYPE_EXT) { - const char *sub_name = subprog_name(env, subprog); + const char *sub_name = bpf_subprog_name(env, subprog); struct bpf_subprog_arg_info *arg; struct bpf_reg_state *reg; @@ -18656,7 +18679,7 @@ static int do_check_subprogs(struct bpf_verifier_env *env) return ret; } else if (env->log.level & BPF_LOG_LEVEL) { verbose(env, "Func#%d ('%s') is safe for any args that match its prototype\n", - i, subprog_name(env, i)); + i, bpf_subprog_name(env, i)); } /* We verified new global subprog, it might have called some @@ -20188,6 +20211,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, ret = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size); if (ret) goto err_free_env; + ret = bpf_diag_init(env); + if (ret) + goto err_prep; if (env->signature) { ret = bpf_prog_calc_tag(env->prog); if (ret < 0) @@ -20478,6 +20504,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, kvfree(env->scc_info); kvfree(env->succ); kvfree(env->gotox_tmp_buf); + bpf_diag_free(env); kvfree(env); return ret; } From daf8248701b621d8df7ea134793dbb750d4887c0 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:45:58 +0200 Subject: [PATCH 325/373] bpf: Add verifier diagnostic event log Add an environment-owned diagnostic history for verifier reports. Event payloads keep the user-facing branch history shape, while storage lives in bpf_verifier_env and follows the active verifier path. Grow the event array geometrically up to a 64 MiB limit. Once storage reaches the limit, or an allocation fails, overwrite the oldest event so diagnostics retain the newest useful suffix without adding per-event metadata. Represent saved positions as absolute logical sequence numbers. A restore truncates to a retained position. If its prefix has already been evicted, clear the abandoned suffix and preserve the missing-history position. This keeps marks stable across rotation without increasing their size. Add the branch event renderer and branch recording. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-4-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 130 +++++++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 3 + kernel/bpf/verifier.c | 21 +++++++ 3 files changed, 154 insertions(+) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 815aa7938b50..8f21b46adeca 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -22,8 +22,24 @@ #define BPF_DIAG_TAB_WIDTH 8 #define BPF_DIAG_FMT_CHUNK_SIZE (PAGE_SIZE - sizeof(struct diag_fmt_chunk)) #define BPF_DIAG_FMT_BUF_SIZE 256 +#define BPF_DIAG_EVENT_LOG_MAX_SIZE (64U << 20) #define DISASM_LINE_LEN 160 +enum bpf_diag_history_kind { + BPF_DIAG_HISTORY_BRANCH, +}; + +struct bpf_diag_history_event { + u32 insn_idx : 24; + u32 kind : 8; + u8 in_lineage : 1; + union { + struct { + bool cond_true; + } branch; + }; +}; + struct disasm_line { char text[DISASM_LINE_LEN]; int idx; @@ -46,12 +62,23 @@ struct diag_fmt_mark { size_t len; }; +struct bpf_diag_log { + struct bpf_diag_history_event *events; + /* Sequence number of the oldest retained event on the active path. */ + u64 first_seq; + u32 cnt; + u32 cap; + u32 head; + bool growth_failed; +}; + struct bpf_diag_scratch { struct bpf_linfo_source source_lines[BPF_DIAG_CONTEXT_CNT]; struct disasm_line disasm_lines[BPF_DIAG_CONTEXT_CNT]; }; struct bpf_diag { + struct bpf_diag_log log; struct bpf_diag_scratch scratch; struct list_head fmt_chunks; }; @@ -191,6 +218,7 @@ void bpf_diag_free(struct bpf_verifier_env *env) return; diag_fmt_restore(env, (struct diag_fmt_mark){}); + kvfree(diag->log.events); kfree(diag); env->diag = NULL; } @@ -207,6 +235,95 @@ static void diag_write(struct bpf_verifier_env *env, const char *fmt, ...) va_end(args); } +static u64 log_end(const struct bpf_diag_log *log) +{ + return log->first_seq + log->cnt; +} + +static u32 log_pos(const struct bpf_diag_log *log, u32 idx) +{ + u32 pos = log->head + idx; + + return pos < log->cap ? pos : pos - log->cap; +} + +u64 bpf_diag_event_log_save(struct bpf_verifier_env *env) +{ + struct bpf_diag *diag = env->diag; + + return diag ? log_end(&diag->log) : 0; +} + +void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos) +{ + struct bpf_diag *diag = env->diag; + struct bpf_diag_log *log; + u64 end_seq; + + if (!diag) + return; + + log = &diag->log; + end_seq = log_end(log); + if (WARN_ON_ONCE(log_pos > end_seq)) + log_pos = end_seq; + + /* + * A deep abandoned path may have rotated away the shared prefix. In + * that case, restart with an empty retained suffix and remember that + * every event before the restored mark is unavailable. + */ + if (log_pos <= log->first_seq) { + log->first_seq = log_pos; + log->head = 0; + log->cnt = 0; + return; + } + + log->cnt = log_pos - log->first_seq; +} + +static void diag_append_history(struct bpf_verifier_env *env, + const struct bpf_diag_history_event *event) +{ + struct bpf_diag_history_event *events; + struct bpf_diag *diag = env->diag; + struct bpf_diag_log *log; + u32 cap, max_events; + + if (!diag) + return; + log = &diag->log; + + if (log->cnt < log->cap) { + log->events[log_pos(log, log->cnt++)] = *event; + return; + } + + max_events = BPF_DIAG_EVENT_LOG_MAX_SIZE / sizeof(*events); + if (log->growth_failed || log->cap == max_events) + goto rotate; + + cap = min(log->cap ? log->cap * 2 : 64, max_events); + events = kvrealloc(log->events, array_size(cap, sizeof(*events)), GFP_KERNEL_ACCOUNT); + if (!events) { + log->growth_failed = true; + goto rotate; + } + log->events = events; + log->cap = cap; + log->events[log->cnt++] = *event; + return; + +rotate: + if (log->cap) { + log->events[log->head++] = *event; + if (log->head == log->cap) + log->head = 0; + } + log->first_seq++; +} + static void diag_print_wrapped_prefixed(struct bpf_verifier_env *env, const char *first_prefix, const char *next_prefix, const char *text) { @@ -535,3 +652,16 @@ static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const ch out_restore: diag_fmt_restore(env, mark); } + +void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true) +{ + struct bpf_diag_history_event event = { + .insn_idx = insn_idx, + .kind = BPF_DIAG_HISTORY_BRANCH, + .branch = { + .cond_true = cond_true, + }, + }; + + diag_append_history(env, &event); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index ba268b589ac9..6eda2fd65ee1 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -16,6 +16,9 @@ char *bpf_diag_fmt_buf(struct bpf_verifier_env *env, size_t size); const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list args) __printf(2, 0); const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); +u64 bpf_diag_event_log_save(struct bpf_verifier_env *env); +void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos); void bpf_diag_free(struct bpf_verifier_env *env); +void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); #endif /* __BPF_DIAGNOSTICS_H */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2f330230f8d5..60dcb87a2417 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17439,6 +17439,27 @@ static int do_check(struct bpf_verifier_env *env) state->last_insn_idx = env->prev_insn_idx; state->insn_idx = env->insn_idx; + /* + * Record the incoming edge so active and queued paths use the same + * branch-recording path. A zero-offset conditional has identical + * successors, so its outcome cannot be reconstructed from the edge. + */ + if (!state->speculative && prev_insn_idx >= 0 && prev_insn_idx < insn_cnt) { + struct bpf_insn *prev_insn = &insns[prev_insn_idx]; + int fallthrough_idx = prev_insn_idx + 1; + int branch_idx = prev_insn_idx + bpf_jmp_offset(prev_insn) + 1; + u8 class = BPF_CLASS(prev_insn->code); + u8 opcode = BPF_OP(prev_insn->code); + + if ((class == BPF_JMP || class == BPF_JMP32) && + opcode != BPF_JA && opcode != BPF_CALL && opcode != BPF_EXIT && + opcode <= BPF_JCOND && branch_idx != fallthrough_idx) { + if (env->insn_idx == branch_idx) + bpf_diag_record_branch(env, prev_insn_idx, true); + else if (env->insn_idx == fallthrough_idx) + bpf_diag_record_branch(env, prev_insn_idx, false); + } + } if (bpf_is_prune_point(env, env->insn_idx)) { err = bpf_is_state_visited(env, env->insn_idx); From a6debd5f25c9c79f534074e9cb460cf6495d6daf Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:45:59 +0200 Subject: [PATCH 326/373] bpf: Prune verifier diagnostics when switching paths Save the diagnostic event-log position with each verifier stack entry and reset the environment-owned stream together with the normal verifier log when a queued state is popped. Also reset the diagnostic stream after successful subprogram verification even when level-2 logging preserves the normal verifier log. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-5-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 60dcb87a2417..db644690ac4b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -194,6 +194,7 @@ struct bpf_verifier_stack_elem { struct bpf_verifier_stack_elem *next; /* length of verifier log at the time this state was pushed on stack */ u32 log_pos; + u64 diag_log_pos; }; #define BPF_COMPLEXITY_LIMIT_JMP_SEQ 8192 @@ -1700,6 +1701,7 @@ static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx, err = bpf_copy_verifier_state(cur, &head->st); if (err) return err; + bpf_diag_event_log_restore(env, head->diag_log_pos); } if (pop_log) bpf_vlog_reset(&env->log, head->log_pos); @@ -1743,6 +1745,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, elem->prev_insn_idx = prev_insn_idx; elem->next = env->head; elem->log_pos = env->log.end_pos; + elem->diag_log_pos = bpf_diag_event_log_save(env); env->head = elem; env->stack_size++; err = bpf_copy_verifier_state(&elem->st, cur); @@ -2264,6 +2267,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, elem->prev_insn_idx = prev_insn_idx; elem->next = env->head; elem->log_pos = env->log.end_pos; + elem->diag_log_pos = bpf_diag_event_log_save(env); env->head = elem; env->stack_size++; if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) { @@ -18635,8 +18639,11 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) ret = do_check(env); out: account_current_path(env); - if (!ret && pop_log) - bpf_vlog_reset(&env->log, 0); + if (!ret) { + if (pop_log) + bpf_vlog_reset(&env->log, 0); + bpf_diag_event_log_restore(env, 0); + } free_states(env); /* From af4ea6e20fff383cdc2f01b9a372b4c7a0abf5ff Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:00 +0200 Subject: [PATCH 327/373] bpf: Track verifier register diagnostic events Record material register and outgoing stack argument changes so diagnostics can explain how a value reached its current type, bounds, or unreadable state. Store old and new register types, scalar ranges, tnum value and mask, map and BTF type identity, and basic operand metadata in the environment-owned diagnostic event stream. Record invalidations when packet data moves, references are released, or borrowed references leave their protected region. Register-scoped history starts at the latest matching modification and then shows later branch outcomes. Also record fixed stack spills and overwrites, and tag register fills from stack so register-scoped history can follow value flow through spilled stack slots. The type_is_map_ptr() helper previously lived as a static function in kernel/bpf/log.c since commit 0c95c9fdb696 ("bpf: emit map name in register state if applicable and available"). Move it verbatim to include/linux/bpf_verifier.h as a static inline, next to the other type classifiers, so diagnostics.c can reuse it without duplicating the case list. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-6-memxor@gmail.com Signed-off-by: Eduard Zingerman --- include/linux/bpf_verifier.h | 17 ++ kernel/bpf/diagnostics.c | 356 +++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 23 +++ kernel/bpf/log.c | 11 -- kernel/bpf/verifier.c | 131 +++++++++++-- 5 files changed, 515 insertions(+), 23 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 579a288bc8de..bc2af02547fe 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -354,6 +354,11 @@ struct bpf_func_state { * 0 = main function, 1 = first callee. */ u32 frameno; + /* + * Unique diagnostic identity for this function invocation. Frame depth is + * reused after returns, while this ID is preserved across state clones. + */ + u32 diag_frame_id; /* subprog number == index within subprog_info * zero == main subprog */ @@ -1351,6 +1356,18 @@ static inline bool type_is_non_owning_ref(u32 type) return type_is_ptr_alloc_obj(type) && type_flag(type) & NON_OWN_REF; } +static inline bool type_is_map_ptr(enum bpf_reg_type type) +{ + switch (base_type(type)) { + case CONST_PTR_TO_MAP: + case PTR_TO_MAP_KEY: + case PTR_TO_MAP_VALUE: + return true; + default: + return false; + } +} + static inline bool type_is_pkt_pointer(enum bpf_reg_type type) { type = base_type(type); diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 8f21b46adeca..2e8e75815581 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -25,8 +25,83 @@ #define BPF_DIAG_EVENT_LOG_MAX_SIZE (64U << 20) #define DISASM_LINE_LEN 160 +enum bpf_diag_mod_target_kind { + BPF_DIAG_MOD_TARGET_NONE, + BPF_DIAG_MOD_TARGET_REG, + BPF_DIAG_MOD_TARGET_STACK_ARG, + BPF_DIAG_MOD_TARGET_STACK_SLOT, + BPF_DIAG_MOD_TARGET_STACK_RANGE, +}; + +struct bpf_diag_mod_target { + u32 frame_id; + union { + struct { + s16 min_off; + s16 max_off; + } range; + u16 spi; + u8 regno; + u8 stack_arg; + }; + u8 frameno; + u8 kind; +}; + +static struct bpf_diag_mod_target diag_reg_target(u32 frame_id, u8 frameno, u8 regno) +{ + return (struct bpf_diag_mod_target){ + .frame_id = frame_id, + .frameno = frameno, + .kind = BPF_DIAG_MOD_TARGET_REG, + .regno = regno, + }; +} + +static struct bpf_diag_mod_target diag_stack_arg_target(u32 frame_id, u8 frameno, u8 slot) +{ + return (struct bpf_diag_mod_target){ + .frame_id = frame_id, + .frameno = frameno, + .kind = BPF_DIAG_MOD_TARGET_STACK_ARG, + .stack_arg = slot, + }; +} + +static struct bpf_diag_mod_target diag_stack_slot_target(u32 frame_id, u8 frameno, u16 spi) +{ + return (struct bpf_diag_mod_target){ + .frame_id = frame_id, + .frameno = frameno, + .kind = BPF_DIAG_MOD_TARGET_STACK_SLOT, + .spi = spi, + }; +} + +static struct bpf_diag_mod_target diag_stack_range_target(u32 frame_id, u8 frameno, + s16 min_off, s16 max_off) +{ + return (struct bpf_diag_mod_target){ + .frame_id = frame_id, + .frameno = frameno, + .kind = BPF_DIAG_MOD_TARGET_STACK_RANGE, + .range.min_off = min_off, + .range.max_off = max_off, + }; +} + +struct bpf_diag_reg_snapshot { + u32 type; + u32 btf_id; + const struct bpf_map *map_ptr; + const struct btf *btf; + struct tnum var_off; + struct cnum64 r64; +}; + enum bpf_diag_history_kind { BPF_DIAG_HISTORY_BRANCH, + BPF_DIAG_HISTORY_MOD, }; struct bpf_diag_history_event { @@ -37,6 +112,13 @@ struct bpf_diag_history_event { struct { bool cond_true; } branch; + struct { + struct bpf_diag_mod_target target; + struct bpf_diag_mod_target origin; + struct bpf_diag_reg_snapshot old, new; + u8 reason; + bool origin_valid; + } mod; }; }; @@ -77,10 +159,22 @@ struct bpf_diag_scratch { struct disasm_line disasm_lines[BPF_DIAG_CONTEXT_CNT]; }; +struct bpf_diag_mod_scope { + struct bpf_reg_state target_reg_snapshot; + struct bpf_diag_mod_target target; + struct bpf_diag_mod_target origin; + enum bpf_diag_mod_reason reason; + u32 insn_idx; + bool active; + bool origin_valid; +}; + struct bpf_diag { struct bpf_diag_log log; struct bpf_diag_scratch scratch; struct list_head fmt_chunks; + struct bpf_diag_mod_scope mod; + u32 frame_id_gen; }; bool bpf_diag_enabled(const struct bpf_verifier_env *env) @@ -103,6 +197,12 @@ int bpf_diag_init(struct bpf_verifier_env *env) return 0; } +void bpf_diag_init_frame(struct bpf_verifier_env *env, struct bpf_func_state *state) +{ + if (env->diag) + state->diag_frame_id = ++env->diag->frame_id_gen; +} + static char *diag_fmt_alloc(struct bpf_verifier_env *env, size_t size) { struct bpf_diag *diag = env->diag; @@ -359,6 +459,28 @@ static void diag_print_wrapped_prefixed(struct bpf_verifier_env *env, const char } } +const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id) +{ + char *buf = bpf_diag_fmt_buf(env, BPF_DIAG_FMT_BUF_SIZE); + size_t len; + int ret; + + if (!buf) + return ""; + + buf[0] = '\0'; + ret = btf_type_name_to_buf(btf, type_id, buf, BPF_DIAG_FMT_BUF_SIZE); + if (ret < 0 || !buf[0]) { + scnprintf(buf, BPF_DIAG_FMT_BUF_SIZE, "BTF type ID %u", type_id); + return buf; + } + + len = strlen(buf); + if (len && buf[len - 1] == '{') + buf[len - 1] = '\0'; + return buf; +} + static int diag_line_width(unsigned int line) { int width = 1; @@ -665,3 +787,237 @@ void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool con diag_append_history(env, &event); } + +static void diag_snapshot_reg(struct bpf_diag_reg_snapshot *snapshot, + const struct bpf_reg_state *reg) +{ + snapshot->type = reg->type; + if (type_is_map_ptr(reg->type)) + snapshot->map_ptr = reg->map_ptr; + if (base_type(reg->type) == PTR_TO_BTF_ID && reg->btf && reg->btf_id) { + snapshot->btf_id = reg->btf_id; + snapshot->btf = reg->btf; + } + snapshot->var_off = reg->var_off; + snapshot->r64 = reg->r64; +} + +static bool diag_mod_insn_origin(struct bpf_verifier_env *env, u32 insn_idx, + const struct bpf_diag_mod_target *target, + struct bpf_diag_mod_target *origin) +{ + const struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; + u8 class = BPF_CLASS(insn->code); + const struct bpf_func_state *state; + + if (target->kind == BPF_DIAG_MOD_TARGET_REG && (class == BPF_ALU || class == BPF_ALU64) && + BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_X) { + *origin = diag_reg_target(target->frame_id, target->frameno, insn->src_reg); + return true; + } + + if ((target->kind != BPF_DIAG_MOD_TARGET_STACK_ARG && + target->kind != BPF_DIAG_MOD_TARGET_STACK_SLOT) || + class != BPF_STX) + return false; + + state = env->cur_state->frame[env->cur_state->curframe]; + *origin = diag_reg_target(state->diag_frame_id, state->frameno, insn->src_reg); + return true; +} + +static bool diag_mod_keeps_lineage(struct bpf_verifier_env *env, + const struct bpf_diag_history_event *event) +{ + const struct bpf_insn *insn; + u8 class; + + if (event->mod.reason != BPF_DIAG_MOD_WRITE || + event->mod.target.kind != BPF_DIAG_MOD_TARGET_REG) + return false; + + insn = &env->prog->insnsi[event->insn_idx]; + class = BPF_CLASS(insn->code); + if (class != BPF_ALU && class != BPF_ALU64) + return false; + + switch (BPF_OP(insn->code)) { + case BPF_ADD: + case BPF_SUB: + case BPF_MUL: + case BPF_OR: + case BPF_AND: + case BPF_LSH: + case BPF_RSH: + case BPF_ARSH: + case BPF_XOR: + case BPF_NEG: + case BPF_END: + return true; + default: + return false; + } +} + +static void diag_record_mod(struct bpf_verifier_env *env, u32 insn_idx, + struct bpf_diag_mod_target target, + enum bpf_diag_mod_reason reason, + const struct bpf_reg_state *old_reg, + const struct bpf_reg_state *new_reg, + const struct bpf_diag_mod_target *origin) +{ + struct bpf_diag_history_event event = { + .insn_idx = insn_idx, + .kind = BPF_DIAG_HISTORY_MOD, + .mod = { + .target = target, + .reason = reason, + }, + }; + + if (old_reg) + diag_snapshot_reg(&event.mod.old, old_reg); + if (new_reg) + diag_snapshot_reg(&event.mod.new, new_reg); + if (origin) { + event.mod.origin = *origin; + event.mod.origin_valid = true; + } else if (diag_mod_insn_origin(env, insn_idx, &target, &event.mod.origin)) { + event.mod.origin_valid = true; + } + if (old_reg && new_reg && + (reason == BPF_DIAG_MOD_WRITE || reason == BPF_DIAG_MOD_SPILL) && + !memcmp(&event.mod.old, &event.mod.new, sizeof(event.mod.old)) && + !event.mod.origin_valid && + diag_mod_keeps_lineage(env, &event)) + return; + + diag_append_history(env, &event); +} + +static struct bpf_reg_state *target_to_reg(struct bpf_verifier_env *env, + const struct bpf_diag_mod_target *target) +{ + struct bpf_verifier_state *vstate = env->cur_state; + struct bpf_func_state *state; + + state = target->frameno <= vstate->curframe ? vstate->frame[target->frameno] : NULL; + + if (!state) + return NULL; + if (state->diag_frame_id != target->frame_id) + return NULL; + + switch (target->kind) { + case BPF_DIAG_MOD_TARGET_REG: + if (target->regno >= MAX_BPF_REG) + return NULL; + return &state->regs[target->regno]; + case BPF_DIAG_MOD_TARGET_STACK_ARG: + if (target->stack_arg >= state->out_stack_arg_cnt) + return NULL; + return &state->stack_arg_regs[target->stack_arg]; + case BPF_DIAG_MOD_TARGET_STACK_SLOT: + if (target->spi >= state->allocated_stack / BPF_REG_SIZE) + return NULL; + return &state->stack[target->spi].spilled_ptr; + default: + return NULL; + } +} + +static bool reg_to_target(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, + struct bpf_diag_mod_target *target) +{ + struct bpf_verifier_state *vstate = env->cur_state; + unsigned long addr = (unsigned long)reg; + int frame; + + for (frame = 0; frame <= vstate->curframe; frame++) { + struct bpf_func_state *state = vstate->frame[frame]; + unsigned long start, end; + u32 nslots = state->allocated_stack / BPF_REG_SIZE; + int spi; + + start = (unsigned long)state->regs; + end = (unsigned long)(state->regs + MAX_BPF_REG); + if (addr >= start && addr < end) { + *target = diag_reg_target(state->diag_frame_id, state->frameno, + reg - state->regs); + return true; + } + + start = (unsigned long)state->stack_arg_regs; + end = (unsigned long)(state->stack_arg_regs + state->out_stack_arg_cnt); + if (state->out_stack_arg_cnt && addr >= start && addr < end) { + *target = diag_stack_arg_target(state->diag_frame_id, state->frameno, + reg - state->stack_arg_regs); + return true; + } + + start = (unsigned long)state->stack; + end = (unsigned long)(state->stack + nslots); + if (nslots && addr >= start && addr < end) { + spi = ((const char *)reg - (const char *)state->stack) / + sizeof(*state->stack); + *target = diag_stack_slot_target(state->diag_frame_id, state->frameno, spi); + return true; + } + } + return false; +} + +void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, + const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason) +{ + struct bpf_diag *diag = env->diag; + + if (!diag) + return; + diag->mod.active = reg_to_target(env, reg, &diag->mod.target); + if (!diag->mod.active) + return; + diag->mod.target_reg_snapshot = *reg; + diag->mod.insn_idx = env->insn_idx; + diag->mod.reason = reason; + diag->mod.origin_valid = origin && reg_to_target(env, origin, &diag->mod.origin); +} + +void bpf_diag_mod_end(struct bpf_verifier_env *env) +{ + struct bpf_diag *diag = env->diag; + const struct bpf_reg_state *new_reg; + + if (!diag || !diag->mod.active) + return; + diag->mod.active = false; + /* + * Resolve the target again because the enclosing function state's stack + * may have been reallocated while the modification was in progress. + */ + new_reg = target_to_reg(env, &diag->mod.target); + if (!new_reg) + return; + diag_record_mod(env, diag->mod.insn_idx, diag->mod.target, diag->mod.reason, + &diag->mod.target_reg_snapshot, new_reg, + diag->mod.origin_valid ? &diag->mod.origin : NULL); +} + +void bpf_diag_record_scrub(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, + enum bpf_diag_mod_reason reason) +{ + struct bpf_diag_mod_target target; + + if (!env->diag || reg->type == NOT_INIT || !reg_to_target(env, reg, &target)) + return; + diag_record_mod(env, env->insn_idx, target, reason, reg, NULL, NULL); +} + +void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env, + const struct bpf_func_state *state, s16 min_off, s16 max_off, + enum bpf_diag_mod_reason reason) +{ + diag_record_mod(env, env->insn_idx, + diag_stack_range_target(state->diag_frame_id, state->frameno, min_off, max_off), + reason, NULL, NULL, NULL); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index 6eda2fd65ee1..c4e44b86e89d 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -8,17 +8,40 @@ #include #include +struct bpf_func_state; +struct bpf_reg_state; struct bpf_verifier_env; +struct btf; + +enum bpf_diag_mod_reason { + BPF_DIAG_MOD_WRITE, + BPF_DIAG_MOD_SPILL, + BPF_DIAG_MOD_VAR_WRITE, + BPF_DIAG_MOD_REF_RELEASE, + BPF_DIAG_MOD_PKT_DATA_CHANGE, + BPF_DIAG_MOD_NON_OWN_REF, + BPF_DIAG_MOD_CALLER_SAVED, +}; bool bpf_diag_enabled(const struct bpf_verifier_env *env); int bpf_diag_init(struct bpf_verifier_env *env); +void bpf_diag_init_frame(struct bpf_verifier_env *env, struct bpf_func_state *state); char *bpf_diag_fmt_buf(struct bpf_verifier_env *env, size_t size); const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list args) __printf(2, 0); const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); +const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id); u64 bpf_diag_event_log_save(struct bpf_verifier_env *env); void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos); void bpf_diag_free(struct bpf_verifier_env *env); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); +void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, + const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); +void bpf_diag_mod_end(struct bpf_verifier_env *env); +void bpf_diag_record_scrub(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, + enum bpf_diag_mod_reason reason); +void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env, + const struct bpf_func_state *state, s16 min_off, s16 max_off, + enum bpf_diag_mod_reason reason); #endif /* __BPF_DIAGNOSTICS_H */ diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c index b740fa73ee26..589770ca3d3a 100644 --- a/kernel/bpf/log.c +++ b/kernel/bpf/log.c @@ -615,17 +615,6 @@ static void print_scalar_ranges(struct bpf_verifier_env *env, } } -static bool type_is_map_ptr(enum bpf_reg_type t) { - switch (base_type(t)) { - case CONST_PTR_TO_MAP: - case PTR_TO_MAP_KEY: - case PTR_TO_MAP_VALUE: - return true; - default: - return false; - } -} - /* * _a stands for append, was shortened to avoid multiline statements below. * This macro is used to output a comma separated list of attributes. diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index db644690ac4b..a5929e40f18d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1792,6 +1792,17 @@ static const int caller_saved[CALLER_SAVED_REGS] = { BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5 }; +static void bpf_diag_record_caller_saved(struct bpf_verifier_env *env, + struct bpf_reg_state *regs) +{ + int i; + + for (i = 1; i < CALLER_SAVED_REGS; i++) { + bpf_diag_record_scrub(env, ®s[caller_saved[i]], + BPF_DIAG_MOD_CALLER_SAVED); + } +} + /* This helper doesn't clear reg->id */ static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm) { @@ -2245,6 +2256,7 @@ static void init_func_state(struct bpf_verifier_env *env, { state->callsite = callsite; state->frameno = frameno; + bpf_diag_init_frame(env, state); state->subprogno = subprogno; state->callback_ret_range = retval_range(0, 0); init_reg_state(env, state); @@ -3362,6 +3374,7 @@ static void save_register_state(struct bpf_verifier_env *env, { int i; + bpf_diag_mod_begin(env, &state->stack[spi].spilled_ptr, reg, BPF_DIAG_MOD_SPILL); state->stack[spi].spilled_ptr = *reg; for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--) @@ -3370,6 +3383,8 @@ static void save_register_state(struct bpf_verifier_env *env, /* size < 8 bytes spill */ for (; i; i--) mark_stack_slot_misc(env, &state->stack[spi].slot_type[i - 1]); + + bpf_diag_mod_end(env); } static bool is_bpf_st_mem(struct bpf_insn *insn) @@ -3506,6 +3521,9 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, } else { u8 type = STACK_MISC; + if (bpf_is_spilled_reg(&state->stack[spi])) + bpf_diag_record_scrub(env, &state->stack[spi].spilled_ptr, + BPF_DIAG_MOD_WRITE); scrub_special_slot(state, spi); /* when we zero initialize stack slots mark them as such */ @@ -3666,6 +3684,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, if (err) return err; } + bpf_diag_record_scrub_stack(env, state, min_off, max_off, + BPF_DIAG_MOD_VAR_WRITE); return 0; } @@ -3758,6 +3778,12 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, mark_stack_slot_scratched(env, spi); check_fastcall_stack_contract(env, state, env->insn_idx, off); + /* + * Refine the in-progress load record's origin to the source stack slot. + */ + if (dst_regno >= 0) + bpf_diag_mod_begin(env, &state->regs[dst_regno], reg, BPF_DIAG_MOD_WRITE); + if (bpf_is_spilled_reg(®_state->stack[spi])) { u8 spill_size = 1; @@ -4051,14 +4077,17 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s if (spi + 1 > subprog->max_out_stack_arg_cnt) subprog->max_out_stack_arg_cnt = spi + 1; + arg = &state->stack_arg_regs[spi]; + bpf_diag_mod_begin(env, arg, value_reg, BPF_DIAG_MOD_WRITE); + if (value_reg) { state->stack_arg_regs[spi] = *value_reg; } else { /* BPF_ST: store immediate, treat as scalar */ - arg = &state->stack_arg_regs[spi]; arg->type = SCALAR_VALUE; __mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm); } + bpf_diag_mod_end(env); state->no_stack_arg_load = true; return bpf_push_jmp_history(env, env->cur_state, INSN_F_STACK_ARG_ACCESS, spi, 0, 0); @@ -4091,7 +4120,9 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st caller = vstate->frame[vstate->curframe - 1]; arg = &caller->stack_arg_regs[spi]; cur = vstate->frame[vstate->curframe]; + bpf_diag_mod_begin(env, &cur->regs[dst_regno], arg, BPF_DIAG_MOD_WRITE); cur->regs[dst_regno] = *arg; + bpf_diag_mod_end(env); return bpf_push_jmp_history(env, env->cur_state, INSN_F_STACK_ARG_ACCESS, spi, 0, 0); } @@ -6426,15 +6457,19 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, src_reg_type = regs[insn->src_reg].type; - /* Check if (src_reg + off) is readable. The state of dst_reg will be - * updated by this call. + /* + * check_stack_read_fixed_off() may refine the modification's origin to + * the source stack slot. */ + bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE); err = check_mem_access(env, env->insn_idx, regs + insn->src_reg, argno_from_reg(insn->src_reg), insn->off, BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, strict_alignment_once, is_ldsx); err = err ?: save_aux_ptr_type(env, src_reg_type, allow_trust_mismatch); err = err ?: reg_bounds_sanity_check(env, ®s[insn->dst_reg], ctx); + if (!err) + bpf_diag_mod_end(env); return err; } @@ -6540,10 +6575,14 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, */ err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, BPF_SIZE(insn->code), BPF_READ, -1, true, false); - if (!err && load_reg >= 0) + if (!err && load_reg >= 0) { + bpf_diag_mod_begin(env, cur_regs(env) + load_reg, NULL, BPF_DIAG_MOD_WRITE); err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, BPF_SIZE(insn->code), BPF_READ, load_reg, true, false); + if (!err) + bpf_diag_mod_end(env); + } if (err) return err; @@ -8945,8 +8984,10 @@ static void clear_all_pkt_pointers(struct bpf_verifier_env *env) struct bpf_reg_state *reg; bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ - if (reg_is_pkt_pointer_any(reg) || reg_is_dynptr_slice_pkt(reg)) + if (reg_is_pkt_pointer_any(reg) || reg_is_dynptr_slice_pkt(reg)) { + bpf_diag_record_scrub(env, reg, BPF_DIAG_MOD_PKT_DATA_CHANGE); mark_reg_invalid(env, reg); + } })); } @@ -9062,10 +9103,25 @@ static int release_reference(struct bpf_verifier_env *env, int id) return err; } + /* + * A dynptr occupies two stack slots that invalidate_dynptr() + * clears together. Record both scrubs before invalidating it. + */ + if (stack && stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR) { + struct bpf_stack_state *dyn_stack = stack; + + if (reg->dynptr.first_slot) + dyn_stack--; + bpf_diag_record_scrub(env, &dyn_stack[0].spilled_ptr, + BPF_DIAG_MOD_REF_RELEASE); + bpf_diag_record_scrub(env, &dyn_stack[1].spilled_ptr, + BPF_DIAG_MOD_REF_RELEASE); + invalidate_dynptr(env, dyn_stack); + continue; + } + bpf_diag_record_scrub(env, reg, BPF_DIAG_MOD_REF_RELEASE); if (!stack || stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL) mark_reg_invalid(env, reg); - else if (stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR) - invalidate_dynptr(env, stack); })); } @@ -9078,8 +9134,10 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) struct bpf_reg_state *reg; bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({ - if (type_is_non_owning_ref(reg->type)) + if (type_is_non_owning_ref(reg->type)) { + bpf_diag_record_scrub(env, reg, BPF_DIAG_MOD_NON_OWN_REF); mark_reg_invalid(env, reg); + } })); } @@ -9092,8 +9150,10 @@ static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env) bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({ if (reg->type & MEM_RCU) { + bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE); reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL); reg->type |= PTR_UNTRUSTED; + bpf_diag_mod_end(env); } })); } @@ -9110,9 +9170,11 @@ static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id) if (reg->id != id) continue; if ((reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) { + bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE); reg->id = 0; reg->type &= ~MEM_ALLOC; reg->type |= MEM_RCU; + bpf_diag_mod_end(env); } })); @@ -9124,6 +9186,8 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, { int i; + bpf_diag_record_caller_saved(env, regs); + /* after the call registers r0 - r5 were scratched */ for (i = 0; i < CALLER_SAVED_REGS; i++) { bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); @@ -9131,13 +9195,15 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, } } -static void invalidate_outgoing_stack_args(const struct bpf_verifier_env *env, +static void invalidate_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *state) { int i, nslots = state->out_stack_arg_cnt; - for (i = 0; i < nslots; i++) + for (i = 0; i < nslots; i++) { + bpf_diag_record_scrub(env, &state->stack_arg_regs[i], BPF_DIAG_MOD_CALLER_SAVED); bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]); + } } typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env, @@ -9436,6 +9502,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return err; if (bpf_subprog_is_global(env, subprog)) { const char *sub_name = bpf_subprog_name(env, subprog); + bool returns_void; if (env->cur_state->active_locks) { verbose(env, "global function calls are not allowed while holding a lock,\n" @@ -9458,16 +9525,22 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (env->log.level & BPF_LOG_LEVEL) verbose(env, "Func#%d ('%s') is global and assumed valid.\n", subprog, sub_name); + returns_void = subprog_returns_void(env, subprog); if (env->subprog_info[subprog].changes_pkt_data) clear_all_pkt_pointers(env); /* mark global subprog for verifying after main prog */ subprog_aux(env, subprog)->called = true; + if (returns_void) + bpf_diag_record_scrub(env, &caller->regs[BPF_REG_0], BPF_DIAG_MOD_CALLER_SAVED); + else + bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); /* All non-void global functions return a 64-bit SCALAR_VALUE. */ - if (!subprog_returns_void(env, subprog)) { + if (!returns_void) { mark_reg_unknown(env, caller->regs, BPF_REG_0); + bpf_diag_mod_end(env); } if (env->subprog_info[subprog].might_throw) { @@ -9502,6 +9575,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (err) return err; + bpf_diag_record_scrub(env, &caller->regs[BPF_REG_0], BPF_DIAG_MOD_CALLER_SAVED); clear_caller_saved_regs(env, caller->regs); /* and go analyze first insn of the callee */ @@ -9865,7 +9939,9 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) } } else { /* return to the caller whatever r0 had in the callee */ + bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE); caller->regs[BPF_REG_0] = *r0; + bpf_diag_mod_end(env); } /* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite, @@ -10518,12 +10594,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn return err; /* reset caller saved regs */ + bpf_diag_record_caller_saved(env, regs); for (i = 0; i < CALLER_SAVED_REGS; i++) { bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); } invalidate_outgoing_stack_args(env, cur_func(env)); + bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); /* update return register (already marked as written above) */ ret_type = fn->ret_type; ret_flag = type_flag(ret_type); @@ -10672,6 +10750,8 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (err) return err; + bpf_diag_mod_end(env); + /* * In order for a release of any of the original or cast pointers * to invalidate all other pointers, reuse the same reference id for @@ -10688,6 +10768,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn __mark_reg_known_zero(r0); r0->type = SCALAR_VALUE; + bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); regs[BPF_REG_0].type &= ~PTR_MAYBE_NULL; regs[BPF_REG_0].id = meta.ref_obj.id; } else if (is_acquire_function(func_id, meta.map.ptr)) { @@ -10706,6 +10787,8 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (err) return err; + bpf_diag_mod_end(env); + err = check_map_func_compatibility(env, meta.map.ptr, func_id); if (err) return err; @@ -13211,6 +13294,8 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } + bpf_diag_record_caller_saved(env, regs); + bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); for (i = 0; i < CALLER_SAVED_REGS; i++) { u32 regno = caller_saved[i]; @@ -13362,6 +13447,12 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, caller_info->stack_arg_cnt = stack_arg_cnt; } + /* + * Record R0 before process_iter_next_call() snapshots the alternate + * iterator path's diagnostic position. + */ + bpf_diag_mod_end(env); + if (bpf_is_iter_next_kfunc(&meta)) { err = process_iter_next_call(env, insn_idx, &meta); if (err) @@ -15004,6 +15095,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) u8 opcode = BPF_OP(insn->code); int err; + bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE); + if (opcode == BPF_END || opcode == BPF_NEG) { /* check src operand */ err = check_reg_arg(env, insn->dst_reg, SRC_OP); @@ -15177,7 +15270,12 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) return err; } - return reg_bounds_sanity_check(env, ®s[insn->dst_reg], "alu"); + err = reg_bounds_sanity_check(env, ®s[insn->dst_reg], "alu"); + if (err) + return err; + + bpf_diag_mod_end(env); + return 0; } static void find_good_pkt_pointers(struct bpf_verifier_state *vstate, @@ -16271,11 +16369,13 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) return err; dst_reg = ®s[insn->dst_reg]; + bpf_diag_mod_begin(env, dst_reg, NULL, BPF_DIAG_MOD_WRITE); if (insn->src_reg == 0) { u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm; dst_reg->type = SCALAR_VALUE; __mark_reg_known(®s[insn->dst_reg], imm); + bpf_diag_mod_end(env); return 0; } @@ -16299,6 +16399,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) verifier_bug(env, "pseudo btf id: unexpected dst reg type"); return -EFAULT; } + bpf_diag_mod_end(env); return 0; } @@ -16318,6 +16419,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) dst_reg->type = PTR_TO_FUNC; dst_reg->subprogno = subprogno; + bpf_diag_mod_end(env); return 0; } @@ -16328,6 +16430,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) if (map->map_type == BPF_MAP_TYPE_ARENA) { __mark_reg_unknown(env, dst_reg); dst_reg->map_ptr = map; + bpf_diag_mod_end(env); return 0; } __mark_reg_known(dst_reg, aux->map_off); @@ -16345,6 +16448,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EFAULT; } + bpf_diag_mod_end(env); return 0; } @@ -16423,6 +16527,8 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return err; /* reset caller saved regs to unreadable */ + bpf_diag_record_caller_saved(env, regs); + bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); for (i = 0; i < CALLER_SAVED_REGS; i++) { bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); @@ -16433,6 +16539,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * Already marked as written above. */ mark_reg_unknown(env, regs, BPF_REG_0); + bpf_diag_mod_end(env); /* * See bpf_gen_ld_abs() which emits a hidden BPF_EXIT with r0=0 * which must be explored by the verifier when in a subprog. From 9ecd70304e28985af297726cd961a33a6fec5f67 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:01 +0200 Subject: [PATCH 328/373] bpf: Track verifier reference diagnostic events Add reference acquire and release events to diagnostic history so Resource Lifetime Safety reports can show the lifetime of a specific reference id along the path. Record acquisitions after the verifier assigns the reference id. Record releases only after release_reference_nomark() succeeds, including the kptr_xchg RCU conversion path and owning-to-non-owning conversion path that consume an owning reference. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-7-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 28 ++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 2 ++ kernel/bpf/verifier.c | 32 +++++++++++++++++++++++++------- 3 files changed, 55 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 2e8e75815581..ddeaff1e90b7 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -102,6 +102,8 @@ struct bpf_diag_reg_snapshot { enum bpf_diag_history_kind { BPF_DIAG_HISTORY_BRANCH, BPF_DIAG_HISTORY_MOD, + BPF_DIAG_HISTORY_REF_ACQUIRE, + BPF_DIAG_HISTORY_REF_RELEASE, }; struct bpf_diag_history_event { @@ -119,6 +121,9 @@ struct bpf_diag_history_event { u8 reason; bool origin_valid; } mod; + struct { + u32 ref_id; + } ref; }; }; @@ -1021,3 +1026,26 @@ void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env, diag_stack_range_target(state->diag_frame_id, state->frameno, min_off, max_off), reason, NULL, NULL, NULL); } + +static void diag_record_ref(struct bpf_verifier_env *env, u32 insn_idx, u8 kind, u32 ref_id) +{ + struct bpf_diag_history_event event = { + .insn_idx = insn_idx, + .kind = kind, + .ref = { + .ref_id = ref_id, + }, + }; + + diag_append_history(env, &event); +} + +void bpf_diag_record_ref_acquire(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id) +{ + diag_record_ref(env, insn_idx, BPF_DIAG_HISTORY_REF_ACQUIRE, ref_id); +} + +void bpf_diag_record_ref_release(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id) +{ + diag_record_ref(env, insn_idx, BPF_DIAG_HISTORY_REF_RELEASE, ref_id); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index c4e44b86e89d..d17b498a3f66 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -43,5 +43,7 @@ void bpf_diag_record_scrub(struct bpf_verifier_env *env, const struct bpf_reg_st void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env, const struct bpf_func_state *state, s16 min_off, s16 max_off, enum bpf_diag_mod_reason reason); +void bpf_diag_record_ref_acquire(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id); +void bpf_diag_record_ref_release(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id); #endif /* __BPF_DIAGNOSTICS_H */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a5929e40f18d..8e32fa5fa30a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -205,7 +205,8 @@ struct bpf_verifier_stack_elem { #define BPF_PRIV_STACK_MIN_SIZE 64 static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id); -static int release_reference_nomark(struct bpf_verifier_state *state, int id); +static int __release_reference_nomark(struct bpf_verifier_state *state, int id); +static int release_reference_nomark(struct bpf_verifier_env *env, int id); static int release_reference(struct bpf_verifier_env *env, int id); static void invalidate_non_owning_refs(struct bpf_verifier_env *env); static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env); @@ -1418,6 +1419,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int par s->type = REF_TYPE_PTR; s->id = ++env->id_gen; s->parent_id = parent_id; + bpf_diag_record_ref_acquire(env, insn_idx, s->id); return s->id; } @@ -9017,7 +9019,7 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range reg->range = AT_PKT_END; } -static int release_reference_nomark(struct bpf_verifier_state *state, int id) +static int __release_reference_nomark(struct bpf_verifier_state *state, int id) { int i; @@ -9032,6 +9034,16 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int id) return -EINVAL; } +static int release_reference_nomark(struct bpf_verifier_env *env, int id) +{ + int err; + + err = __release_reference_nomark(env->cur_state, id); + if (!err) + bpf_diag_record_ref_release(env, env->insn_idx, id); + return err; +} + static int idstack_push(struct bpf_idmap *idmap, u32 id) { int i; @@ -9074,8 +9086,10 @@ static int release_reference(struct bpf_verifier_env *env, int id) if (err) return err; - if (find_reference_state(vstate, id)) - WARN_ON_ONCE(release_reference_nomark(vstate, id)); + if (find_reference_state(vstate, id)) { + err = release_reference_nomark(env, id); + WARN_ON_ONCE(err); + } while ((id = idstack_pop(idstack))) { /* @@ -9164,7 +9178,9 @@ static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id) struct bpf_reg_state *reg; int err; - err = release_reference_nomark(env->cur_state, id); + err = release_reference_nomark(env, id); + if (err) + return err; bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ if (reg->id != id) @@ -11757,8 +11773,10 @@ static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id) { struct bpf_func_state *unused; struct bpf_reg_state *reg; + int err; - WARN_ON_ONCE(release_reference_nomark(env->cur_state, id)); + err = release_reference_nomark(env, id); + WARN_ON_ONCE(err); bpf_for_each_reg_in_vstate(env->cur_state, unused, reg, ({ if (reg->id == id) { @@ -15890,7 +15908,7 @@ static void mark_ptr_or_null_regs(struct bpf_verifier_state *vstate, u32 regno, * No one could have freed the reference state before * doing the NULL check. */ - WARN_ON_ONCE(release_reference_nomark(vstate, id)); + WARN_ON_ONCE(__release_reference_nomark(vstate, id)); bpf_for_each_reg_in_vstate(vstate, state, reg, ({ mark_ptr_or_null_reg(state, reg, id, is_null); From 956a66e5c33fb53003ca2bc043a90ff0b671b3a5 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:02 +0200 Subject: [PATCH 329/373] bpf: Track verifier context diagnostic events Record verifier context transitions in the diagnostic history so later reports can anchor causal paths to the critical section that made an operation invalid. This covers lock, IRQ, RCU, and preempt regions without adding any new verifier error reports. Category-specific commits decide where those recorded events should be rendered. Use context depth when selecting scoped history so nested regions anchor at the outer active region, and fall back to the earliest retained event when the matching entry was pruned. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-8-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 39 +++++++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 12 ++++++++++++ kernel/bpf/verifier.c | 28 +++++++++++++++++++++++----- 3 files changed, 74 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index ddeaff1e90b7..15bca8a02a48 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -104,6 +104,7 @@ enum bpf_diag_history_kind { BPF_DIAG_HISTORY_MOD, BPF_DIAG_HISTORY_REF_ACQUIRE, BPF_DIAG_HISTORY_REF_RELEASE, + BPF_DIAG_HISTORY_CONTEXT, }; struct bpf_diag_history_event { @@ -124,6 +125,11 @@ struct bpf_diag_history_event { struct { u32 ref_id; } ref; + struct { + u32 depth; + u8 kind; + bool enter; + } ctx; }; }; @@ -388,6 +394,19 @@ void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos) log->cnt = log_pos - log->first_seq; } +u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state) +{ + u32 depth = 0; + int i; + + for (i = 0; i < state->acquired_refs; i++) { + if (state->refs[i].type == REF_TYPE_IRQ) + depth++; + } + + return depth; +} + static void diag_append_history(struct bpf_verifier_env *env, const struct bpf_diag_history_event *event) { @@ -1049,3 +1068,23 @@ void bpf_diag_record_ref_release(struct bpf_verifier_env *env, u32 insn_idx, u32 { diag_record_ref(env, insn_idx, BPF_DIAG_HISTORY_REF_RELEASE, ref_id); } + +void bpf_diag_record_context(struct bpf_verifier_env *env, u32 insn_idx, + enum bpf_diag_context_kind ctx_kind, bool enter, u32 depth) +{ + /* + * Keep leave events so context rendering can stop at a depth-zero exit + * and show nested-region depth accurately for the active path. + */ + struct bpf_diag_history_event event = { + .insn_idx = insn_idx, + .kind = BPF_DIAG_HISTORY_CONTEXT, + .ctx = { + .kind = ctx_kind, + .enter = enter, + .depth = depth, + }, + }; + + diag_append_history(env, &event); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index d17b498a3f66..ed64776736c6 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -11,6 +11,7 @@ struct bpf_func_state; struct bpf_reg_state; struct bpf_verifier_env; +struct bpf_verifier_state; struct btf; enum bpf_diag_mod_reason { @@ -23,6 +24,14 @@ enum bpf_diag_mod_reason { BPF_DIAG_MOD_CALLER_SAVED, }; +enum bpf_diag_context_kind { + BPF_DIAG_CONTEXT_NONE, + BPF_DIAG_CONTEXT_RCU, + BPF_DIAG_CONTEXT_PREEMPT, + BPF_DIAG_CONTEXT_IRQ, + BPF_DIAG_CONTEXT_LOCK, +}; + bool bpf_diag_enabled(const struct bpf_verifier_env *env); int bpf_diag_init(struct bpf_verifier_env *env); void bpf_diag_init_frame(struct bpf_verifier_env *env, struct bpf_func_state *state); @@ -33,6 +42,7 @@ const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __p const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id); u64 bpf_diag_event_log_save(struct bpf_verifier_env *env); void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos); +u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state); void bpf_diag_free(struct bpf_verifier_env *env); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, @@ -45,5 +55,7 @@ void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env, enum bpf_diag_mod_reason reason); void bpf_diag_record_ref_acquire(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id); void bpf_diag_record_ref_release(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id); +void bpf_diag_record_context(struct bpf_verifier_env *env, u32 insn_idx, + enum bpf_diag_context_kind ctx_kind, bool enter, u32 depth); #endif /* __BPF_DIAGNOSTICS_H */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8e32fa5fa30a..1f2a7f480ce3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1045,7 +1045,7 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s } static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx); -static int release_irq_state(struct bpf_verifier_state *state, int id); +static int release_irq_state(struct bpf_verifier_env *env, int id); static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, @@ -1104,7 +1104,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r return -EINVAL; } - err = release_irq_state(env->cur_state, st->id); + err = release_irq_state(env, st->id); WARN_ON_ONCE(err && err != -EACCES); if (err) { int insn_idx = 0; @@ -1439,6 +1439,8 @@ static int acquire_lock_state(struct bpf_verifier_env *env, int insn_idx, enum r state->active_locks++; state->active_lock_id = id; state->active_lock_ptr = ptr; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_LOCK, true, + state->active_locks); return 0; } @@ -1454,6 +1456,8 @@ static int acquire_irq_state(struct bpf_verifier_env *env, int insn_idx) s->id = ++env->id_gen; state->active_irq_id = s->id; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_IRQ, true, + bpf_diag_irq_depth(state)); return s->id; } @@ -1495,8 +1499,9 @@ static bool reg_is_referenced(struct bpf_verifier_env *env, const struct bpf_reg return find_reference_state(env->cur_state, reg->id); } -static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr) +static int release_lock_state(struct bpf_verifier_env *env, int type, int id, void *ptr) { + struct bpf_verifier_state *state = env->cur_state; void *prev_ptr = NULL; u32 prev_id = 0; int i; @@ -1509,6 +1514,8 @@ static int release_lock_state(struct bpf_verifier_state *state, int type, int id /* Reassign active lock (id, ptr). */ state->active_lock_id = prev_id; state->active_lock_ptr = prev_ptr; + bpf_diag_record_context(env, env->insn_idx, BPF_DIAG_CONTEXT_LOCK, + false, state->active_locks); return 0; } if (state->refs[i].type & REF_TYPE_LOCK_MASK) { @@ -1519,8 +1526,9 @@ static int release_lock_state(struct bpf_verifier_state *state, int type, int id return -EINVAL; } -static int release_irq_state(struct bpf_verifier_state *state, int id) +static int release_irq_state(struct bpf_verifier_env *env, int id) { + struct bpf_verifier_state *state = env->cur_state; u32 prev_id = 0; int i; @@ -1533,6 +1541,8 @@ static int release_irq_state(struct bpf_verifier_state *state, int id) if (state->refs[i].id == id) { release_reference_state(state, i); state->active_irq_id = prev_id; + bpf_diag_record_context(env, env->insn_idx, BPF_DIAG_CONTEXT_IRQ, + false, bpf_diag_irq_depth(state)); return 0; } else { prev_id = state->refs[i].id; @@ -7181,7 +7191,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state verbose(env, "%s_unlock cannot be out of order\n", lock_str); return -EINVAL; } - if (release_lock_state(cur, type, reg->id, ptr)) { + if (release_lock_state(env, type, reg->id, ptr)) { verbose(env, "%s_unlock of different lock\n", lock_str); return -EINVAL; } @@ -13242,22 +13252,30 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (rcu_lock) { env->cur_state->active_rcu_locks++; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, true, + env->cur_state->active_rcu_locks); } else if (rcu_unlock) { if (env->cur_state->active_rcu_locks == 0) { verbose(env, "unmatched rcu read unlock (kernel function %s)\n", func_name); return -EINVAL; } env->cur_state->active_rcu_locks--; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_RCU, false, + env->cur_state->active_rcu_locks); if (!in_rcu_cs(env)) invalidate_rcu_protected_refs(env); } else if (preempt_disable) { env->cur_state->active_preempt_locks++; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, true, + env->cur_state->active_preempt_locks); } else if (preempt_enable) { if (env->cur_state->active_preempt_locks == 0) { verbose(env, "unmatched attempt to enable preemption (kernel function %s)\n", func_name); return -EINVAL; } env->cur_state->active_preempt_locks--; + bpf_diag_record_context(env, insn_idx, BPF_DIAG_CONTEXT_PREEMPT, false, + env->cur_state->active_preempt_locks); if (!in_rcu_cs(env)) invalidate_rcu_protected_refs(env); } From d63284e62b3185cafe40d5fab17245ee60b6cffe Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:03 +0200 Subject: [PATCH 330/373] bpf: Report Register Type Safety errors Augment selected register-state verifier failures with Register Type Safety reports. The existing verbose verifier messages remain in place; the new reports add reason, source context, causal path, and suggestions. Cover invalid pointer dereferences, unreadable registers, missing outgoing stack arguments for bpf2bpf and kfunc calls, and rejected pointer arithmetic. Use scoped diagnostic history so reports start from the latest relevant value change and then show later branch outcomes. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-9-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 854 ++++++++++++++++++ kernel/bpf/diagnostics.h | 18 + kernel/bpf/verifier.c | 128 ++- .../selftests/bpf/progs/verifier_uninit.c | 1 + 4 files changed, 986 insertions(+), 15 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 15bca8a02a48..02399cad2fb0 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -15,9 +15,13 @@ #include "disasm.h" #include "diagnostics.h" +#define REGISTER_TYPE_SAFETY "Register Type Safety" + #define BPF_DIAG_TEXT_WIDTH 100 +#define BPF_DIAG_TEXT_INDENT " " #define BPF_DIAG_CONTEXT 2 #define BPF_DIAG_CONTEXT_CNT (1 + BPF_DIAG_CONTEXT * 2) +#define BPF_DIAG_HISTORY_RENDER_MAX 64 #define BPF_DIAG_SOURCE_LANE_WIDTH 88 #define BPF_DIAG_TAB_WIDTH 8 #define BPF_DIAG_FMT_CHUNK_SIZE (PAGE_SIZE - sizeof(struct diag_fmt_chunk)) @@ -133,6 +137,28 @@ struct bpf_diag_history_event { }; }; +enum bpf_diag_history_scope { + BPF_DIAG_HISTORY_SCOPE_REG, + BPF_DIAG_HISTORY_SCOPE_STACK_ARG, + BPF_DIAG_HISTORY_SCOPE_REF, + BPF_DIAG_HISTORY_SCOPE_CONTEXT, +}; + +struct bpf_diag_history_opts { + enum bpf_diag_history_scope scope; + u32 frame_id; + u32 frameno; + int regno; + int stack_arg_slot; + u32 ref_id; + enum bpf_diag_context_kind ctx_kind; + u32 ctx_depth; +}; + +static void diag_print_history(struct bpf_verifier_env *env, + const struct bpf_diag_history_opts *opts); +static bool diag_target_matches(const struct bpf_diag_mod_target *event_target, + const struct bpf_diag_mod_target *target); struct disasm_line { char text[DISASM_LINE_LEN]; int idx; @@ -505,6 +531,26 @@ const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf return buf; } +static void diag_vprint_indented(struct bpf_verifier_env *env, const char *fmt, va_list args) + __printf(2, 0); + +static void diag_vprint_indented(struct bpf_verifier_env *env, const char *fmt, va_list args) +{ + char *buf; + + if (!bpf_diag_enabled(env)) + return; + + buf = kvasprintf(GFP_KERNEL_ACCOUNT, fmt, args); + if (!buf) { + diag_write(env, "%s\n", BPF_DIAG_TEXT_INDENT); + return; + } + + diag_print_wrapped_prefixed(env, BPF_DIAG_TEXT_INDENT, BPF_DIAG_TEXT_INDENT, buf); + kfree(buf); +} + static int diag_line_width(unsigned int line) { int width = 1; @@ -663,6 +709,47 @@ static void bpf_diag_header(struct bpf_verifier_env *env, const char *category, diag_write(env, "\nVerification failed: %s: %c%s\n", category, first, problem + 1); } +static void diag_reason(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); +static void diag_suggestion(struct bpf_verifier_env *env, const char *fmt, ...) + __printf(2, 3); + +static void diag_section(struct bpf_verifier_env *env, const char *title) +{ + if (!bpf_diag_enabled(env)) + return; + + diag_write(env, "\n%s:\n", title); +} + +static void diag_reason(struct bpf_verifier_env *env, const char *fmt, ...) +{ + va_list args; + + if (!bpf_diag_enabled(env)) + return; + + diag_section(env, "Reason"); + + va_start(args, fmt); + diag_vprint_indented(env, fmt, args); + va_end(args); +} + +static void diag_suggestion(struct bpf_verifier_env *env, const char *fmt, ...) +{ + va_list args; + + if (!bpf_diag_enabled(env)) + return; + + diag_section(env, "Suggestion"); + + va_start(args, fmt); + diag_vprint_indented(env, fmt, args); + va_end(args); + diag_write(env, "\n"); +} + static void diag_print_source_annotation(struct bpf_verifier_env *env, int line_width, int indent, const char *label, const char *msg) { @@ -799,6 +886,284 @@ static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const ch diag_fmt_restore(env, mark); } +static const struct bpf_func_state *diag_current_frame(const struct bpf_verifier_env *env) +{ + return env->cur_state->frame[env->cur_state->curframe]; +} + +void bpf_diag_register_type(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *problem, const char *reason, const char *suggestion) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_REG, + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + .regno = regno, + }; + + bpf_diag_header(env, REGISTER_TYPE_SAFETY, problem); + diag_reason(env, "%s", reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s", problem); + + if (regno >= 0) + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + +const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type) +{ + switch (base_type(type)) { + case NOT_INIT: + return "an uninitialized value"; + case SCALAR_VALUE: + return "an integer scalar"; + case PTR_TO_CTX: + return "a context pointer"; + case PTR_TO_STACK: + return "a stack pointer"; + case PTR_TO_MAP_VALUE: + if (type_may_be_null(type)) + return "a nullable map value pointer"; + return "a map value pointer"; + case PTR_TO_MEM: + if (type_may_be_null(type)) + return "a nullable memory pointer"; + return "a memory pointer"; + case PTR_TO_BTF_ID: + if (type_may_be_null(type)) + return "a nullable kernel object pointer"; + if (type_is_non_owning_ref(type)) + return "a borrowed allocated object pointer"; + if (type_is_ptr_alloc_obj(type)) + return "an owned allocated object pointer"; + if (type_flag(type) & PTR_UNTRUSTED) + return "an untrusted kernel object pointer"; + return "a kernel object pointer"; + default: + return reg_type_str(env, type); + } +} + +static const char *diag_arg_ordinal(int argno) +{ + switch (argno) { + case 1: + return "first"; + case 2: + return "second"; + case 3: + return "third"; + case 4: + return "fourth"; + case 5: + return "fifth"; + case 6: + return "sixth"; + case 7: + return "seventh"; + case 8: + return "eighth"; + case 9: + return "ninth"; + case 10: + return "tenth"; + case 11: + return "eleventh"; + case 12: + return "twelfth"; + default: + return NULL; + } +} + +void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *reg_name, const struct bpf_reg_state *reg, + enum bpf_diag_invalid_deref_kind kind, s64 offset) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_REG, + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + .regno = regno, + }; + const char *type_name = bpf_diag_reg_type_plain(env, reg->type); + + bpf_diag_header(env, REGISTER_TYPE_SAFETY, "invalid dereference"); + + switch (kind) { + case BPF_DIAG_DEREF_SCALAR: + diag_reason(env, "%s is an integer scalar here, not a pointer to memory.", + reg_name); + break; + case BPF_DIAG_DEREF_NULLABLE_PTR: + diag_reason( + env, "%s may be NULL here (%s). The program could dereference NULL on this path, so the verifier cannot prove this access is safe.", + reg_name, type_name); + break; + case BPF_DIAG_DEREF_MODIFIED_PTR: + diag_reason( + env, "%s has offset %lld here, but this pointer type must be dereferenced in its original form.", + reg_name, offset); + break; + case BPF_DIAG_DEREF_INVALID_PTR: + default: + diag_reason( + env, "%s has type %s here, which is not valid for this memory access.", + reg_name, type_name); + break; + } + + diag_section(env, "At"); + if (kind == BPF_DIAG_DEREF_MODIFIED_PTR) + bpf_diag_source(env, insn_idx, "error", + "dereference requires the original %s pointer", type_name); + else + bpf_diag_source(env, insn_idx, "error", "invalid dereference of %s (%s)", + reg_name, type_name); + + if (regno >= 0) + diag_print_history(env, &opts); + + switch (kind) { + case BPF_DIAG_DEREF_NULLABLE_PTR: + diag_suggestion( + env, "Add a NULL check before the access and dereference the pointer only on the non-NULL path."); + break; + case BPF_DIAG_DEREF_MODIFIED_PTR: + diag_suggestion( + env, "Preserve the original pointer in another register, or use only offsets this pointer type permits before dereferencing it."); + break; + case BPF_DIAG_DEREF_SCALAR: + case BPF_DIAG_DEREF_INVALID_PTR: + default: + diag_suggestion( + env, "Preserve a pointer-valued register where needed, or reload and revalidate the pointer after scalar arithmetic, helper calls, or other operations that can invalidate it."); + break; + } +} + +void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int regno) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_REG, + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + .regno = regno, + }; + const struct bpf_diag_log *log = env->diag ? &env->diag->log : NULL; + struct bpf_diag_mod_target target; + bool invalidated = false; + int i; + + target = diag_reg_target(opts.frame_id, opts.frameno, regno); + for (i = log ? log->cnt : 0; i > 0; i--) { + const struct bpf_diag_history_event *event; + + event = &log->events[log_pos(log, i - 1)]; + + if (event->kind != BPF_DIAG_HISTORY_MOD || + !diag_target_matches(&event->mod.target, &target)) + continue; + invalidated = event->mod.new.type == NOT_INIT; + break; + } + + bpf_diag_header(env, REGISTER_TYPE_SAFETY, "unreadable register"); + if (invalidated) + diag_reason( + env, "R%d is not readable here. A previous operation invalidated this register, so the verifier cannot use it as an input.", + regno); + else if (log && !log->first_seq) + diag_reason(env, + "R%d has never been initialized on this path, so the verifier cannot use it as an input.", + regno); + else + diag_reason( + env, "R%d is not readable here. It may never have been initialized, or an earlier operation may have invalidated it.", + regno); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "R%d is not readable", regno); + + if (regno >= 0) + diag_print_history(env, &opts); + + if (invalidated) + diag_suggestion( + env, "Avoid using the register after it is invalidated, or initialize it again before this instruction."); + else if (log && !log->first_seq) + diag_suggestion(env, "Initialize R%d on every path before this instruction.", regno); + else + diag_suggestion( + env, "Initialize the register on every path, or initialize it again after any operation that invalidates it."); +} + +static int diag_stack_argno(u8 slot) +{ + return MAX_BPF_FUNC_REG_ARGS + slot + 1; +} + +static void diag_format_stack_arg(char *buf, size_t size, u8 slot, const char *arg_name) +{ + int argno = diag_stack_argno(slot); + const char *ordinal = diag_arg_ordinal(argno); + + if (ordinal && arg_name) + scnprintf(buf, size, "outgoing stack argument %u (%s argument, %s)", slot + 1, + ordinal, arg_name); + else if (ordinal) + scnprintf(buf, size, "outgoing stack argument %u (%s argument)", slot + 1, ordinal); + else if (arg_name) + scnprintf(buf, size, "outgoing stack argument %u (%s)", slot + 1, arg_name); + else + scnprintf(buf, size, "outgoing stack argument %u", slot + 1); +} + +void bpf_diag_stack_arg_uninit(struct bpf_verifier_env *env, u32 insn_idx, int nargs, + int stack_arg_slot, const char *callee_name, + const char *arg_name) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_STACK_ARG, + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + .stack_arg_slot = stack_arg_slot, + }; + const char *arg_buf; + + arg_buf = bpf_diag_fmt_buf(env, BPF_DIAG_FMT_BUF_SIZE); + if (arg_buf) + diag_format_stack_arg((char *)arg_buf, BPF_DIAG_FMT_BUF_SIZE, stack_arg_slot, + arg_name); + else + arg_buf = ""; + bpf_diag_header(env, REGISTER_TYPE_SAFETY, "missing stack argument"); + if (callee_name && *callee_name) + diag_reason( + env, "Function %s expects %d arguments, but %s is not initialized at this call.", + callee_name, nargs, arg_buf); + else + diag_reason( + env, "The callee expects %d arguments, but %s is not initialized at this call.", + nargs, arg_buf); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s is not initialized", arg_buf); + + if (stack_arg_slot >= 0) + diag_print_history(env, &opts); + + diag_suggestion( + env, "Write the outgoing stack argument after any operation that may invalidate stored pointer values, and before making this call."); +} + void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true) { struct bpf_diag_history_event event = { @@ -1088,3 +1453,492 @@ void bpf_diag_record_context(struct bpf_verifier_env *env, u32 insn_idx, diag_append_history(env, &event); } + +static int diag_history_context_start_idx(const struct bpf_diag_log *log, + const struct bpf_diag_history_opts *opts) +{ + int i; + + if (!opts->ctx_depth) + return 0; + + /* Find the most recent outermost entry, or a depth-zero exit. */ + for (i = log->cnt; i > 0; i--) { + const struct bpf_diag_history_event *event; + + event = &log->events[log_pos(log, i - 1)]; + + if (event->kind != BPF_DIAG_HISTORY_CONTEXT || event->ctx.kind != opts->ctx_kind) + continue; + + if (event->ctx.enter && event->ctx.depth == 1) + return i - 1; + if (!event->ctx.enter && event->ctx.depth == 0) + return 0; + } + + return 0; +} + +struct bpf_diag_history_filter { + const struct bpf_diag_history_opts *opts; + u32 lineage_start; + bool lineage_valid; +}; + +static bool diag_target_matches(const struct bpf_diag_mod_target *event_target, + const struct bpf_diag_mod_target *target) +{ + int slot_off; + + if (event_target->frame_id != target->frame_id || event_target->frameno != target->frameno) + return false; + + if (event_target->kind == BPF_DIAG_MOD_TARGET_STACK_RANGE && + target->kind == BPF_DIAG_MOD_TARGET_STACK_SLOT) { + slot_off = -(target->spi + 1) * BPF_REG_SIZE; + return event_target->range.min_off < slot_off + BPF_REG_SIZE && + event_target->range.max_off > slot_off; + } + + if (event_target->kind != target->kind) + return false; + + switch (target->kind) { + case BPF_DIAG_MOD_TARGET_REG: + return event_target->regno == target->regno; + case BPF_DIAG_MOD_TARGET_STACK_ARG: + return event_target->stack_arg == target->stack_arg; + case BPF_DIAG_MOD_TARGET_STACK_SLOT: + return event_target->spi == target->spi; + default: + return false; + } +} + +static void diag_build_lineage(struct bpf_verifier_env *env, struct bpf_diag_log *log, + struct bpf_diag_history_filter *filter) +{ + const struct bpf_diag_history_opts *opts = filter->opts; + struct bpf_diag_mod_target target; + int i; + + for (i = 0; i < log->cnt; i++) + log->events[log_pos(log, i)].in_lineage = false; + + if (opts->scope == BPF_DIAG_HISTORY_SCOPE_REG) + target = diag_reg_target(opts->frame_id, opts->frameno, opts->regno); + else if (opts->scope == BPF_DIAG_HISTORY_SCOPE_STACK_ARG) + target = diag_stack_arg_target(opts->frame_id, opts->frameno, + opts->stack_arg_slot); + else + return; + + /* + * Find the nearest mutation of the active target. A fill or spill changes + * the target to its origin, so the same walk follows register/stack + * lineage recursively until it reaches the write that created the value. + */ + for (i = log->cnt; i > 0; i--) { + struct bpf_diag_history_event *event; + + event = &log->events[log_pos(log, i - 1)]; + if (event->kind != BPF_DIAG_HISTORY_MOD || + !diag_target_matches(&event->mod.target, &target)) + continue; + + event->in_lineage = true; + filter->lineage_start = i - 1; + filter->lineage_valid = true; + + if (event->mod.origin_valid) { + target = event->mod.origin; + continue; + } + if (event->mod.reason != BPF_DIAG_MOD_WRITE && + event->mod.reason != BPF_DIAG_MOD_SPILL) + continue; + if (diag_mod_keeps_lineage(env, event)) + continue; + break; + } +} + +static int diag_history_start_idx(const struct bpf_diag_log *log, + const struct bpf_diag_history_filter *filter) +{ + const struct bpf_diag_history_opts *opts = filter->opts; + int i; + + if (opts->scope == BPF_DIAG_HISTORY_SCOPE_CONTEXT) + return diag_history_context_start_idx(log, opts); + if (filter->lineage_valid) + return filter->lineage_start; + if (opts->scope != BPF_DIAG_HISTORY_SCOPE_REF) + return 0; + + for (i = log->cnt; i > 0; i--) { + const struct bpf_diag_history_event *event; + + event = &log->events[log_pos(log, i - 1)]; + if (event->kind == BPF_DIAG_HISTORY_REF_ACQUIRE && + event->ref.ref_id == opts->ref_id) + return i - 1; + } + + return 0; +} + +static bool diag_history_event_visible(const struct bpf_diag_history_event *event, + const struct bpf_diag_history_filter *filter) +{ + const struct bpf_diag_history_opts *opts = filter->opts; + + switch (event->kind) { + case BPF_DIAG_HISTORY_BRANCH: + return true; + case BPF_DIAG_HISTORY_MOD: + return filter->lineage_valid && event->in_lineage; + case BPF_DIAG_HISTORY_REF_ACQUIRE: + case BPF_DIAG_HISTORY_REF_RELEASE: + return opts->scope == BPF_DIAG_HISTORY_SCOPE_REF && + event->ref.ref_id == opts->ref_id; + case BPF_DIAG_HISTORY_CONTEXT: + return opts->scope == BPF_DIAG_HISTORY_SCOPE_CONTEXT && + event->ctx.kind == opts->ctx_kind; + default: + return false; + } +} + +static const char *diag_s64_bound_name(s64 value) +{ + if (value == S64_MIN) + return "S64_MIN"; + if (value == S64_MAX) + return "S64_MAX"; + return NULL; +} + +static const char *diag_u64_bound_name(u64 value) +{ + if (value == U64_MAX) + return "U64_MAX"; + return NULL; +} + +static const char *diag_s64_str(struct bpf_verifier_env *env, s64 value) +{ + return diag_s64_bound_name(value) ?: bpf_diag_fmt(env, "%lld", value); +} + +static const char *diag_u64_str(struct bpf_verifier_env *env, u64 value) +{ + return diag_u64_bound_name(value) ?: bpf_diag_fmt(env, "%llu", value); +} + +static bool diag_cnum64_unknown(struct cnum64 range) +{ + return cnum64_smin(range) == S64_MIN && cnum64_smax(range) == S64_MAX && + cnum64_umin(range) == 0 && cnum64_umax(range) == U64_MAX; +} + +static bool diag_snapshot_unknown(const struct bpf_diag_reg_snapshot *snapshot) +{ + return tnum_is_unknown(snapshot->var_off) && diag_cnum64_unknown(snapshot->r64); +} + +static const char *diag_scalar_range(struct bpf_verifier_env *env, struct cnum64 range) +{ + return bpf_diag_fmt(env, "signed range [%s, %s], unsigned range [%s, %s]", + diag_s64_str(env, cnum64_smin(range)), + diag_s64_str(env, cnum64_smax(range)), + diag_u64_str(env, cnum64_umin(range)), + diag_u64_str(env, cnum64_umax(range))); +} + +static const char *diag_var_offset(struct bpf_verifier_env *env, + const struct bpf_diag_reg_snapshot *snapshot) +{ + if (tnum_is_const(snapshot->var_off)) + return bpf_diag_fmt(env, "at offset %lld", (s64)snapshot->var_off.value); + + if (diag_snapshot_unknown(snapshot)) + return bpf_diag_fmt(env, "with unknown offset"); + + return bpf_diag_fmt(env, + "with variable offset: known bits %#llx, unknown mask %#llx, %s", + snapshot->var_off.value, snapshot->var_off.mask, + diag_scalar_range(env, snapshot->r64)); +} + +static const char *diag_reg_map_name(const struct bpf_map *map) +{ + if (!map || !map->name[0]) + return NULL; + + return map->name; +} + +static const char *diag_reg_snapshot(struct bpf_verifier_env *env, + const struct bpf_diag_reg_snapshot *snapshot) +{ + const char *type_name = reg_type_str(env, snapshot->type); + const char *offset = diag_var_offset(env, snapshot); + const char *btf = snapshot->btf && snapshot->btf_id ? + bpf_diag_fmt_btf_type(env, snapshot->btf, snapshot->btf_id) : NULL; + const char *map_name; + + if (snapshot->type == SCALAR_VALUE) { + if (tnum_is_const(snapshot->var_off)) + return bpf_diag_fmt(env, "integer scalar value %lld", + (s64)snapshot->var_off.value); + if (diag_snapshot_unknown(snapshot)) + return bpf_diag_fmt(env, "integer scalar with unknown value"); + if (cnum64_is_const(snapshot->r64)) + return bpf_diag_fmt(env, "integer scalar value %lld", + cnum64_smin(snapshot->r64)); + return bpf_diag_fmt(env, "integer scalar with %s", + diag_scalar_range(env, snapshot->r64)); + } + + if (snapshot->type == NOT_INIT) + return bpf_diag_fmt(env, "uninitialized value"); + + if (base_type(snapshot->type) == PTR_TO_CTX) + return bpf_diag_fmt(env, "context pointer %s", offset); + + if (base_type(snapshot->type) == PTR_TO_STACK) + return bpf_diag_fmt(env, "stack pointer %s", offset); + + if (base_type(snapshot->type) == PTR_TO_MAP_VALUE) { + const char *kind = type_may_be_null(snapshot->type) ? "nullable map value" : + "map value"; + + map_name = diag_reg_map_name(snapshot->map_ptr); + if (map_name) + return bpf_diag_fmt(env, "%s from %s %s", kind, map_name, offset); + return bpf_diag_fmt(env, "%s %s", kind, offset); + } + + if (base_type(snapshot->type) == CONST_PTR_TO_MAP) { + map_name = diag_reg_map_name(snapshot->map_ptr); + if (map_name) + return bpf_diag_fmt(env, "map pointer for map %s", map_name); + return bpf_diag_fmt(env, "map pointer"); + } + + if (type_is_non_owning_ref(snapshot->type)) { + if (btf) + return bpf_diag_fmt(env, "borrowed allocated object pointer type=%s", btf); + return bpf_diag_fmt(env, "borrowed allocated object pointer"); + } + + if (type_is_ptr_alloc_obj(snapshot->type)) { + if (btf) + return bpf_diag_fmt(env, "owned allocated object pointer type=%s", btf); + return bpf_diag_fmt(env, "owned allocated object pointer"); + } + + if (base_type(snapshot->type) == PTR_TO_BTF_ID && btf) + return bpf_diag_fmt(env, "%s type=%s %s", type_name, btf, offset); + + return bpf_diag_fmt(env, "%s %s", type_name, offset); +} + +static const char *diag_mod_target_desc(struct bpf_verifier_env *env, + const struct bpf_diag_mod_target *target) +{ + switch (target->kind) { + case BPF_DIAG_MOD_TARGET_REG: + return bpf_diag_fmt(env, "R%u", target->regno); + case BPF_DIAG_MOD_TARGET_STACK_ARG: + return bpf_diag_fmt(env, "stack arg%d", diag_stack_argno(target->stack_arg)); + case BPF_DIAG_MOD_TARGET_STACK_SLOT: + return bpf_diag_fmt(env, "stack slot fp%d", -(target->spi + 1) * BPF_REG_SIZE); + default: + return "value"; + } +} + +static void diag_print_mod(struct bpf_verifier_env *env, const struct bpf_diag_history_event *event) +{ + const struct bpf_diag_mod_target *target = &event->mod.target; + const char *target_desc, *reason = NULL, *old, *new; + const char *label = "update"; + + if (target->kind == BPF_DIAG_MOD_TARGET_STACK_RANGE) { + bpf_diag_source( + env, event->insn_idx, "invalidated", + "variable-offset stack write may affect bytes fp%d through fp%d", + target->range.min_off, target->range.max_off - 1); + return; + } + + old = diag_reg_snapshot(env, &event->mod.old); + new = diag_reg_snapshot(env, &event->mod.new); + target_desc = diag_mod_target_desc(env, target); + + switch (event->mod.reason) { + case BPF_DIAG_MOD_REF_RELEASE: + reason = target->kind == BPF_DIAG_MOD_TARGET_REG ? "resource release invalidated " + "this pointer" : + "resource release invalidated " + "this value"; + break; + case BPF_DIAG_MOD_PKT_DATA_CHANGE: + reason = "packet data may have moved"; + break; + case BPF_DIAG_MOD_NON_OWN_REF: + reason = "leaving the protected region invalidated this borrowed pointer"; + break; + case BPF_DIAG_MOD_CALLER_SAVED: + reason = target->kind == BPF_DIAG_MOD_TARGET_STACK_ARG ? + "call invalidated this outgoing stack argument" : + "call invalidated this caller-saved register"; + break; + case BPF_DIAG_MOD_WRITE: + if (target->kind == BPF_DIAG_MOD_TARGET_STACK_SLOT) + reason = "a later stack write overwrote this spilled value"; + break; + case BPF_DIAG_MOD_SPILL: + label = "spilled"; + break; + case BPF_DIAG_MOD_VAR_WRITE: + default: + break; + } + + if (reason) { + bpf_diag_source(env, event->insn_idx, "invalidated", + "%s: %s; previous value was %s", target_desc, reason, old); + return; + } + + bpf_diag_source(env, event->insn_idx, label, "%s changed from %s to %s", target_desc, + old, new); +} + +static void diag_print_ref_event(struct bpf_verifier_env *env, + const struct bpf_diag_history_event *event) +{ + const char *label; + + label = event->kind == BPF_DIAG_HISTORY_REF_ACQUIRE ? "acquired" : "released"; + bpf_diag_source(env, event->insn_idx, label, "owned resource (id=%u)", + event->ref.ref_id); +} + +static const char *diag_context_name(enum bpf_diag_context_kind kind) +{ + switch (kind) { + case BPF_DIAG_CONTEXT_RCU: + return "RCU read lock region"; + case BPF_DIAG_CONTEXT_PREEMPT: + return "non-preemptible region"; + case BPF_DIAG_CONTEXT_IRQ: + return "IRQ-disabled region"; + case BPF_DIAG_CONTEXT_LOCK: + return "lock region"; + case BPF_DIAG_CONTEXT_NONE: + default: + return "context"; + } +} + +static void diag_print_context_event(struct bpf_verifier_env *env, + const struct bpf_diag_history_event *event) +{ + bpf_diag_source(env, event->insn_idx, "context", "%s %s; depth is now %u", + event->ctx.enter ? "entered" : "left", + diag_context_name(event->ctx.kind), event->ctx.depth); +} + +static void diag_print_history(struct bpf_verifier_env *env, + const struct bpf_diag_history_opts *opts) +{ + const struct bpf_diag_history_event *event; + struct bpf_diag_history_filter filter = { + .opts = opts, + }; + struct bpf_diag_log *log; + struct diag_fmt_mark mark; + bool first = true; + int start_idx; + u32 i, visible_cnt = 0, visible_idx = 0; + + if (!bpf_diag_enabled(env)) + return; + + if (!env->diag) + return; + log = &env->diag->log; + + diag_build_lineage(env, log, &filter); + + start_idx = diag_history_start_idx(log, &filter); + for (i = start_idx; i < log->cnt; i++) { + event = &log->events[log_pos(log, i)]; + if (diag_history_event_visible(event, &filter)) + visible_cnt++; + } + + if (!visible_cnt && !log->first_seq && opts->scope == BPF_DIAG_HISTORY_SCOPE_STACK_ARG) + return; + + diag_section(env, "Causal path"); + mark = diag_fmt_save(env); + for (i = start_idx; i < log->cnt; i++) { + event = &log->events[log_pos(log, i)]; + if (!diag_history_event_visible(event, &filter)) + continue; + + diag_fmt_restore(env, mark); + if (visible_cnt > BPF_DIAG_HISTORY_RENDER_MAX && + visible_idx >= BPF_DIAG_HISTORY_RENDER_MAX / 2 && + visible_idx < visible_cnt - BPF_DIAG_HISTORY_RENDER_MAX / 2) { + if (visible_idx++ != BPF_DIAG_HISTORY_RENDER_MAX / 2) + continue; + if (!first) + diag_write(env, "\n"); + first = false; + diag_write(env, " %u intermediate causal-history events omitted\n", + visible_cnt - BPF_DIAG_HISTORY_RENDER_MAX); + continue; + } + visible_idx++; + + if (!first) + diag_write(env, "\n"); + first = false; + + switch (event->kind) { + case BPF_DIAG_HISTORY_BRANCH: + bpf_diag_source(env, event->insn_idx, "branch", + "took the %s branch of this conditional, goto %s", + event->branch.cond_true ? "true" : "false", + event->branch.cond_true ? "followed" : "not followed"); + break; + case BPF_DIAG_HISTORY_MOD: + diag_print_mod(env, event); + break; + case BPF_DIAG_HISTORY_REF_ACQUIRE: + case BPF_DIAG_HISTORY_REF_RELEASE: + diag_print_ref_event(env, event); + break; + case BPF_DIAG_HISTORY_CONTEXT: + diag_print_context_event(env, event); + break; + default: + break; + } + } + + if (!visible_cnt) + diag_write(env, " no retained diagnostic events on this path\n"); + if (log->first_seq) + diag_write(env, " %llu older causal-history event%s not retained because diagnostic " + "event storage reached capacity\n", + log->first_seq, log->first_seq == 1 ? "" : "s"); + diag_fmt_restore(env, mark); +} diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index ed64776736c6..d2355c46dad1 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -4,6 +4,7 @@ #ifndef __BPF_DIAGNOSTICS_H #define __BPF_DIAGNOSTICS_H +#include #include #include #include @@ -32,6 +33,13 @@ enum bpf_diag_context_kind { BPF_DIAG_CONTEXT_LOCK, }; +enum bpf_diag_invalid_deref_kind { + BPF_DIAG_DEREF_SCALAR, + BPF_DIAG_DEREF_NULLABLE_PTR, + BPF_DIAG_DEREF_MODIFIED_PTR, + BPF_DIAG_DEREF_INVALID_PTR, +}; + bool bpf_diag_enabled(const struct bpf_verifier_env *env); int bpf_diag_init(struct bpf_verifier_env *env); void bpf_diag_init_frame(struct bpf_verifier_env *env, struct bpf_func_state *state); @@ -40,10 +48,20 @@ const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list __printf(2, 0); const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3); const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id); +const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type); u64 bpf_diag_event_log_save(struct bpf_verifier_env *env); void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos); u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state); void bpf_diag_free(struct bpf_verifier_env *env); +void bpf_diag_register_type(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *problem, const char *reason, const char *suggestion); +void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *reg_name, const struct bpf_reg_state *reg, + enum bpf_diag_invalid_deref_kind kind, s64 offset); +void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int regno); +void bpf_diag_stack_arg_uninit(struct bpf_verifier_env *env, u32 insn_idx, int nargs, + int stack_arg_slot, const char *callee_name, + const char *arg_name); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1f2a7f480ce3..962eb7b37e6b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3130,6 +3130,7 @@ static int __check_reg_arg(struct bpf_verifier_env *env, struct bpf_reg_state *r /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { verbose(env, "R%d !read_ok\n", regno); + bpf_diag_unreadable_reg(env, env->insn_idx, regno); return -EACCES; } /* We don't need to worry about FP liveness because it's read-only */ @@ -4149,7 +4150,8 @@ static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx) } static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller, - int nargs) + int nargs, const char *callee_name, const struct btf *btf, + const struct btf_param *args) { int i, spi; @@ -4157,8 +4159,14 @@ static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_fu spi = i - MAX_BPF_FUNC_REG_ARGS; if (spi >= caller->out_stack_arg_cnt || caller->stack_arg_regs[spi].type == NOT_INIT) { + const char *arg_name = NULL; + + if (args && args[i].name_off) + arg_name = btf_name_by_offset(btf, args[i].name_off); verbose(env, "callee expects %d args, stack arg%d is not initialized\n", nargs, spi + 1); + bpf_diag_stack_arg_uninit(env, env->insn_idx, nargs, spi, + callee_name, arg_name); return -EFAULT; } } @@ -4313,6 +4321,9 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, if (!fixed_off_ok && reg->var_off.value != 0) { verbose(env, "dereference of modified %s ptr %s off=%lld disallowed\n", reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value); + bpf_diag_invalid_deref(env, env->insn_idx, reg_from_argno(argno), + reg_arg_name(env, argno), reg, + BPF_DIAG_DEREF_MODIFIED_PTR, reg->var_off.value); return -EACCES; } @@ -6258,6 +6269,9 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b if (type_may_be_null(reg->type)) { verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), reg_type_str(env, reg->type)); + bpf_diag_invalid_deref(env, insn_idx, reg_from_argno(argno), + reg_arg_name(env, argno), reg, + BPF_DIAG_DEREF_NULLABLE_PTR, 0); return -EACCES; } @@ -6408,8 +6422,16 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b if (t == BPF_READ && value_regno >= 0) mark_reg_unknown(env, regs, value_regno); } else { + enum bpf_diag_invalid_deref_kind kind = BPF_DIAG_DEREF_INVALID_PTR; + verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), reg_type_str(env, reg->type)); + if (reg->type == SCALAR_VALUE) + kind = BPF_DIAG_DEREF_SCALAR; + else if (type_may_be_null(reg->type)) + kind = BPF_DIAG_DEREF_NULLABLE_PTR; + bpf_diag_invalid_deref(env, insn_idx, reg_from_argno(argno), + reg_arg_name(env, argno), reg, kind, 0); return -EACCES; } @@ -9297,20 +9319,28 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, struct bpf_func_state *caller = cur_func(env); struct bpf_verifier_log *log = &env->log; struct ref_obj_desc ref_obj = {}; + const struct btf_param *args; + const struct btf_type *func, *func_proto; u32 i; int ret, err; ret = btf_prepare_func_args(env, subprog); if (ret) { if (bpf_in_stack_arg_cnt(sub) > 0) { - err = check_outgoing_stack_args(env, caller, sub->arg_cnt); + err = check_outgoing_stack_args(env, caller, sub->arg_cnt, + bpf_subprog_name(env, subprog), + NULL, NULL); if (err) return err; } return ret; } - ret = check_outgoing_stack_args(env, caller, sub->arg_cnt); + func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id); + func_proto = btf_type_by_id(btf, func->type); + args = btf_params(func_proto); + ret = check_outgoing_stack_args(env, caller, sub->arg_cnt, + bpf_subprog_name(env, subprog), btf, args); if (ret) return ret; @@ -12191,7 +12221,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me args = (const struct btf_param *)(meta->func_proto + 1); nargs = btf_type_vlen(meta->func_proto); - ret = check_outgoing_stack_args(env, caller, nargs); + ret = check_outgoing_stack_args(env, caller, nargs, func_name, btf, args); if (ret) return ret; @@ -13872,9 +13902,8 @@ static int sanitize_check_bounds(struct bpf_verifier_env *env, * If we return -EACCES, caller may want to try again treating pointer as a * scalar. So we only emit a diagnostic if !env->allow_ptr_leaks. */ -static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, - struct bpf_insn *insn, - const struct bpf_reg_state *ptr_reg, +static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn *insn, + u32 ptr_regno, const struct bpf_reg_state *ptr_reg, const struct bpf_reg_state *off_reg) { struct bpf_verifier_state *vstate = env->cur_state; @@ -13886,6 +13915,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_sanitize_info info = {}; u8 opcode = BPF_OP(insn->code); u32 dst = insn->dst_reg; + const char *reason; int ret, bounds_ret; dst_reg = ®s[dst]; @@ -13909,12 +13939,24 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, verbose(env, "R%d 32-bit pointer arithmetic prohibited\n", dst); + reason = bpf_diag_fmt( + env, "R%d holds %s. 32-bit ALU operations on pointers discard pointer tracking, so the verifier cannot keep the result as a safe pointer.", + ptr_regno, bpf_diag_reg_type_plain(env, ptr_reg->type)); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "32-bit pointer arithmetic", reason, + "Use a 64-bit ALU instruction with an allowed, bounded scalar offset."); return -EACCES; } if (ptr_reg->type & PTR_MAYBE_NULL) { verbose(env, "R%d pointer arithmetic on %s prohibited, null-check it first\n", dst, reg_type_str(env, ptr_reg->type)); + reason = bpf_diag_fmt( + env, "R%d may be NULL (%s). Pointer arithmetic is allowed only after the program proves the pointer is non-NULL on this path.", + ptr_regno, reg_type_str(env, ptr_reg->type)); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer arithmetic before NULL check", reason, + "Make sure that a NULL check precedes any arithmetic performed on the pointer."); return -EACCES; } @@ -13944,6 +13986,12 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, default: verbose(env, "R%d pointer arithmetic on %s prohibited\n", dst, reg_type_str(env, ptr_reg->type)); + reason = bpf_diag_fmt( + env, "R%d holds %s. This pointer kind does not allow offset arithmetic.", + ptr_regno, bpf_diag_reg_type_plain(env, ptr_reg->type)); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer arithmetic is not allowed", reason, + "Do not change this pointer's offset; use it only in operations accepted for its kind."); return -EACCES; } @@ -13961,9 +14009,25 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, if (base_type(ptr_reg->type) == PTR_TO_MEM && (ptr_reg->type & PTR_UNTRUSTED)) return 0; - if (!check_reg_sane_offset_scalar(env, off_reg, ptr_reg->type) || - !check_reg_sane_offset_ptr(env, ptr_reg, ptr_reg->type)) + if (!check_reg_sane_offset_scalar(env, off_reg, ptr_reg->type)) { + reason = bpf_diag_fmt( + env, "The scalar offset used with R%d is unbounded or outside the verifier's safe pointer-offset range [-%u, %u].", + ptr_regno, BPF_MAX_VAR_OFF, BPF_MAX_VAR_OFF); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer offset is not safe", reason, + "Clamp or bounds-check the scalar offset before applying it to the pointer."); return -EINVAL; + } + if (!check_reg_sane_offset_ptr(env, ptr_reg, ptr_reg->type)) { + reason = bpf_diag_fmt( + env, "R%d already has an offset outside the verifier's safe range [-%u, %u] for %s.", + ptr_regno, BPF_MAX_VAR_OFF, BPF_MAX_VAR_OFF, + bpf_diag_reg_type_plain(env, ptr_reg->type)); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer offset is not safe", reason, + "Keep the base pointer within the verifier's allowed offset range before applying more arithmetic."); + return -EINVAL; + } /* pointer types do not carry 32-bit bounds at the moment. */ __mark_reg32_unbounded(dst_reg); @@ -14006,6 +14070,13 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, /* scalar -= pointer. Creates an unknown scalar */ verbose(env, "R%d tried to subtract pointer from scalar\n", dst); + reason = bpf_diag_fmt( + env, "This operation subtracts pointer register R%d from scalar register R%d. " + "The verifier only tracks pointer-minus-scalar arithmetic for allowed pointer types.", + ptr_regno, dst); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer subtracted from scalar", reason, + "Keep the pointer as the base; only add or subtract bounded scalars when permitted."); return -EACCES; } /* We don't allow subtraction from FP, because (according to @@ -14015,6 +14086,12 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, if (ptr_reg->type == PTR_TO_STACK) { verbose(env, "R%d subtraction from stack pointer prohibited\n", dst); + reason = bpf_diag_fmt( + env, "R%d is a stack pointer. The verifier does not allow BPF_SUB to move stack pointers.", + ptr_regno); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "subtraction from stack pointer", reason, + "Use addition from R10 to form stack addresses within the tracked stack frame."); return -EACCES; } dst_reg->r64 = cnum64_add(ptr_reg->r64, cnum64_negate(off_reg->r64)); @@ -14040,16 +14117,38 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, /* bitwise ops on pointers are troublesome, prohibit. */ verbose(env, "R%d bitwise operator %s on pointer prohibited\n", dst, bpf_alu_string[opcode >> 4]); + reason = bpf_diag_fmt( + env, "R%d holds %s. Bitwise operator %s would destroy the pointer value the verifier is tracking.", + ptr_regno, bpf_diag_reg_type_plain(env, ptr_reg->type), + bpf_alu_string[opcode >> 4]); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "bitwise operation on pointer", reason, + "Do bitwise operations on scalar values, not on pointer-valued registers."); return -EACCES; default: /* other operators (e.g. MUL,LSH) produce non-pointer results */ verbose(env, "R%d pointer arithmetic with %s operator prohibited\n", dst, bpf_alu_string[opcode >> 4]); + reason = bpf_diag_fmt( + env, "R%d holds %s. Operator %s is not one of the limited pointer arithmetic operations the verifier can track.", + ptr_regno, bpf_diag_reg_type_plain(env, ptr_reg->type), + bpf_alu_string[opcode >> 4]); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "invalid pointer arithmetic operator", reason, + "Use only verifier-supported addition or subtraction with a bounded scalar offset, or perform this operation on a scalar value."); return -EACCES; } - if (!check_reg_sane_offset_ptr(env, dst_reg, ptr_reg->type)) + if (!check_reg_sane_offset_ptr(env, dst_reg, ptr_reg->type)) { + reason = bpf_diag_fmt( + env, "After this arithmetic, R%d would be outside the verifier's safe offset range [-%u, %u] for %s.", + dst, BPF_MAX_VAR_OFF, BPF_MAX_VAR_OFF, + bpf_diag_reg_type_plain(env, ptr_reg->type)); + bpf_diag_register_type( + env, env->insn_idx, ptr_regno, "pointer offset is not safe", reason, + "Tighten the scalar bounds before the arithmetic so the resulting pointer remains within the allowed range."); return -EINVAL; + } reg_bounds_sync(dst_reg); bounds_ret = sanitize_check_bounds(env, insn, dst_reg); if (bounds_ret == -EACCES) @@ -15021,15 +15120,15 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, if (err) return err; off_reg = *dst_reg; - return adjust_ptr_min_max_vals(env, insn, src_reg, &off_reg); + return adjust_ptr_min_max_vals(env, insn, insn->src_reg, src_reg, + &off_reg); } } else if (ptr_reg) { /* pointer += scalar */ err = mark_chain_precision(env, insn->src_reg); if (err) return err; - return adjust_ptr_min_max_vals(env, insn, - dst_reg, src_reg); + return adjust_ptr_min_max_vals(env, insn, insn->dst_reg, dst_reg, src_reg); } else if (dst_reg->precise) { /* if dst_reg is precise, src_reg should be precise as well */ err = mark_chain_precision(env, insn->src_reg); @@ -15044,8 +15143,7 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, __mark_reg_known(&off_reg, insn->imm); src_reg = &off_reg; if (ptr_reg) /* pointer += K */ - return adjust_ptr_min_max_vals(env, insn, - ptr_reg, src_reg); + return adjust_ptr_min_max_vals(env, insn, insn->dst_reg, ptr_reg, src_reg); } /* Got here implies adding two SCALAR_VALUEs */ diff --git a/tools/testing/selftests/bpf/progs/verifier_uninit.c b/tools/testing/selftests/bpf/progs/verifier_uninit.c index 7718cd7d19ce..691018a46049 100644 --- a/tools/testing/selftests/bpf/progs/verifier_uninit.c +++ b/tools/testing/selftests/bpf/progs/verifier_uninit.c @@ -9,6 +9,7 @@ SEC("socket") __description("read uninitialized register") __failure __msg("R2 !read_ok") +__msg("R2 has never been initialized on this path") __failure_unpriv __naked void read_uninitialized_register(void) { From 2bdc90f5319451d825466375b4c0d7fc1e52c501 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:04 +0200 Subject: [PATCH 331/373] bpf: Report Memory Safety bounds errors Augment selected memory-range verifier failures with Memory Safety reports while preserving the existing terse verifier messages for compatibility. Cover stack spill corruption, uninitialized stack reads, variable stack helper accesses, and check_mem_region_access() range-proof failures. The bounds report spells out the required offset + access_size <= object_size proof with concrete values and uses scoped diagnostic history for causal context. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-10-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 78 ++++++++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 6 ++++ kernel/bpf/verifier.c | 78 +++++++++++++++++++++++++++++++++++++--- 3 files changed, 158 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 02399cad2fb0..058574a1411e 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -16,6 +17,7 @@ #include "diagnostics.h" #define REGISTER_TYPE_SAFETY "Register Type Safety" +#define MEMORY_SAFETY "Memory Safety" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -1164,6 +1166,18 @@ void bpf_diag_stack_arg_uninit(struct bpf_verifier_env *env, u32 insn_idx, int n env, "Write the outgoing stack argument after any operation that may invalidate stored pointer values, and before making this call."); } +void bpf_diag_memory(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion) +{ + bpf_diag_header(env, MEMORY_SAFETY, problem); + diag_reason(env, "%s", reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s", problem); + + diag_suggestion(env, "%s", suggestion); +} + void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true) { struct bpf_diag_history_event event = { @@ -1657,6 +1671,70 @@ static const char *diag_scalar_range(struct bpf_verifier_env *env, struct cnum64 diag_u64_str(env, cnum64_umax(range))); } +const char *bpf_diag_fmt_s64_sum(struct bpf_verifier_env *env, s64 value, int addend) +{ + s64 sum; + + if (check_add_overflow(value, (s64)addend, &sum)) + return bpf_diag_fmt(env, "%lld plus %d (%s)", value, addend, + addend < 0 ? "below S64_MIN" : "above S64_MAX"); + + return bpf_diag_fmt(env, "%lld", sum); +} + +static const char *diag_access_offset(struct bpf_verifier_env *env, int off, + const struct bpf_reg_state *reg) +{ + if (tnum_is_const(reg->var_off)) + return bpf_diag_fmt(env, "constant %s", + bpf_diag_fmt_s64_sum(env, (s64)reg->var_off.value, off)); + + if (tnum_is_unknown(reg->var_off) && diag_cnum64_unknown(reg->r64)) + return bpf_diag_fmt(env, "unbounded"); + + if (off) + return bpf_diag_fmt(env, + "variable: known bits %#llx, unknown mask %#llx, plus fixed offset %d; %s", + (u64)reg->var_off.value, reg->var_off.mask, off, + diag_scalar_range(env, reg->r64)); + return bpf_diag_fmt(env, "variable: known bits %#llx, unknown mask %#llx; %s", + (u64)reg->var_off.value, reg->var_off.mask, + diag_scalar_range(env, reg->r64)); +} + +void bpf_diag_mem_bounds(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *reg_name, const char *type_name, const char *proof, + int off, int size, u32 mem_size, const struct bpf_reg_state *reg) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_REG, + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + .regno = regno, + }; + const char *offset_desc; + + if (!bpf_diag_enabled(env)) + return; + + offset_desc = diag_access_offset(env, off, reg); + + bpf_diag_header(env, MEMORY_SAFETY, "access outside bounds"); + diag_reason( + env, "The verifier cannot prove offset + access_size <= object_size. Here, %s. %s is %s; offset is %s; access_size is %d; object_size is %u.", + proof, reg_name, type_name, offset_desc, size, mem_size); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "access may be outside object bounds"); + + if (regno >= 0) + diag_print_history(env, &opts); + + diag_suggestion( + env, "Add or adjust a bounds check that proves offset + access_size stays within the object."); +} + static const char *diag_var_offset(struct bpf_verifier_env *env, const struct bpf_diag_reg_snapshot *snapshot) { diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index d2355c46dad1..b5feda71de3e 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -15,6 +15,7 @@ struct bpf_verifier_env; struct bpf_verifier_state; struct btf; +const char *bpf_diag_fmt_s64_sum(struct bpf_verifier_env *env, s64 value, int addend); enum bpf_diag_mod_reason { BPF_DIAG_MOD_WRITE, BPF_DIAG_MOD_SPILL, @@ -62,6 +63,11 @@ void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int reg void bpf_diag_stack_arg_uninit(struct bpf_verifier_env *env, u32 insn_idx, int nargs, int stack_arg_slot, const char *callee_name, const char *arg_name); +void bpf_diag_memory(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion); +void bpf_diag_mem_bounds(struct bpf_verifier_env *env, u32 insn_idx, int regno, + const char *reg_name, const char *type_name, const char *proof, + int off, int size, u32 mem_size, const struct bpf_reg_state *reg); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 962eb7b37e6b..86c4212de5aa 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3470,7 +3470,16 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, bpf_is_spilled_reg(&state->stack[spi]) && !bpf_is_spilled_scalar_reg(&state->stack[spi]) && size != BPF_REG_SIZE) { + const char *reason; + verbose(env, "attempt to corrupt spilled pointer on stack\n"); + reason = bpf_diag_fmt(env, + "This store writes %d bytes at stack offset %d into a stack slot that currently holds a spilled pointer. " + "Partial writes to spilled pointers are rejected because they can corrupt pointer metadata and leak kernel pointers.", + size, off); + bpf_diag_memory( + env, insn_idx, "stack spill corruption", reason, + "Write the full 8-byte spilled pointer slot, or use a separate stack slot for scalar data before overwriting only part of it."); return -EACCES; } @@ -3762,6 +3771,21 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env, return 0; } +static void bpf_diag_stack_read_uninit(struct bpf_verifier_env *env, int off, int i, + int size) +{ + const char *reason; + + reason = bpf_diag_fmt(env, + "This rejected read uses %d bytes at stack offset %d, but byte %d in that range is uninitialized on this path. " + "Programs loaded with CAP_PERFMON can be allowed to read uninitialized stack bytes, but this program is being rejected without that allowance.", + size, off, i); + bpf_diag_memory( + env, env->insn_idx, "uninitialized stack read", reason, + "Initialize every byte in the stack range before reading it, adjust the offset and size so the read covers only initialized bytes, " + "or load with CAP_PERFMON if uninitialized stack reads are intended."); +} + /* Read the stack at 'off' and put the results into the register indicated by * 'dst_regno'. It handles reg filling if the addressed stack slot is a * spilled reg. @@ -3851,6 +3875,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, } else { verbose(env, "invalid read from stack off %d+%d size %d\n", off, i, size); + bpf_diag_stack_read_uninit(env, off, i, size); } return -EACCES; } @@ -3909,6 +3934,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, } else { verbose(env, "invalid read from stack off %d+%d size %d\n", off, i, size); + bpf_diag_stack_read_uninit(env, off, i, size); } return -EACCES; } @@ -4001,11 +4027,19 @@ static int check_stack_read(struct bpf_verifier_env *env, * check_stack_read_fixed_off). */ if (dst_regno < 0 && var_off) { + const char *reason; char tn_buf[48]; tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); verbose(env, "variable offset stack pointer cannot be passed into helper function; var_off=%s off=%d size=%d\n", tn_buf, off, size); + reason = bpf_diag_fmt(env, + "The helper would access the stack through variable offset %s plus fixed offset %d and size %d. " + "Helper stack memory arguments require a constant stack offset and a precise initialized range.", + tn_buf, off, size); + bpf_diag_memory( + env, env->insn_idx, "variable stack access", reason, + "Use a fixed stack offset for helper memory arguments, or copy the needed bytes into a fixed stack slot first."); return -EACCES; } /* Variable offset is prohibited for unprivileged mode for simplicity @@ -4247,6 +4281,9 @@ static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_ int off, int size, u32 mem_size, bool zero_size_allowed) { + const char *proof = ""; + const char *start; + s64 max_start, max_end; int err; /* We may have adjusted the register pointing to memory region, so we @@ -4265,14 +4302,28 @@ static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_ reg_smin(reg) + off < 0)) { verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n", reg_arg_name(env, argno)); - return -EACCES; + err = -EACCES; + if (bpf_diag_enabled(env)) { + start = bpf_diag_fmt_s64_sum(env, reg_smin(reg), off); + proof = bpf_diag_fmt( + env, "the minimal bound for a memory access is a negative value: %s", + start); + } + goto report_error; } + err = __check_mem_access(env, reg, argno, reg_smin(reg) + off, size, mem_size, zero_size_allowed); if (err) { verbose(env, "%s min value is outside of the allowed memory range\n", reg_arg_name(env, argno)); - return err; + if (bpf_diag_enabled(env)) { + start = bpf_diag_fmt_s64_sum(env, reg_smin(reg), off); + proof = bpf_diag_fmt( + env, "the minimal bound for a memory access is %s and is outside of the object of size %u", + start, mem_size); + } + goto report_error; } /* If we haven't set a max value then we need to bail since we can't be @@ -4282,17 +4333,36 @@ static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_ if (reg_umax(reg) >= BPF_MAX_VAR_OFF) { verbose(env, "%s unbounded memory access, make sure to bounds check any such access\n", reg_arg_name(env, argno)); - return -EACCES; + err = -EACCES; + if (bpf_diag_enabled(env)) + proof = bpf_diag_fmt( + env, "the maximal bound for a memory access is %llu and exceeds maximum allowed offset of %u", + reg_umax(reg), BPF_MAX_VAR_OFF); + goto report_error; } + err = __check_mem_access(env, reg, argno, reg_umax(reg) + off, size, mem_size, zero_size_allowed); if (err) { verbose(env, "%s max value is outside of the allowed memory range\n", reg_arg_name(env, argno)); - return err; + if (bpf_diag_enabled(env)) { + max_start = (s64)reg_umax(reg) + off; + max_end = max_start + size; + proof = bpf_diag_fmt( + env, "the maximal bound for a memory access is %lld: start %lld + access_size %d, beyond object_size %u", + max_end, max_start, size, mem_size); + } + goto report_error; } return 0; + +report_error: + bpf_diag_mem_bounds(env, env->insn_idx, reg_from_argno(argno), + reg_arg_name(env, argno), reg_type_str(env, reg->type), proof, + off, size, mem_size, reg); + return err; } static int __check_ptr_off_reg(struct bpf_verifier_env *env, From 5d5764627555f6beda39bc03af56428dec0c4582 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:05 +0200 Subject: [PATCH 332/373] bpf: Report Resource Lifetime reference leaks Augment selected Resource Lifetime Safety failures with structured diagnostics while preserving the existing verifier messages. Report unreleased references from check_reference_leak() using reference-scoped diagnostic history, and add state reports for dynptr, iterator, lock, and IRQ-flag lifetime misuse. IRQ restore mismatch and out-of-order diagnostics use IRQ context-scoped history when an IRQ-disabled region is active, so retained save/restore context is still visible after per-state history removal. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-11-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 91 ++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.h | 9 ++++ kernel/bpf/verifier.c | 103 ++++++++++++++++++++++++++++++++++++--- 3 files changed, 195 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 058574a1411e..5d20ea9e470e 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -18,6 +18,7 @@ #define REGISTER_TYPE_SAFETY "Register Type Safety" #define MEMORY_SAFETY "Memory Safety" +#define RESOURCE_LIFETIME_SAFETY "Resource Lifetime Safety" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -1735,6 +1736,96 @@ void bpf_diag_mem_bounds(struct bpf_verifier_env *env, u32 insn_idx, int regno, env, "Add or adjust a bounds check that proves offset + access_size stays within the object."); } +static const char *diag_lock_name(const struct bpf_reference_state *lock) +{ + switch (lock->type) { + case REF_TYPE_LOCK: + return "bpf_spin_lock"; + case REF_TYPE_RES_LOCK: + return "resource spin lock"; + case REF_TYPE_RES_LOCK_IRQ: + return "IRQ-saving resource spin lock"; + default: + return "lock"; + } +} + +static void diag_res_report(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason) +{ + bpf_diag_header(env, RESOURCE_LIFETIME_SAFETY, problem); + diag_reason(env, "%s", reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s", problem); +} + +void bpf_diag_res(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion) +{ + diag_res_report(env, insn_idx, problem, reason); + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_lock(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion, + const struct bpf_reference_state *active_lock) +{ + diag_res_report(env, insn_idx, problem, reason); + + if (active_lock) { + diag_section(env, "Active lock"); + bpf_diag_source(env, active_lock->insn_idx, "acquired", + "active %s has verifier identity %d", + diag_lock_name(active_lock), active_lock->id); + } + + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_irq(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion, u32 depth) +{ + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_CONTEXT, + .ctx_kind = BPF_DIAG_CONTEXT_IRQ, + .ctx_depth = depth, + }; + + bpf_diag_header(env, RESOURCE_LIFETIME_SAFETY, problem); + diag_reason(env, "%s", reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s", problem); + + if (depth) + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_leak(struct bpf_verifier_env *env, u32 ref_id, u32 alloc_insn, u32 fail_insn) +{ + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_REF, + .ref_id = ref_id, + }; + + bpf_diag_header(env, RESOURCE_LIFETIME_SAFETY, "unreleased resource"); + diag_reason( + env, "Owned resource (id=%u) was acquired at instruction %u and still needs to be released before this exit path.", + ref_id, alloc_insn); + + diag_section(env, "At"); + bpf_diag_source(env, fail_insn, "error", + "owned resource (id=%u) still needs release", ref_id); + + diag_print_history(env, &opts); + + diag_suggestion( + env, "Release or transfer ownership of the acquired resource on every path before the program exits."); +} + static const char *diag_var_offset(struct bpf_verifier_env *env, const struct bpf_diag_reg_snapshot *snapshot) { diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index b5feda71de3e..66dd2bb655b7 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -9,6 +9,7 @@ #include #include +struct bpf_reference_state; struct bpf_func_state; struct bpf_reg_state; struct bpf_verifier_env; @@ -68,6 +69,14 @@ void bpf_diag_memory(struct bpf_verifier_env *env, u32 insn_idx, const char *pro void bpf_diag_mem_bounds(struct bpf_verifier_env *env, u32 insn_idx, int regno, const char *reg_name, const char *type_name, const char *proof, int off, int size, u32 mem_size, const struct bpf_reg_state *reg); +void bpf_diag_res(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion); +void bpf_diag_lock(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion, + const struct bpf_reference_state *active_lock); +void bpf_diag_irq(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, + const char *reason, const char *suggestion, u32 depth); +void bpf_diag_leak(struct bpf_verifier_env *env, u32 ref_id, u32 alloc_insn, u32 fail_insn); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 86c4212de5aa..186176973c3c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -816,6 +816,10 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) && dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) { verbose(env, "cannot overwrite referenced dynptr\n"); + bpf_diag_res( + env, env->insn_idx, "referenced dynptr overwrite", + "This stack slot contains a dynptr that owns or protects a referenced resource. Overwriting the last dynptr for that resource would lose the verifier-tracked release path.", + "Release or clone the dynptr so another live dynptr still tracks the referenced resource before overwriting this stack slot."); return -EINVAL; } @@ -1098,9 +1102,19 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r if (st->irq.kfunc_class != kfunc_class) { const char *flag_kfunc = st->irq.kfunc_class == IRQ_NATIVE_KFUNC ? "native" : "lock"; const char *used_kfunc = kfunc_class == IRQ_NATIVE_KFUNC ? "native" : "lock"; + const char *reason; verbose(env, "irq flag acquired by %s kfuncs cannot be restored with %s kfuncs\n", flag_kfunc, used_kfunc); + reason = bpf_diag_fmt(env, + "This IRQ flag was saved by %s IRQ kfuncs, but the restore call " + "belongs to the %s IRQ kfunc family. Save and restore operations " + "must use the same family.", + flag_kfunc, used_kfunc); + bpf_diag_irq(env, env->insn_idx, "IRQ flag restore mismatch", reason, + "Restore the flag with the matching IRQ restore kfunc for the save " + "operation that created it.", + bpf_diag_irq_depth(env->cur_state)); return -EINVAL; } @@ -1118,6 +1132,11 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r verbose(env, "cannot restore irq state out of order, expected id=%d acquired at insn_idx=%d\n", env->cur_state->active_irq_id, insn_idx); + bpf_diag_irq(env, env->insn_idx, "IRQ flag restore out of order", + "IRQ-disabled regions must be restored in last-in, first-out order, " + "but this restore does not match the currently active IRQ flag.", + "Restore nested IRQ flags in the reverse order they were saved.", + bpf_diag_irq_depth(env->cur_state)); return err; } @@ -7183,6 +7202,7 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK; const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin"; struct bpf_verifier_state *cur = env->cur_state; + struct bpf_reference_state *lock; bool is_const = tnum_is_const(reg->var_off); bool is_irq = flags & PROCESS_LOCK_IRQ; u64 val = reg->var_off.value; @@ -7232,14 +7252,25 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state ptr = btf; if (!is_res_lock && cur->active_locks) { - if (find_lock_state(env->cur_state, REF_TYPE_LOCK, 0, NULL)) { + lock = find_lock_state(cur, REF_TYPE_LOCK, 0, NULL); + if (lock) { verbose(env, "Locking two bpf_spin_locks are not allowed\n"); + bpf_diag_lock( + env, env->insn_idx, "nested spin lock", + "This path already holds a bpf_spin_lock. The verifier allows only one regular BPF spin lock at a time.", + "Unlock the current bpf_spin_lock before taking another one.", lock); return -EINVAL; } } else if (is_res_lock && cur->active_locks) { - if (find_lock_state(env->cur_state, REF_TYPE_RES_LOCK | REF_TYPE_RES_LOCK_IRQ, reg->id, ptr)) { + lock = find_lock_state(cur, REF_TYPE_RES_LOCK | REF_TYPE_RES_LOCK_IRQ, + reg->id, ptr); + if (lock) { verbose(env, "Acquiring the same lock again, AA deadlock detected\n"); + bpf_diag_lock( + env, env->insn_idx, "recursive resource spin lock", + "This path already holds the same resource spin lock. Taking it again would deadlock.", + "Avoid reacquiring the same resource spin lock before it is unlocked.", lock); return -EINVAL; } } @@ -7266,6 +7297,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state if (!cur->active_locks) { verbose(env, "%s_unlock without taking a lock\n", lock_str); + bpf_diag_res( + env, env->insn_idx, "unlock without lock", + "This unlock operation has no matching active lock on the current path.", + "Take the matching lock before this unlock, or remove the unmatched unlock path."); return -EINVAL; } @@ -7275,16 +7310,35 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state type = REF_TYPE_RES_LOCK; else type = REF_TYPE_LOCK; - if (!find_lock_state(cur, type, reg->id, ptr)) { + + lock = find_lock_state(cur, type, reg->id, ptr); + if (!lock) { verbose(env, "%s_unlock of different lock\n", lock_str); + lock = find_lock_state(cur, REF_TYPE_LOCK_MASK, cur->active_lock_id, + cur->active_lock_ptr); + bpf_diag_lock( + env, env->insn_idx, "unlock of a different lock", + "This unlock does not match any active lock with the same tracked identity on the current path.", + "Unlock the same lock object that was most recently acquired.", lock); return -EINVAL; } if (reg->id != cur->active_lock_id || ptr != cur->active_lock_ptr) { verbose(env, "%s_unlock cannot be out of order\n", lock_str); + lock = find_lock_state(cur, REF_TYPE_LOCK_MASK, cur->active_lock_id, + cur->active_lock_ptr); + bpf_diag_lock( + env, env->insn_idx, "unlock out of order", + "Locks must be released in last-in, first-out order, but this unlock does not match the currently active lock.", + "Release nested locks in the reverse order they were acquired.", lock); return -EINVAL; } if (release_lock_state(env, type, reg->id, ptr)) { verbose(env, "%s_unlock of different lock\n", lock_str); + bpf_diag_lock( + env, env->insn_idx, "unlock of a different lock", + "The verifier could not release a lock state matching this unlock operation.", + "Pass the same lock object and lock kind that were used for the matching lock operation.", + lock); return -EINVAL; } if (!in_rcu_cs(env)) @@ -7462,6 +7516,10 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat if (!is_dynptr_reg_valid_uninit(env, reg)) { verbose(env, "Dynptr has to be an uninitialized dynptr\n"); + bpf_diag_res( + env, insn_idx, "dynptr is already initialized", + "This kfunc constructs a dynptr and requires an uninitialized dynptr stack slot, but the selected slot already holds dynptr state.", + "Use a fresh stack dynptr slot, or release/destroy the existing dynptr before reusing the slot."); return -EINVAL; } @@ -7478,21 +7536,29 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */ if (reg->type == CONST_PTR_TO_DYNPTR && (arg_type & OBJ_RELEASE)) { verbose(env, "CONST_PTR_TO_DYNPTR cannot be released\n"); + bpf_diag_res( + env, insn_idx, "const dynptr release", + "This release operation was given a const dynptr. Const dynptr values are verifier-provided views and cannot be released by the program.", + "Release only mutable dynptrs that the program initialized or reserved."); return -EINVAL; } if (!is_dynptr_reg_valid_init(env, reg)) { verbose(env, "Expected an initialized dynptr as %s\n", reg_arg_name(env, argno)); + bpf_diag_res( + env, insn_idx, "uninitialized dynptr use", + "This operation requires an initialized dynptr, but the stack slot does not currently hold a valid dynptr on this path.", + "Initialize the dynptr on every path before this call, and avoid overwriting or releasing it before this use."); return -EINVAL; } /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */ if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) { - verbose(env, - "Expected a dynptr of type %s as %s\n", - dynptr_type_str(arg_to_dynptr_type(arg_type)), - reg_arg_name(env, argno)); + enum bpf_dynptr_type expected_type = arg_to_dynptr_type(arg_type); + + verbose(env, "Expected a dynptr of type %s as %s\n", + dynptr_type_str(expected_type), reg_arg_name(env, argno)); return -EINVAL; } @@ -7579,6 +7645,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state * if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) { verbose(env, "expected uninitialized iter_%s as %s\n", iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); + bpf_diag_res( + env, insn_idx, "iterator is already initialized", + "Iterator creation requires an uninitialized iterator stack object, but this stack range already contains iterator state.", + "Use a fresh iterator stack slot, or destroy the existing iterator before reusing the slot."); return -EINVAL; } @@ -7603,6 +7673,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state * case -EINVAL: verbose(env, "expected an initialized iter_%s as %s\n", iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); + bpf_diag_res( + env, insn_idx, "uninitialized iterator use", + "This iterator operation requires an initialized iterator state object, but the stack range does not contain a live iterator on this path.", + "Call the matching iterator new kfunc on every path before calling next or destroy, and do not destroy the iterator before this use."); return err; case -EPROTO: verbose(env, "expected an RCU CS when using %s\n", meta->func_name); @@ -9475,7 +9549,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, if (ret) return ret; - ret = process_dynptr_func(env, reg, argno, -1, arg->arg_type, &ref_obj, NULL); + ret = process_dynptr_func(env, reg, argno, env->insn_idx, arg->arg_type, + &ref_obj, NULL); if (ret) return ret; } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) { @@ -10273,6 +10348,7 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi continue; verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", state->refs[i].id, state->refs[i].insn_idx); + bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, env->insn_idx); refs_lingering = true; } return refs_lingering ? -EINVAL : 0; @@ -11823,6 +11899,12 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state * if (!is_irq_flag_reg_valid_uninit(env, reg)) { verbose(env, "expected uninitialized irq flag as %s\n", reg_arg_name(env, argno)); + bpf_diag_res(env, env->insn_idx, "IRQ flag is already initialized", + "Saving IRQ state requires an uninitialized stack slot for " + "the IRQ flag, but this slot already contains tracked IRQ " + "flag state.", + "Use a fresh stack slot for this save operation, or restore " + "the existing IRQ flag before reusing the slot."); return -EINVAL; } @@ -11839,6 +11921,11 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state * if (err) { verbose(env, "expected an initialized irq flag as %s\n", reg_arg_name(env, argno)); + bpf_diag_res(env, env->insn_idx, "uninitialized IRQ flag restore", + "Restoring IRQ state requires a stack slot that was " + "initialized by a matching IRQ save operation on this path.", + "Pass the same stack slot that was previously initialized by " + "the matching IRQ save kfunc."); return err; } From 66e2727395dd994e26ace31d331013acc9ce8f2e Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:06 +0200 Subject: [PATCH 333/373] bpf: Report Call Type Safety argument errors Augment selected helper and kfunc argument-contract failures with Call Type Safety reports. Keep the existing terse verifier messages and add reason, source context, causal register or stack-argument history, and targeted suggestions. Cover helper register-type mismatch, helper and kfunc non-NULL pointer requirements, release-helper ownership requirements, scalar and constant kfunc arguments, trusted and RCU pointer contracts, kfunc memory arguments, memory/length pairs, refcounted kptrs, constant strings, and IRQ flag stack arguments. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-12-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 46 ++ kernel/bpf/diagnostics.h | 3 + kernel/bpf/verifier.c | 414 +++++++++++++++--- .../selftests/bpf/progs/verifier_map_in_map.c | 1 + 4 files changed, 398 insertions(+), 66 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 5d20ea9e470e..99784d465881 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -19,6 +19,7 @@ #define REGISTER_TYPE_SAFETY "Register Type Safety" #define MEMORY_SAFETY "Memory Safety" #define RESOURCE_LIFETIME_SAFETY "Resource Lifetime Safety" +#define CALL_TYPE_SAFETY "Call Type Safety" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -983,6 +984,51 @@ static const char *diag_arg_ordinal(int argno) } } +void bpf_diag_call_type(struct bpf_verifier_env *env, u32 insn_idx, int argno, int regno, + int stack_arg_slot, const char *call_name, const char *arg_name, + const char *reason, const char *suggestion) +{ + const struct bpf_func_state *frame = diag_current_frame(env); + struct bpf_diag_history_opts opts = { + .frame_id = frame->diag_frame_id, + .frameno = frame->frameno, + }; + const char *ordinal = diag_arg_ordinal(argno); + const char *arg_desc; + bool print_history = true; + + if (regno >= 0) { + opts.scope = BPF_DIAG_HISTORY_SCOPE_REG; + opts.regno = regno; + } else if (stack_arg_slot >= 0) { + opts.scope = BPF_DIAG_HISTORY_SCOPE_STACK_ARG; + opts.stack_arg_slot = stack_arg_slot; + } else { + print_history = false; + } + + if (ordinal && arg_name) + arg_desc = bpf_diag_fmt(env, "%s argument (%s)", ordinal, arg_name); + else if (ordinal) + arg_desc = bpf_diag_fmt(env, "%s argument", ordinal); + else if (arg_name) + arg_desc = bpf_diag_fmt(env, "argument %s", arg_name); + else + arg_desc = "argument"; + + bpf_diag_header(env, CALL_TYPE_SAFETY, "invalid call argument"); + diag_reason(env, "The %s to %s does not satisfy the verifier contract: %s.", + arg_desc, call_name, reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "invalid %s for %s", arg_desc, call_name); + + if (print_history) + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, const char *reg_name, const struct bpf_reg_state *reg, enum bpf_diag_invalid_deref_kind kind, s64 offset) diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index 66dd2bb655b7..4b85a7ad2019 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -77,6 +77,9 @@ void bpf_diag_lock(struct bpf_verifier_env *env, u32 insn_idx, const char *probl void bpf_diag_irq(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, const char *reason, const char *suggestion, u32 depth); void bpf_diag_leak(struct bpf_verifier_env *env, u32 ref_id, u32 alloc_insn, u32 fail_insn); +void bpf_diag_call_type(struct bpf_verifier_env *env, u32 insn_idx, int argno, int regno, + int stack_arg_slot, const char *call_name, const char *arg_name, + const char *reason, const char *suggestion); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 186176973c3c..30e48d4b02f9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -890,26 +890,29 @@ static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_re return true; } +static enum bpf_dynptr_type dynptr_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg) +{ + struct bpf_func_state *state; + int spi; + + if (reg->type == CONST_PTR_TO_DYNPTR) + return reg->dynptr.type; + + spi = dynptr_get_spi(env, reg); + if (spi < 0) + return BPF_DYNPTR_TYPE_INVALID; + state = bpf_func(env, reg); + return state->stack[spi].spilled_ptr.dynptr.type; +} + static bool is_dynptr_type_expected(struct bpf_verifier_env *env, struct bpf_reg_state *reg, enum bpf_arg_type arg_type) { - struct bpf_func_state *state = bpf_func(env, reg); - enum bpf_dynptr_type dynptr_type; - int spi; - /* ARG_PTR_TO_DYNPTR takes any type of dynptr */ if (arg_type == ARG_PTR_TO_DYNPTR) return true; - dynptr_type = arg_to_dynptr_type(arg_type); - if (reg->type == CONST_PTR_TO_DYNPTR) { - return reg->dynptr.type == dynptr_type; - } else { - spi = dynptr_get_spi(env, reg); - if (spi < 0) - return false; - return state->stack[spi].spilled_ptr.dynptr.type == dynptr_type; - } + return dynptr_reg_type(env, reg) == arg_to_dynptr_type(arg_type); } static void __mark_reg_known_zero(struct bpf_reg_state *reg); @@ -6923,14 +6926,17 @@ static int check_stack_range_initialized( return 0; } -static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - int access_size, enum bpf_access_type access_type, - bool zero_size_allowed, - struct bpf_call_arg_meta *meta) +static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, + argno_t argno, int access_size, + enum bpf_access_type access_type, bool zero_size_allowed, + struct bpf_call_arg_meta *meta, bool *known_memory) { struct bpf_reg_state *regs = cur_regs(env); u32 *max_access; + if (known_memory) + *known_memory = true; + switch (base_type(reg->type)) { case PTR_TO_PACKET: case PTR_TO_PACKET_META: @@ -7000,6 +7006,8 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_ if (zero_size_allowed && access_size == 0 && bpf_register_is_null(reg)) return 0; + if (known_memory && base_type(reg->type) != PTR_TO_CTX) + *known_memory = false; verbose(env, "%s type=%s ", reg_arg_name(env, argno), reg_type_str(env, reg->type)); @@ -7008,6 +7016,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_ } } +enum bpf_mem_size_failure { + BPF_MEM_SIZE_FAIL_NONE, + BPF_MEM_SIZE_FAIL_MEMORY, + BPF_MEM_SIZE_FAIL_SIZE, +}; + /* verify arguments to helpers or kfuncs consisting of a pointer and an access * size. * @@ -7018,10 +7032,14 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno, u32 access_type, bool zero_size_allowed, - struct bpf_call_arg_meta *meta) + struct bpf_call_arg_meta *meta, + enum bpf_mem_size_failure *failure) { int err = 0; + if (failure) + *failure = BPF_MEM_SIZE_FAIL_NONE; + /* This is used to refine r0 return value bounds for helpers * that enforce this value as an upper bound on return values. * See do_refine_retval_range() for helpers that can refine @@ -7043,27 +7061,32 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, if (reg_smin(size_reg) < 0) { verbose(env, "%s min value is negative, either use unsigned or 'var &= const'\n", reg_arg_name(env, size_argno)); - return -EACCES; + err = -EACCES; + goto size_error; } if (reg_umin(size_reg) == 0 && !zero_size_allowed) { verbose(env, "%s invalid zero-sized read: u64=[%lld,%lld]\n", reg_arg_name(env, size_argno), reg_umin(size_reg), reg_umax(size_reg)); - return -EACCES; + err = -EACCES; + goto size_error; } if (reg_umax(size_reg) >= BPF_MAX_VAR_SIZ) { verbose(env, "%s unbounded memory access, use 'var &= const' or 'if (var < const)'\n", reg_arg_name(env, size_argno)); - return -EACCES; + err = -EACCES; + goto size_error; } if (access_type & BPF_READ) err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), - BPF_READ, zero_size_allowed, meta); + BPF_READ, zero_size_allowed, meta, NULL); if (!err && access_type & BPF_WRITE) err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), - BPF_WRITE, zero_size_allowed, meta); + BPF_WRITE, zero_size_allowed, meta, NULL); + if (err && failure) + *failure = BPF_MEM_SIZE_FAIL_MEMORY; if (!err) { int regno = reg_from_argno(size_argno); @@ -7075,16 +7098,23 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, } return err; + +size_error: + if (failure) + *failure = BPF_MEM_SIZE_FAIL_SIZE; + return err; } static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, u32 mem_size, enum bpf_access_type access_type, - struct bpf_call_arg_meta *meta) + struct bpf_call_arg_meta *meta, bool *known_memory) { int size, err = 0; if (bpf_register_is_null(reg)) return 0; + if (known_memory) + *known_memory = true; if (mem_size > S32_MAX) { verbose(env, "%s memory size %u is too large\n", @@ -7099,9 +7129,11 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg size = (!meta && base_type(reg->type) == PTR_TO_STACK) ? -(int)mem_size : mem_size; if (access_type & BPF_READ) - err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta); + err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, meta, + known_memory); if (!err && (access_type & BPF_WRITE)) - err = check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta); + err = check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, meta, + known_memory); return err; } @@ -7461,6 +7493,12 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, return 0; } +static void bpf_diag_call_arg(struct bpf_verifier_env *env, u32 insn_idx, argno_t argno, + const char *call_name, const char *reason, const char *suggestion); +__printf(6, 7) static void bpf_diag_call_arg_fmt(struct bpf_verifier_env *env, u32 insn_idx, + argno_t argno, const char *call_name, + const char *suggestion, const char *fmt, ...); + /* * Validate dynptr arguments for helper, kfunc and subprog. * @@ -7485,7 +7523,8 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, * and checked dynamically during runtime. */ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, - argno_t argno, int insn_idx, enum bpf_arg_type arg_type, + argno_t argno, int insn_idx, const char *call_name, + enum bpf_arg_type arg_type, struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr) { int spi, err = 0; @@ -7494,6 +7533,11 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat verbose(env, "%s expected pointer to stack or const struct bpf_dynptr\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt( + env, insn_idx, argno, call_name, + "Pass the address of a stack dynptr object, or use a const dynptr pointer returned by the verifier-supported path.", + "a dynptr argument must be a pointer to a dynptr stack slot or a verifier-provided const struct bpf_dynptr, but %s is %s", + reg_arg_name(env, argno), bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } @@ -7556,9 +7600,15 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */ if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) { enum bpf_dynptr_type expected_type = arg_to_dynptr_type(arg_type); + enum bpf_dynptr_type actual_type = dynptr_reg_type(env, reg); verbose(env, "Expected a dynptr of type %s as %s\n", dynptr_type_str(expected_type), reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt( + env, insn_idx, argno, call_name, + "Use a dynptr constructor that matches this operation, or call an operation that accepts the dynptr's current type.", + "the dynptr is initialized with backing object type %s, but this operation expects dynptr type %s", + dynptr_type_str(actual_type), dynptr_type_str(expected_type)); return -EINVAL; } @@ -7622,6 +7672,11 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state * if (reg->type != PTR_TO_STACK) { verbose(env, "%s expected pointer to an iterator on stack\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt( + env, insn_idx, argno, meta->func_name, + "Pass the address of a stack iterator object for iterator new, next, and destroy calls.", + "iterator state must live in verifier-tracked stack memory, but %s is %s", + reg_arg_name(env, argno), bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } @@ -7635,6 +7690,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state * if (btf_id < 0) { verbose(env, "expected valid iter pointer as %s\n", reg_arg_name(env, argno)); + bpf_diag_call_arg( + env, insn_idx, argno, meta->func_name, + "the kfunc expects a recognized iterator state pointer, but this argument does not match a valid iterator type", + "Pass the exact iterator state type expected by this kfunc."); return -EINVAL; } t = btf_type_by_id(meta->btf, btf_id); @@ -8099,13 +8158,70 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_PTR_TO_DYNPTR] = &dynptr_types, }; +static void bpf_diag_call_arg(struct bpf_verifier_env *env, u32 insn_idx, argno_t argno, + const char *call_name, const char *reason, + const char *suggestion) +{ + int arg = arg_from_argno(argno); + int regno = reg_from_argno(argno); + int stack_slot = -1; + + if (arg < 0 && regno >= BPF_REG_1 && regno <= BPF_REG_5) + arg = regno; + if (arg > MAX_BPF_FUNC_REG_ARGS) + stack_slot = arg - MAX_BPF_FUNC_REG_ARGS - 1; + + bpf_diag_call_type(env, insn_idx, arg, regno, stack_slot, + call_name && *call_name ? call_name : "call", + reg_arg_name(env, argno), reason, suggestion); +} + +static const char *bpf_diag_arg_name(struct bpf_verifier_env *env, argno_t argno) +{ + return bpf_diag_fmt(env, "%s", reg_arg_name(env, argno)); +} + +__printf(6, 7) static void bpf_diag_call_arg_fmt(struct bpf_verifier_env *env, u32 insn_idx, + argno_t argno, const char *call_name, + const char *suggestion, const char *fmt, ...) +{ + const char *reason; + va_list args; + + va_start(args, fmt); + reason = bpf_diag_vfmt(env, fmt, args); + va_end(args); + + bpf_diag_call_arg(env, insn_idx, argno, call_name, reason, suggestion); +} + +static const char *bpf_diag_expected_reg_types(struct bpf_verifier_env *env, + const enum bpf_reg_type *types, int count) +{ + size_t len = 0, size = 1; + char *buf; + int i; + + for (i = 0; i < count; i++) + size += strlen(reg_type_str(env, types[i])) + (i ? 2 : 0); + + buf = bpf_diag_fmt_buf(env, size); + if (!buf) + return ""; + + for (i = 0; i < count; i++) + len += scnprintf(buf + len, size - len, "%s%s", i ? ", " : "", + reg_type_str(env, types[i])); + return buf; +} + static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, - enum bpf_arg_type arg_type, - const u32 *arg_btf_id, - struct bpf_call_arg_meta *meta) + enum bpf_arg_type arg_type, const u32 *arg_btf_id, + struct bpf_call_arg_meta *meta, const char *call_name) { enum bpf_reg_type expected, type = reg->type; const struct bpf_reg_types *compatible; + const char *actual, *accepted; int i, j, err; compatible = compatible_reg_types[base_type(arg_type)]; @@ -8152,6 +8268,12 @@ static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *re for (j = 0; j + 1 < i; j++) verbose(env, "%s, ", reg_type_str(env, compatible->types[j])); verbose(env, "%s\n", reg_type_str(env, compatible->types[j])); + actual = bpf_diag_fmt(env, "%s", reg_type_str(env, reg->type)); + accepted = bpf_diag_expected_reg_types(env, compatible->types, i); + bpf_diag_call_arg_fmt(env, env->insn_idx, argno, call_name, + "Pass a value with one of the accepted pointer or scalar types for this call.", + "it has type %s, but this argument accepts %s", + actual, accepted); return -EACCES; found: @@ -8188,6 +8310,10 @@ static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *re (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) { verbose(env, "Possibly NULL pointer passed to helper %s\n", reg_arg_name(env, argno)); + bpf_diag_call_arg( + env, env->insn_idx, argno, call_name, + "the pointer may be NULL, but this call requires a non-NULL pointer", + "Add a NULL check and make the call only on the non-NULL path."); return -EACCES; } @@ -8574,7 +8700,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) arg_btf_id = fn->arg_btf_id[arg]; - err = check_reg_type(env, reg, argno, arg_type, arg_btf_id, meta); + err = check_reg_type(env, reg, argno, arg_type, arg_btf_id, meta, + func_id_name(meta->func_id)); if (err) return err; @@ -8587,6 +8714,10 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) { verbose(env, "release helper %s expects referenced PTR_TO_BTF_ID passed to %s\n", func_id_name(meta->func_id), reg_arg_name(env, argno)); + bpf_diag_call_arg( + env, insn_idx, argno, func_id_name(meta->func_id), + "release helpers require a value that owns a live resource returned by a matching acquire helper", + "Pass the resource-owning pointer returned by the matching acquire helper, and avoid calling the release helper after ownership has already been transferred or released."); return -EINVAL; } @@ -8615,7 +8746,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return -EFAULT; } key_size = meta->map.ptr->key_size; - err = check_helper_mem_access(env, reg, argno, key_size, BPF_READ, false, NULL); + err = check_helper_mem_access(env, reg, argno, key_size, BPF_READ, false, NULL, + NULL); if (err) return err; if (can_elide_value_nullness(meta->map.ptr)) { @@ -8652,7 +8784,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, err = check_helper_mem_access(env, reg, argno, meta->map.ptr->value_size, arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, - false, meta); + false, meta, NULL); break; case ARG_PTR_TO_PERCPU_BTF_ID: if (!reg->btf_id) { @@ -8694,7 +8826,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, */ if (arg_type & MEM_FIXED_SIZE) { err = check_mem_reg(env, reg, argno_from_reg(regno), fn->arg_size[arg], - arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, meta); + arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, meta, NULL); if (err) return err; if (arg_type & MEM_ALIGNED) @@ -8705,17 +8837,17 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), argno, fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, - false, meta); + false, meta, NULL); break; case ARG_MEM_SIZE_OR_ZERO: err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), argno, fn->arg_type[arg - 1] & MEM_WRITE ? BPF_WRITE : BPF_READ, - true, meta); + true, meta, NULL); break; case ARG_PTR_TO_DYNPTR: - err = process_dynptr_func(env, reg, argno, insn_idx, arg_type, &meta->ref_obj, - &meta->dynptr); + err = process_dynptr_func(env, reg, argno, insn_idx, func_id_name(meta->func_id), + arg_type, &meta->ref_obj, &meta->dynptr); if (err) return err; break; @@ -9523,7 +9655,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE); if (ret < 0) return ret; - if (check_mem_reg(env, reg, argno, arg->mem_size, BPF_READ | BPF_WRITE, NULL)) + if (check_mem_reg(env, reg, argno, arg->mem_size, BPF_READ | BPF_WRITE, NULL, + NULL)) return -EINVAL; if (!(arg->arg_type & PTR_MAYBE_NULL) && (type_may_be_null(reg->type) || bpf_register_is_null(reg))) { @@ -9549,7 +9682,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, if (ret) return ret; - ret = process_dynptr_func(env, reg, argno, env->insn_idx, arg->arg_type, + ret = process_dynptr_func(env, reg, argno, env->insn_idx, + bpf_subprog_name(env, subprog), arg->arg_type, &ref_obj, NULL); if (ret) return ret; @@ -9561,7 +9695,8 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, continue; memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */ - err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta); + err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta, + bpf_subprog_name(env, subprog)); err = err ?: check_func_arg_reg_off(env, reg, argno, arg->arg_type); if (err) return err; @@ -12392,7 +12527,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me argno_t argno = argno_from_arg(i + 1); int regno = reg_from_argno(argno); bool btf_id_fixed_off_ok = true; - u32 ref_id, type_size; + u32 ref_id = args[i].type, type_size; int kf_arg_type = meta->fn->arg_type[i]; if (is_kfunc_arg_prog_aux(btf, &args[i])) { @@ -12416,28 +12551,42 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me t = btf_type_skip_modifiers(btf, args[i].type, NULL); - if (btf_type_is_ptr(t) && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && - !type_may_be_null(kf_arg_type)) { - verbose(env, "Possibly NULL pointer passed to trusted %s\n", - reg_arg_name(env, argno)); - return -EACCES; - } - - if (regno == meta->release_regno && !is_kfunc_arg_dynptr(meta->btf, &args[i]) && - !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) { - verbose(env, "release kfunc %s expects referenced PTR_TO_BTF_ID passed to %s\n", - func_name, reg_arg_name(env, argno)); - return -EINVAL; - } - - if (reg_is_referenced(env, reg)) - update_ref_obj(&meta->ref_obj, reg); - if (btf_type_is_ptr(t)) { ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id); ref_tname = btf_name_by_offset(btf, ref_t->name_off); } + if (btf_type_is_ptr(t) && + (bpf_register_is_null(reg) || type_may_be_null(reg->type)) && + !type_may_be_null(kf_arg_type)) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); + verbose(env, "Possibly NULL pointer passed to trusted %s\n", + reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Add a NULL check and call the kfunc only on the non-NULL path.", + "the pointer may be NULL, but this kfunc requires a non-NULL pointer to %s", + expected_type); + return -EACCES; + } + + if (regno == meta->release_regno && !is_kfunc_arg_dynptr(meta->btf, &args[i]) && + !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); + verbose(env, "release kfunc %s expects referenced PTR_TO_BTF_ID passed to %s\n", + func_name, reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass the resource-owning pointer returned by the matching acquire kfunc, and avoid calling the release kfunc after ownership has already been transferred or released.", + "release kfuncs require a resource-owning value of type %s returned by a matching acquire kfunc", + expected_type); + return -EINVAL; + } + + if (reg_is_referenced(env, reg)) + update_ref_obj(&meta->ref_obj, reg); if (bpf_register_is_null(reg) && type_may_be_null(kf_arg_type)) continue; @@ -12498,35 +12647,67 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me case KF_ARG_CONST: if (reg->type != SCALAR_VALUE) { verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass an integer scalar value for this argument, not a pointer or resource object.", + "the kfunc expects an integer scalar, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } ret = process_const_arg(env, reg, argno, meta); - if (ret < 0) + if (ret < 0) { + if (ret == -EINVAL) + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a compile-time constant or a value the verifier can prove is constant at this call.", + "the kfunc requires this scalar argument to be a verifier-known constant, but %s is variable on this path", + reg_arg_name(env, argno)); return ret; + } break; case KF_ARG_ANYTHING: if (reg->type != SCALAR_VALUE) { verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass an integer scalar value for this argument, not a pointer or resource object.", + "the kfunc expects an integer scalar, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } break; case KF_ARG_CONST_ALLOC_SIZE_OR_ZERO: if (reg->type != SCALAR_VALUE) { verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass an integer scalar value for this argument, not a pointer or resource object.", + "the kfunc expects an integer scalar, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } if (is_kfunc_arg_scalar_with_name(btf, &args[i], "rdonly_buf_size")) meta->r0_rdonly = true; ret = process_const_alloc_mem_size(env, reg, argno, &meta->ret_mem); - if (ret < 0) + if (ret < 0) { + if (ret == -EINVAL) + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a verifier-known constant size for this kfunc buffer argument.", + "the kfunc uses this argument as a return-buffer size, but %s is invalid or variable on this path", + reg_arg_name(env, argno)); return ret; + } break; case KF_ARG_PTR_TO_CTX: if (reg->type != PTR_TO_CTX) { verbose(env, "%s expected pointer to ctx, but got %s\n", reg_arg_name(env, argno), reg_type_str(env, reg->type)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass the original program context pointer or preserve it before modifying registers.", + "the kfunc expects a context pointer, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } @@ -12560,10 +12741,19 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me } else { verbose(env, "%s expected pointer to allocated object\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a pointer returned by the matching BPF object allocation path.", + "the kfunc expects an allocated object pointer, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } if (!reg_is_referenced(env, reg)) { verbose(env, "allocated object must be referenced\n"); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass the owned object pointer before it is released or transferred.", + "the allocated object pointer in %s must still carry verifier-tracked ownership, but this pointer no longer owns a live resource", + reg_arg_name(env, argno)); return -EINVAL; } if (meta->btf == btf_vmlinux) { @@ -12600,8 +12790,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me dynptr_arg_type |= (unsigned int)get_dynptr_type_flag(parent_type); } - ret = process_dynptr_func(env, reg, argno, insn_idx, dynptr_arg_type, - &meta->ref_obj, &meta->dynptr); + ret = process_dynptr_func(env, reg, argno, insn_idx, func_name, + dynptr_arg_type, &meta->ref_obj, &meta->dynptr); if (ret < 0) return ret; break; @@ -12716,13 +12906,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (!is_trusted_reg(env, reg) || bpf_type_has_unsafe_modifiers(reg->type)) { if (!is_kfunc_rcu(meta)) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); verbose(env, "%s must be referenced or trusted\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a pointer acquired from a verifier-tracked source, or call this kfunc only inside the required protection if it accepts RCU pointers.", + "the kfunc requires a trusted or resource-owning pointer to %s, but %s is %s", + expected_type, + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } if (!is_rcu_reg(reg)) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); verbose(env, "%s must be a rcu pointer\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Use this kfunc with a pointer that is valid in an RCU read lock region.", + "the kfunc requires an RCU-protected pointer to %s, but %s is %s", + expected_type, + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } } @@ -12735,6 +12943,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); + const char *expected_type; verbose(env, "%s is %s expected %s %s", reg_arg_name(env, argno), reg_type_str(env, reg->type), @@ -12742,6 +12951,12 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (reg2btf_type != NOT_INIT) verbose(env, " or %s", reg_type_str(env, reg2btf_type)); verbose(env, "\n"); + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a verifier-tracked pointer to the expected kernel object type, not a pointer to stack storage or another memory buffer.", + "the kfunc expects a pointer to %s, but this argument is %s and cannot be used as that kernel object pointer", + expected_type, + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } @@ -12753,6 +12968,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me fallthrough; case KF_ARG_PTR_TO_MEM: if (kf_arg_type & MEM_FIXED_SIZE) { + bool known_memory; + resolve_ret = btf_resolve_size(btf, ref_t, &type_size); if (IS_ERR(resolve_ret)) { verbose(env, "%s reference type('%s %s') size cannot be determined: %ld\n", @@ -12760,9 +12977,28 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me ref_tname, PTR_ERR(resolve_ret)); return -EINVAL; } - ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, meta); - if (ret < 0) + ret = check_mem_reg(env, reg, argno, type_size, BPF_READ | BPF_WRITE, + meta, &known_memory); + if (ret < 0) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); + if (known_memory) + bpf_diag_call_arg_fmt( + env, insn_idx, argno, func_name, + "Pass memory with at least the required number of accessible bytes and suitable read and write access.", + "the kfunc expects %u bytes of memory for %s, but the verifier cannot prove that %s provides a readable and writable range of that size", + type_size, expected_type, + bpf_diag_reg_type_plain(env, reg->type)); + else + bpf_diag_call_arg_fmt( + env, insn_idx, argno, func_name, + "Pass stack, map, context, or other verifier-known memory of the expected type and size, not an integer cast to a pointer.", + "the kfunc expects %u bytes of memory for %s, but it is %s and not verifier-known memory", + type_size, expected_type, + bpf_diag_reg_type_plain(env, reg->type)); return ret; + } } break; case KF_ARG_CONST_MEM_SIZE: @@ -12775,9 +13011,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me struct bpf_reg_state *buff_reg = get_func_arg_reg(caller, regs, i - 1); struct bpf_reg_state *size_reg = reg; argno_t buff_argno = argno_from_arg(i); + enum bpf_mem_size_failure failure; if (reg->type != SCALAR_VALUE) { verbose(env, "%s is not a scalar\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass an integer scalar length for this memory argument.", + "the kfunc expects a scalar memory size, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } @@ -12785,11 +13027,34 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me break; ret = check_mem_size_reg(env, buff_reg, size_reg, buff_argno, argno, - BPF_READ | BPF_WRITE, true, meta); + BPF_READ | BPF_WRITE, true, meta, &failure); if (ret < 0) { + const char *buff_arg, *size_arg; + + buff_arg = bpf_diag_arg_name(env, buff_argno); + size_arg = bpf_diag_arg_name(env, argno); verbose(env, "%s and ", reg_arg_name(env, buff_argno)); verbose(env, "%s memory, len pair leads to invalid memory access\n", reg_arg_name(env, argno)); + if (failure == BPF_MEM_SIZE_FAIL_MEMORY) { + bpf_diag_call_arg_fmt(env, insn_idx, buff_argno, func_name, + "Pass a stack, map, context, or other verifier-known memory pointer, and keep the paired length within that object.", + "it is the memory pointer in a memory/length pair with %s, but %s does not describe verifier-readable memory for the requested length", + size_arg, buff_arg); + } else if (failure == BPF_MEM_SIZE_FAIL_SIZE) { + if (reg_smin(size_reg) < 0) + bpf_diag_call_arg_fmt( + env, insn_idx, argno, func_name, + "Constrain the memory size to a non-negative value smaller than BPF_MAX_VAR_SIZ before this call.", + "the memory size in %s may be negative because its signed minimum is %lld", + size_arg, reg_smin(size_reg)); + else + bpf_diag_call_arg_fmt( + env, insn_idx, argno, func_name, + "Constrain the memory size to a non-negative value smaller than BPF_MAX_VAR_SIZ before this call.", + "the memory size in %s may reach %llu bytes, but variable memory accesses must stay below %u bytes", + size_arg, reg_umax(size_reg), BPF_MAX_VAR_SIZ); + } return ret; } break; @@ -12803,8 +13068,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me break; case KF_ARG_PTR_TO_REFCOUNTED_KPTR: if (!type_is_ptr_alloc_obj(reg->type)) { + const char *expected_type; + + expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); verbose(env, "%s is neither owning or non-owning ref\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a pointer returned by the matching BPF object allocation or lookup operation for this kfunc.", + "the kfunc expects a pointer to BPF-managed refcounted object type %s, but this argument is not such an object pointer", + expected_type); return -EINVAL; } if (!type_is_non_owning_ref(reg->type)) @@ -12829,6 +13101,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (reg->type != PTR_TO_MAP_VALUE) { verbose(env, "%s doesn't point to a const string\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a constant string pointer that the verifier recognizes, such as a string stored in a read-only map value.", + "the kfunc expects a pointer to a constant string stored in verifier-known memory, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } ret = check_arg_const_str(env, reg, argno); @@ -12869,6 +13146,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (reg->type != PTR_TO_STACK) { verbose(env, "%s doesn't point to an irq flag on stack\n", reg_arg_name(env, argno)); + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass the same stack slot used by bpf_local_irq_save() or bpf_res_spin_lock_irqsave().", + "the kfunc expects a stack pointer to an IRQ flag slot, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } ret = process_irq_flag(env, reg, argno, meta); diff --git a/tools/testing/selftests/bpf/progs/verifier_map_in_map.c b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c index 7918646e5bfc..d3be69a9a755 100644 --- a/tools/testing/selftests/bpf/progs/verifier_map_in_map.c +++ b/tools/testing/selftests/bpf/progs/verifier_map_in_map.c @@ -155,6 +155,7 @@ l0_%=: r0 = 0; \ SEC("socket") __description("forgot null checking on the inner map pointer") __failure __msg("R1 type=map_ptr_or_null expected=map_ptr") +__msg("map_ptr_or_null, but this argument accepts map_ptr") __failure_unpriv __naked void on_the_inner_map_pointer(void) { From 99a6a288a82bf00ba0b01e72bf85164e848d1d18 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:07 +0200 Subject: [PATCH 334/373] bpf: Report Execution Context Safety errors Augment selected sleepability and critical-section failures with Execution Context Safety reports. Keep the existing verifier messages and add source context, path history, and suggestions tied to the active context. Use the context history recorded earlier to anchor causal paths to lock, IRQ, RCU, and preempt regions instead of unrelated register updates. Cover global calls while holding a lock, sleepable global function calls, sleepable helpers, sleepable kfunc calls from disallowed contexts, operations that exit while a context is still active, and unmatched context exits. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260815064612.378577-13-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 165 ++++++++++++++++++++++++++++++++++++++- kernel/bpf/diagnostics.h | 9 +++ kernel/bpf/verifier.c | 44 +++++++++++ 3 files changed, 217 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 99784d465881..c69160f656e9 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -20,6 +20,7 @@ #define MEMORY_SAFETY "Memory Safety" #define RESOURCE_LIFETIME_SAFETY "Resource Lifetime Safety" #define CALL_TYPE_SAFETY "Call Type Safety" +#define EXECUTION_CONTEXT_SAFETY "Execution Context Safety" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -163,6 +164,7 @@ static void diag_print_history(struct bpf_verifier_env *env, const struct bpf_diag_history_opts *opts); static bool diag_target_matches(const struct bpf_diag_mod_target *event_target, const struct bpf_diag_mod_target *target); +static const char *diag_context_name(enum bpf_diag_context_kind kind); struct disasm_line { char text[DISASM_LINE_LEN]; int idx; @@ -1029,6 +1031,167 @@ void bpf_diag_call_type(struct bpf_verifier_env *env, u32 insn_idx, int argno, i diag_suggestion(env, "%s", suggestion); } +static const char *diag_context_constraint(enum bpf_diag_context_kind kind) +{ + switch (kind) { + case BPF_DIAG_CONTEXT_RCU: + return "RCU read-side critical sections cannot call operations that may sleep"; + case BPF_DIAG_CONTEXT_PREEMPT: + return "preemption-disabled code cannot call operations that may sleep"; + case BPF_DIAG_CONTEXT_IRQ: + return "IRQ-disabled code cannot call operations that may sleep"; + case BPF_DIAG_CONTEXT_LOCK: + return "code holding a BPF spin lock cannot call operations that may sleep"; + case BPF_DIAG_CONTEXT_NONE: + default: + return NULL; + } +} + +static const char *diag_active_context(struct bpf_verifier_env *env, u32 depth, + const char *context) +{ + if (depth == 1) + return bpf_diag_fmt(env, "an active %s (depth 1)", context); + return bpf_diag_fmt(env, "%u active %ss (depth %u)", depth, context, depth); +} + +static u32 diag_context_depth(struct bpf_verifier_env *env, enum bpf_diag_context_kind kind) +{ + switch (kind) { + case BPF_DIAG_CONTEXT_RCU: + return env->cur_state->active_rcu_locks; + case BPF_DIAG_CONTEXT_PREEMPT: + return env->cur_state->active_preempt_locks; + case BPF_DIAG_CONTEXT_IRQ: + return bpf_diag_irq_depth(env->cur_state); + case BPF_DIAG_CONTEXT_LOCK: + return env->cur_state->active_locks; + case BPF_DIAG_CONTEXT_NONE: + default: + return 0; + } +} + +void bpf_diag_ctx_forbidden(struct bpf_verifier_env *env, u32 insn_idx, + const char *operation, const char *suggestion) +{ + struct bpf_diag_history_opts opts; + enum bpf_diag_context_kind ctx_kind; + const char *constraint, *context; + u32 depth; + + if (env->cur_state->active_rcu_locks) + ctx_kind = BPF_DIAG_CONTEXT_RCU; + else if (env->cur_state->active_preempt_locks) + ctx_kind = BPF_DIAG_CONTEXT_PREEMPT; + else if (env->cur_state->active_irq_id) + ctx_kind = BPF_DIAG_CONTEXT_IRQ; + else if (env->cur_state->active_locks) + ctx_kind = BPF_DIAG_CONTEXT_LOCK; + else + ctx_kind = BPF_DIAG_CONTEXT_NONE; + + depth = diag_context_depth(env, ctx_kind); + opts = (struct bpf_diag_history_opts) { + .scope = BPF_DIAG_HISTORY_SCOPE_CONTEXT, + .ctx_kind = ctx_kind, + .ctx_depth = depth, + }; + constraint = diag_context_constraint(ctx_kind); + context = diag_context_name(ctx_kind); + + bpf_diag_header(env, EXECUTION_CONTEXT_SAFETY, + "operation is not allowed in this context"); + if (constraint) { + if (depth) { + diag_reason( + env, "The operation %s cannot be used in %s because %s. This path is still inside %s.", + operation, context, constraint, diag_active_context(env, depth, context)); + } else { + diag_reason(env, "The operation %s cannot be used in %s because %s.", + operation, context, constraint); + } + } else { + diag_reason(env, "The operation %s cannot be used in %s.", operation, + context); + } + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s is not allowed in %s", operation, + context); + + if (ctx_kind != BPF_DIAG_CONTEXT_NONE) + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_ctx_active(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + enum bpf_diag_context_kind ctx_kind, const char *suggestion) +{ + u32 depth = diag_context_depth(env, ctx_kind); + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_CONTEXT, + .ctx_kind = ctx_kind, + .ctx_depth = depth, + }; + const char *context = diag_context_name(ctx_kind); + + bpf_diag_header(env, EXECUTION_CONTEXT_SAFETY, + "operation is not allowed in this context"); + diag_reason( + env, "The operation %s cannot be used while this path is still inside %s. Leave the region before this operation.", + operation, diag_active_context(env, depth, context)); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s is not allowed before leaving %s", + operation, context); + + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_ctx_required(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + enum bpf_diag_context_kind ctx_kind, const char *suggestion) +{ + const char *context = diag_context_name(ctx_kind); + + bpf_diag_header(env, EXECUTION_CONTEXT_SAFETY, "required context is not active"); + diag_reason(env, "The operation %s requires an active %s, but this path is outside one.", + operation, context); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s requires %s", operation, context); + + diag_suggestion(env, "%s", suggestion); +} + +void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx, + const char *operation, enum bpf_diag_context_kind ctx_kind, + const char *suggestion) +{ + struct bpf_diag_history_opts opts = { + .scope = BPF_DIAG_HISTORY_SCOPE_CONTEXT, + .ctx_kind = ctx_kind, + }; + const char *context = diag_context_name(ctx_kind); + + bpf_diag_header(env, EXECUTION_CONTEXT_SAFETY, "unmatched context exit"); + diag_reason( + env, "The operation %s tries to leave %s, but this path has no active %s to leave. The current depth is 0.", + operation, context, context); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s has no matching enter on this path", + operation); + + diag_print_history(env, &opts); + + diag_suggestion(env, "%s", suggestion); +} + void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, const char *reg_name, const struct bpf_reg_state *reg, enum bpf_diag_invalid_deref_kind kind, s64 offset) @@ -2057,7 +2220,7 @@ static const char *diag_context_name(enum bpf_diag_context_kind kind) return "lock region"; case BPF_DIAG_CONTEXT_NONE: default: - return "context"; + return "non-sleepable program"; } } diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index 4b85a7ad2019..95bc654e5b3e 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -80,6 +80,15 @@ void bpf_diag_leak(struct bpf_verifier_env *env, u32 ref_id, u32 alloc_insn, u32 void bpf_diag_call_type(struct bpf_verifier_env *env, u32 insn_idx, int argno, int regno, int stack_arg_slot, const char *call_name, const char *arg_name, const char *reason, const char *suggestion); +void bpf_diag_ctx_forbidden(struct bpf_verifier_env *env, u32 insn_idx, + const char *operation, const char *suggestion); +void bpf_diag_ctx_active(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + enum bpf_diag_context_kind ctx_kind, const char *suggestion); +void bpf_diag_ctx_required(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + enum bpf_diag_context_kind ctx_kind, const char *suggestion); +void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx, + const char *operation, enum bpf_diag_context_kind ctx_kind, + const char *suggestion); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 30e48d4b02f9..0f126f090a47 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7739,6 +7739,9 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state * return err; case -EPROTO: verbose(env, "expected an RCU CS when using %s\n", meta->func_name); + bpf_diag_ctx_required( + env, insn_idx, meta->func_name, BPF_DIAG_CONTEXT_RCU, + "Wrap iterator use in bpf_rcu_read_lock() and bpf_rcu_read_unlock(), keeping all exit paths balanced."); return err; default: return err; @@ -9838,17 +9841,24 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return err; if (bpf_subprog_is_global(env, subprog)) { const char *sub_name = bpf_subprog_name(env, subprog); + const char *operation; bool returns_void; if (env->cur_state->active_locks) { verbose(env, "global function calls are not allowed while holding a lock,\n" "use static function instead\n"); + operation = bpf_diag_fmt(env, "global function %s()", sub_name); + bpf_diag_ctx_active(env, *insn_idx, operation, BPF_DIAG_CONTEXT_LOCK, + "Release the lock before calling the global function, or use a static function instead."); return -EINVAL; } if (env->subprog_info[subprog].might_sleep && !in_sleepable_context(env)) { verbose(env, "sleepable global function %s() called in %s\n", sub_name, non_sleepable_context_description(env)); + operation = bpf_diag_fmt(env, "sleepable global function %s()", sub_name); + bpf_diag_ctx_forbidden(env, *insn_idx, operation, + "Move the call outside the critical section, or use a non-sleepable function."); return -EINVAL; } @@ -10495,6 +10505,8 @@ static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit if (check_lock && env->cur_state->active_locks) { verbose(env, "%s cannot be used inside bpf_spin_lock-ed region\n", prefix); + bpf_diag_ctx_active(env, env->insn_idx, prefix, BPF_DIAG_CONTEXT_LOCK, + "Release the BPF spin lock before this operation on every path."); return -EINVAL; } @@ -10506,16 +10518,23 @@ static int check_resource_leak(struct bpf_verifier_env *env, bool exception_exit if (check_lock && env->cur_state->active_irq_id) { verbose(env, "%s cannot be used inside bpf_local_irq_save-ed region\n", prefix); + bpf_diag_ctx_active(env, env->insn_idx, prefix, BPF_DIAG_CONTEXT_IRQ, + "Restore the saved IRQ state before this operation on every path."); return -EINVAL; } if (check_lock && env->cur_state->active_rcu_locks) { verbose(env, "%s cannot be used inside bpf_rcu_read_lock-ed region\n", prefix); + bpf_diag_ctx_active(env, env->insn_idx, prefix, BPF_DIAG_CONTEXT_RCU, + "Call bpf_rcu_read_unlock() before this operation on every path."); return -EINVAL; } if (check_lock && env->cur_state->active_preempt_locks) { verbose(env, "%s cannot be used inside bpf_preempt_disable-ed region\n", prefix); + bpf_diag_ctx_active( + env, env->insn_idx, prefix, BPF_DIAG_CONTEXT_PREEMPT, + "Call bpf_preempt_enable() before this operation on every path."); return -EINVAL; } @@ -10697,6 +10716,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn enum bpf_type_flag ret_flag; struct bpf_reg_state *regs; struct bpf_call_arg_meta meta; + const char *operation; int insn_idx = *insn_idx_p; bool changes_data; int i, err, func_id; @@ -10744,6 +10764,10 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (fn->might_sleep && !in_sleepable_context(env)) { verbose(env, "sleepable helper %s#%d in %s\n", func_id_name(func_id), func_id, non_sleepable_context_description(env)); + operation = bpf_diag_fmt(env, "sleepable helper %s#%d", + func_id_name(func_id), func_id); + bpf_diag_ctx_forbidden(env, insn_idx, operation, + "Move the helper call outside the critical section, or use a non-sleepable helper."); return -EINVAL; } @@ -13591,6 +13615,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, const struct btf_type *t, *ptr_type; struct bpf_call_arg_meta meta; struct bpf_insn_aux_data *insn_aux; + const char *operation; int err, insn_idx = *insn_idx_p; u32 i, nargs, ptr_type_id; struct bpf_kfunc_desc *desc; @@ -13656,6 +13681,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, sleepable = bpf_is_kfunc_sleepable(&meta); if (sleepable && !in_sleepable(env)) { verbose(env, "program must be sleepable to call sleepable kfunc %s\n", func_name); + operation = bpf_diag_fmt(env, "sleepable kfunc %s", func_name); + bpf_diag_ctx_forbidden(env, insn_idx, operation, + "Mark the program sleepable if the program type allows it, or use a non-sleepable kfunc."); return -EACCES; } @@ -13726,6 +13754,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } else if (rcu_unlock) { if (env->cur_state->active_rcu_locks == 0) { verbose(env, "unmatched rcu read unlock (kernel function %s)\n", func_name); + bpf_diag_ctx_underflow( + env, insn_idx, func_name, BPF_DIAG_CONTEXT_RCU, + "Remove the extra bpf_rcu_read_unlock() call, or ensure this path first enters an RCU read lock region."); return -EINVAL; } env->cur_state->active_rcu_locks--; @@ -13740,6 +13771,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } else if (preempt_enable) { if (env->cur_state->active_preempt_locks == 0) { verbose(env, "unmatched attempt to enable preemption (kernel function %s)\n", func_name); + bpf_diag_ctx_underflow( + env, insn_idx, func_name, BPF_DIAG_CONTEXT_PREEMPT, + "Remove the extra bpf_preempt_enable() call, or ensure this path first disables preemption."); return -EINVAL; } env->cur_state->active_preempt_locks--; @@ -13752,6 +13786,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (sleepable && !in_sleepable_context(env)) { verbose(env, "kernel func %s is sleepable within %s\n", func_name, non_sleepable_context_description(env)); + operation = bpf_diag_fmt(env, "sleepable kfunc %s", func_name); + bpf_diag_ctx_forbidden(env, insn_idx, operation, + "Move the kfunc call outside the critical section, or use a non-sleepable kfunc."); return -EACCES; } @@ -13762,6 +13799,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (is_kfunc_rcu_protected(&meta) && !in_rcu_cs(env)) { verbose(env, "kernel func %s requires RCU critical section protection\n", func_name); + bpf_diag_ctx_required( + env, insn_idx, func_name, BPF_DIAG_CONTEXT_RCU, + "Call this kfunc between bpf_rcu_read_lock() and bpf_rcu_read_unlock(), keeping all exit paths balanced."); return -EACCES; } @@ -18039,6 +18079,10 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state) !kfunc_spin_allowed(env, insn->imm, insn->off))) { verbose(env, "function calls are not allowed while holding a lock\n"); + bpf_diag_ctx_active( + env, env->insn_idx, + "function call", BPF_DIAG_CONTEXT_LOCK, + "Release the BPF spin lock before making this call, or move the call outside the locked region."); return -EINVAL; } } From a8f4278353947d019c990a77d574dfd4d1dc9b46 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:08 +0200 Subject: [PATCH 335/373] bpf: Report Program Structure CFG errors Augment selected whole-program and subprogram CFG validation failures with Program Structure reports. These errors are structural rather than path-dependent, so the reports focus on source and instruction context instead of causal history. Cover direct and indirect jumps outside the program or current subprogram, unprivileged backedges, missing and out-of-range jump tables, targets in the second half of an ldimm64, unreachable instructions, subprogram fallthrough, and recursive bpf2bpf call graph edges. Format long jump-range reasons directly in diagnostics.c, and keep the fallthrough suggestion aligned with the verifier check by suggesting exit or explicit jumps. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-14-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/cfg.c | 35 +++++++++++++++++++++++++++++++++++ kernel/bpf/diagnostics.c | 19 +++++++++++++++++++ kernel/bpf/diagnostics.h | 3 +++ kernel/bpf/verifier.c | 16 ++++++++++++++++ 4 files changed, 73 insertions(+) diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index 818f7afac83a..0f13c13f4133 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -5,6 +5,8 @@ #include #include +#include "diagnostics.h" + #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) /* non-recursive DFS pseudo code @@ -112,6 +114,10 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) if (w < 0 || w >= env->prog->len) { verbose_linfo(env, t, "%d: ", t); verbose(env, "jump out of range from insn %d to %d\n", t, w); + bpf_diag_program_structure( + env, t, "jump out of range", "Keep branch targets inside the program.", + "Instruction %d jumps to instruction %d, but the program only contains instructions 0 through %d.", + t, w, env->prog->len - 1); return -EINVAL; } @@ -135,6 +141,11 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) verbose_linfo(env, t, "%d: ", t); verbose_linfo(env, w, "%d: ", w); verbose(env, "back-edge from insn %d to %d\n", t, w); + bpf_diag_program_structure( + env, t, "back-edge is not allowed", + "Load with privileges that allow this back-edge, or rewrite the control flow so it does not branch backward.", + "Instruction %d branches back to instruction %d. This program is being rejected without the privilege needed for this back-edge.", + t, w); return -EINVAL; } else if (insn_state[w] == EXPLORED) { /* forward- or cross-edge */ @@ -315,6 +326,11 @@ static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env, if (!jt) { verbose(env, "no jump tables found for subprog starting at %u\n", subprog_start); + bpf_diag_program_structure( + env, subprog_start, "missing jump table", + "Make sure subprograms containing gotox instructions are accompanied by jump tables referencing these subprograms.", + "No jump table was found for the subprogram that starts at instruction %u.", + subprog_start); return ERR_PTR(-EINVAL); } @@ -342,6 +358,11 @@ create_jt(int t, struct bpf_verifier_env *env) if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) { verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]\n", t, subprog_start, subprog_end); + bpf_diag_program_structure( + env, t, "jump table target out of range", + "Keep every jump-table target inside the same subprogram.", + "The jump table for instruction %d points outside subprogram range [%u,%u).", + t, subprog_start, subprog_end); kvfree(jt); return ERR_PTR(-EINVAL); } @@ -373,6 +394,11 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env) w = jt->items[i]; if (w < 0 || w >= env->prog->len) { verbose(env, "indirect jump out of range from insn %d to %d\n", t, w); + bpf_diag_program_structure( + env, t, "indirect jump out of range", + "Keep indirect jump targets inside the program.", + "Instruction %d can jump indirectly to instruction %d, but the program only contains instructions 0 through %d.", + t, w, env->prog->len - 1); return -EINVAL; } @@ -623,12 +649,21 @@ int bpf_check_cfg(struct bpf_verifier_env *env) if (insn_state[i] != EXPLORED) { verbose(env, "unreachable insn %d\n", i); + bpf_diag_program_structure( + env, i, "unreachable instruction", + "Remove the unreachable instruction or add valid control flow that reaches it.", + "Instruction %d is not reachable from the program entry point.", i); ret = -EINVAL; goto err_free; } if (bpf_is_ldimm64(insn)) { if (insn_state[i + 1] != 0) { verbose(env, "jump into the middle of ldimm64 insn %d\n", i); + bpf_diag_program_structure( + env, i, "jump into ldimm64 immediate", + "Target the first instruction of the ldimm64 pair, or restructure the jump target.", + "Control flow reaches the second half of the ldimm64 instruction pair that starts at instruction %d.", + i); ret = -EINVAL; goto err_free; } diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index c69160f656e9..9fc1f8cf7312 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -21,6 +21,7 @@ #define RESOURCE_LIFETIME_SAFETY "Resource Lifetime Safety" #define CALL_TYPE_SAFETY "Call Type Safety" #define EXECUTION_CONTEXT_SAFETY "Execution Context Safety" +#define PROGRAM_STRUCTURE "Program Structure" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -1192,6 +1193,24 @@ void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx, diag_suggestion(env, "%s", suggestion); } +void bpf_diag_program_structure(struct bpf_verifier_env *env, u32 insn_idx, + const char *problem, const char *suggestion, + const char *reason_fmt, ...) +{ + va_list args; + + bpf_diag_header(env, PROGRAM_STRUCTURE, problem); + diag_section(env, "Reason"); + + va_start(args, reason_fmt); + diag_vprint_indented(env, reason_fmt, args); + va_end(args); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "%s", problem); + + diag_suggestion(env, "%s", suggestion); +} void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, const char *reg_name, const struct bpf_reg_state *reg, enum bpf_diag_invalid_deref_kind kind, s64 offset) diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index 95bc654e5b3e..ab082d2d6e37 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -89,6 +89,9 @@ void bpf_diag_ctx_required(struct bpf_verifier_env *env, u32 insn_idx, const cha void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, enum bpf_diag_context_kind ctx_kind, const char *suggestion); +void bpf_diag_program_structure(struct bpf_verifier_env *env, u32 insn_idx, + const char *problem, const char *suggestion, + const char *reason_fmt, ...) __printf(5, 6); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0f126f090a47..6a886650cc40 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -3020,6 +3020,12 @@ static int check_subprogs(struct bpf_verifier_env *env) off = i + bpf_jmp_offset(&insn[i]) + 1; if (off < subprog_start || off >= subprog_end) { verbose(env, "jump out of range from insn %d to %d\n", i, off); + bpf_diag_program_structure( + env, i, "jump out of range", + "Keep branch targets within the same subprogram, or use an explicit subprogram call.", + "Instruction %d jumps to instruction %d, but subprogram %d only contains instructions %d through %d. " + "A branch target must stay inside the same subprogram.", + i, off, cur_subprog, subprog_start, subprog_end - 1); return -EINVAL; } next: @@ -3032,6 +3038,11 @@ static int check_subprogs(struct bpf_verifier_env *env) code != (BPF_JMP32 | BPF_JA) && code != (BPF_JMP | BPF_JA)) { verbose(env, "last insn is not an exit or jmp\n"); + bpf_diag_program_structure( + env, i, "subprogram can fall through", + "End each subprogram with an exit or an explicit jump that keeps control flow inside the subprogram.", + "Subprogram %d reaches its last instruction %d without an exit or jump, so control could continue into the next subprogram.", + cur_subprog, i); return -EINVAL; } subprog_start = subprog_end; @@ -3104,6 +3115,11 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env) verbose(env, "recursive call from %s() to %s()\n", bpf_subprog_name(env, cur), bpf_subprog_name(env, callee)); + bpf_diag_program_structure( + env, idx, "recursive subprogram call", + "Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.", + "This bpf2bpf call would make the subprogram call graph recursive. " + "The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis."); ret = -EINVAL; goto out; } From ac545b00ca56368b8acb498107ad4f1d91a5245d Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 15 Aug 2026 08:46:09 +0200 Subject: [PATCH 336/373] bpf: Report Policy helper and kfunc errors Augment selected helper and kfunc allowability failures with Policy reports. These reports explain which requested operation is forbidden and why, without adding path history for non-path-dependent policy checks. Cover unprivileged bpf2bpf and kfunc use, helper program-type restrictions, GPL-only helpers, helper-specific allow callbacks, kfunc allowability, and destructive kfunc capability checks. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260815064612.378577-15-memxor@gmail.com Signed-off-by: Eduard Zingerman --- kernel/bpf/diagnostics.c | 14 ++++++++++++++ kernel/bpf/diagnostics.h | 2 ++ kernel/bpf/verifier.c | 33 ++++++++++++++++++++++++++++++++- 3 files changed, 48 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 9fc1f8cf7312..33b7d9e8e2c3 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -22,6 +22,7 @@ #define CALL_TYPE_SAFETY "Call Type Safety" #define EXECUTION_CONTEXT_SAFETY "Execution Context Safety" #define PROGRAM_STRUCTURE "Program Structure" +#define POLICY "Policy" #define BPF_DIAG_TEXT_WIDTH 100 #define BPF_DIAG_TEXT_INDENT " " @@ -1211,6 +1212,19 @@ void bpf_diag_program_structure(struct bpf_verifier_env *env, u32 insn_idx, diag_suggestion(env, "%s", suggestion); } + +void bpf_diag_policy(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + const char *reason, const char *suggestion) +{ + bpf_diag_header(env, POLICY, "operation is not allowed"); + diag_reason(env, "The %s is not allowed: %s.", operation, reason); + + diag_section(env, "At"); + bpf_diag_source(env, insn_idx, "error", "policy check failed for %s", operation); + + diag_suggestion(env, "%s", suggestion); +} + void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno, const char *reg_name, const struct bpf_reg_state *reg, enum bpf_diag_invalid_deref_kind kind, s64 offset) diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h index ab082d2d6e37..d1b79945008a 100644 --- a/kernel/bpf/diagnostics.h +++ b/kernel/bpf/diagnostics.h @@ -92,6 +92,8 @@ void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx, void bpf_diag_program_structure(struct bpf_verifier_env *env, u32 insn_idx, const char *problem, const char *suggestion, const char *reason_fmt, ...) __printf(5, 6); +void bpf_diag_policy(struct bpf_verifier_env *env, u32 insn_idx, const char *operation, + const char *reason, const char *suggestion); void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true); void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6a886650cc40..0bb7ee95c8bd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2924,6 +2924,10 @@ static int add_subprogs(struct bpf_verifier_env *env) if (!env->bpf_capable) { verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n"); + bpf_diag_policy( + env, i, "BPF-to-BPF function call", + "loading or calling other BPF functions requires CAP_BPF or CAP_SYS_ADMIN", + "Load this program with the required capability, or avoid BPF-to-BPF function calls in unprivileged programs."); return -EPERM; } @@ -2976,6 +2980,10 @@ static int add_kfuncs(struct bpf_verifier_env *env) if (!env->bpf_capable) { verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n"); + bpf_diag_policy( + env, i, "kernel function call", + "calling kernel functions requires CAP_BPF or CAP_SYS_ADMIN", + "Load this program with the required capability, or avoid kernel function calls in unprivileged programs."); return -EPERM; } @@ -10748,17 +10756,31 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (err) { verbose(env, "program of this type cannot use helper %s#%d\n", func_id_name(func_id), func_id); + operation = bpf_diag_fmt(env, "helper %s#%d", func_id_name(func_id), func_id); + bpf_diag_policy( + env, insn_idx, operation, "this program type does not allow the helper", + "Use a helper allowed for this program type, or move the logic to a compatible program type."); return err; } /* eBPF programs must be GPL compatible to use GPL-ed functions */ if (!env->prog->gpl_compatible && fn->gpl_only) { verbose(env, "cannot call GPL-restricted function from non-GPL compatible program\n"); + operation = bpf_diag_fmt(env, "helper %s#%d", func_id_name(func_id), func_id); + bpf_diag_policy( + env, insn_idx, operation, + "this helper is restricted to GPL-compatible programs", + "Use a GPL-compatible license, or replace the helper with one that is available to non-GPL programs."); return -EINVAL; } if (fn->allowed && !fn->allowed(env->prog)) { verbose(env, "helper call is not allowed in probe\n"); + operation = bpf_diag_fmt(env, "helper %s#%d", func_id_name(func_id), func_id); + bpf_diag_policy( + env, insn_idx, operation, + "the helper-specific policy callback rejected this program", + "Use the helper only from an allowed attach point or program configuration."); return -EINVAL; } @@ -13643,8 +13665,13 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return 0; err = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta); - if (err == -EACCES && meta.func_name) + if (err == -EACCES && meta.func_name) { verbose(env, "calling kernel function %s is not allowed\n", meta.func_name); + operation = bpf_diag_fmt(env, "kfunc %s", meta.func_name); + bpf_diag_policy( + env, insn_idx, operation, "this program cannot call the kfunc", + "Use a kfunc allowed for this program type and attach point, or change the program context."); + } if (err) return err; desc_btf = meta.btf; @@ -13691,6 +13718,10 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, if (is_kfunc_destructive(&meta) && !capable(CAP_SYS_BOOT)) { verbose(env, "destructive kfunc calls require CAP_SYS_BOOT capability\n"); + operation = bpf_diag_fmt(env, "destructive kfunc %s", meta.func_name); + bpf_diag_policy( + env, insn_idx, operation, "destructive kfuncs require CAP_SYS_BOOT", + "Load the program with CAP_SYS_BOOT, or avoid destructive kfuncs."); return -EACCES; } From 5bd369c05576ec12f62f02c65b576bf0bf074131 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 13 Aug 2026 11:05:36 +0000 Subject: [PATCH 337/373] net: Add connect_socket() helper Add a helper that connects an existing socket while invoking the LSM hook. Reuse it in __sys_connect_file() to avoid duplicating the connect logic. Other socket operations have equivalent helpers that trigger the appropriate LSM hooks that can be reused, this one was the only one missing. This will be used in the next commit for a new BPF kfunc that needs to connect a socket and trigger the LSM hook. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Jiayuan Chen Reviewed-by: Kuniyuki Iwashima Acked-by: Song Liu Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260813110540.103550-2-mahe.tardy@gmail.com --- include/linux/socket.h | 2 ++ net/socket.c | 32 ++++++++++++++++++-------------- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index 2a8d7b14f1d1..5a5eb1250103 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -461,6 +461,8 @@ extern struct file *__sys_socket_file(int family, int type, int protocol); extern int __sys_bind(int fd, struct sockaddr __user *umyaddr, int addrlen); extern int __sys_bind_socket(struct socket *sock, struct sockaddr_storage *address, int addrlen); +int connect_socket(struct socket *sock, struct sockaddr_storage *addr, + int addrlen, int flags); extern int __sys_connect_file(struct file *file, struct sockaddr_storage *addr, int addrlen, int file_flags); extern int __sys_connect(int fd, struct sockaddr __user *uservaddr, diff --git a/net/socket.c b/net/socket.c index 63c69a0fa74e..5b02e6217c68 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2103,6 +2103,20 @@ SYSCALL_DEFINE3(accept, int, fd, struct sockaddr __user *, upeer_sockaddr, return __sys_accept4(fd, upeer_sockaddr, upeer_addrlen, 0); } +int connect_socket(struct socket *sock, struct sockaddr_storage *address, + int addrlen, int flags) +{ + int err; + + err = security_socket_connect(sock, (struct sockaddr *)address, + addrlen); + if (err) + return err; + + return READ_ONCE(sock->ops)->connect(sock, (struct sockaddr_unsized *)address, + addrlen, flags); +} + /* * Attempt to connect to a socket with the server address. The address * is in user space so we verify it is OK and move it to kernel space. @@ -2119,23 +2133,13 @@ int __sys_connect_file(struct file *file, struct sockaddr_storage *address, int addrlen, int file_flags) { struct socket *sock; - int err; sock = sock_from_file(file); - if (!sock) { - err = -ENOTSOCK; - goto out; - } + if (!sock) + return -ENOTSOCK; - err = - security_socket_connect(sock, (struct sockaddr *)address, addrlen); - if (err) - goto out; - - err = READ_ONCE(sock->ops)->connect(sock, (struct sockaddr_unsized *)address, - addrlen, sock->file->f_flags | file_flags); -out: - return err; + return connect_socket(sock, address, addrlen, + sock->file->f_flags | file_flags); } int __sys_connect(int fd, struct sockaddr __user *uservaddr, int addrlen) From 7ae4eb14c5f9d9bf0e0feabeab206151b1280512 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 13 Aug 2026 11:05:37 +0000 Subject: [PATCH 338/373] bpf: Add ksock kfuncs Add BPF kfuncs that allow BPF LSM programs to create and use sockets for sending data. This provides a mechanism for BPF programs to emit telemetry. For this first patch set, it's restricted to SOCK_DGRAM socket types with IPPROTO_UDP protocol but could be easily extended to SOCK_STREAM and IPPROTO_TCP in the future. The API consists of five kfuncs: bpf_ksock_create() - Create a socket (sleepable) bpf_ksock_connect() - Connect socket to remote address (sleepable) bpf_ksock_send() - Send data through the socket (sleepable) bpf_ksock_acquire() - Acquire a reference to a socket context bpf_ksock_release() - Release a reference (cleanup via queue_rcu_work since sock_release sleeps) The setup kfuncs bpf_ksock_create, bpf_ksock_connect, can be called from SYSCALL programs only. While bpf_ksock_acquire, bpf_ksock_release and bpf_ksock_send can be called from SYSCALL and LSM programs. The implementation follows the established kfunc lifecycle pattern (create/acquire/release with refcounting, kptr map storage, dtor registration). The kernel socket is wrapped in a refcounted bpf_ksock struct. Cleanup is deferred via queue_rcu_work() because sock_release() may sleep. The kfuncs are only compiled when CONFIG_INET is enabled, as they specifically support AF_INET and AF_INET6 sockets. The socket operations go through the expected LSM hooks instead of by-passing them like many kernel sockets since those are created by BPF programs and thus system users. Thus, the bpf_ksock_send() kfunc, which is exposed to LSM progs has a verifier filter protection to avoid recursion so that the whole bpf_kfunc_set kfunc set cannot be called in a program attached to security_socket_sendmsg(). Also, because of the LSM checks, we prevent the use of the kfuncs from asynchronous workqueue as the current value would then be invalid. In bpf_ksock_create(), we copy the arg values to avoid TOCTOU races since the kfunc can sleep and the arg values could be stored in a map that could be re-written by BPF progs or even userspace programs if the map is mmaped. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Jiayuan Chen Acked-by: Stanislav Fomichev Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260813110540.103550-3-mahe.tardy@gmail.com --- include/linux/bpf_ksock.h | 36 +++++ kernel/bpf/verifier.c | 3 + net/core/Makefile | 3 + net/core/bpf_ksock.c | 328 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 370 insertions(+) create mode 100644 include/linux/bpf_ksock.h create mode 100644 net/core/bpf_ksock.c diff --git a/include/linux/bpf_ksock.h b/include/linux/bpf_ksock.h new file mode 100644 index 000000000000..cb387fb75e43 --- /dev/null +++ b/include/linux/bpf_ksock.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2026 Isovalent */ + +#ifndef _BPF_KSOCK_H +#define _BPF_KSOCK_H + +#include +#include +#include + +/** + * struct bpf_ksock_create_opts - BPF kernel socket creation parameters + * @family: Address family: AF_INET or AF_INET6. + * @type: Socket type: only SOCK_DGRAM supported for now. + * @protocol: Protocol number (e.g. IPPROTO_UDP), or 0 for the default protocol + * of the given type. + * @reserved: Must be zero. Reserved for future use. + */ +struct bpf_ksock_create_opts { + __u8 family; + __u8 type; + __u8 protocol; + __u8 reserved; +}; + +/** + * union bpf_ksock_addr - IPv4 or IPv6 socket address + * @sin: IPv4 socket address. + * @sin6: IPv6 socket address. + */ +union bpf_ksock_addr { + struct sockaddr_in sin; + struct sockaddr_in6 sin6; +}; + +#endif /* _BPF_KSOCK_H */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0bb7ee95c8bd..d17f14b35b79 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4564,6 +4564,9 @@ BTF_ID(struct, task_struct) #ifdef CONFIG_CRYPTO BTF_ID(struct, bpf_crypto_ctx) #endif +#ifdef CONFIG_INET +BTF_ID(struct, bpf_ksock) +#endif BTF_SET_END(rcu_protected_types) static bool rcu_protected_object(const struct btf *btf, u32 btf_id) diff --git a/net/core/Makefile b/net/core/Makefile index b3fdcb4e355f..c20e520358b8 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -44,6 +44,9 @@ obj-$(CONFIG_FAILOVER) += failover.o obj-$(CONFIG_NET_SOCK_MSG) += skmsg.o obj-$(CONFIG_BPF_SYSCALL) += sock_map.o obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o +ifdef CONFIG_INET +obj-$(CONFIG_BPF_SYSCALL) += bpf_ksock.o +endif obj-$(CONFIG_OF) += of_net.o obj-$(CONFIG_NET_TEST) += net_test.o obj-$(CONFIG_NET_DEVMEM) += devmem.o diff --git a/net/core/bpf_ksock.c b/net/core/bpf_ksock.c new file mode 100644 index 000000000000..e9943aeeadd3 --- /dev/null +++ b/net/core/bpf_ksock.c @@ -0,0 +1,328 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2026 Isovalent */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * struct bpf_ksock - refcounted BPF kernel socket context + * @sock: The underlying kernel socket. + * @usage: Reference counter. + * @rwork: RCU work for deferred cleanup (sock_release may sleep). + */ +struct bpf_ksock { + struct socket *sock; + refcount_t usage; + struct rcu_work rwork; +}; + +static void ksock_release_work_fn(struct work_struct *work) +{ + struct bpf_ksock *ks; + + ks = container_of(to_rcu_work(work), struct bpf_ksock, rwork); + sock_release(ks->sock); + kfree(ks); +} + +static bool bpf_ksock_has_user_task_context(void) +{ + /* + * Task work can run from do_exit() after exit_nsproxy_namespaces() + * cleared current->nsproxy, while current is still not a kthread. + */ + return !(current->flags & PF_KTHREAD) && current->nsproxy; +} + +__bpf_kfunc_start_defs(); + +/** + * bpf_ksock_create() - Create a BPF kernel socket. + * + * Allocates and creates a kernel socket. + * + * The returned context must either be stored in a map as a kptr, or + * freed with bpf_ksock_release(). + * + * This function may sleep (sock_create), so it can only be used + * in sleepable BPF programs (SYSCALL). + * It cannot be called from a BPF workqueue callback because that callback + * does not retain the invoking task's namespace or security context. + * + * @opts: Pointer to struct bpf_ksock_create_opts with socket parameters. + * @opts__sz: Size of the opts struct. + * @err__uninit: Integer to store error code when NULL is returned. + */ +__bpf_kfunc struct bpf_ksock * +bpf_ksock_create(const struct bpf_ksock_create_opts *opts, u32 opts__sz, + int *err__uninit) +{ + struct bpf_ksock_create_opts opts_copy; + struct bpf_ksock *ks; + int err; + + /* + * sock_create() derives the network namespace, credentials, and cgroup + * from current. Kernel threads, including BPF workqueue callbacks, do + * not carry the context of the task that invoked the BPF program. + */ + if (!bpf_ksock_has_user_task_context()) { + err = -EOPNOTSUPP; + goto err_out; + } + + if (!opts || opts__sz != sizeof(struct bpf_ksock_create_opts)) { + err = -EINVAL; + goto err_out; + } + + opts_copy = (struct bpf_ksock_create_opts){ + .family = READ_ONCE(opts->family), + .type = READ_ONCE(opts->type), + .protocol = READ_ONCE(opts->protocol), + .reserved = READ_ONCE(opts->reserved), + }; + + if (opts_copy.reserved) { + err = -EINVAL; + goto err_out; + } + + if (opts_copy.family != AF_INET && opts_copy.family != AF_INET6) { + err = -EAFNOSUPPORT; + goto err_out; + } + + if (opts_copy.type != SOCK_DGRAM) { + err = -EPROTONOSUPPORT; + goto err_out; + } + + if (opts_copy.protocol != IPPROTO_UDP && opts_copy.protocol != 0) { + err = -EPROTONOSUPPORT; + goto err_out; + } + + ks = kzalloc_obj(*ks); + if (!ks) { + err = -ENOMEM; + goto err_out; + } + + /* + * Use the normal current-task socket path so LSM/cgroup policy, + * socket labels, and the active netns reference match a socket(2) + * created by the BPF program's caller. + */ + err = sock_create(opts_copy.family, opts_copy.type, opts_copy.protocol, + &ks->sock); + if (err) + goto err_free; + + ks->sock->sk->sk_rcvbuf = SOCK_MIN_RCVBUF; + ks->sock->sk->sk_userlocks |= SOCK_RCVBUF_LOCK; + + refcount_set(&ks->usage, 1); + put_unaligned(0, err__uninit); + return ks; + +err_free: + kfree(ks); +err_out: + put_unaligned(err, err__uninit); + return NULL; +} + +/** + * bpf_ksock_connect() - Connect a BPF kernel socket to a remote address. + * @ks: The BPF kernel socket context. + * @addr: Pointer to an IPv4 or IPv6 socket address. + * @addr__sz: Size of the address union. + * + * Connects the socket to the specified remote address and port. + * + * This function may sleep while connecting the socket, so it can only be used + * in sleepable BPF programs (SYSCALL). + * + * Return: 0 on success, negative errno on error. + */ +__bpf_kfunc int bpf_ksock_connect(struct bpf_ksock *ks, + const union bpf_ksock_addr *addr, + u32 addr__sz) +{ + struct sockaddr_storage sa; + int addrlen; + + if (!bpf_ksock_has_user_task_context()) + return -EOPNOTSUPP; + + if (!addr || addr__sz != sizeof(*addr)) + return -EINVAL; + + /* Kfunc memory arguments may be unaligned. */ + memcpy(&sa, addr, sizeof(*addr)); + + switch (sa.ss_family) { + case AF_INET: + addrlen = sizeof(struct sockaddr_in); + break; + case AF_INET6: + addrlen = sizeof(struct sockaddr_in6); + break; + default: + return -EAFNOSUPPORT; + } + + return connect_socket(ks->sock, &sa, addrlen, 0); +} + +/** + * bpf_ksock_acquire() - Acquire a reference to a BPF kernel socket. + * @ks: The BPF kernel socket context to acquire. Must be a + * trusted pointer (e.g. RCU-protected kptr from a map). + * + * The acquired context must either be stored in a map as a kptr, or + * freed with bpf_ksock_release(). + */ +__bpf_kfunc struct bpf_ksock *bpf_ksock_acquire(struct bpf_ksock *ks) +{ + if (!refcount_inc_not_zero(&ks->usage)) + return NULL; + return ks; +} + +/** + * bpf_ksock_release() - Release a BPF kernel socket. + * @ks: The BPF kernel socket context to release. + * + * When the final reference is released, the socket is cleaned up via + * queue_rcu_work() (since sock_release may sleep). + */ +__bpf_kfunc void bpf_ksock_release(struct bpf_ksock *ks) +{ + if (refcount_dec_and_test(&ks->usage)) { + INIT_RCU_WORK(&ks->rwork, ksock_release_work_fn); + queue_rcu_work(system_dfl_wq, &ks->rwork); + } +} + +__bpf_kfunc void bpf_ksock_release_dtor(void *ks) +{ + bpf_ksock_release(ks); +} +CFI_NOSEAL(bpf_ksock_release_dtor); + +/** + * bpf_ksock_send() - Send data through a BPF kernel socket. + * @ks: The BPF kernel socket context. Must be an acquired reference. + * @data: Pointer to the data to send. + * @data__sz: Size of the data to send. + * + * Sends data on a connected socket, best-effort and nonblocking. This may sleep + * (kernel_sendmsg), so it can only be called from sleepable BPF programs. + * + * Return: Number of bytes sent on success, negative errno on error. + */ +__bpf_kfunc int bpf_ksock_send(struct bpf_ksock *ks, const void *data, + u32 data__sz) +{ + struct msghdr msg = { + .msg_flags = MSG_DONTWAIT, + }; + struct kvec iov = { + .iov_base = (void *)data, + .iov_len = data__sz, + }; + int ret; + + if (!bpf_ksock_has_user_task_context()) + return -EOPNOTSUPP; + + ret = kernel_sendmsg(ks->sock, &msg, &iov, 1, data__sz); + + return ret; +} + +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(ksock_init_kfunc_btf_ids) +BTF_ID_FLAGS(func, bpf_ksock_create, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_ksock_connect, KF_SLEEPABLE) +BTF_KFUNCS_END(ksock_init_kfunc_btf_ids) + +static const struct btf_kfunc_id_set ksock_init_kfunc_set = { + .owner = THIS_MODULE, + .set = &ksock_init_kfunc_btf_ids, +}; + +BTF_KFUNCS_START(ksock_kfunc_btf_ids) +BTF_ID_FLAGS(func, bpf_ksock_release, KF_RELEASE) +BTF_ID_FLAGS(func, bpf_ksock_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_ksock_send, KF_SLEEPABLE) +BTF_KFUNCS_END(ksock_kfunc_btf_ids) + +#ifdef CONFIG_BPF_LSM +BTF_ID_LIST_SINGLE(bpf_lsm_socket_sendmsg_id, func, bpf_lsm_socket_sendmsg) +#endif + +static int bpf_ksock_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id) +{ + if (!btf_id_set8_contains(&ksock_kfunc_btf_ids, kfunc_id)) + return 0; + + if (prog->type == BPF_PROG_TYPE_SYSCALL) + return 0; + +#ifdef CONFIG_BPF_LSM + if (prog->type == BPF_PROG_TYPE_LSM && + prog->aux->attach_btf_id != bpf_lsm_socket_sendmsg_id[0]) + return 0; +#endif + + return -EACCES; +} + +static const struct btf_kfunc_id_set ksock_kfunc_set = { + .owner = THIS_MODULE, + .set = &ksock_kfunc_btf_ids, + .filter = bpf_ksock_kfunc_filter, +}; + +BTF_ID_LIST(bpf_ksock_dtor_ids) +BTF_ID(struct, bpf_ksock) +BTF_ID(func, bpf_ksock_release_dtor) + +static int __init bpf_ksock_kfunc_init(void) +{ + int ret; + const struct btf_id_dtor_kfunc bpf_ksock_dtors[] = { + { + .btf_id = bpf_ksock_dtor_ids[0], + .kfunc_btf_id = bpf_ksock_dtor_ids[1], + }, + }; + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, + &ksock_init_kfunc_set); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, + &ksock_kfunc_set); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LSM, + &ksock_kfunc_set); + return ret ?: register_btf_id_dtor_kfuncs(bpf_ksock_dtors, + ARRAY_SIZE(bpf_ksock_dtors), + THIS_MODULE); +} + +late_initcall(bpf_ksock_kfunc_init); From c7838e3dc61a1e3e0a44f73267ffd33fa2d5e083 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 13 Aug 2026 11:05:38 +0000 Subject: [PATCH 339/373] selftests/bpf: Add ksock kfunc test Add a selftest that exercises the ksock kfuncs end-to-end. One syscall BPF setup program creates a ksock context and connects the socket. Another LSM sleepable BPF program looks up the context and send test data. The userspace harness creates a network namespace and a new socket on loopback, run the setup syscall prog and send LSM BPF prog then check that the userspace socket received the data from BPF. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Reviewed-by: Jiayuan Chen Acked-by: Stanislav Fomichev Link: https://lore.kernel.org/bpf/20260813110540.103550-4-mahe.tardy@gmail.com --- .../testing/selftests/bpf/prog_tests/ksock.c | 124 ++++++++++++++++++ .../selftests/bpf/progs/ksock_common.h | 78 +++++++++++ tools/testing/selftests/bpf/progs/ksock_lsm.c | 72 ++++++++++ 3 files changed, 274 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/ksock.c create mode 100644 tools/testing/selftests/bpf/progs/ksock_common.h create mode 100644 tools/testing/selftests/bpf/progs/ksock_lsm.c diff --git a/tools/testing/selftests/bpf/prog_tests/ksock.c b/tools/testing/selftests/bpf/prog_tests/ksock.c new file mode 100644 index 000000000000..05d7b7424aee --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/ksock.c @@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Isovalent */ + +#include + +#include "test_progs.h" +#include "network_helpers.h" +#include "ksock_lsm.skel.h" + +#define NS_TEST "ksock_lsm_ns" +#define RECV_PORT 7777 +#define RECV_TIMEOUT_SEC 5 + +struct ksock_test_env { + struct nstoken *nstoken; + int rfd; +}; + +static bool ksock_test_env_setup(struct ksock_test_env *env) +{ + struct sockaddr_in addr = { + .sin_family = AF_INET, + .sin_addr.s_addr = htonl(INADDR_LOOPBACK), + .sin_port = htons(RECV_PORT), + }; + struct timeval tv = { .tv_sec = RECV_TIMEOUT_SEC }; + int err; + + memset(env, 0, sizeof(*env)); + env->rfd = -1; + + if (!ASSERT_OK(make_netns(NS_TEST), "make_netns")) + goto fail; + + env->nstoken = open_netns(NS_TEST); + if (!ASSERT_OK_PTR(env->nstoken, "open_netns")) + goto fail; + + env->rfd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (!ASSERT_OK_FD(env->rfd, "receiver socket")) + goto fail; + + err = bind(env->rfd, (struct sockaddr *)&addr, sizeof(addr)); + if (!ASSERT_OK(err, "bind receiver")) + goto fail; + + err = setsockopt(env->rfd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + if (!ASSERT_OK(err, "set rcvtimeo")) + goto fail; + + return true; + +fail: + return false; +} + +void test_ksock_lsm(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct ksock_test_env env; + struct sockaddr_in trigger_addr = { + .sin_family = AF_INET, + .sin_addr.s_addr = htonl(INADDR_LOOPBACK), + }; + struct ksock_lsm *skel; + char recv_data[sizeof(skel->data->send_data)] = {}; + ssize_t n; + int tfd = -1; + int err; + + skel = ksock_lsm__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel open_and_load")) + return; + + if (!ksock_test_env_setup(&env)) + goto fail; + + /* Step 1: Run the setup SYSCALL prog to create the ksock */ + skel->bss->ipv4_remote = htonl(INADDR_LOOPBACK); + skel->bss->remote_port = RECV_PORT; + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.ksock_setup), + &opts); + if (!ASSERT_OK(err, "ksock_setup run")) + goto fail; + if (!ASSERT_OK(opts.retval, "ksock_setup retval")) + goto fail; + + /* Step 2: Attach LSM prog and trigger socket_bind from userspace */ + skel->links.ksock_socket_bind = + bpf_program__attach_lsm(skel->progs.ksock_socket_bind); + if (!ASSERT_OK_PTR(skel->links.ksock_socket_bind, + "attach socket_bind lsm")) + goto fail; + + tfd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); + if (!ASSERT_OK_FD(tfd, "trigger socket")) + goto fail; + + skel->bss->target_pid = getpid(); + err = bind(tfd, (struct sockaddr *)&trigger_addr, sizeof(trigger_addr)); + skel->bss->target_pid = 0; + if (!ASSERT_OK(err, "trigger bind")) + goto fail; + + /* Step 3: Verify the LSM hook sent the notification */ + if (!ASSERT_EQ(skel->data->send_ret, sizeof(skel->data->send_data), + "LSM send bytes")) + goto fail; + + n = recvfrom(env.rfd, recv_data, sizeof(recv_data), 0, NULL, NULL); + if (ASSERT_EQ(n, sizeof(recv_data), "recvfrom len")) + ASSERT_MEMEQ(recv_data, skel->data->send_data, sizeof(recv_data), + "payload match"); + +fail: + if (tfd >= 0) + close(tfd); + if (env.rfd >= 0) + close(env.rfd); + if (env.nstoken) + close_netns(env.nstoken); + remove_netns(NS_TEST); + ksock_lsm__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/ksock_common.h b/tools/testing/selftests/bpf/progs/ksock_common.h new file mode 100644 index 000000000000..01edaeb9fdd4 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/ksock_common.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Isovalent */ + +#ifndef _KSOCK_COMMON_H +#define _KSOCK_COMMON_H + +#include "errno.h" + +#define SOCK_DGRAM 2 +#define IPPROTO_UDP 17 + +struct bpf_ksock *bpf_ksock_create(const struct bpf_ksock_create_opts *opts, + u32 opts__sz, int *err__uninit) __ksym; +int bpf_ksock_connect(struct bpf_ksock *ks, const union bpf_ksock_addr *addr, + u32 addr__sz) __ksym; +struct bpf_ksock *bpf_ksock_acquire(struct bpf_ksock *ks) __ksym; +void bpf_ksock_release(struct bpf_ksock *ks) __ksym; +int bpf_ksock_send(struct bpf_ksock *ks, const void *data, u32 data__sz) __ksym; +void bpf_rcu_read_lock(void) __ksym; +void bpf_rcu_read_unlock(void) __ksym; + +struct __ksock_ctx_value { + struct bpf_ksock __kptr * ctx; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct __ksock_ctx_value); + __uint(max_entries, 1); +} __ksock_ctx_map SEC(".maps"); + +static inline struct __ksock_ctx_value *ksock_ctx_value_lookup(void) +{ + u32 key = 0; + + return bpf_map_lookup_elem(&__ksock_ctx_map, &key); +} + +static inline struct bpf_ksock *ksock_ctx_get(void) +{ + struct __ksock_ctx_value *v; + struct bpf_ksock *ks = NULL, *tmp; + + v = ksock_ctx_value_lookup(); + if (!v) + return NULL; + + bpf_rcu_read_lock(); + tmp = v->ctx; + if (tmp) + ks = bpf_ksock_acquire(tmp); + bpf_rcu_read_unlock(); + + return ks; +} + +static inline int ksock_ctx_insert(struct bpf_ksock *ctx) +{ + struct __ksock_ctx_value *v; + struct bpf_ksock *old; + + v = ksock_ctx_value_lookup(); + if (!v) { + bpf_ksock_release(ctx); + return -ENOENT; + } + + old = bpf_kptr_xchg(&v->ctx, ctx); + if (old) { + bpf_ksock_release(old); + return -EEXIST; + } + + return 0; +} + +#endif /* _KSOCK_COMMON_H */ diff --git a/tools/testing/selftests/bpf/progs/ksock_lsm.c b/tools/testing/selftests/bpf/progs/ksock_lsm.c new file mode 100644 index 000000000000..9808451098ef --- /dev/null +++ b/tools/testing/selftests/bpf/progs/ksock_lsm.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Isovalent */ + +#include "vmlinux.h" +#include +#include +#include +#include "bpf_tracing_net.h" +#include "ksock_common.h" + +char send_data[32] = "hello from bpf ksock"; + +__be32 ipv4_remote; +__u16 remote_port; +int target_pid; +int send_ret = -1; + +SEC("syscall") +int ksock_setup(void *ctx) +{ + struct bpf_ksock_create_opts create_opts = {}; + union bpf_ksock_addr addr = {}; + struct bpf_ksock *ks; + int err = 0; + + create_opts.family = AF_INET; + create_opts.type = SOCK_DGRAM; + create_opts.protocol = IPPROTO_UDP; + + ks = bpf_ksock_create(&create_opts, sizeof(create_opts), &err); + if (!ks) + return err; + + addr.sin.sin_family = AF_INET; + addr.sin.sin_port = bpf_htons(remote_port); + addr.sin.sin_addr.s_addr = ipv4_remote; + + err = bpf_ksock_connect(ks, &addr, sizeof(addr)); + if (err) { + bpf_ksock_release(ks); + return err; + } + + err = ksock_ctx_insert(ks); + if (err && err != -EEXIST) + return err; + return 0; +} + +SEC("lsm.s/socket_bind") +int BPF_PROG(ksock_socket_bind, struct socket *sock, struct sockaddr *address, + int addrlen, int ret) +{ + struct bpf_ksock *ks; + u32 pid = bpf_get_current_pid_tgid() >> 32; + + if (ret || pid != target_pid) + return ret; + + ks = ksock_ctx_get(); + if (!ks) { + send_ret = -ENOENT; + return ret; + } + + send_ret = bpf_ksock_send(ks, send_data, sizeof(send_data)); + bpf_ksock_release(ks); + + return ret; +} + +char __license[] SEC("license") = "GPL"; From 7b0dfbf5776bf75fadcb59fce1417c187bbb9943 Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 13 Aug 2026 11:05:39 +0000 Subject: [PATCH 340/373] selftests/bpf: Test forbidden bpf_ksock_send() LSM attach The bpf_ksock_send() kfunc eventually calls security_socket_sendmsg(), thus creating a possible recursion if a program calling the kfunc is attached on that specific hook. A filter is added on the kfunc registration to prevent that at load time from the verifier. This test exercises that the verifier will reject such program on that attach point. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Acked-by: Stanislav Fomichev Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260813110540.103550-5-mahe.tardy@gmail.com --- .../testing/selftests/bpf/prog_tests/ksock.c | 6 ++++ .../selftests/bpf/progs/ksock_lsm_verifier.c | 35 +++++++++++++++++++ 2 files changed, 41 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/ksock_lsm_verifier.c diff --git a/tools/testing/selftests/bpf/prog_tests/ksock.c b/tools/testing/selftests/bpf/prog_tests/ksock.c index 05d7b7424aee..dd6b167623d9 100644 --- a/tools/testing/selftests/bpf/prog_tests/ksock.c +++ b/tools/testing/selftests/bpf/prog_tests/ksock.c @@ -6,6 +6,7 @@ #include "test_progs.h" #include "network_helpers.h" #include "ksock_lsm.skel.h" +#include "ksock_lsm_verifier.skel.h" #define NS_TEST "ksock_lsm_ns" #define RECV_PORT 7777 @@ -122,3 +123,8 @@ void test_ksock_lsm(void) remove_netns(NS_TEST); ksock_lsm__destroy(skel); } + +void test_ksock_lsm_verifier(void) +{ + RUN_TESTS(ksock_lsm_verifier); +} diff --git a/tools/testing/selftests/bpf/progs/ksock_lsm_verifier.c b/tools/testing/selftests/bpf/progs/ksock_lsm_verifier.c new file mode 100644 index 000000000000..fd2ccfdb5802 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/ksock_lsm_verifier.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Isovalent */ + +#include "vmlinux.h" +#include +#include +#include "bpf_misc.h" +#include "ksock_common.h" + +char send_data[11] = "dummy data"; + +SEC("lsm.s/socket_sendmsg") +__description("bpf_ksock_send is rejected from socket_sendmsg LSM hook") +__failure __msg("calling kernel function bpf_ksock_send is not allowed") +int BPF_PROG(ksock_socket_sendmsg, struct socket *sock, struct msghdr *msg, + int size, int ret) +{ + struct __ksock_ctx_value *v; + struct bpf_ksock *ks; + + v = ksock_ctx_value_lookup(); + if (!v) + return ret; + + ks = bpf_kptr_xchg(&v->ctx, NULL); + if (!ks) + return ret; + + bpf_ksock_send(ks, send_data, sizeof(send_data)); + bpf_ksock_release(ks); + + return ret; +} + +char __license[] SEC("license") = "GPL"; From c93cbdb13f995f87b5356329b3fe551c80bb482d Mon Sep 17 00:00:00 2001 From: Mahe Tardy Date: Thu, 13 Aug 2026 11:05:40 +0000 Subject: [PATCH 341/373] selftests/bpf: Add ksock test for async callback guard Because the kfuncs are going through LSM hooks, allowing their use via workqueue callbacks would expose the wrong credentials. This test ensures the kfunc are preventing any use from these contexts. Signed-off-by: Mahe Tardy Signed-off-by: Daniel Borkmann Acked-by: Stanislav Fomichev Acked-by: Song Liu Link: https://lore.kernel.org/bpf/20260813110540.103550-6-mahe.tardy@gmail.com --- .../selftests/bpf/prog_tests/ksock_wq.c | 45 ++++++++++++++ tools/testing/selftests/bpf/progs/ksock_wq.c | 62 +++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/ksock_wq.c create mode 100644 tools/testing/selftests/bpf/progs/ksock_wq.c diff --git a/tools/testing/selftests/bpf/prog_tests/ksock_wq.c b/tools/testing/selftests/bpf/prog_tests/ksock_wq.c new file mode 100644 index 000000000000..d6dc20b8f95b --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/ksock_wq.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Isovalent */ + +#include + +#include "test_progs.h" +#include "ksock_wq.skel.h" + +#define CALLBACK_WAIT_RETRIES 1000 +#define CALLBACK_WAIT_US 1000 + +void test_ksock_wq(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct ksock_wq *skel; + u32 callback_done; + int err, i; + + skel = ksock_wq__open_and_load(); + if (!ASSERT_OK_PTR(skel, "ksock_wq open and load")) + return; + + err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.ksock_wq_start), + &opts); + if (!ASSERT_OK(err, "run ksock_wq_start")) + goto out; + if (!ASSERT_OK(opts.retval, "ksock_wq_start retval")) + goto out; + + for (i = 0; i < CALLBACK_WAIT_RETRIES; i++) { + if (__atomic_load_n(&skel->bss->callback_done, __ATOMIC_ACQUIRE)) + break; + usleep(CALLBACK_WAIT_US); + } + callback_done = __atomic_load_n(&skel->bss->callback_done, + __ATOMIC_ACQUIRE); + if (!ASSERT_EQ(callback_done, 1, "workqueue callback completed")) + goto out; + + ASSERT_EQ(skel->bss->create_err, -EOPNOTSUPP, + "workqueue create rejected"); + +out: + ksock_wq__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/ksock_wq.c b/tools/testing/selftests/bpf/progs/ksock_wq.c new file mode 100644 index 000000000000..16a1873d132e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/ksock_wq.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Isovalent */ + +#include "vmlinux.h" +#include +#include "bpf_experimental.h" +#include "bpf_tracing_net.h" +#include "errno.h" +#include "ksock_common.h" + +struct ksock_wq_value { + struct bpf_wq work; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, u32); + __type(value, struct ksock_wq_value); +} work_map SEC(".maps"); + +int create_err; +u32 callback_done; + +static int ksock_wq_callback(void *map, int *key, void *value) +{ + struct bpf_ksock_create_opts opts = { + .family = AF_INET, + .type = SOCK_DGRAM, + .protocol = IPPROTO_UDP, + }; + struct bpf_ksock *ks; + int err = 0; + + ks = bpf_ksock_create(&opts, sizeof(opts), &err); + if (ks) + bpf_ksock_release(ks); + create_err = err; + __sync_fetch_and_add(&callback_done, 1); + return 0; +} + +SEC("syscall") +int ksock_wq_start(void *ctx) +{ + struct ksock_wq_value *value; + u32 key = 0; + int err; + + value = bpf_map_lookup_elem(&work_map, &key); + if (!value) + return -ENOENT; + err = bpf_wq_init(&value->work, &work_map, 0); + if (err) + return err; + err = bpf_wq_set_callback(&value->work, ksock_wq_callback, 0); + if (err) + return err; + return bpf_wq_start(&value->work, 0); +} + +char __license[] SEC("license") = "GPL"; From 4bc49ae344d65cfcef738f281ac575cf73ca2fc5 Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Sun, 16 Aug 2026 10:56:33 +0000 Subject: [PATCH 342/373] bpf: Check pointer type for all atomic RMW paths Atomic RMW verification records an instruction pointer type only when the current destination is PTR_TO_ARENA. A second path can therefore reach the same instruction with an ordinary pointer without comparing it against the saved arena type. The post-verification fixup uses the saved type to rewrite the instruction to BPF_PROBE_ATOMIC for every path. Record the actual destination type for all atomic RMW paths so the existing mismatch check rejects incompatible uses of one instruction. Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT") Signed-off-by: Yiyang Chen Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-1-4644c1886dbc@mails.tsinghua.edu.cn Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d17f14b35b79..93463caf5c9a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6739,11 +6739,9 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, if (err) return err; - if (is_arena_reg(env, insn->dst_reg)) { - err = save_aux_ptr_type(env, PTR_TO_ARENA, false); - if (err) - return err; - } + err = save_aux_ptr_type(env, dst_reg->type, false); + if (err) + return err; /* Check whether we can write into the same memory. */ err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, BPF_SIZE(insn->code), BPF_WRITE, -1, true, false); From 5ab9fbeca8f7da1b40697cf4e75257b9447f03f2 Mon Sep 17 00:00:00 2001 From: Yiyang Chen Date: Sun, 16 Aug 2026 10:56:34 +0000 Subject: [PATCH 343/373] selftests/bpf: Cover mixed arena and stack atomics Add a verifier test with one atomic RMW instruction reached through PTR_TO_ARENA and PTR_TO_STACK paths. The verifier must reject the shared instruction with the existing incompatible-pointer diagnostic. Signed-off-by: Yiyang Chen Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-2-4644c1886dbc@mails.tsinghua.edu.cn Signed-off-by: Eduard Zingerman --- .../selftests/bpf/progs/verifier_arena.c | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/progs/verifier_arena.c b/tools/testing/selftests/bpf/progs/verifier_arena.c index b241bbcf54a8..815f342eb4b0 100644 --- a/tools/testing/selftests/bpf/progs/verifier_arena.c +++ b/tools/testing/selftests/bpf/progs/verifier_arena.c @@ -635,7 +635,37 @@ int non_arena_ptr_add_to_arena_ptr(void *ctx) return 0; } -#endif +SEC("socket") +__description("arena and stack atomic at the same instruction") +__failure __msg("same insn cannot be used with different pointers") +__arch_x86_64 +__load_if_JITed() +__naked void mixed_arena_stack_atomic(void) +{ + asm volatile (" \ + r1 = %[arena] ll; \ + r6 = r10; \ + r6 += -8; \ + r9 = 0; \ + *(u64 *)(r6 + 0) = r9; \ + r7 = 8192; \ + r7 = addr_space_cast(r7, 0, 1); \ + call %[bpf_get_prandom_u32]; \ + if w0 != 0 goto 1f; \ + r8 = r6; \ + goto 2f; \ +1: r8 = r7; \ +2: r9 = 1; \ + lock *(u64 *)(r8 + 0) += r9; \ + r0 = 0; \ + exit; \ +" : + : __imm_addr(arena), + __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +#endif /* defined(__BPF_FEATURE_ADDR_SPACE_CAST) */ static __noinline u32 __arena *check_arena_arg_nonglobal(u32 __arena *arg) From 09c447564fcac5c531a19e3003d14c9c0a68fd19 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Fri, 14 Aug 2026 23:52:55 +0200 Subject: [PATCH 344/373] bpf: Keep fault protection when merging pointer types When the same BPF_LDX instruction is reached through paths that yield different pointer types, save_aux_ptr_type() merges them into a single type which is later used by bpf_convert_ctx_accesses() to decide whether the load has to be rewritten into a BPF_PROBE_MEM one. Before f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()") the merge only accepted two PTR_TO_BTF_ID pointers and unconditionally fell back to PTR_TO_BTF_ID | PTR_UNTRUSTED, so the merged type was always one that gets the BPF_PROBE_MEM rewrite. However, the mentioned commit widened the merge to also cover a PTR_TO_MEM base and replaced the fallback by a union of the PTR_UNTRUSTED and MEM_RDONLY flags. A union of flags though cannot express the property the later rewrite is built upon, some examples: - PTR_TO_MEM merged with PTR_TO_BTF_ID | PTR_UNTRUSTED gets PTR_TO_MEM | PTR_UNTRUSTED but only the MEM_RDONLY variant is valid - PTR_TO_MEM merged with a plain PTR_TO_BTF_ID gets PTR_TO_MEM dropping the rewrite the latter type would have gotten - PTR_TO_MEM | MEM_RDONLY merged with a plain PTR_TO_BTF_ID gets PTR_TO_MEM | MEM_RDONLY which is not rewritten either since only its PTR_UNTRUSTED variant is In all three cases a program can take the unsafe path at runtime with a NULL or otherwise bad pointer and panic the kernel on the faulting load: BUG: kernel NULL pointer dereference, address: 0000000000000038 RIP: 0010:bpf_prog_77531a87032eeaf1_mixed_mem_btf_id_type+0x4b/0x65 Call Trace: bpf_test_run+0x20b/0x460 bpf_prog_test_run_skb+0x650/0xbe0 __sys_bpf+0xb96/0x3140 __x64_sys_bpf+0x2c/0x40 do_syscall_64+0xba/0x590 Kernel panic - not syncing: Fatal exception in interrupt Note that the last two shapes have to be fixed right here, otherwise the merged type retains nothing which marks the load as fault prone, thus no rule in bpf_convert_ctx_accesses() can recover it. Fix it by normalizing the merged type instead. Reuse it in is_load_acq_unsafe() to avoid open coding, and trim the overly verbose comment which is more of an implementation detail of bpf_convert_ctx_accesses() anyway. Fixes: f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()") Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260814215301.709827-1-daniel@iogearbox.net --- include/linux/bpf_verifier.h | 10 ++++++++ kernel/bpf/verifier.c | 49 +++++++++++++++++------------------- 2 files changed, 33 insertions(+), 26 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index bc2af02547fe..ae0274a4bf35 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1308,6 +1308,16 @@ static inline u32 type_flag(u32 type) return type & ~BPF_BASE_TYPE_MASK; } +static inline bool bpf_may_fault_on_deref(enum bpf_reg_type type) +{ + /* + * The pointer types which must not be dereferenced without fault + * protection, that is, the ones bpf_convert_ctx_accesses() has to + * turn a BPF_LDX into a BPF_PROBE_MEM one for. + */ + return type == PTR_TO_BTF_ID || (type_flag(type) & PTR_UNTRUSTED); +} + static inline bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog) { int i; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 93463caf5c9a..9ec23a9c6592 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5025,19 +5025,11 @@ static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno, * A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the * verifier, unlike a regular BPF_LDX. The JIT would emit a plain load * with no exception table entry, so a fault (e.g. NULL deref) crashes - * the kernel instead of being handled. - * - * Reject the source pointer types that a BPF_LDX would have had that - * fault protection applied to, i.e. the ones bpf_convert_ctx_accesses() - * turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID and any PTR_UNTRUSTED - * pointer (untrusted btf ids, untrusted MEM_ALLOC, rdonly untrusted - * memory). A PTR_TRUSTED pointer is not among them, is not converted, - * and stays allowed. Same for the other flagged PTR_TO_BTF_ID variants - * (MEM_ALLOC, MEM_RCU, ...), hence the exact match on the base type. + * the kernel instead of being handled. Reject the source pointer types + * that would have needed that protection, the remaining ones stay + * allowed. */ - return insn->imm == BPF_LOAD_ACQ && - (reg->type == PTR_TO_BTF_ID || - (type_flag(reg->type) & PTR_UNTRUSTED)); + return insn->imm == BPF_LOAD_ACQ && bpf_may_fault_on_deref(reg->type); } /* Return false if @regno contains a pointer whose type isn't supported for @@ -17862,11 +17854,24 @@ static bool is_ptr_to_mem(enum bpf_reg_type type) return base_type(type) == PTR_TO_MEM; } +static enum bpf_reg_type merge_ptr_types(enum bpf_reg_type type_a, + enum bpf_reg_type type_b) +{ + bool to_mem = is_ptr_to_mem(type_a) || is_ptr_to_mem(type_b); + enum bpf_reg_type type_merged = to_mem ? PTR_TO_MEM : PTR_TO_BTF_ID; + + if (bpf_may_fault_on_deref(type_a) || bpf_may_fault_on_deref(type_b)) + type_merged |= to_mem ? MEM_RDONLY | PTR_UNTRUSTED : + PTR_UNTRUSTED; + else + type_merged |= ((type_a | type_b) & MEM_RDONLY); + return type_merged; +} + static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type, bool allow_trust_mismatch) { enum bpf_reg_type *prev_type = &env->insn_aux_data[env->insn_idx].ptr_type; - enum bpf_reg_type merged_type; if (*prev_type == NOT_INIT) { /* Saw a valid insn @@ -17887,20 +17892,12 @@ static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type typ is_ptr_to_mem_or_btf_id(*prev_type)) { /* * Have to support a use case when one path through - * the program yields TRUSTED pointer while another - * is UNTRUSTED. Fallback to UNTRUSTED to generate - * BPF_PROBE_MEM/BPF_PROBE_MEMSX. - * Same behavior of MEM_RDONLY flag. + * the program yields a TRUSTED pointer while another + * is UNTRUSTED. Merge them into a type which keeps + * the BPF_PROBE_MEM/BPF_PROBE_MEMSX rewrite when + * either side needs it. */ - if (is_ptr_to_mem(type) || is_ptr_to_mem(*prev_type)) - merged_type = PTR_TO_MEM; - else - merged_type = PTR_TO_BTF_ID; - if ((type & PTR_UNTRUSTED) || (*prev_type & PTR_UNTRUSTED)) - merged_type |= PTR_UNTRUSTED; - if ((type & MEM_RDONLY) || (*prev_type & MEM_RDONLY)) - merged_type |= MEM_RDONLY; - *prev_type = merged_type; + *prev_type = merge_ptr_types(type, *prev_type); } else { verbose(env, "same insn cannot be used with different pointers\n"); return -EINVAL; From f438ba7a4c3efa627dc91a132c4653725723a6bc Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Fri, 14 Aug 2026 23:52:56 +0200 Subject: [PATCH 345/373] bpf: Treat a fault prone PTR_TO_MEM as a pointer type mismatch reg_type_mismatch_ok() enumerates the pointer types which must not silently share a BPF_LDX with a different one, since the type recorded for the insn drives a rewrite in bpf_convert_ctx_accesses(). f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()") added PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED as another type in need of one, namely the BPF_PROBE_MEM rewrite, but did not add it there. Fix it by adding the missing case to reg_type_mismatch_ok(), so that a PTR_TO_MEM which may fault on deref is not mismatch ok anymore. The triage in save_aux_ptr_type() then merges them. Fixes: f2362a57aeff ("bpf: allow void* cast using bpf_rdonly_cast()") Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260814215301.709827-2-daniel@iogearbox.net --- kernel/bpf/verifier.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9ec23a9c6592..ad3310b55b02 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17815,6 +17815,8 @@ static bool reg_type_mismatch_ok(enum bpf_reg_type type) case PTR_TO_BTF_ID: case PTR_TO_ARENA: return false; + case PTR_TO_MEM: + return !bpf_may_fault_on_deref(type); default: return true; } From ee9ad135b2087f9335eed97d062f5853e70f89fe Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Fri, 14 Aug 2026 23:52:57 +0200 Subject: [PATCH 346/373] bpf: Reject a store through a fault prone pointer check_ptr_to_btf_access() allows the program to store before the default BTF access path gets to reject a non read access. ac65c710cc64 ("bpf: Reject writes through untrusted BTF pointers") closed that for a PTR_UNTRUSTED pointer, but a bare PTR_TO_BTF_ID may fault on a dereference just the same and is let through. A BPF_LDX gets the BPF_PROBE_MEM rewrite in bpf_convert_ctx_accesses() and a bad address is handled, but a BPF_STX does not and cannot, there is no probed store to rewrite. The store is emitted as a plain one without an exception table entry and a bad address panics the kernel. A bpf_qdisc program can reach this, bpf_qdisc_btf_struct_access() permits a write to Qdisc::limit and Qdisc::next_sched is a plain struct Qdisc pointer which the walk turns into the compat type: struct Qdisc *next = sch->next_sched; next->limit = 1000; BUG: kernel NULL pointer dereference, address: 0000000000000014 RIP: 0010:bpf_prog_c6e14e7f32c8e325_bpf_fifo_enqueue+0x3a/0x12b Code: [...] bf e8 03 00 00 <89> 7e 14 41 8b 7f 14 [...] Kernel panic - not syncing: Fatal exception in interrupt Fix by widen the check to bpf_may_fault_on_deref() so that it covers both. Fixes: 27ae7997a661 ("bpf: Introduce BPF_PROG_TYPE_STRUCT_OPS") Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260814215301.709827-3-daniel@iogearbox.net --- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ad3310b55b02..58a128a8d8d0 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5988,7 +5988,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, return -EACCES; } - if (atype != BPF_READ && (type_flag(reg->type) & PTR_UNTRUSTED)) { + if (atype != BPF_READ && bpf_may_fault_on_deref(reg->type)) { verbose(env, "only read is supported\n"); return -EACCES; } From d99bda7f017b47aff45accbb321facba9f7dd799 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Fri, 14 Aug 2026 23:52:58 +0200 Subject: [PATCH 347/373] bpf: Rewrite any fault prone load out of a mem or btf_id pointer bpf_convert_ctx_accesses() turns a BPF_LDX into a BPF_PROBE_MEM one by matching the type recorded for the insn against a list of exact pointer types. The list cannot keep up with the flag combinations the verifier produces, and a type which is missing from it ends up as a plain load without an exception table entry, so a bad address panics the kernel instead of being handled. Two such types exist today and are reachable: - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_ALLOC | NON_OWN_REF - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU Rather than adding the two, just drop the list and state the property itself in the default case of the switch. This is a superset of what the list matched, the untrusted PTR_TO_MEM does not have to carry MEM_RDONLY for it anymore, and it stays in sync with the verifier side which uses the same match in save_aux_ptr_type() and reg_type_mismatch_ok(). Assert that a fault prone type which does not get the rewrite for whatever reason is rejected at load time rather than left to fault at runtime to catch any future cases. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Fixes: 6fcd486b3a0a ("bpf: Refactor RCU enforcement in the verifier.") Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260814215301.709827-4-daniel@iogearbox.net --- include/linux/bpf_verifier.h | 11 +++++++++ kernel/bpf/fixups.c | 47 ++++++++++++++++++++---------------- kernel/bpf/verifier.c | 15 ++---------- 3 files changed, 39 insertions(+), 34 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index ae0274a4bf35..5fad59fdab0d 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1308,6 +1308,17 @@ static inline u32 type_flag(u32 type) return type & ~BPF_BASE_TYPE_MASK; } +static inline bool bpf_is_ptr_to_mem_or_btf_id(enum bpf_reg_type type) +{ + switch (base_type(type)) { + case PTR_TO_MEM: + case PTR_TO_BTF_ID: + return true; + default: + return false; + } +} + static inline bool bpf_may_fault_on_deref(enum bpf_reg_type type) { /* diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 70f22eb63ed5..65b441e4a351 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -812,6 +812,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) for (i = 0; i < insn_cnt; i++, insn++) { bpf_convert_ctx_access_t convert_ctx_access; + enum bpf_reg_type ptr_type; u8 mode; if (env->insn_aux_data[i + delta].nospec) { @@ -904,7 +905,8 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) continue; } - switch ((int)env->insn_aux_data[i + delta].ptr_type) { + ptr_type = env->insn_aux_data[i + delta].ptr_type; + switch ((int)ptr_type) { case PTR_TO_CTX: if (!ops->convert_ctx_access) continue; @@ -920,26 +922,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) case PTR_TO_XDP_SOCK: convert_ctx_access = bpf_xdp_sock_convert_ctx_access; break; - case PTR_TO_BTF_ID: - case PTR_TO_BTF_ID | PTR_UNTRUSTED: - /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike - * PTR_TO_BTF_ID, and an active referenced id, but the same cannot - * be said once it is marked PTR_UNTRUSTED, hence we must handle - * any faults for loads into such types. BPF_WRITE is disallowed - * for this case. - */ - case PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED: - case PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED: - if (type == BPF_READ) { - if (BPF_MODE(insn->code) == BPF_MEM) - insn->code = BPF_LDX | BPF_PROBE_MEM | - BPF_SIZE((insn)->code); - else - insn->code = BPF_LDX | BPF_PROBE_MEMSX | - BPF_SIZE((insn)->code); - env->prog->aux->num_exentries++; - } - continue; case PTR_TO_ARENA: if (BPF_MODE(insn->code) == BPF_MEMSX) { if (!bpf_jit_supports_insn(insn, true)) { @@ -953,6 +935,29 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) env->prog->aux->num_exentries++; continue; default: + /* + * A pointer which may fault on a dereference must not + * be loaded from without fault protection, hence turn + * the BPF_LDX into a BPF_PROBE_MEM one so that a bad + * address is handled rather than panicking the kernel. + * A store through one is rejected earlier, there is no + * probed counterpart to rewrite it into. + */ + if (bpf_is_ptr_to_mem_or_btf_id(ptr_type) && + bpf_may_fault_on_deref(ptr_type) && + type == BPF_READ) { + if (BPF_MODE(insn->code) == BPF_MEM) + insn->code = BPF_LDX | BPF_PROBE_MEM | + BPF_SIZE(insn->code); + else + insn->code = BPF_LDX | BPF_PROBE_MEMSX | + BPF_SIZE(insn->code); + env->prog->aux->num_exentries++; + continue; + } + if (verifier_bug_if(bpf_may_fault_on_deref(ptr_type), env, + "access to a fault prone pointer is not rewritten as a probed one")) + return -EFAULT; continue; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 58a128a8d8d0..9f833e913e43 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17840,17 +17840,6 @@ static bool reg_type_mismatch(enum bpf_reg_type src, enum bpf_reg_type prev) !reg_type_mismatch_ok(prev)); } -static bool is_ptr_to_mem_or_btf_id(enum bpf_reg_type type) -{ - switch (base_type(type)) { - case PTR_TO_MEM: - case PTR_TO_BTF_ID: - return true; - default: - return false; - } -} - static bool is_ptr_to_mem(enum bpf_reg_type type) { return base_type(type) == PTR_TO_MEM; @@ -17890,8 +17879,8 @@ static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type typ * Reject it. */ if (allow_trust_mismatch && - is_ptr_to_mem_or_btf_id(type) && - is_ptr_to_mem_or_btf_id(*prev_type)) { + bpf_is_ptr_to_mem_or_btf_id(type) && + bpf_is_ptr_to_mem_or_btf_id(*prev_type)) { /* * Have to support a use case when one path through * the program yields a TRUSTED pointer while another From 3c3d2c09ec4e11bf7f5419163643b3b02d3f5add Mon Sep 17 00:00:00 2001 From: Michal Luczaj Date: Thu, 13 Aug 2026 14:41:59 +0200 Subject: [PATCH 348/373] bpf: Extract shared reqsk-to-listener upgrade __bpf_sk_lookup() and bpf_sk_lookup() duplicate the same sk_to_full_sk() reqsk-to-listener upgrade. Extract it into a helper. Leave the currently unreachable WARN_ONCE as a defensive assert. No functional change. Signed-off-by: Michal Luczaj Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Reviewed-by: Jakub Sitnicki Link: https://lore.kernel.org/bpf/20260813-sockmap-lookup-get-ref-v1-1-31f5d55f44ac@rbox.co --- net/core/filter.c | 62 ++++++++++++++++++++--------------------------- 1 file changed, 26 insertions(+), 36 deletions(-) diff --git a/net/core/filter.c b/net/core/filter.c index 3423734124a5..79cbef41af59 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -7167,6 +7167,28 @@ __bpf_skc_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, return sk; } +static struct sock * +bpf_sk_lookup_full_sk(struct sock *sk) +{ + struct sock *sk2 = sk_to_full_sk(sk); + + /* + * sk_to_full_sk() may return sk->rsk_listener, make sure the original + * sk sock refcnt is decremented to prevent a request_sock leak. + */ + if (sk2 != sk) { + sock_gen_put(sk); + /* Ensure there is no need to bump sk2 refcnt. */ + if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) { + WARN_ONCE(1, "Found non-RCU, unreferenced socket!"); + return NULL; + } + sk = sk2; + } + + return sk; +} + static struct sock * __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id, @@ -7175,24 +7197,8 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net, ifindex, proto, netns_id, flags, sdif); - - if (sk) { - struct sock *sk2 = sk_to_full_sk(sk); - - /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk - * sock refcnt is decremented to prevent a request_sock leak. - */ - if (sk2 != sk) { - sock_gen_put(sk); - /* Ensure there is no need to bump sk2 refcnt */ - if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) { - WARN_ONCE(1, "Found non-RCU, unreferenced socket!"); - return NULL; - } - sk = sk2; - } - } - + if (sk) + sk = bpf_sk_lookup_full_sk(sk); return sk; } @@ -7221,24 +7227,8 @@ bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len, { struct sock *sk = bpf_skc_lookup(skb, tuple, len, proto, netns_id, flags); - - if (sk) { - struct sock *sk2 = sk_to_full_sk(sk); - - /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk - * sock refcnt is decremented to prevent a request_sock leak. - */ - if (sk2 != sk) { - sock_gen_put(sk); - /* Ensure there is no need to bump sk2 refcnt */ - if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) { - WARN_ONCE(1, "Found non-RCU, unreferenced socket!"); - return NULL; - } - sk = sk2; - } - } - + if (sk) + sk = bpf_sk_lookup_full_sk(sk); return sk; } From 34e0eb763becfee4487a7105e3204d9fc486be93 Mon Sep 17 00:00:00 2001 From: Michal Luczaj Date: Thu, 13 Aug 2026 14:42:00 +0200 Subject: [PATCH 349/373] bpf, sockmap: Use sock_hold() instead of refcount_inc_not_zero() in lookup psock's hold on the looked up socket isn't dropped until sk_psock_drop() -> queue_rcu_work() -> sk_psock_destroy() runs, which happens only after the entry is unlinked and an RCU grace period elapses. Since the lookup runs under RCU, a non-NULL result guarantees sk_refcnt >= 1: refcount_inc_not_zero() can never fail here. Use sock_hold() instead. Signed-off-by: Michal Luczaj Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Reviewed-by: Jakub Sitnicki Link: https://lore.kernel.org/bpf/20260813-sockmap-lookup-get-ref-v1-2-31f5d55f44ac@rbox.co --- net/core/sock_map.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 9efbd8ca7db8..ca49bc7f8687 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -392,8 +392,8 @@ static void *sock_map_lookup(struct bpf_map *map, void *key) sk = __sock_map_lookup_elem(map, *(u32 *)key); if (!sk) return NULL; - if (sk_is_refcounted(sk) && !refcount_inc_not_zero(&sk->sk_refcnt)) - return NULL; + if (sk_is_refcounted(sk)) + sock_hold(sk); return sk; } @@ -1218,8 +1218,8 @@ static void *sock_hash_lookup(struct bpf_map *map, void *key) sk = __sock_hash_lookup_elem(map, key); if (!sk) return NULL; - if (sk_is_refcounted(sk) && !refcount_inc_not_zero(&sk->sk_refcnt)) - return NULL; + if (sk_is_refcounted(sk)) + sock_hold(sk); return sk; } From c238ff829eb14c6bb014ee44dbc32ec25b78aa7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexis=20Lothor=C3=A9=20=28eBPF=20Foundation=29?= Date: Fri, 14 Aug 2026 09:11:27 +0200 Subject: [PATCH 350/373] selftests/bpf: Fix comment style in network_helpers.c MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BPF subsystem requires multi-line comments to have the opening /* start on its own line. Update multi-line comments in network_helpers.c to follow this requirement. Signed-off-by: Alexis Lothoré (eBPF Foundation) Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260814-tc_tunnel_flaky-v5-1-5b93d030c42c@bootlin.com --- tools/testing/selftests/bpf/network_helpers.c | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/bpf/network_helpers.c b/tools/testing/selftests/bpf/network_helpers.c index db935a9d9fc1..cdf2d7d3ab32 100644 --- a/tools/testing/selftests/bpf/network_helpers.c +++ b/tools/testing/selftests/bpf/network_helpers.c @@ -424,7 +424,8 @@ int make_sockaddr(int family, const char *addr_str, __u16 port, *len = sizeof(*sin6); return 0; } else if (family == AF_UNIX) { - /* Note that we always use abstract unix sockets to avoid having + /* + * Note that we always use abstract unix sockets to avoid having * to clean up leftover files. */ struct sockaddr_un *sun = (void *)addr; @@ -865,7 +866,8 @@ static bool is_ethernet(const u_char *packet) memcpy(&arphdr_type, packet + 8, 2); arphdr_type = ntohs(arphdr_type); - /* Except the following cases, the protocol type contains the + /* + * Except the following cases, the protocol type contains the * Ethernet protocol type for the packet. * * https://www.tcpdump.org/linktypes/LINKTYPE_LINUX_SLL2.html @@ -1033,19 +1035,22 @@ static void *traffic_monitor_thread(void *arg) if (!packet) continue; - /* According to the man page of pcap_dump(), first argument + /* + * According to the man page of pcap_dump(), first argument * is the pcap_dumper_t pointer even it's argument type is * u_char *. */ pcap_dump((u_char *)dumper, &header, packet); - /* Not sure what other types of packets look like. Here, we + /* + * Not sure what other types of packets look like. Here, we * parse only Ethernet and compatible packets. */ if (!is_ethernet(packet)) continue; - /* Skip SLL2 header + /* + * Skip SLL2 header * https://www.tcpdump.org/linktypes/LINKTYPE_LINUX_SLL2.html * * Although the document doesn't mention that, the payload @@ -1079,7 +1084,8 @@ static void *traffic_monitor_thread(void *arg) return NULL; } -/* Prepare the pcap handle to capture packets. +/* + * Prepare the pcap handle to capture packets. * * This pcap is non-blocking and immediate mode is enabled to receive * captured packets as soon as possible. The snaplen is set to 1024 bytes @@ -1150,7 +1156,8 @@ static void encode_test_name(char *buf, size_t len, const char *test_name, const #define PCAP_DIR "/tmp/tmon_pcap" -/* Start to monitor the network traffic in the given network namespace. +/* + * Start to monitor the network traffic in the given network namespace. * * netns: the name of the network namespace to monitor. If NULL, the * current network namespace is monitored. @@ -1255,7 +1262,8 @@ static void traffic_monitor_release(struct tmonitor_ctx *ctx) free(ctx); } -/* Stop the network traffic monitor. +/* + * Stop the network traffic monitor. * * ctx: the context returned by traffic_monitor_start() */ From 5fe7007aed9ad069b2bd77e5d0c875c64f5c0269 Mon Sep 17 00:00:00 2001 From: Junseo Lim Date: Tue, 11 Aug 2026 13:41:49 +0900 Subject: [PATCH 351/373] lwt_bpf: Restore reserved headroom after xmit program ip_finish_output2() expands an skb to LL_RESERVED_SPACE(dev) before LWT xmit. An LWT_XMIT BPF program can then modify the skb head and still return BPF_OK, so bpf_xmit() rechecks the remaining headroom before the skb continues to neighbour output. That recheck uses dst->dev->hard_header_len. This is not enough for the neighbour cached-header path: neigh_hh_output() copies the cached hardware header using the aligned hh_cache size, HH_DATA_MOD for short headers or HH_DATA_ALIGN(hh_len) otherwise. On Ethernet, hard_header_len is 14 but the cached copy needs 16 bytes. If an LWT_XMIT BPF program calls bpf_skb_change_head(skb, 1, 0), the skb can still have 15 bytes of headroom after the program. The existing check accepts that, after which neigh_hh_output() hits its headroom warning and drops the skb. Use LL_RESERVED_SPACE(dst->dev) in the post-BPF headroom check to match the reservation made before LWT xmit. Fixes: 3a0af8fd61f9 ("bpf: BPF for lightweight tunnel infrastructure") Reported-by: Sechang Lim Suggested-by: Daniel Borkmann Signed-off-by: Junseo Lim Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260811044149.118235-1-zirajs7@gmail.com --- net/core/lwt_bpf.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/net/core/lwt_bpf.c b/net/core/lwt_bpf.c index 652952d416f2..da49364ec63d 100644 --- a/net/core/lwt_bpf.c +++ b/net/core/lwt_bpf.c @@ -167,10 +167,10 @@ static int bpf_output(struct net *net, struct sock *sk, struct sk_buff *skb) return dst->lwtstate->orig_output(net, sk, skb); } -static int xmit_check_hhlen(struct sk_buff *skb, int hh_len) +static int xmit_check_headroom(struct sk_buff *skb, int hroom) { - if (skb_headroom(skb) < hh_len) { - int nhead = HH_DATA_ALIGN(hh_len - skb_headroom(skb)); + if (skb_headroom(skb) < hroom) { + int nhead = hroom - skb_headroom(skb); if (pskb_expand_head(skb, nhead, 0, GFP_ATOMIC)) return -ENOMEM; @@ -282,7 +282,7 @@ static int bpf_xmit(struct sk_buff *skb) bpf = bpf_lwt_lwtunnel(dst->lwtstate); if (bpf->xmit.prog) { - int hh_len = dst->dev->hard_header_len; + int hroom = LL_RESERVED_SPACE(dst->dev); __be16 proto = skb->protocol; int ret; @@ -298,9 +298,12 @@ static int bpf_xmit(struct sk_buff *skb) return -EINVAL; } /* If the header was expanded, headroom might be too - * small for L2 header to come, expand as needed. + * small for the L2 header to come, expand as needed. + * neigh_hh_output() copies the cached header in + * HH_DATA_MOD aligned chunks, so match the reservation + * made before LWT xmit. */ - ret = xmit_check_hhlen(skb, hh_len); + ret = xmit_check_headroom(skb, hroom); if (unlikely(ret)) return ret; From 84473a7e1813a2da7b759ab1d098a84998c8d3f5 Mon Sep 17 00:00:00 2001 From: Junseo Lim Date: Wed, 12 Aug 2026 18:16:54 +0900 Subject: [PATCH 352/373] bpf: Disallow bpf_{g,s}etsockopt() in cgroup UNIX getname hooks _bpf_setsockopt() and _bpf_getsockopt() call sock_owned_by_me() for full sockets, so these helpers expect the socket lock to be held. BPF_CGROUP_UNIX_GETPEERNAME and BPF_CGROUP_UNIX_GETSOCKNAME run BPF programs without acquiring the socket lock. A program attached to either hook can therefore trigger the sock_owned_by_me() warning by calling bpf_setsockopt() or bpf_getsockopt(). Disallow bpf_setsockopt() and bpf_getsockopt() for CGROUP_UNIX_GETPEERNAME and CGROUP_UNIX_GETSOCKNAME. Fixes: 859051dd165e ("bpf: Implement cgroup sockaddr hooks for unix sockets") Reported-by: Sechang Lim Signed-off-by: Junseo Lim Signed-off-by: Daniel Borkmann Reviewed-by: Kuniyuki Iwashima Link: https://lore.kernel.org/bpf/20260812091654.244752-1-zirajs7@gmail.com --- net/core/filter.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/net/core/filter.c b/net/core/filter.c index 79cbef41af59..4a38c9b33bc9 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -8428,10 +8428,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) case BPF_CGROUP_UNIX_SENDMSG: case BPF_CGROUP_INET4_GETPEERNAME: case BPF_CGROUP_INET6_GETPEERNAME: - case BPF_CGROUP_UNIX_GETPEERNAME: case BPF_CGROUP_INET4_GETSOCKNAME: case BPF_CGROUP_INET6_GETSOCKNAME: - case BPF_CGROUP_UNIX_GETSOCKNAME: return &bpf_sock_addr_setsockopt_proto; default: return NULL; @@ -8451,10 +8449,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) case BPF_CGROUP_UNIX_SENDMSG: case BPF_CGROUP_INET4_GETPEERNAME: case BPF_CGROUP_INET6_GETPEERNAME: - case BPF_CGROUP_UNIX_GETPEERNAME: case BPF_CGROUP_INET4_GETSOCKNAME: case BPF_CGROUP_INET6_GETSOCKNAME: - case BPF_CGROUP_UNIX_GETSOCKNAME: return &bpf_sock_addr_getsockopt_proto; default: return NULL; From 77877bf570ffb9756a20e602fba81ca0155468d0 Mon Sep 17 00:00:00 2001 From: Tiezhu Yang Date: Thu, 13 Aug 2026 15:09:06 +0800 Subject: [PATCH 353/373] selftests/bpf: Enable timed may_goto tests for LoongArch Enable stream_cond_break, may_goto_interaction, and verifier_may_goto_1 tests for LoongArch, aligning with recent architectural infrastructure support (timed may_goto and arch_bpf_stack_walk JIT). With this patch, the following tests passed on LoongArch: sudo ./test_progs -a stream_success/stream_cond_break sudo ./test_progs -a verifier_bpf_fastcall/may_goto_interaction sudo ./test_progs -a verifier_may_goto_1 Signed-off-by: Tiezhu Yang Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260813070906.5164-1-yangtiezhu@loongson.cn --- tools/testing/selftests/bpf/progs/stream.c | 1 + tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c | 1 + tools/testing/selftests/bpf/progs/verifier_may_goto_1.c | 5 +++++ 3 files changed, 7 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/stream.c b/tools/testing/selftests/bpf/progs/stream.c index 00a37933e411..8e8e1339dc74 100644 --- a/tools/testing/selftests/bpf/progs/stream.c +++ b/tools/testing/selftests/bpf/progs/stream.c @@ -65,6 +65,7 @@ __arch_x86_64 __arch_arm64 __arch_s390x __arch_riscv64 +__arch_loongarch __success __retval(0) __stderr("ERROR: Timeout detected for may_goto instruction") __stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}") diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index 4cfaa6b4ab40..328cf630210a 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -665,6 +665,7 @@ __naked void may_goto_interaction_x86_64(void) SEC("raw_tp") __arch_arm64 __arch_riscv64 +__arch_loongarch __log_level(4) __msg("subprog 0 (may_goto_interaction) main {{.*}} stack 24") /* may_goto counter at -24 */ diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c index 0e211f030d0d..db7e30da234f 100644 --- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c +++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c @@ -12,6 +12,7 @@ __arch_x86_64 __arch_s390x __arch_arm64 __arch_riscv64 +__arch_loongarch __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -33,6 +34,7 @@ __arch_x86_64 __arch_s390x __arch_arm64 __arch_riscv64 +__arch_loongarch __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -56,6 +58,7 @@ __arch_x86_64 __arch_s390x __arch_arm64 __arch_riscv64 +__arch_loongarch __xlated("0: r0 = 1") __xlated("1: exit") __success @@ -83,6 +86,7 @@ __arch_x86_64 __arch_s390x __arch_arm64 __arch_riscv64 +__arch_loongarch __xlated("0: *(u64 *)(r10 -16) = 65535") __xlated("1: *(u64 *)(r10 -8) = 0") __xlated("2: r12 = *(u64 *)(r10 -16)") @@ -120,6 +124,7 @@ __arch_x86_64 __arch_s390x __arch_arm64 __arch_riscv64 +__arch_loongarch __success __retval(0) __naked void timed_may_goto_preserves_regs(void) From 5e4bcad6171d4baf426e49a39580cdb79254ea36 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:10 +0100 Subject: [PATCH 354/373] bpf: Name the enum for BPF_FUNC_skb_adjust_room flags The existing anonymous enum for BPF_FUNC_skb_adjust_room flags is named to enum bpf_adj_room_flags to enable CO-RE (Compile Once - Run Everywhere) lookups in BPF programs. Co-developed-by: Max Tottenham Co-developed-by: Anna Glasgall Signed-off-by: Max Tottenham Signed-off-by: Anna Glasgall Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/bpf/20260812083115.73100-2-nhudson@akamai.com --- include/uapi/linux/bpf.h | 2 +- tools/include/uapi/linux/bpf.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index ffd96e8b920b..f6c9dc856858 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6283,7 +6283,7 @@ enum { }; /* BPF_FUNC_skb_adjust_room flags. */ -enum { +enum bpf_adj_room_flags { BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0), BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1), BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2), diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index ffd96e8b920b..f6c9dc856858 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6283,7 +6283,7 @@ enum { }; /* BPF_FUNC_skb_adjust_room flags. */ -enum { +enum bpf_adj_room_flags { BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0), BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1), BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2), From 7b2ea1151e04d3506030db8734c48c5c2bec1392 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:11 +0100 Subject: [PATCH 355/373] bpf: Refactor masks for ADJ_ROOM flags and encap validation Refactor the helper masks for bpf_skb_adjust_room() flags to simplify validation logic and introduce: - BPF_F_ADJ_ROOM_ENCAP_MASK - BPF_F_ADJ_ROOM_DECAP_MASK Refactor existing validation checks in bpf_skb_net_shrink() and bpf_skb_adjust_room() to use the new masks (no behavior change). This is in preparation for supporting the new decap flags. Co-developed-by: Max Tottenham Co-developed-by: Anna Glasgall Signed-off-by: Max Tottenham Signed-off-by: Anna Glasgall Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/bpf/20260812083115.73100-3-nhudson@akamai.com --- net/core/filter.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/net/core/filter.c b/net/core/filter.c index 4a38c9b33bc9..00b7ebe4a02b 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3572,14 +3572,19 @@ static u32 bpf_skb_net_base_len(const struct sk_buff *skb) #define BPF_F_ADJ_ROOM_DECAP_L3_MASK (BPF_F_ADJ_ROOM_DECAP_L3_IPV4 | \ BPF_F_ADJ_ROOM_DECAP_L3_IPV6) -#define BPF_F_ADJ_ROOM_MASK (BPF_F_ADJ_ROOM_FIXED_GSO | \ - BPF_F_ADJ_ROOM_ENCAP_L3_MASK | \ +#define BPF_F_ADJ_ROOM_ENCAP_MASK (BPF_F_ADJ_ROOM_ENCAP_L3_MASK | \ BPF_F_ADJ_ROOM_ENCAP_L4_GRE | \ BPF_F_ADJ_ROOM_ENCAP_L4_UDP | \ BPF_F_ADJ_ROOM_ENCAP_L2_ETH | \ BPF_F_ADJ_ROOM_ENCAP_L2( \ - BPF_ADJ_ROOM_ENCAP_L2_MASK) | \ - BPF_F_ADJ_ROOM_DECAP_L3_MASK) + BPF_ADJ_ROOM_ENCAP_L2_MASK)) + +#define BPF_F_ADJ_ROOM_DECAP_MASK (BPF_F_ADJ_ROOM_DECAP_L3_MASK) + +#define BPF_F_ADJ_ROOM_MASK (BPF_F_ADJ_ROOM_FIXED_GSO | \ + BPF_F_ADJ_ROOM_ENCAP_MASK | \ + BPF_F_ADJ_ROOM_DECAP_MASK | \ + BPF_F_ADJ_ROOM_NO_CSUM_RESET) static int bpf_skb_net_grow(struct sk_buff *skb, u32 off, u32 len_diff, u64 flags) @@ -3702,8 +3707,8 @@ static int bpf_skb_net_shrink(struct sk_buff *skb, u32 off, u32 len_diff, bool decap = flags & BPF_F_ADJ_ROOM_DECAP_L3_MASK; int ret; - if (unlikely(flags & ~(BPF_F_ADJ_ROOM_FIXED_GSO | - BPF_F_ADJ_ROOM_DECAP_L3_MASK | + if (unlikely(flags & ~(BPF_F_ADJ_ROOM_DECAP_MASK | + BPF_F_ADJ_ROOM_FIXED_GSO | BPF_F_ADJ_ROOM_NO_CSUM_RESET))) return -EINVAL; @@ -3802,8 +3807,7 @@ BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, u32 off; int ret; - if (unlikely(flags & ~(BPF_F_ADJ_ROOM_MASK | - BPF_F_ADJ_ROOM_NO_CSUM_RESET))) + if (unlikely(flags & ~BPF_F_ADJ_ROOM_MASK)) return -EINVAL; if (unlikely(len_diff_abs > 0xfffU)) return -EFAULT; @@ -3822,20 +3826,20 @@ BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, return -ENOTSUPP; } - if (flags & BPF_F_ADJ_ROOM_DECAP_L3_MASK) { + if (flags & BPF_F_ADJ_ROOM_DECAP_MASK) { if (!shrink) return -EINVAL; - switch (flags & BPF_F_ADJ_ROOM_DECAP_L3_MASK) { - case BPF_F_ADJ_ROOM_DECAP_L3_IPV4: - len_min = sizeof(struct iphdr); - break; - case BPF_F_ADJ_ROOM_DECAP_L3_IPV6: - len_min = sizeof(struct ipv6hdr); - break; - default: + /* Reject mutually exclusive decap flag pairs. */ + if ((flags & BPF_F_ADJ_ROOM_DECAP_L3_MASK) == + BPF_F_ADJ_ROOM_DECAP_L3_MASK) return -EINVAL; - } + + if (flags & BPF_F_ADJ_ROOM_DECAP_L3_IPV4) + len_min = sizeof(struct iphdr); + + if (flags & BPF_F_ADJ_ROOM_DECAP_L3_IPV6) + len_min = sizeof(struct ipv6hdr); } len_cur = skb->len - skb_network_offset(skb); From da199070bc6209bf5db970f8f84c7db4210fe511 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:12 +0100 Subject: [PATCH 356/373] bpf: Add BPF_F_ADJ_ROOM_DECAP_* flags for tunnel decapsulation Add new bpf_skb_adjust_room() decapsulation flags: - BPF_F_ADJ_ROOM_DECAP_L4_GRE - BPF_F_ADJ_ROOM_DECAP_L4_UDP - BPF_F_ADJ_ROOM_DECAP_IPXIP4 - BPF_F_ADJ_ROOM_DECAP_IPXIP6 These flags let BPF programs describe which tunnel layer is being removed, so later changes can update tunnel-related GSO state accordingly during decapsulation. This patch only introduces the UAPI flag definitions and helper documentation. Co-developed-by: Max Tottenham Co-developed-by: Anna Glasgall Signed-off-by: Max Tottenham Signed-off-by: Anna Glasgall Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/bpf/20260812083115.73100-4-nhudson@akamai.com --- include/uapi/linux/bpf.h | 34 ++++++++++++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 64 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index f6c9dc856858..732b35cc08d1 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3038,8 +3038,34 @@ union bpf_attr { * * * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**, * **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**: - * Indicate the new IP header version after decapsulating the outer - * IP header. Used when the inner and outer IP versions are different. + * Indicate the new IP header version after decapsulating the + * outer IP header. Used when the inner and outer IP versions + * are different. These flags only trigger a protocol change + * without clearing any tunnel-specific GSO flags. + * + * * **BPF_F_ADJ_ROOM_DECAP_L4_GRE**: + * Clear GRE tunnel GSO flags (SKB_GSO_GRE and SKB_GSO_GRE_CSUM) + * when decapsulating a GRE tunnel. + * + * * **BPF_F_ADJ_ROOM_DECAP_L4_UDP**: + * Clear UDP tunnel GSO flags (SKB_GSO_UDP_TUNNEL and + * SKB_GSO_UDP_TUNNEL_CSUM) when decapsulating a UDP tunnel. + * + * * **BPF_F_ADJ_ROOM_DECAP_IPXIP4**: + * Clear IPIP/SIT tunnel GSO flag (SKB_GSO_IPXIP4) when decapsulating + * a tunnel with an outer IPv4 header (IPv4-in-IPv4 or IPv6-in-IPv4). + * + * * **BPF_F_ADJ_ROOM_DECAP_IPXIP6**: + * Clear IPv6 encapsulation tunnel GSO flag (SKB_GSO_IPXIP6) when + * decapsulating a tunnel with an outer IPv6 header (IPv6-in-IPv6 + * or IPv4-in-IPv6). + * + * When using the decapsulation flags above, the skb->encapsulation + * flag is automatically cleared if all tunnel-specific GSO flags + * (SKB_GSO_UDP_TUNNEL, SKB_GSO_UDP_TUNNEL_CSUM, SKB_GSO_GRE, + * SKB_GSO_GRE_CSUM, SKB_GSO_IPXIP4, SKB_GSO_IPXIP6) have been + * removed from the packet. This handles cases where all tunnel + * layers have been decapsulated. * * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers @@ -6293,6 +6319,10 @@ enum bpf_adj_room_flags { BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6), BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7), BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8), + BPF_F_ADJ_ROOM_DECAP_L4_GRE = (1ULL << 9), + BPF_F_ADJ_ROOM_DECAP_L4_UDP = (1ULL << 10), + BPF_F_ADJ_ROOM_DECAP_IPXIP4 = (1ULL << 11), + BPF_F_ADJ_ROOM_DECAP_IPXIP6 = (1ULL << 12), }; enum { diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f6c9dc856858..732b35cc08d1 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3038,8 +3038,34 @@ union bpf_attr { * * * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**, * **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**: - * Indicate the new IP header version after decapsulating the outer - * IP header. Used when the inner and outer IP versions are different. + * Indicate the new IP header version after decapsulating the + * outer IP header. Used when the inner and outer IP versions + * are different. These flags only trigger a protocol change + * without clearing any tunnel-specific GSO flags. + * + * * **BPF_F_ADJ_ROOM_DECAP_L4_GRE**: + * Clear GRE tunnel GSO flags (SKB_GSO_GRE and SKB_GSO_GRE_CSUM) + * when decapsulating a GRE tunnel. + * + * * **BPF_F_ADJ_ROOM_DECAP_L4_UDP**: + * Clear UDP tunnel GSO flags (SKB_GSO_UDP_TUNNEL and + * SKB_GSO_UDP_TUNNEL_CSUM) when decapsulating a UDP tunnel. + * + * * **BPF_F_ADJ_ROOM_DECAP_IPXIP4**: + * Clear IPIP/SIT tunnel GSO flag (SKB_GSO_IPXIP4) when decapsulating + * a tunnel with an outer IPv4 header (IPv4-in-IPv4 or IPv6-in-IPv4). + * + * * **BPF_F_ADJ_ROOM_DECAP_IPXIP6**: + * Clear IPv6 encapsulation tunnel GSO flag (SKB_GSO_IPXIP6) when + * decapsulating a tunnel with an outer IPv6 header (IPv6-in-IPv6 + * or IPv4-in-IPv6). + * + * When using the decapsulation flags above, the skb->encapsulation + * flag is automatically cleared if all tunnel-specific GSO flags + * (SKB_GSO_UDP_TUNNEL, SKB_GSO_UDP_TUNNEL_CSUM, SKB_GSO_GRE, + * SKB_GSO_GRE_CSUM, SKB_GSO_IPXIP4, SKB_GSO_IPXIP6) have been + * removed from the packet. This handles cases where all tunnel + * layers have been decapsulated. * * A call to this helper is susceptible to change the underlying * packet buffer. Therefore, at load time, all checks on pointers @@ -6293,6 +6319,10 @@ enum bpf_adj_room_flags { BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6), BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7), BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8), + BPF_F_ADJ_ROOM_DECAP_L4_GRE = (1ULL << 9), + BPF_F_ADJ_ROOM_DECAP_L4_UDP = (1ULL << 10), + BPF_F_ADJ_ROOM_DECAP_IPXIP4 = (1ULL << 11), + BPF_F_ADJ_ROOM_DECAP_IPXIP6 = (1ULL << 12), }; enum { From 3a39c214fd2c3dd8266649e7f9f85ca1439eb738 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:13 +0100 Subject: [PATCH 357/373] bpf: Allow new DECAP flags and add guard rails Add checks to require shrink-only decap, reject conflicting decap flag combinations, and verify removed length is sufficient for claimed header decapsulation. Co-developed-by: Max Tottenham Co-developed-by: Anna Glasgall Signed-off-by: Max Tottenham Signed-off-by: Anna Glasgall Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/bpf/20260812083115.73100-5-nhudson@akamai.com --- net/core/filter.c | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/net/core/filter.c b/net/core/filter.c index 00b7ebe4a02b..3ba653ab3239 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include @@ -3572,6 +3573,12 @@ static u32 bpf_skb_net_base_len(const struct sk_buff *skb) #define BPF_F_ADJ_ROOM_DECAP_L3_MASK (BPF_F_ADJ_ROOM_DECAP_L3_IPV4 | \ BPF_F_ADJ_ROOM_DECAP_L3_IPV6) +#define BPF_F_ADJ_ROOM_DECAP_L4_MASK (BPF_F_ADJ_ROOM_DECAP_L4_UDP | \ + BPF_F_ADJ_ROOM_DECAP_L4_GRE) + +#define BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK (BPF_F_ADJ_ROOM_DECAP_IPXIP4 | \ + BPF_F_ADJ_ROOM_DECAP_IPXIP6) + #define BPF_F_ADJ_ROOM_ENCAP_MASK (BPF_F_ADJ_ROOM_ENCAP_L3_MASK | \ BPF_F_ADJ_ROOM_ENCAP_L4_GRE | \ BPF_F_ADJ_ROOM_ENCAP_L4_UDP | \ @@ -3579,7 +3586,9 @@ static u32 bpf_skb_net_base_len(const struct sk_buff *skb) BPF_F_ADJ_ROOM_ENCAP_L2( \ BPF_ADJ_ROOM_ENCAP_L2_MASK)) -#define BPF_F_ADJ_ROOM_DECAP_MASK (BPF_F_ADJ_ROOM_DECAP_L3_MASK) +#define BPF_F_ADJ_ROOM_DECAP_MASK (BPF_F_ADJ_ROOM_DECAP_L3_MASK | \ + BPF_F_ADJ_ROOM_DECAP_L4_MASK | \ + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK) #define BPF_F_ADJ_ROOM_MASK (BPF_F_ADJ_ROOM_FIXED_GSO | \ BPF_F_ADJ_ROOM_ENCAP_MASK | \ @@ -3827,6 +3836,8 @@ BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, } if (flags & BPF_F_ADJ_ROOM_DECAP_MASK) { + u32 len_decap_min = 0; + if (!shrink) return -EINVAL; @@ -3835,6 +3846,37 @@ BPF_CALL_4(bpf_skb_adjust_room, struct sk_buff *, skb, s32, len_diff, BPF_F_ADJ_ROOM_DECAP_L3_MASK) return -EINVAL; + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_MASK) == + BPF_F_ADJ_ROOM_DECAP_L4_MASK) + return -EINVAL; + + if ((flags & BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK) == + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK) + return -EINVAL; + + /* Reject mutually exclusive decap tunnel type flags. */ + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_MASK) && + (flags & BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK)) + return -EINVAL; + + if (flags & BPF_F_ADJ_ROOM_DECAP_L4_MASK) + len_decap_min += bpf_skb_net_base_len(skb); + + if (flags & BPF_F_ADJ_ROOM_DECAP_L4_UDP) + len_decap_min += sizeof(struct udphdr); + + if (flags & BPF_F_ADJ_ROOM_DECAP_L4_GRE) + len_decap_min += sizeof(struct gre_base_hdr); + + if (flags & BPF_F_ADJ_ROOM_DECAP_IPXIP4) + len_decap_min += sizeof(struct iphdr); + + if (flags & BPF_F_ADJ_ROOM_DECAP_IPXIP6) + len_decap_min += sizeof(struct ipv6hdr); + + if (len_diff_abs < len_decap_min) + return -EINVAL; + if (flags & BPF_F_ADJ_ROOM_DECAP_L3_IPV4) len_min = sizeof(struct iphdr); From ec20dee2f2c4796c0f7c0a7d8a3a3e8a9e9da7a4 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:14 +0100 Subject: [PATCH 358/373] bpf: Clear decap state on skb_adjust_room shrink path On shrink in bpf_skb_adjust_room(), apply decapsulation state updates according to BPF_F_ADJ_ROOM_DECAP_* flags. For GSO skbs, clear only the tunnel gso_type bits that correspond to the requested decap layer: - DECAP_L4_UDP: SKB_GSO_UDP_TUNNEL{,_CSUM} - DECAP_L4_GRE: SKB_GSO_GRE{,_CSUM} - DECAP_IPXIP4: SKB_GSO_IPXIP4 - DECAP_IPXIP6: SKB_GSO_IPXIP6 Then clear skb->encapsulation only if no tunnel GSO bits remain, keeping encapsulation set for cases such as ESP-in-UDP where tunnel state remains. For non-GSO skbs, there are no tunnel GSO bits to consult, so clear skb->encapsulation directly when DECAP_L4_* or DECAP_IPXIP_* flags are set. This keeps decap state handling consistent between GSO and non-GSO packets. Co-developed-by: Max Tottenham Co-developed-by: Anna Glasgall Signed-off-by: Max Tottenham Signed-off-by: Anna Glasgall Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Reviewed-by: Willem de Bruijn Link: https://lore.kernel.org/bpf/20260812083115.73100-6-nhudson@akamai.com --- net/core/filter.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/net/core/filter.c b/net/core/filter.c index 3ba653ab3239..61940e753552 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3754,9 +3754,48 @@ static int bpf_skb_net_shrink(struct sk_buff *skb, u32 off, u32 len_diff, if (!(flags & BPF_F_ADJ_ROOM_FIXED_GSO)) skb_increase_gso_size(shinfo, len_diff); + /* Selective GSO flag clearing based on decap type. + * Only clear the flags for the tunnel layer being removed. + */ + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_UDP) && + (shinfo->gso_type & (SKB_GSO_UDP_TUNNEL | + SKB_GSO_UDP_TUNNEL_CSUM))) + shinfo->gso_type &= ~(SKB_GSO_UDP_TUNNEL | + SKB_GSO_UDP_TUNNEL_CSUM); + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_GRE) && + (shinfo->gso_type & (SKB_GSO_GRE | SKB_GSO_GRE_CSUM))) + shinfo->gso_type &= ~(SKB_GSO_GRE | + SKB_GSO_GRE_CSUM); + if ((flags & BPF_F_ADJ_ROOM_DECAP_IPXIP4) && + (shinfo->gso_type & SKB_GSO_IPXIP4)) + shinfo->gso_type &= ~SKB_GSO_IPXIP4; + if ((flags & BPF_F_ADJ_ROOM_DECAP_IPXIP6) && + (shinfo->gso_type & SKB_GSO_IPXIP6)) + shinfo->gso_type &= ~SKB_GSO_IPXIP6; + + /* Clear encapsulation flag only when no tunnel GSO flags remain */ + if (flags & (BPF_F_ADJ_ROOM_DECAP_L4_MASK | + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK)) { + if (!(shinfo->gso_type & (SKB_GSO_UDP_TUNNEL | + SKB_GSO_UDP_TUNNEL_CSUM | + SKB_GSO_GRE | + SKB_GSO_GRE_CSUM | + SKB_GSO_IPXIP4 | + SKB_GSO_IPXIP6 | + SKB_GSO_ESP))) + if (skb->encapsulation) + skb->encapsulation = 0; + } + /* Header must be checked, and gso_segs recomputed. */ shinfo->gso_type |= SKB_GSO_DODGY; shinfo->gso_segs = 0; + } else { + /* For non-GSO packets, clear encapsulation if decap flags are set */ + if ((flags & (BPF_F_ADJ_ROOM_DECAP_L4_MASK | + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK)) && + skb->encapsulation) + skb->encapsulation = 0; } return 0; From adb771973026efe54627bcbe927e7205d04d6c68 Mon Sep 17 00:00:00 2001 From: Nick Hudson Date: Wed, 12 Aug 2026 09:31:15 +0100 Subject: [PATCH 359/373] selftests/bpf: tc_tunnel - validate decap GSO and encapsulation state tc_tunnel only partially validated decap state and missed some tunnel cases. In particular, IPXIP decap checks were not exercised for IPIP/SIT paths, and non-GSO decap encapsulation state was not verified. Tighten the test by: - setting DECAP_IPXIP4/6 flags for IPIP/SIT/IP6 decap paths based on the outer tunnel header family; - requiring needed DECAP enum values via CO-RE enum existence checks so missing kernel support fails fast; - validating post-decap tunnel state for both GSO and non-GSO packets: expected gso_type bits must be cleared and skb->encapsulation must match remaining tunnel flags; - removing forced TSO disable in the test harness so GSO validation is exercised. This improves coverage for decap tunnel-state regressions and ensures sit_none/ipip-style paths are checked correctly. Signed-off-by: Nick Hudson Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260812083115.73100-7-nhudson@akamai.com --- .../selftests/bpf/prog_tests/test_tc_tunnel.c | 1 - .../selftests/bpf/progs/test_tc_tunnel.c | 91 +++++++++++++++++-- 2 files changed, 84 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/test_tc_tunnel.c b/tools/testing/selftests/bpf/prog_tests/test_tc_tunnel.c index 1aa7c9463980..67ba27d69347 100644 --- a/tools/testing/selftests/bpf/prog_tests/test_tc_tunnel.c +++ b/tools/testing/selftests/bpf/prog_tests/test_tc_tunnel.c @@ -438,7 +438,6 @@ static int setup(void) SYS(fail_close_ns_client, "ip link add %s type veth peer name %s", "veth1 mtu 1500 netns " CLIENT_NS " address " MAC_ADDR_VETH1, "veth2 mtu 1500 netns " SERVER_NS " address " MAC_ADDR_VETH2); - SYS(fail_close_ns_client, "ethtool -K veth1 tso off"); SYS(fail_close_ns_client, "ip link set veth1 up"); nstoken_server = open_netns(SERVER_NS); if (!ASSERT_OK_PTR(nstoken_server, "open server ns")) diff --git a/tools/testing/selftests/bpf/progs/test_tc_tunnel.c b/tools/testing/selftests/bpf/progs/test_tc_tunnel.c index 7376df405a6b..853bca962910 100644 --- a/tools/testing/selftests/bpf/progs/test_tc_tunnel.c +++ b/tools/testing/selftests/bpf/progs/test_tc_tunnel.c @@ -6,6 +6,7 @@ #include #include +#include #include "bpf_tracing_net.h" #include "bpf_compiler.h" @@ -37,6 +38,22 @@ struct vxlanhdr___local { #define EXTPROTO_VXLAN 0x1 +#define SKB_GSO_UDP_TUNNEL_MASK (SKB_GSO_UDP_TUNNEL | \ + SKB_GSO_UDP_TUNNEL_CSUM) + +#define SKB_GSO_TUNNEL_MASK (SKB_GSO_UDP_TUNNEL_MASK | \ + SKB_GSO_GRE | \ + SKB_GSO_GRE_CSUM | \ + SKB_GSO_IPXIP4 | \ + SKB_GSO_IPXIP6 | \ + SKB_GSO_ESP) + +#define BPF_F_ADJ_ROOM_DECAP_L4_MASK (BPF_F_ADJ_ROOM_DECAP_L4_UDP | \ + BPF_F_ADJ_ROOM_DECAP_L4_GRE) + +#define BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK (BPF_F_ADJ_ROOM_DECAP_IPXIP4 | \ + BPF_F_ADJ_ROOM_DECAP_IPXIP6) + #define VXLAN_FLAGS bpf_htonl(1<<27) #define VNI_ID 1 #define VXLAN_VNI bpf_htonl(VNI_ID << 8) @@ -589,9 +606,12 @@ int __encap_ip6vxlan_eth(struct __sk_buff *skb) return TC_ACT_OK; } -static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) +static int decap_internal(struct __sk_buff *skb, int off, int len, char proto, + __u64 ipxip_flag) { __u64 flags = BPF_F_ADJ_ROOM_FIXED_GSO; + struct sk_buff *kskb; + struct skb_shared_info *shinfo; struct ipv6_opt_hdr ip6_opt_hdr; struct gre_hdr greh; struct udphdr udph; @@ -599,10 +619,12 @@ static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) switch (proto) { case IPPROTO_IPIP: - flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV4; + flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV4 | + ipxip_flag; break; case IPPROTO_IPV6: - flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV6; + flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV6 | + ipxip_flag; break; case NEXTHDR_DEST: if (bpf_skb_load_bytes(skb, off + len, &ip6_opt_hdr, @@ -610,10 +632,12 @@ static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) return TC_ACT_OK; switch (ip6_opt_hdr.nexthdr) { case IPPROTO_IPIP: - flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV4; + flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV4 | + ipxip_flag; break; case IPPROTO_IPV6: - flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV6; + flags |= BPF_F_ADJ_ROOM_DECAP_L3_IPV6 | + ipxip_flag; break; default: return TC_ACT_OK; @@ -621,6 +645,11 @@ static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) break; case IPPROTO_GRE: olen += sizeof(struct gre_hdr); + if (!bpf_core_enum_value_exists(enum bpf_adj_room_flags, + BPF_F_ADJ_ROOM_DECAP_L4_GRE)) + return TC_ACT_SHOT; + flags |= BPF_F_ADJ_ROOM_DECAP_L4_GRE; + if (bpf_skb_load_bytes(skb, off + len, &greh, sizeof(greh)) < 0) return TC_ACT_OK; switch (bpf_ntohs(greh.protocol)) { @@ -634,6 +663,10 @@ static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) break; case IPPROTO_UDP: olen += sizeof(struct udphdr); + if (!bpf_core_enum_value_exists(enum bpf_adj_room_flags, + BPF_F_ADJ_ROOM_DECAP_L4_UDP)) + return TC_ACT_SHOT; + flags |= BPF_F_ADJ_ROOM_DECAP_L4_UDP; if (bpf_skb_load_bytes(skb, off + len, &udph, sizeof(udph)) < 0) return TC_ACT_OK; switch (bpf_ntohs(udph.dest)) { @@ -655,6 +688,40 @@ static int decap_internal(struct __sk_buff *skb, int off, int len, char proto) if (bpf_skb_adjust_room(skb, -olen, BPF_ADJ_ROOM_MAC, flags)) return TC_ACT_SHOT; + kskb = bpf_cast_to_kern_ctx(skb); + shinfo = bpf_core_cast(kskb->head + kskb->end, struct skb_shared_info); + if (shinfo->gso_size) { + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_UDP) && + (shinfo->gso_type & SKB_GSO_UDP_TUNNEL_MASK)) + return TC_ACT_SHOT; + + if ((flags & BPF_F_ADJ_ROOM_DECAP_L4_GRE) && + (shinfo->gso_type & (SKB_GSO_GRE | SKB_GSO_GRE_CSUM))) + return TC_ACT_SHOT; + + if ((flags & BPF_F_ADJ_ROOM_DECAP_IPXIP4) && + (shinfo->gso_type & SKB_GSO_IPXIP4)) + return TC_ACT_SHOT; + + if ((flags & BPF_F_ADJ_ROOM_DECAP_IPXIP6) && + (shinfo->gso_type & SKB_GSO_IPXIP6)) + return TC_ACT_SHOT; + + if (flags & (BPF_F_ADJ_ROOM_DECAP_L4_MASK | + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK)) { + if ((shinfo->gso_type & SKB_GSO_TUNNEL_MASK) && + !kskb->encapsulation) + return TC_ACT_SHOT; + if (!(shinfo->gso_type & SKB_GSO_TUNNEL_MASK) && + kskb->encapsulation) + return TC_ACT_SHOT; + } + } else if ((flags & (BPF_F_ADJ_ROOM_DECAP_L4_MASK | + BPF_F_ADJ_ROOM_DECAP_IPXIP_MASK)) && + kskb->encapsulation) { + return TC_ACT_SHOT; + } + return TC_ACT_OK; } @@ -662,6 +729,10 @@ static int decap_ipv4(struct __sk_buff *skb) { struct iphdr iph_outer; + if (!bpf_core_enum_value_exists(enum bpf_adj_room_flags, + BPF_F_ADJ_ROOM_DECAP_IPXIP4)) + return TC_ACT_SHOT; + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_outer, sizeof(iph_outer)) < 0) return TC_ACT_OK; @@ -670,19 +741,25 @@ static int decap_ipv4(struct __sk_buff *skb) return TC_ACT_OK; return decap_internal(skb, ETH_HLEN, sizeof(iph_outer), - iph_outer.protocol); + iph_outer.protocol, + BPF_F_ADJ_ROOM_DECAP_IPXIP4); } static int decap_ipv6(struct __sk_buff *skb) { struct ipv6hdr iph_outer; + if (!bpf_core_enum_value_exists(enum bpf_adj_room_flags, + BPF_F_ADJ_ROOM_DECAP_IPXIP6)) + return TC_ACT_SHOT; + if (bpf_skb_load_bytes(skb, ETH_HLEN, &iph_outer, sizeof(iph_outer)) < 0) return TC_ACT_OK; return decap_internal(skb, ETH_HLEN, sizeof(iph_outer), - iph_outer.nexthdr); + iph_outer.nexthdr, + BPF_F_ADJ_ROOM_DECAP_IPXIP6); } SEC("tc") From 1b5aacd5b2419b0790e955e466d389a61c79b4b1 Mon Sep 17 00:00:00 2001 From: Junseo Lim Date: Tue, 11 Aug 2026 23:19:07 +0900 Subject: [PATCH 360/373] bpf: Reject negative optlen in cgroup getsockopt hook A cgroup getsockopt BPF program can shrink ctx->optlen after the kernel getsockopt handler has run. The kernel-buffer variant, used by TCP_ZEROCOPY_RECEIVE, only rejects values larger than the original length. If BPF writes a negative optlen, that value is accepted and propagated back to the TCP getsockopt code. It can then be passed to copy_to_sockptr() as a size_t and trigger the hardened usercopy bytes > INT_MAX warning. Reject negative ctx.optlen in __cgroup_bpf_run_filter_getsockopt_kern(), matching the lower-bound validation already present in the sockptr-based getsockopt hook. Fixes: 9cacf81f8161 ("bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE") Reported-by: Sechang Lim Signed-off-by: Junseo Lim Signed-off-by: Daniel Borkmann Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/187a4d756275aaaee5d65eecb63c1477b3b66554.1786448307.git.zirajs7@gmail.com --- kernel/bpf/cgroup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 8fbc942a1cc3..149672c76c49 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -2260,7 +2260,7 @@ int __cgroup_bpf_run_filter_getsockopt_kern(struct sock *sk, int level, if (ret < 0) return ret; - if (ctx.optlen > *optlen) + if (ctx.optlen > *optlen || ctx.optlen < 0) return -EFAULT; /* BPF programs can shrink the buffer, export the modifications. From 6b0835ac79b2e43a7948c911add88246baea0a98 Mon Sep 17 00:00:00 2001 From: Junseo Lim Date: Tue, 11 Aug 2026 23:19:08 +0900 Subject: [PATCH 361/373] selftests/bpf: Exercise negative optlen in cgroup getsockopt hook Add a cgroup getsockopt selftest that sets ctx->optlen to -1. Use TCP_ZEROCOPY_RECEIVE to exercise the kernel-buffer getsockopt hook. The userspace-visible result is -EFAULT on both patched and unpatched kernels, so the return value alone cannot distinguish the bug. The test still exercises the kernel-buffer getsockopt path with a negative ctx->optlen, which reproduces the hardened usercopy warning on unpatched kernels. Signed-off-by: Junseo Lim Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/4dbdcda23b2f2be06c5659f8102cd6bd036825b3.1786448307.git.zirajs7@gmail.com --- .../selftests/bpf/prog_tests/sockopt.c | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/sockopt.c b/tools/testing/selftests/bpf/prog_tests/sockopt.c index eaac83a7f388..6c96f2d9fccf 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockopt.c +++ b/tools/testing/selftests/bpf/prog_tests/sockopt.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include #include "cgroup_helpers.h" @@ -283,6 +284,27 @@ static struct sockopt_test { .error = EFAULT_GETSOCKOPT, .io_uring_support = true, }, + { + .descr = "getsockopt: deny negative ctx->optlen in TCP_ZEROCOPY_RECEIVE", + .insns = { + /* ctx->optlen = -1 */ + BPF_MOV64_IMM(BPF_REG_0, -1), + BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, + offsetof(struct bpf_sockopt, optlen)), + + /* return 1 */ + BPF_MOV64_IMM(BPF_REG_0, 1), + BPF_EXIT_INSN(), + }, + .attach_type = BPF_CGROUP_GETSOCKOPT, + .expected_attach_type = BPF_CGROUP_GETSOCKOPT, + + .get_level = IPPROTO_TCP, + .get_optname = TCP_ZEROCOPY_RECEIVE, + .get_optlen = sizeof(struct tcp_zerocopy_receive), + + .error = EFAULT_GETSOCKOPT, + }, { .descr = "getsockopt: ignore >PAGE_SIZE optlen", .insns = { From b26c0b2dd5195f3397e7f64fefc9c190ebba7204 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:31 +0200 Subject: [PATCH 362/373] bpf: Preserve R0 lineage across helper calls check_helper_call() clears all caller-saved registers before taking the diagnostic snapshot of R0. This records NOT_INIT as the old state for every helper return and loses the lineage of the value held in R0 before the call. bpf_diag_record_caller_saved() deliberately skips R0 because the paired modification scope is responsible for it. Open the R0 modification scope before clearing caller-saved registers, matching the kfunc, ld_abs, and subprogram call paths. Reported-by: Sashiko Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260815073833.A93A91F000E9@smtp.kernel.org Link: https://lore.kernel.org/bpf/48e6f021b89562f68850fe21ef8c78719819b04cf9c4e4f50bc791937d37ace8@mail.kernel.org Link: https://lore.kernel.org/bpf/20260816015746.2632990-4-memxor@gmail.com --- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9f833e913e43..4adc13584818 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10987,13 +10987,13 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn /* reset caller saved regs */ bpf_diag_record_caller_saved(env, regs); + bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); for (i = 0; i < CALLER_SAVED_REGS; i++) { bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); } invalidate_outgoing_stack_args(env, cur_func(env)); - bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE); /* update return register (already marked as written above) */ ret_type = fn->ret_type; ret_flag = type_flag(ret_type); From 09a0c2d678643aa8362ed83ea21b3a21566b318a Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:33 +0200 Subject: [PATCH 363/373] bpf: Use canonical stack argument names in diagnostics The main diagnostic identifies the first outgoing stack slot as stack argument 1 and the sixth function argument. The causal history instead labels the same value as stack arg6, making it look like a different slot. Render causal-history targets in the verifier's canonical stack-argument location form. The first outgoing slot is now shown as *(R11-8), matching reg_arg_name(), while the main diagnostic retains its fuller slot and ordinal description. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/eb1be5327d136b7e5bd6d68e76fef6de20c40790.camel@gmail.com Link: https://lore.kernel.org/bpf/20260816015746.2632990-6-memxor@gmail.com --- kernel/bpf/diagnostics.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index 33b7d9e8e2c3..b24be9df5fab 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -2164,7 +2164,7 @@ static const char *diag_mod_target_desc(struct bpf_verifier_env *env, case BPF_DIAG_MOD_TARGET_REG: return bpf_diag_fmt(env, "R%u", target->regno); case BPF_DIAG_MOD_TARGET_STACK_ARG: - return bpf_diag_fmt(env, "stack arg%d", diag_stack_argno(target->stack_arg)); + return bpf_diag_fmt(env, "*(R11-%u)", (target->stack_arg + 1) * BPF_REG_SIZE); case BPF_DIAG_MOD_TARGET_STACK_SLOT: return bpf_diag_fmt(env, "stack slot fp%d", -(target->spi + 1) * BPF_REG_SIZE); default: From cc782c7ad0f7416f4fcf90bf15064313cbb8a7c9 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:34 +0200 Subject: [PATCH 364/373] bpf: Correct kfunc argument diagnostics The Call Type Safety diagnostics mishandle three kfunc argument classes. BTF type ID 0 represents void, but btf_show_name() also uses zero to end type traversal. A pointer that resolves to void therefore loses its pointee name and is rendered as "()". End traversal directly for concrete terminal types, but resolve referenced types before testing for ID zero, and name the void terminal type explicitly. Format the complete parameter pointer type for nullable kfunc arguments, so void pointers are reported as (void *). Also add the missing structured report when an __szk memory-size argument is not a verifier-known constant. Describe the generic bpf_refcount_acquire() contract without deriving an object type from its void pointer prototype. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/668871823f90f69896d3db27b56db2f53e481162.camel@gmail.com Link: https://lore.kernel.org/bpf/20260816015746.2632990-7-memxor@gmail.com --- kernel/bpf/btf.c | 8 ++++---- kernel/bpf/verifier.c | 22 +++++++++++++--------- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 5b9d767895c9..6967d48bba49 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -1169,19 +1169,19 @@ static const char *btf_show_name(struct btf_show *show) id = t->type; break; default: - id = 0; - break; + goto resolved; } + t = btf_type_skip_qualifiers(show->btf, id); if (!id) break; - t = btf_type_skip_qualifiers(show->btf, id); } /* We may not be able to represent this type; bail to be safe */ if (i == BTF_SHOW_MAX_ITER) return ""; +resolved: if (!name) - name = btf_name_by_offset(show->btf, t->name_off); + name = btf_type_is_void(t) ? "void" : btf_name_by_offset(show->btf, t->name_off); switch (BTF_INFO_KIND(t->info)) { case BTF_KIND_STRUCT: diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4adc13584818..a25f3c94976a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12616,12 +12616,12 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me !type_may_be_null(kf_arg_type)) { const char *expected_type; - expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); + expected_type = bpf_diag_fmt_btf_type(env, btf, args[i].type); verbose(env, "Possibly NULL pointer passed to trusted %s\n", reg_arg_name(env, argno)); bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, "Add a NULL check and call the kfunc only on the non-NULL path.", - "the pointer may be NULL, but this kfunc requires a non-NULL pointer to %s", + "the pointer may be NULL, but this kfunc requires a non-NULL value of type %s", expected_type); return -EACCES; } @@ -13058,8 +13058,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me break; case KF_ARG_CONST_MEM_SIZE: ret = process_const_arg(env, reg, argno, meta); - if (ret < 0) + if (ret < 0) { + if (ret == -EINVAL) + bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, + "Pass a compile-time constant or a value the verifier can prove is constant at this call.", + "the kfunc requires this memory size to be a verifier-known constant, but %s is variable on this path", + reg_arg_name(env, argno)); return ret; + } fallthrough; case KF_ARG_MEM_SIZE: { @@ -13123,15 +13129,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me break; case KF_ARG_PTR_TO_REFCOUNTED_KPTR: if (!type_is_ptr_alloc_obj(reg->type)) { - const char *expected_type; - - expected_type = bpf_diag_fmt_btf_type(env, btf, ref_id); verbose(env, "%s is neither owning or non-owning ref\n", reg_arg_name(env, argno)); bpf_diag_call_arg_fmt(env, insn_idx, argno, func_name, - "Pass a pointer returned by the matching BPF object allocation or lookup operation for this kfunc.", - "the kfunc expects a pointer to BPF-managed refcounted object type %s, but this argument is not such an object pointer", - expected_type); + "Pass an owning or non-owning pointer to a BPF-managed object containing a bpf_refcount field.", + "the kfunc expects a pointer to a BPF-managed refcounted object, but %s is %s", + reg_arg_name(env, argno), + bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } if (!type_is_non_owning_ref(reg->type)) From b03bb4a597f91403b6397bb792f5bb4ef4032dd6 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:35 +0200 Subject: [PATCH 365/373] selftests/bpf: Test kfunc argument diagnostics Extend existing negative kfunc programs to assert that BTF void is rendered as void and that variable __szk arguments receive a structured constant-size diagnostic. Also pass a context pointer to bpf_refcount_acquire() and verify that the report describes the generic refcounted-object contract and the actual argument type. Retain the legacy verbose-message assertions. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/3eda33675965763aa9b2e6a5784f32b34a6a83988a55fbea98b0dbd0cf3b088d@mail.kernel.org Link: https://lore.kernel.org/bpf/20260816015746.2632990-8-memxor@gmail.com --- tools/testing/selftests/bpf/progs/dynptr_fail.c | 2 ++ tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c index beaa73dc35f5..1cd61d72c166 100644 --- a/tools/testing/selftests/bpf/progs/dynptr_fail.c +++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c @@ -1590,6 +1590,7 @@ __u32 hdr_size = sizeof(struct ethhdr); /* Can't pass in variable-sized len to bpf_dynptr_slice */ SEC("?tc") __failure __msg("must be a known constant") +__msg("requires this memory size to be a verifier-known constant") int dynptr_slice_var_len1(struct __sk_buff *skb) { struct bpf_dynptr ptr; @@ -1609,6 +1610,7 @@ int dynptr_slice_var_len1(struct __sk_buff *skb) /* Can't pass in variable-sized len to bpf_dynptr_slice */ SEC("?tc") __failure __msg("must be a known constant") +__msg("requires this memory size to be a verifier-known constant") int dynptr_slice_var_len2(struct __sk_buff *skb) { char buffer[sizeof(struct ethhdr)] = {}; diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c index 024ef2aae200..eaaed0859f94 100644 --- a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c @@ -63,6 +63,7 @@ long rbtree_refcounted_node_ref_escapes(void *ctx) SEC("?tc") __failure __msg("Possibly NULL pointer passed to trusted R1") +__msg("requires a non-NULL value of type (void *)") long refcount_acquire_maybe_null(void *ctx) { struct node_acquire *n, *m; @@ -80,6 +81,14 @@ long refcount_acquire_maybe_null(void *ctx) return 0; } +SEC("?tc") +__failure __msg("R1 is neither owning or non-owning ref") +__msg("expects a pointer to a BPF-managed refcounted object, but R1 is a context pointer") +long refcount_acquire_non_object(void *ctx) +{ + return bpf_refcount_acquire(ctx) != NULL; +} + SEC("?tc") __failure __msg("Unreleased reference id=3 alloc_insn={{[0-9]+}}") long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) From 6bd520a6e3b66010e6e87ef77a1756c6b6ce30b1 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:39 +0200 Subject: [PATCH 366/373] bpf: Preserve source attribution without source text GCC emits BTF line records with a file name and line number, but leaves the source line string empty. bpf_diag_source() currently treats that empty string as if the complete line record were unavailable, so diagnostics fall back to an instruction number and discard the function, file, and line attribution. Print the available source location before deciding whether source context can be rendered. When source text is absent, omit only the source context and retain the diagnostic annotation and instruction context. Fixes: b9c5d822f677 ("bpf: Add source and instruction diagnostic context") Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260816015746.2632990-12-memxor@gmail.com --- kernel/bpf/diagnostics.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c index b24be9df5fab..b682fd2be443 100644 --- a/kernel/bpf/diagnostics.c +++ b/kernel/bpf/diagnostics.c @@ -833,11 +833,9 @@ static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const ch linfo = bpf_find_linfo(env->prog, insn_idx); if (btf && linfo) bpf_get_linfo_source(btf, linfo, &src); - if (!src.file || !*src.file || !src.line || !*src.line) { + if (!src.file || !*src.file) { diag_write(env, " insn %u\n", insn_idx); - diag_print_source_annotation(env, 0, 0, label, msg); - diag_print_insn_context(env, insn_idx, disasm_lines); - goto out_restore; + goto out_annotation; } subprog = bpf_find_containing_subprog(env, insn_idx); @@ -847,6 +845,8 @@ static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const ch diag_write(env, " %s @ %s:%d:%d\n", func, src.file, src.line_num, src.line_col); else diag_write(env, " %s:%d:%d\n", src.file, src.line_num, src.line_col); + if (!src.line || !*src.line) + goto out_annotation; start_line = src.line_num - BPF_DIAG_CONTEXT; end_line = src.line_num + BPF_DIAG_CONTEXT; @@ -889,7 +889,11 @@ static void bpf_diag_source(struct bpf_verifier_env *env, u32 insn_idx, const ch diag_print_source_annotation(env, width, indent, label, msg); } diag_print_insn_context(env, insn_idx, disasm_lines); + goto out_restore; +out_annotation: + diag_print_source_annotation(env, 0, 0, label, msg); + diag_print_insn_context(env, insn_idx, disasm_lines); out_restore: diag_fmt_restore(env, mark); } From fc009f4658224734e6859b8f67a681ecac5a2b22 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:41 +0200 Subject: [PATCH 367/373] bpf: Distinguish function references in policy diagnostics add_subprogs() rejects both BPF-to-BPF calls and BPF_PSEUDO_FUNC loads for unprivileged programs. The latter loads a subprogram address for use as a callback, but its Policy report currently describes it as a function call and suggests avoiding calls that the program does not contain. Select the operation and suggestion from the instruction kind. Preserve the existing call wording for BPF_PSEUDO_CALL, and describe BPF_PSEUDO_FUNC as a BPF function reference. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/bpf/d02e6a6d3b2dc43a207b8ba836ce62497b250dede9252e7409c5212201c794b7@mail.kernel.org Link: https://lore.kernel.org/bpf/20260816015746.2632990-14-memxor@gmail.com --- kernel/bpf/verifier.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a25f3c94976a..e421ea2b80c3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2912,6 +2912,7 @@ static int add_subprogs(struct bpf_verifier_env *env) struct bpf_subprog_info *subprog = env->subprog_info; int i, ret, insn_cnt = env->prog->len, ex_cb_insn; struct bpf_insn *insn = env->prog->insnsi; + const char *operation, *suggestion; /* Add entry function. */ ret = add_subprog(env, 0); @@ -2923,11 +2924,18 @@ static int add_subprogs(struct bpf_verifier_env *env) continue; if (!env->bpf_capable) { + if (bpf_pseudo_func(insn)) { + operation = "BPF function reference"; + suggestion = "Load this program with the required capability, or avoid BPF function references in unprivileged programs."; + } else { + operation = "BPF-to-BPF function call"; + suggestion = "Load this program with the required capability, or avoid BPF-to-BPF function calls in unprivileged programs."; + } verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n"); bpf_diag_policy( - env, i, "BPF-to-BPF function call", + env, i, operation, "loading or calling other BPF functions requires CAP_BPF or CAP_SYS_ADMIN", - "Load this program with the required capability, or avoid BPF-to-BPF function calls in unprivileged programs."); + suggestion); return -EPERM; } From 6ab6a94c4f7915fbaa0f072c5f042df398faa2d6 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sun, 16 Aug 2026 03:57:42 +0200 Subject: [PATCH 368/373] selftests/bpf: Test pseudo-function policy diagnostics Load a socket-filter program that passes a callback to bpf_loop() without making a BPF-to-BPF call. Verify that the privileged load succeeds and the unprivileged Policy report identifies the BPF function reference at its ldimm64 instruction. Also reject the inaccurate BPF-to-BPF call wording in the portion of the log covered by the structured report. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/bpf/d02e6a6d3b2dc43a207b8ba836ce62497b250dede9252e7409c5212201c794b7@mail.kernel.org Link: https://lore.kernel.org/bpf/20260816015746.2632990-15-memxor@gmail.com --- .../selftests/bpf/progs/verifier_unpriv.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c index 42de5cff7e52..3069e70fbcbd 100644 --- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c +++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c @@ -96,6 +96,24 @@ __naked void pseudo_btf_id_log_masks_address(void) : __clobber_all); } +static int pseudo_func_callback(__u32 index, void *ctx) +{ + return 0; +} + +SEC("socket") +__description("unpriv: pseudo function policy diagnostic") +__success __failure_unpriv +__msg_unpriv("loading/calling other bpf or kernel functions") +__not_msg_unpriv("BPF-to-BPF function call") +__msg_unpriv("policy check failed for BPF function reference") +__msg_unpriv("avoid BPF function references in unprivileged") +int unpriv_pseudo_func_policy(void *ctx) +{ + bpf_loop(1, pseudo_func_callback, NULL, 0); + return 0; +} + SEC("socket") __description("unpriv: return pointer") __success __failure_unpriv __msg_unpriv("R0 leaks addr") From 3d9393ff98ca9a132c7a0778436f5ed856fb3dc6 Mon Sep 17 00:00:00 2001 From: Andrii Nakryiko Date: Fri, 14 Aug 2026 16:20:17 -0700 Subject: [PATCH 369/373] selftests/bpf: Retry stat generation in cgroup_iter_memcg Each cgroup_iter_memcg subtest touches 1024 pages and expects the matching memcg counter to be non-zero. On a host with many CPUs it reads zero instead: test_anon:FAIL:final anon mapped val: actual 0 <= expected 0 memcg stats are cached per-cpu and only become visible once the periodic flusher runs (FLUSH_TIME, 2s), or once pending updates cross MEMCG_CHARGE_BATCH * num_online_cpus(). That threshold is 512 pages at 8 CPUs but 8192 at 128, so a single pass no longer reaches it and bpf_mem_cgroup_flush_stats() returns without flushing anything. Retry the stat generation, sleeping in between, so that a flusher cycle is always covered. Sleep before dropping the mapping, so that a flusher cycle landing in the sleep observes the mapped state. nr_anon_mapped and nr_file_mapped are rmap gauges, and unmapping first would post a matching negative delta for the flusher to aggregate to a net zero. test_file asserts on both nr_file_pages and nr_file_mapped, which have different lifetimes, as page cache pages outlive the mapping. Retry while either one is still zero. Fixes: 6bce6ddbe634 ("bpf: selftests: selftests for memcg stat kfuncs") Signed-off-by: Andrii Nakryiko Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260814232017.2839959-1-andrii@kernel.org --- .../bpf/prog_tests/cgroup_iter_memcg.c | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_memcg.c b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_memcg.c index b7c18d590b99..5a1e08d39a06 100644 --- a/tools/testing/selftests/bpf/prog_tests/cgroup_iter_memcg.c +++ b/tools/testing/selftests/bpf/prog_tests/cgroup_iter_memcg.c @@ -10,6 +10,17 @@ #include "cgroup_iter_memcg.h" #include "cgroup_iter_memcg.skel.h" +/* + * memcg stats are cached per-cpu and only become visible once the periodic + * flusher runs (FLUSH_TIME, 2s), or once pending updates cross + * MEMCG_CHARGE_BATCH * num_online_cpus(). That threshold grows with the CPU + * count, so on a large machine a single pass does not reach it and + * bpf_mem_cgroup_flush_stats() returns without flushing anything. Retry for + * long enough to cover a flusher cycle. + */ +#define MEMCG_STAT_RETRIES 16 +#define MEMCG_STAT_RETRY_DELAY_US (250 * 1000) + static int read_stats(struct bpf_link *link) { int fd, ret = 0; @@ -35,11 +46,13 @@ static int read_stats(struct bpf_link *link) static void test_anon(struct bpf_link *link, struct memcg_query *memcg_query) { + int retries = 0; void *map; size_t len; len = sysconf(_SC_PAGESIZE) * 1024; +retry: /* * Increase memcg anon usage by mapping and writing * to a new anon region. @@ -53,6 +66,12 @@ static void test_anon(struct bpf_link *link, struct memcg_query *memcg_query) if (!ASSERT_OK(read_stats(link), "read stats")) goto cleanup; + if (!memcg_query->nr_anon_mapped && ++retries < MEMCG_STAT_RETRIES) { + usleep(MEMCG_STAT_RETRY_DELAY_US); + munmap(map, len); + goto retry; + } + ASSERT_GT(memcg_query->nr_anon_mapped, 0, "final anon mapped val"); cleanup: @@ -61,6 +80,7 @@ static void test_anon(struct bpf_link *link, struct memcg_query *memcg_query) static void test_file(struct bpf_link *link, struct memcg_query *memcg_query) { + int retries = 0; void *map; size_t len; char *path; @@ -76,6 +96,7 @@ static void test_file(struct bpf_link *link, struct memcg_query *memcg_query) fd = open(path, O_CREAT | O_RDWR, 0644); if (!ASSERT_OK_FD(fd, "open fd")) return; +retry: if (!ASSERT_OK(ftruncate(fd, len), "ftruncate")) goto cleanup_fd; @@ -88,6 +109,13 @@ static void test_file(struct bpf_link *link, struct memcg_query *memcg_query) if (!ASSERT_OK(read_stats(link), "read stats")) goto cleanup_map; + if ((!memcg_query->nr_file_pages || !memcg_query->nr_file_mapped) && + ++retries < MEMCG_STAT_RETRIES) { + usleep(MEMCG_STAT_RETRY_DELAY_US); + munmap(map, len); + goto retry; + } + ASSERT_GT(memcg_query->nr_file_pages, 0, "final file value"); ASSERT_GT(memcg_query->nr_file_mapped, 0, "final file mapped value"); @@ -100,6 +128,7 @@ static void test_file(struct bpf_link *link, struct memcg_query *memcg_query) static void test_shmem(struct bpf_link *link, struct memcg_query *memcg_query) { + int retries = 0; size_t len; int fd; @@ -113,12 +142,18 @@ static void test_shmem(struct bpf_link *link, struct memcg_query *memcg_query) if (!ASSERT_OK_FD(fd, "memfd_create")) return; +retry: if (!ASSERT_OK(fallocate(fd, 0, 0, len), "fallocate")) goto cleanup; if (!ASSERT_OK(read_stats(link), "read stats")) goto cleanup; + if (!memcg_query->nr_shmem && ++retries < MEMCG_STAT_RETRIES) { + usleep(MEMCG_STAT_RETRY_DELAY_US); + goto retry; + } + ASSERT_GT(memcg_query->nr_shmem, 0, "final shmem value"); cleanup: @@ -127,11 +162,13 @@ static void test_shmem(struct bpf_link *link, struct memcg_query *memcg_query) static void test_pgfault(struct bpf_link *link, struct memcg_query *memcg_query) { + int retries = 0; void *map; size_t len; len = sysconf(_SC_PAGESIZE) * 1024; +retry: /* Create region to use for triggering a page fault. */ map = mmap(NULL, len, PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (!ASSERT_NEQ(map, MAP_FAILED, "mmap anon")) @@ -143,6 +180,12 @@ static void test_pgfault(struct bpf_link *link, struct memcg_query *memcg_query) if (!ASSERT_OK(read_stats(link), "read stats")) goto cleanup; + if (!memcg_query->pgfault && ++retries < MEMCG_STAT_RETRIES) { + usleep(MEMCG_STAT_RETRY_DELAY_US); + munmap(map, len); + goto retry; + } + ASSERT_GT(memcg_query->pgfault, 0, "final pgfault val"); cleanup: From 9d19ca5d0e8b4a3f4b2eaa14e86a25f1c93ff35b Mon Sep 17 00:00:00 2001 From: Changwoo Min Date: Tue, 18 Aug 2026 01:02:49 +0900 Subject: [PATCH 370/373] selftests/bpf: Remove duplicate copies of the arena spinlock qnodes bpf_arena_spin_lock.h defines its 64KB qnodes array in the header, so every translation unit including it emits a copy. __weak makes them all resolve to one instance, but bpftool gen object merges only the symbols and concatenates each input's .addr_space.1 bytes, leaving the surplus copies unreferenced in the linked object. libarena links ten such units, so nine copies were dead weight (bytes): object before after ----------------------------------------------------- .addr_space.1 in libarena.bpf.o 676200 86376 libarena.skel.h 2100123 892371 libarena_asan.skel.h 2641124 1466477 Declare qnodes in the header and let each program define it once: libarena in src/common.bpf.c, and the arena_spin_lock test beside the lock it guards. Tested with test_progs -t arena_spin_lock and -t libarena. Signed-off-by: Changwoo Min Reviewed-by: Emil Tsalapatis Link: https://lore.kernel.org/bpf/20260817160249.655916-1-changwoo@igalia.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/libarena/include/bpf_arena_spin_lock.h | 7 +------ tools/testing/selftests/bpf/libarena/src/common.bpf.c | 7 +++++++ tools/testing/selftests/bpf/progs/arena_spin_lock.c | 7 +++++++ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h b/tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h index ae6b72d15bb6..71d9db610263 100644 --- a/tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h +++ b/tools/testing/selftests/bpf/libarena/include/bpf_arena_spin_lock.h @@ -103,12 +103,7 @@ struct arena_qnode { #define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) #define _Q_PENDING_VAL (1U << _Q_PENDING_OFFSET) -/* - * The qnodes are marked __weak so we can define them in the header - * while still ensuring all compilation units use the same struct - * instance. - */ -struct arena_qnode __weak __arena __hidden qnodes[_Q_MAX_CPUS][_Q_MAX_NODES]; +extern struct arena_qnode __arena __hidden qnodes[_Q_MAX_CPUS][_Q_MAX_NODES]; static inline u32 encode_tail(int cpu, int idx) { diff --git a/tools/testing/selftests/bpf/libarena/src/common.bpf.c b/tools/testing/selftests/bpf/libarena/src/common.bpf.c index 569f0f64d518..41b1de3452fe 100644 --- a/tools/testing/selftests/bpf/libarena/src/common.bpf.c +++ b/tools/testing/selftests/bpf/libarena/src/common.bpf.c @@ -7,6 +7,13 @@ struct buddy __arena buddy; volatile u32 zero = 0; +/* + * Storage for the queue nodes declared by bpf_arena_spin_lock.h. Each program + * linking the arena spinlock provides exactly one definition, so that the array + * is emitted once rather than once per translation unit. + */ +struct arena_qnode __arena __hidden qnodes[_Q_MAX_CPUS][_Q_MAX_NODES]; + int arena_fls(__u64 word) { if (!word) diff --git a/tools/testing/selftests/bpf/progs/arena_spin_lock.c b/tools/testing/selftests/bpf/progs/arena_spin_lock.c index cf7cda79c16c..92e75ec3844c 100644 --- a/tools/testing/selftests/bpf/progs/arena_spin_lock.c +++ b/tools/testing/selftests/bpf/progs/arena_spin_lock.c @@ -23,6 +23,13 @@ int cs_count; #if defined(ENABLE_ATOMICS_TESTS) && defined(__BPF_FEATURE_ADDR_SPACE_CAST) arena_spinlock_t __arena lock; int test_skip = 1; + +/* + * Storage for the queue nodes declared by bpf_arena_spin_lock.h. Each program + * linking the arena spinlock provides exactly one definition; libarena's lives + * in libarena/src/common.bpf.c. + */ +struct arena_qnode __arena __hidden qnodes[_Q_MAX_CPUS][_Q_MAX_NODES]; #else int test_skip = 2; #endif From 78d8e5b375c013826e08649f3cb6ad50babe5a9a Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Mon, 17 Aug 2026 16:10:13 +0200 Subject: [PATCH 371/373] selftests/bpf: Add tests for pointer type merge at a shared load Cover the ways in which the type recorded for a shared load used to lose the BPF_PROBE_MEM rewrite which would then trigger a NULL deref if not handled properly. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t mem_rdonly_untrusted [...] #242/1 mem_rdonly_untrusted/btf_id_to_ptr_mem:OK #242/2 mem_rdonly_untrusted/ldx_is_ok_bad_addr:OK #242/3 mem_rdonly_untrusted/ldx_is_ok_good_addr:OK #242/4 mem_rdonly_untrusted/offset_not_tracked:OK #242/5 mem_rdonly_untrusted/stx_not_ok:OK #242/6 mem_rdonly_untrusted/atomic_not_ok:OK #242/7 mem_rdonly_untrusted/atomic_rmw_not_ok:OK #242/8 mem_rdonly_untrusted/kfunc_param_not_ok:OK #242/9 mem_rdonly_untrusted/mixed_mem_type:OK #242/10 mem_rdonly_untrusted/mixed_mem_untrusted_btf_id_type:OK #242/11 mem_rdonly_untrusted/mixed_mem_btf_id_type:OK #242/12 mem_rdonly_untrusted/mixed_rdonly_mem_btf_id_type:OK #242/13 mem_rdonly_untrusted/mixed_mem_mem_type:OK #242/14 mem_rdonly_untrusted/mixed_map_value_mem_type:OK #242/15 mem_rdonly_untrusted/mixed_stack_mem_type:OK #242/16 mem_rdonly_untrusted/diff_size_access:OK #242/17 mem_rdonly_untrusted/misaligned_access:OK #242/18 mem_rdonly_untrusted/null_check:OK #242/19 mem_rdonly_untrusted/ldx_is_ok_commuted_addr:OK #242/20 mem_rdonly_untrusted/helper_param_not_ok:OK #242 mem_rdonly_untrusted:OK Summary: 1/20 PASSED, 0 SKIPPED, 0/0 FAILED Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260817141015.878071-1-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/progs/mem_rdonly_untrusted.c | 234 ++++++++++++++++++ 1 file changed, 234 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c index b91271d4caa4..3e0d4f687aaa 100644 --- a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c +++ b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c @@ -3,6 +3,7 @@ #include #include #include "bpf_misc.h" +#include "bpf_kfuncs.h" #include "../test_kmods/bpf_testmod_kfunc.h" SEC("tp_btf/sys_enter") @@ -164,6 +165,239 @@ int mixed_mem_type(void *ctx) return *p; } +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 4096); +} ringbuf SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, u32); + __type(value, u64); +} array SEC(".maps"); + +char dynptr_data[8]; + +int zero; + +SEC("socket") +__success +__log_level(2) +__msg("r8 = *(u64 *)(r7 +0){{.*}}R7=untrusted_ptr_sock") +__msg("r8 = *(u64 *)(r7 +0){{.*}}R7=ringbuf_mem") +__retval(0) +int mixed_mem_untrusted_btf_id_type(void *ctx) +{ + u64 *p, *q, v; + + p = bpf_ringbuf_reserve(&ringbuf, sizeof(*p), 0); + if (!p) + return 1; + *p = 42; + q = bpf_rdonly_cast(0, bpf_core_type_id_kernel(struct sock)); + /* + * The load below is reached with PTR_TO_MEM | MEM_RINGBUF on one + * path and with PTR_TO_BTF_ID | PTR_UNTRUSTED on the other. The + * merged type has to keep the BPF_PROBE_MEM rewrite, otherwise + * the NULL deref taken at runtime panics the kernel instead of + * returning 0. + */ + asm volatile ( + "r7 = %[p];" + "if %[zero] != 0 goto +1;" + "r7 = %[q];" + "r8 = *(u64 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [q]"r"(q), + [zero]"r"(zero) + : "r7", "r8"); + bpf_ringbuf_discard(p, 0); + return v; +} + +SEC("socket") +__success +__log_level(2) +__msg("r8 = *(u32 *)(r7 +0){{.*}}R7=ptr_nameidata") +__msg("r8 = *(u32 *)(r7 +0){{.*}}R7=ringbuf_mem") +__retval(0) +int mixed_mem_btf_id_type(void *ctx) +{ + struct task_struct *task; + u32 *p, *q; + u64 v; + + p = bpf_ringbuf_reserve(&ringbuf, sizeof(*p), 0); + if (!p) + return 1; + *p = 42; + task = bpf_get_current_task_btf(); + /* + * A plain BTF pointer walk yields a bare PTR_TO_BTF_ID, and + * task->nameidata is NULL unless the task currently is in the + * middle of a path lookup. + */ + q = (u32 *)&task->nameidata->flags; + /* + * Same as above, except that the other path yields a bare + * PTR_TO_BTF_ID. Merging it with PTR_TO_MEM used to drop the + * BPF_PROBE_MEM rewrite the bare PTR_TO_BTF_ID would have + * gotten on its own. + */ + asm volatile ( + "r7 = %[p];" + "if %[zero] != 0 goto +1;" + "r7 = %[q];" + "r8 = *(u32 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [q]"r"(q), + [zero]"r"(zero) + : "r7", "r8"); + bpf_ringbuf_discard(p, 0); + return v; +} + +SEC("socket") +__success +__log_level(2) +__msg("r8 = *(u32 *)(r7 +0){{.*}}R7=ptr_nameidata") +__msg("r8 = *(u32 *)(r7 +0){{.*}}R7=rdonly_mem") +__retval(0) +int mixed_rdonly_mem_btf_id_type(void *ctx) +{ + struct task_struct *task; + struct bpf_dynptr dptr; + char buf[sizeof(u32)]; + u32 *p, *q; + u64 v; + + if (bpf_dynptr_from_mem(dynptr_data, sizeof(dynptr_data), 0, &dptr)) + return 1; + p = bpf_dynptr_slice(&dptr, 0, buf, sizeof(buf)); + if (!p) + return 1; + task = bpf_get_current_task_btf(); + q = (u32 *)&task->nameidata->flags; + /* + * Same as above, except that the PTR_TO_MEM side already carries + * MEM_RDONLY. Merging it with a bare PTR_TO_BTF_ID used to yield + * PTR_TO_MEM | MEM_RDONLY, which is not rewritten either since + * only its PTR_UNTRUSTED variant is. + */ + asm volatile ( + "r7 = %[p];" + "if %[zero] != 0 goto +1;" + "r7 = %[q];" + "r8 = *(u32 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [q]"r"(q), + [zero]"r"(zero) + : "r7", "r8"); + return v; +} + +SEC("socket") +__success +__log_level(2) +__msg("r8 = *(u64 *)(r7 +0){{.*}}R7=ringbuf_mem") +__msg("r8 = *(u64 *)(r7 +0){{.*}}R7=rdonly_untrusted_mem") +__retval(0) +int mixed_mem_mem_type(void *ctx) +{ + u64 *p, *q, v; + + p = bpf_ringbuf_reserve(&ringbuf, sizeof(*p), 0); + if (!p) + return 1; + *p = 42; + q = bpf_rdonly_cast(0, 0); + /* + * Both paths are PTR_TO_MEM based, so they used to not trip the + * type mismatch check and skipped the merge altogether, leaving + * the insn with the PTR_TO_MEM | MEM_RINGBUF recorded first and + * hence without the BPF_PROBE_MEM rewrite the other path needs. + */ + asm volatile ( + "r7 = %[q];" + "if %[zero] == 0 goto +1;" + "r7 = %[p];" + "r8 = *(u64 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [q]"r"(q), + [zero]"r"(zero) + : "r7", "r8"); + bpf_ringbuf_discard(p, 0); + return v; +} + +SEC("socket") +__failure +__msg("same insn cannot be used with different pointers") +int mixed_map_value_mem_type(void *ctx) +{ + u64 *p, *q, v; + u32 key = 0; + + p = bpf_map_lookup_elem(&array, &key); + if (!p) + return 1; + q = bpf_rdonly_cast(0, 0); + /* + * PTR_TO_MAP_VALUE is neither PTR_TO_MEM nor PTR_TO_BTF_ID based, + * so it cannot be merged into a type which keeps the BPF_PROBE_MEM + * rewrite the PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED of the other + * path needs. Both bases were mismatch ok, hence the load used to be + * accepted with the PTR_TO_MAP_VALUE recorded and the NULL deref on + * the second path panicked the kernel. + */ + asm volatile ( + "r7 = %[q];" + "if %[zero] == 0 goto +1;" + "r7 = %[p];" + "r8 = *(u64 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [q]"r"(q), + [zero]"r"(zero) + : "r7", "r8"); + return v; +} + +SEC("socket") +__failure +__msg("same insn cannot be used with different pointers") +int mixed_stack_mem_type(void *ctx) +{ + u64 *p = bpf_rdonly_cast(0, 0); + u64 s = 42, v; + + /* + * Same as above, but for a PTR_TO_STACK on the other path. + */ + asm volatile ( + "r7 = %[p];" + "if %[zero] == 0 goto +1;" + "r7 = %[s];" + "r8 = *(u64 *)(r7 + 0);" + "%[v] = r8;" + : [v]"=r"(v) + : [p]"r"(p), + [s]"r"(&s), + [zero]"r"(zero) + : "r7", "r8"); + return v; +} + __attribute__((__aligned__(8))) u8 global[] = { 0x11, 0x22, 0x33, 0x44, From 2b918fe2f11f4fefed34b815004afe4ee352edf1 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Mon, 17 Aug 2026 16:10:14 +0200 Subject: [PATCH 372/373] selftests/bpf: Add tests for fault prone loads out of RCU pointers Cover the two loads which used to lose the BPF_PROBE_MEM rewrite, both reached from an RCU read-side critical section. The purpose of this patch is to assert load success in order to make sure to not trigger verifier_bug_if() on bpf_may_fault_on_deref() due to forgotten rewrite of a probed pointer. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t rcu_read_lock [...] #332/1 rcu_read_lock/success:OK #332/2 rcu_read_lock/rcuptr_acquire:OK #332/3 rcu_read_lock/negative_tests_inproper_region:OK #332/4 rcu_read_lock/negative_tests_rcuptr_misuse:OK #332 rcu_read_lock:OK Summary: 1/4 PASSED, 0 SKIPPED, 0/0 FAILED Signed-off-by: Daniel Borkmann Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260817141015.878071-2-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/rcu_read_lock.c | 2 + .../selftests/bpf/progs/rcu_read_lock.c | 76 +++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c b/tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c index 246eb259c08a..6a07b2b418d1 100644 --- a/tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +++ b/tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c @@ -34,6 +34,8 @@ static void test_success(void) bpf_program__set_autoload(skel->progs.rcu_read_lock_global_subprog, true); bpf_program__set_autoload(skel->progs.rcu_read_lock_subprog_lock, true); bpf_program__set_autoload(skel->progs.rcu_read_lock_subprog_unlock, true); + bpf_program__set_autoload(skel->progs.non_own_ref_untrusted_ld, true); + bpf_program__set_autoload(skel->progs.rcu_untrusted_union_ld, true); err = rcu_read_lock__load(skel); if (!ASSERT_OK(err, "skel_load")) goto out; diff --git a/tools/testing/selftests/bpf/progs/rcu_read_lock.c b/tools/testing/selftests/bpf/progs/rcu_read_lock.c index b4e073168fb1..31d4081c3a9f 100644 --- a/tools/testing/selftests/bpf/progs/rcu_read_lock.c +++ b/tools/testing/selftests/bpf/progs/rcu_read_lock.c @@ -549,3 +549,79 @@ int rcu_read_lock_sleepable_global_subprog_indirect(void *ctx) bpf_rcu_read_unlock(); return 0; } + +struct rcu_node_data { + long key; + struct bpf_rb_node node; +}; + +struct rcu_node_stash { + struct rcu_node_data __kptr *node; +}; + +/* + * Necessary so that LLVM emits BTF for rcu_node_data rather than just a + * fwd reference to it, same as in progs/local_kptr_stash.c. + */ +struct rcu_node_data *just_here_because_btf_bug; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct rcu_node_stash); +} node_stash SEC(".maps"); + +long non_own_ref_key; + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +int non_own_ref_untrusted_ld(void *ctx) +{ + struct rcu_node_stash *stash; + struct rcu_node_data *node; + int key = 0; + + stash = bpf_map_lookup_elem(&node_stash, &key); + if (!stash) + return 0; + bpf_rcu_read_lock(); + node = stash->node; + if (!node) { + bpf_rcu_read_unlock(); + return 0; + } + bpf_rcu_read_unlock(); + /* + * The unlock leaves node as PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED + * | NON_OWN_REF, and the load below has to get the BPF_PROBE_MEM + * rewrite for it, otherwise a bad address panics the kernel. + */ + non_own_ref_key = node->key; + return 0; +} + +long rcu_untrusted_wq_flags; + +SEC("?tp_btf/tcp_probe") +int BPF_PROG(rcu_untrusted_union_ld, struct sock *sk) +{ + struct socket_wq *wq; + + /* + * sk_wq sits in a two member union, so btf_struct_walk() marks the + * pointer PTR_UNTRUSTED, and the __rcu tag on the member adds MEM_RCU + * on top of it. struct sock is not on the __safe_rcu_or_null allow + * list, hence the two stay combined and the load below has to get the + * BPF_PROBE_MEM rewrite for PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU, + * otherwise a bad address panics the kernel. + * + * The __rcu tag only reaches BTF on a clang built kernel, that is, one + * with CONFIG_PAHOLE_HAS_BTF_TAG. On a gcc built kernel the walk yields + * a plain untrusted pointer, which is rewritten either way. + */ + wq = sk->sk_wq; + if (!wq) + return 0; + rcu_untrusted_wq_flags = wq->flags; + return 0; +} From f79066c784022fda83f5936559a1af414e41b603 Mon Sep 17 00:00:00 2001 From: Daniel Borkmann Date: Mon, 17 Aug 2026 16:10:15 +0200 Subject: [PATCH 373/373] selftests/bpf: Add tests for a store on a fault prone qdisc pointer Cover the store which used to be left as a plain BPF_STX without an exception table entry: 1: R1=trusted_ptr_Qdisc() ; struct Qdisc *next = sch->next_sched; 1: (79) r1 = *(u64 *)(r1 +216) ; R1=ptr_Qdisc() ; next->limit = 1000; 3: (63) *(u32 *)(r1 +20) = r2 ; R1=ptr_Qdisc() R2=1000 Assert that it is rejected now. # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t ns_bpf_qdisc [...] #257/1 ns_bpf_qdisc/fifo:OK #257/2 ns_bpf_qdisc/fq:OK #257/3 ns_bpf_qdisc/attach to mq:OK #257/4 ns_bpf_qdisc/attach to non root:OK #257/5 ns_bpf_qdisc/incompl_ops:OK #257/6 ns_bpf_qdisc/invalid_dynptr:OK #257/7 ns_bpf_qdisc/invalid_dynptr_cross_frame:OK #257/8 ns_bpf_qdisc/invalid_dynptr_slice:OK #257/9 ns_bpf_qdisc/untrusted_write:OK #257/10 ns_bpf_qdisc/dynptr_use_after_invalidate_clone:OK #257 ns_bpf_qdisc:OK Summary: 1/10 PASSED, 0 SKIPPED, 0/0 FAILED Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260817141015.878071-3-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi --- .../selftests/bpf/prog_tests/bpf_qdisc.c | 2 + .../progs/bpf_qdisc_fail__untrusted_write.c | 65 +++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bpf_qdisc_fail__untrusted_write.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c b/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c index 77f1c0550c9b..6dbd1487343c 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_qdisc.c @@ -11,6 +11,7 @@ #include "bpf_qdisc_fail__invalid_dynptr.skel.h" #include "bpf_qdisc_fail__invalid_dynptr_slice.skel.h" #include "bpf_qdisc_fail__invalid_dynptr_cross_frame.skel.h" +#include "bpf_qdisc_fail__untrusted_write.skel.h" #include "bpf_qdisc_dynptr_use_after_invalidate_clone.skel.h" #define LO_IFINDEX 1 @@ -230,6 +231,7 @@ void test_ns_bpf_qdisc(void) RUN_TESTS(bpf_qdisc_fail__invalid_dynptr); RUN_TESTS(bpf_qdisc_fail__invalid_dynptr_cross_frame); RUN_TESTS(bpf_qdisc_fail__invalid_dynptr_slice); + RUN_TESTS(bpf_qdisc_fail__untrusted_write); RUN_TESTS(bpf_qdisc_dynptr_use_after_invalidate_clone); } diff --git a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__untrusted_write.c b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__untrusted_write.c new file mode 100644 index 000000000000..688c2a049ae3 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__untrusted_write.c @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include "bpf_experimental.h" +#include "bpf_qdisc_common.h" +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +SEC("struct_ops") +__failure __msg("only read is supported") +int BPF_PROG(untrusted_write, struct sk_buff *skb, struct Qdisc *sch, + struct bpf_sk_buff_ptr *to_free) +{ + struct Qdisc *next = sch->next_sched; + + /* + * sch is trusted, but the walk of next_sched yields a plain + * PTR_TO_BTF_ID which may fault on a dereference. A store through + * it does not get an exception table entry, there is no probed + * store to rewrite it into, hence it has to be rejected before + * bpf_qdisc_btf_struct_access() gets to allow the write to limit. + */ + next->limit = 1000; + + bpf_qdisc_skb_drop(skb, to_free); + return NET_XMIT_DROP; +} + +SEC("struct_ops") +__auxiliary +struct sk_buff *BPF_PROG(bpf_qdisc_test_dequeue, struct Qdisc *sch) +{ + return NULL; +} + +SEC("struct_ops") +__auxiliary +int BPF_PROG(bpf_qdisc_test_init, struct Qdisc *sch, struct nlattr *opt, + struct netlink_ext_ack *extack) +{ + return 0; +} + +SEC("struct_ops") +__auxiliary +void BPF_PROG(bpf_qdisc_test_reset, struct Qdisc *sch) +{ +} + +SEC("struct_ops") +__auxiliary +void BPF_PROG(bpf_qdisc_test_destroy, struct Qdisc *sch) +{ +} + +SEC(".struct_ops") +struct Qdisc_ops test = { + .enqueue = (void *)untrusted_write, + .dequeue = (void *)bpf_qdisc_test_dequeue, + .init = (void *)bpf_qdisc_test_init, + .reset = (void *)bpf_qdisc_test_reset, + .destroy = (void *)bpf_qdisc_test_destroy, + .id = "bpf_qdisc_test", +};