mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-30 13:23:02 -04:00
d99bda7f017b47aff45accbb321facba9f7dd799
1465159 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Daniel Borkmann
|
d99bda7f01 |
bpf: Rewrite any fault prone load out of a mem or btf_id pointer
bpf_convert_ctx_accesses() turns a BPF_LDX into a BPF_PROBE_MEM one by matching the type recorded for the insn against a list of exact pointer types. The list cannot keep up with the flag combinations the verifier produces, and a type which is missing from it ends up as a plain load without an exception table entry, so a bad address panics the kernel instead of being handled. Two such types exist today and are reachable: - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_ALLOC | NON_OWN_REF - PTR_TO_BTF_ID | PTR_UNTRUSTED | MEM_RCU Rather than adding the two, just drop the list and state the property itself in the default case of the switch. This is a superset of what the list matched, the untrusted PTR_TO_MEM does not have to carry MEM_RDONLY for it anymore, and it stays in sync with the verifier side which uses the same match in save_aux_ptr_type() and reg_type_mismatch_ok(). Assert that a fault prone type which does not get the rewrite for whatever reason is rejected at load time rather than left to fault at runtime to catch any future cases. Fixes: |
||
|
Daniel Borkmann
|
ee9ad135b2 |
bpf: Reject a store through a fault prone pointer
check_ptr_to_btf_access() allows the program to store before the default BTF access path gets to reject a non read access. |
||
|
Daniel Borkmann
|
f438ba7a4c |
bpf: Treat a fault prone PTR_TO_MEM as a pointer type mismatch
reg_type_mismatch_ok() enumerates the pointer types which must not silently share a BPF_LDX with a different one, since the type recorded for the insn drives a rewrite in bpf_convert_ctx_accesses(). |
||
|
Daniel Borkmann
|
09c447564f |
bpf: Keep fault protection when merging pointer types
When the same BPF_LDX instruction is reached through paths that yield different pointer types, save_aux_ptr_type() merges them into a single type which is later used by bpf_convert_ctx_accesses() to decide whether the load has to be rewritten into a BPF_PROBE_MEM one. Before |
||
|
Eduard Zingerman
|
8eb1892064 |
Merge branch 'bpf-reject-mixed-arena-and-ordinary-atomic-paths'
Yiyang Chen says: ==================== bpf: Reject mixed arena and ordinary atomic paths Atomic RMW instructions use a single aux pointer type to select their final instruction encoding. The verifier currently records that type only for PTR_TO_ARENA, allowing a second path with an ordinary pointer to reach the same instruction before fixups rewrite it to BPF_PROBE_ATOMIC. Patch 1 records the destination type for every atomic RMW path so the existing pointer mismatch check rejects incompatible uses of one instruction. Patch 2 adds a verifier regression test with PTR_TO_ARENA and PTR_TO_STACK paths converging on one atomic add. ==================== Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-0-4644c1886dbc@mails.tsinghua.edu.cn Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Yiyang Chen
|
5ab9fbeca8 |
selftests/bpf: Cover mixed arena and stack atomics
Add a verifier test with one atomic RMW instruction reached through PTR_TO_ARENA and PTR_TO_STACK paths. The verifier must reject the shared instruction with the existing incompatible-pointer diagnostic. Signed-off-by: Yiyang Chen <chenyy23@mails.tsinghua.edu.cn> Link: https://patch.msgid.link/20260816-bpf-next-038-mixed-atomic-v1-v2-2-4644c1886dbc@mails.tsinghua.edu.cn Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Yiyang Chen
|
4bc49ae344 |
bpf: Check pointer type for all atomic RMW paths
Atomic RMW verification records an instruction pointer type only when the
current destination is PTR_TO_ARENA. A second path can therefore reach the
same instruction with an ordinary pointer without comparing it against the
saved arena type.
The post-verification fixup uses the saved type to rewrite the instruction
to BPF_PROBE_ATOMIC for every path. Record the actual destination type for
all atomic RMW paths so the existing mismatch check rejects incompatible
uses of one instruction.
Fixes:
|
||
|
Mahe Tardy
|
c93cbdb13f |
selftests/bpf: Add ksock test for async callback guard
Because the kfuncs are going through LSM hooks, allowing their use via workqueue callbacks would expose the wrong credentials. This test ensures the kfunc are preventing any use from these contexts. Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Stanislav Fomichev <sdf@fomichev.me> Acked-by: Song Liu <song@kernel.org> Link: https://lore.kernel.org/bpf/20260813110540.103550-6-mahe.tardy@gmail.com |
||
|
Mahe Tardy
|
7b0dfbf577 |
selftests/bpf: Test forbidden bpf_ksock_send() LSM attach
The bpf_ksock_send() kfunc eventually calls security_socket_sendmsg(), thus creating a possible recursion if a program calling the kfunc is attached on that specific hook. A filter is added on the kfunc registration to prevent that at load time from the verifier. This test exercises that the verifier will reject such program on that attach point. Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Stanislav Fomichev <sdf@fomichev.me> Acked-by: Song Liu <song@kernel.org> Link: https://lore.kernel.org/bpf/20260813110540.103550-5-mahe.tardy@gmail.com |
||
|
Mahe Tardy
|
c7838e3dc6 |
selftests/bpf: Add ksock kfunc test
Add a selftest that exercises the ksock kfuncs end-to-end. One syscall BPF setup program creates a ksock context and connects the socket. Another LSM sleepable BPF program looks up the context and send test data. The userspace harness creates a network namespace and a new socket on loopback, run the setup syscall prog and send LSM BPF prog then check that the userspace socket received the data from BPF. Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev> Acked-by: Stanislav Fomichev <sdf@fomichev.me> Link: https://lore.kernel.org/bpf/20260813110540.103550-4-mahe.tardy@gmail.com |
||
|
Mahe Tardy
|
7ae4eb14c5 |
bpf: Add ksock kfuncs
Add BPF kfuncs that allow BPF LSM programs to create and use sockets for
sending data. This provides a mechanism for BPF programs to emit
telemetry. For this first patch set, it's restricted to SOCK_DGRAM
socket types with IPPROTO_UDP protocol but could be easily extended to
SOCK_STREAM and IPPROTO_TCP in the future.
The API consists of five kfuncs:
bpf_ksock_create() - Create a socket (sleepable)
bpf_ksock_connect() - Connect socket to remote address (sleepable)
bpf_ksock_send() - Send data through the socket (sleepable)
bpf_ksock_acquire() - Acquire a reference to a socket context
bpf_ksock_release() - Release a reference (cleanup via
queue_rcu_work since sock_release sleeps)
The setup kfuncs bpf_ksock_create, bpf_ksock_connect, can be called from
SYSCALL programs only. While bpf_ksock_acquire, bpf_ksock_release and
bpf_ksock_send can be called from SYSCALL and LSM programs.
The implementation follows the established kfunc lifecycle pattern
(create/acquire/release with refcounting, kptr map storage, dtor
registration). The kernel socket is wrapped in a refcounted bpf_ksock
struct. Cleanup is deferred via queue_rcu_work() because sock_release()
may sleep.
The kfuncs are only compiled when CONFIG_INET is enabled, as they
specifically support AF_INET and AF_INET6 sockets.
The socket operations go through the expected LSM hooks instead of
by-passing them like many kernel sockets since those are created by BPF
programs and thus system users. Thus, the bpf_ksock_send() kfunc, which
is exposed to LSM progs has a verifier filter protection to avoid
recursion so that the whole bpf_kfunc_set kfunc set cannot be called in
a program attached to security_socket_sendmsg(). Also, because of the
LSM checks, we prevent the use of the kfuncs from asynchronous workqueue
as the current value would then be invalid.
In bpf_ksock_create(), we copy the arg values to avoid TOCTOU races
since the kfunc can sleep and the arg values could be stored in a map
that could be re-written by BPF progs or even userspace programs if the
map is mmaped.
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260813110540.103550-3-mahe.tardy@gmail.com
|
||
|
Mahe Tardy
|
5bd369c055 |
net: Add connect_socket() helper
Add a helper that connects an existing socket while invoking the LSM hook. Reuse it in __sys_connect_file() to avoid duplicating the connect logic. Other socket operations have equivalent helpers that trigger the appropriate LSM hooks that can be reused, this one was the only one missing. This will be used in the next commit for a new BPF kfunc that needs to connect a socket and trigger the LSM hook. Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev> Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com> Acked-by: Song Liu <song@kernel.org> Acked-by: Stanislav Fomichev <sdf@fomichev.me> Link: https://lore.kernel.org/bpf/20260813110540.103550-2-mahe.tardy@gmail.com |
||
|
Eduard Zingerman
|
ce7c9f6c59 |
Merge branch 'redesign-verification-errors'
Kumar Kartikeya Dwivedi says: ==================== Redesign Verification Errors TL;DR: This set reworks verifier error messages to include source and instruction annotations, together with more causal context, making failures easier to understand and more actionable when debugging and repairing BPF programs. Changelog: ---------- v4 -> v5 v4: https://lore.kernel.org/bpf/20260812233326.3575958-1-memxor@gmail.com * Defer Verifier Limit reports and the dependent call-chain allocation guards to follow-up work, reducing the series from 16 to 14 patches. (Eduard) * Make kfunc-name disassembly read-only before module-kfunc metadata is resolved, retain instruction context without usable source metadata, consolidate its fallback, and restrict source discovery to the containing subprogram. (Eduard, Sashiko) * Retain the newest diagnostic history in a bounded 64 MiB rotating buffer, use absolute logical positions across verifier path switches, report evicted shared history, and grow storage geometrically. (Eduard) * Complete active-path history for BPF_LD_IMM64 and atomic fetches, call clobbers and returns, outgoing stack arguments, legacy packet loads, and RCU pointer transitions. (Eduard, Sashiko) * Preserve causal lineage across equal snapshots, nullable pointer-cast branches, and repeated same-depth function invocations using unique diagnostic frame identities. Bound each rendered causal path to the oldest and newest 32 matching events with an omission summary. (Eduard) * Harden diagnostics for malformed release-kfunc signatures, fixed-size argument ranges, and dynptr, iterator, memory-size, and required-RCU failures by reporting the actual offending type or invariant. (Eduard, Sashiko) * Remove unrelated formatting and cross-patch churn, dead or single-use helpers and filter paths, and align helper placement, includes, and commit descriptions with the patches that first need them. (Eduard) v3 -> v4 v3: https://lore.kernel.org/bpf/20260713153910.2556007-1-memxor@gmail.com * Introduce helpers with their first callers and add printf annotations. (Eduard, Sashiko) * Remove "report" from diagnostic function names. (Sashiko) * Reuse bpf_linfo_source and seq_buf, simplify internal names, and use shared formatting storage. (Eduard) * Use compact common event fields and record branches at successor entry. (Eduard) * Bound event storage at 1 MiB, use kvrealloc(), and drop events non-fatally. (Eduard, Sashiko) * Restore diagnostic history only for activated queued states, preserving the active failure trace during cleanup. (Eduard, Sashiko) * Record register changes through begin/end and scrub helpers, deriving targets and origins without caller-saved snapshots. (Eduard) * Store lineage marks on events and rewind shared formatting storage after rendering each event. (Eduard) * Record iterator return values before snapshotting alternate paths. (Sashiko) * Use the current verifier instruction for global-subprogram dynptr errors. (Sashiko) * Use the supplied call name for nullable global-subprogram arguments. (Sashiko) * Describe global calls under locks as a verifier restriction rather than a sleepability failure. (Sashiko) * Keep diagnostic strings unsplit and put long call openings on their own line. (Eduard) * Keep kfunc metadata zeroed before early fetch and allowability failures. (Sashiko) * Drop the Verifier Internal Error report patch. (Eduard) * Distinguish never-initialized registers from invalidated registers. (local review) * Preserve the legacy different-lock verifier message. (local review) * Preserve nullable type qualifiers and stable mismatch formatting. (local review) * Mark truncated call chains with an ellipsis. (local review) v2 -> v3 v2: https://lore.kernel.org/bpf/20260619205934.1312876-1-memxor@gmail.com * Address various comments from Eduard and Sashiko. * Move instruction context from a separate gutter into a new section following source context, since surrounding source lines and BPF instructions do not map one-to-one. * Fix active-path branch reconstruction when switching to queued states, and expand register histories to follow value lineage across spills, fills, stack reads, helper/kfunc clobbers, and dynptr invalidation. * Misc improvements and refinements. v1 -> v2 v1: https://lore.kernel.org/bpf/20260605063412.974640-1-memxor@gmail.com * Reworked diagnostic history from per-verifier-state log to active path log with positions saved and reset when verifier search backtracks. (Eduard) * Moved reusable diagnostic formatting storage into struct bpf_diag under struct bpf_verifier_env, and removed large per-report scratch buffers from verifier stack frames. (Eduard) * Added stack-slot events so diagnostics follow ordinary stack spill/fill value flow and invalidations in register-scoped histories. (Eduard) * Reused existing source and BTF formatting helpers for diagnostics, including bpf_get_linfo_file_line() and btf_type_snprintf_show_name(). (Eduard) * Fixed diagnostic edge cases around signed offset text, BPF_MAX_VAR_OFF reporting, negative-offset clamping, poisoned stack reads, and borrowed-reference invalidations. (Eduard) * Fixed various miscellaneous diagnostic bugs. (Sashiko) * Misc improvements and refinements. --- Motivation ~~~~~~~~~~ The verifier log is the primary interface through which the verifier communicates to the user its verdict on whether a program was accepted or rejected. To aid the debugging of rejection decisions, the verifier also reports the symbolic state of the program at each instruction, across every explored path of the BPF program. Such detailed information is critical to introspect the correctness of verification decisions, and provide insight into why a given program may have failed to load in the kernel. A constant pain point in the BPF ecosystem throughout the years has been the difficulty of debugging verification errors. The human-readable error messages produced in response to a failure in satisfying safety-related constraints are often terse, context-dependent, or insufficient for understanding why a given error may have happened. Users must fall back to the verbose instruction-by-instruction breakdown of how the symbolic state evolved to surface the root cause. For programs with a huge log volume due to high verification complexity, such logs quickly become inscrutable. All of this has made life difficult for users lacking an understanding of how the verifier works, and the various heuristics and idiosyncrasies used by it. In some cases, even seasoned BPF experts spend significant time reverse engineering why a program may have failed, and have to reach into the verifier's source code to form a complete picture of the verification process. Such a steep learning curve and cognitive burden also hurts the speed of BPF development, as the verifier sits right in the middle of the user's iteration loop while they make use of BPF to solve any given problem. Expertise in debugging verifier errors does not scale in terms of teams deploying these programs in production across a diverse set of kernels. Overall, this leads to a poorer developer experience, causes visible user dissatisfaction, and remains a drag on wider BPF adoption. With some of the more recent developments where users increasingly leverage AI tooling [0] to author their code, this bottleneck becomes even more critical to address, since it throttles the much faster iteration loop of AI agents. [0]: https://lwn.net/Articles/1075067 Approach ~~~~~~~~ This series starts moving selected failures from terse terminal messages toward diagnostics that carry the relevant context for a verification failure. The existing verbose log remains the low-level trace. For selected failures, the new report is emitted after this trace and answers the immediate debugging questions: - what verifier rule failed, - why the current state does not satisfy it, - where the failing instruction maps to source, - which earlier branch or state event made this path fail, - what kind of source change would satisfy the verifier. The series adds a text-only diagnostics framework under kernel/bpf and uses it to augment selected verifier errors. Existing verbose(env, ...) messages are kept, so current selftest expectations and existing log consumers continue to see the legacy text. The new report has a uniform outer shape: Verification failed: <category>: <problem> Reason: exact reason for the verification failure, with details At: source and instruction annotation Causal path: compressed branch and verifier-state events relevant for debugging Suggestion: speculation on potential fixes to repair the program The outer shape is shared, but report construction is category-specific. The categories are intentionally broad and reviewable. This revision covers representative cases in Register Type Safety, Memory Safety, Resource Lifetime Safety, Call Type Safety, Execution Context Safety, Program Structure and Policy. It does not attempt to convert every verbose(env, ...) site for now. Additional verbose-only errors can be moved into the same framework incrementally. The following excerpts are copied from this current run on this branch: ./test_progs -j1 \ -a cpumask/test_populate_invalid_destination,\ cpumask/test_alloc_no_release,\ verifier_helper_value_access/via_variable_no_max_check_1,\ verifier_sock/invalidate_pkt_pointers_from_global_func \ -vv They show the old terminal error and the exact new diagnostic report, including the source and instruction annotations. Call Type Safety, cpumask/test_populate_invalid_destination: Legacy: R1 type=scalar expected=fp Diagnostic: Verification failed: Call Type Safety: Invalid call argument Reason: The first argument (R1) to bpf_cpumask_populate does not satisfy the verifier contract: the kfunc expects 24 bytes of memory for (struct bpf_cpumask), but it is an integer scalar and not verifier-known memory. At: test_populate_invalid_destination @ cpumask_failure.c:234:8 Source context: 232 | ... 233 | ... >>> 234 | ret = bpf_cpumask_populate(invalid, &bits, sizeof(bits)); | ^-- error: invalid first argument (R1) for bpf_cpumask_populate 235 | if (!ret) 236 | err = 2; Instruction context: 2 | (b7) r1 = |
||
|
Kumar Kartikeya Dwivedi
|
ac545b00ca |
bpf: Report Policy helper and kfunc errors
Augment selected helper and kfunc allowability failures with Policy reports. These reports explain which requested operation is forbidden and why, without adding path history for non-path-dependent policy checks. Cover unprivileged bpf2bpf and kfunc use, helper program-type restrictions, GPL-only helpers, helper-specific allow callbacks, kfunc allowability, and destructive kfunc capability checks. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-15-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
a8f4278353 |
bpf: Report Program Structure CFG errors
Augment selected whole-program and subprogram CFG validation failures with Program Structure reports. These errors are structural rather than path-dependent, so the reports focus on source and instruction context instead of causal history. Cover direct and indirect jumps outside the program or current subprogram, unprivileged backedges, missing and out-of-range jump tables, targets in the second half of an ldimm64, unreachable instructions, subprogram fallthrough, and recursive bpf2bpf call graph edges. Format long jump-range reasons directly in diagnostics.c, and keep the fallthrough suggestion aligned with the verifier check by suggesting exit or explicit jumps. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-14-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
99a6a288a8 |
bpf: Report Execution Context Safety errors
Augment selected sleepability and critical-section failures with Execution Context Safety reports. Keep the existing verifier messages and add source context, path history, and suggestions tied to the active context. Use the context history recorded earlier to anchor causal paths to lock, IRQ, RCU, and preempt regions instead of unrelated register updates. Cover global calls while holding a lock, sleepable global function calls, sleepable helpers, sleepable kfunc calls from disallowed contexts, operations that exit while a context is still active, and unmatched context exits. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-13-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
66e2727395 |
bpf: Report Call Type Safety argument errors
Augment selected helper and kfunc argument-contract failures with Call Type Safety reports. Keep the existing terse verifier messages and add reason, source context, causal register or stack-argument history, and targeted suggestions. Cover helper register-type mismatch, helper and kfunc non-NULL pointer requirements, release-helper ownership requirements, scalar and constant kfunc arguments, trusted and RCU pointer contracts, kfunc memory arguments, memory/length pairs, refcounted kptrs, constant strings, and IRQ flag stack arguments. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-12-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
5d57646275 |
bpf: Report Resource Lifetime reference leaks
Augment selected Resource Lifetime Safety failures with structured diagnostics while preserving the existing verifier messages. Report unreleased references from check_reference_leak() using reference-scoped diagnostic history, and add state reports for dynptr, iterator, lock, and IRQ-flag lifetime misuse. IRQ restore mismatch and out-of-order diagnostics use IRQ context-scoped history when an IRQ-disabled region is active, so retained save/restore context is still visible after per-state history removal. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-11-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
2bdc90f531 |
bpf: Report Memory Safety bounds errors
Augment selected memory-range verifier failures with Memory Safety reports while preserving the existing terse verifier messages for compatibility. Cover stack spill corruption, uninitialized stack reads, variable stack helper accesses, and check_mem_region_access() range-proof failures. The bounds report spells out the required offset + access_size <= object_size proof with concrete values and uses scoped diagnostic history for causal context. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-10-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
d63284e62b |
bpf: Report Register Type Safety errors
Augment selected register-state verifier failures with Register Type Safety reports. The existing verbose verifier messages remain in place; the new reports add reason, source context, causal path, and suggestions. Cover invalid pointer dereferences, unreadable registers, missing outgoing stack arguments for bpf2bpf and kfunc calls, and rejected pointer arithmetic. Use scoped diagnostic history so reports start from the latest relevant value change and then show later branch outcomes. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-9-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
956a66e5c3 |
bpf: Track verifier context diagnostic events
Record verifier context transitions in the diagnostic history so later reports can anchor causal paths to the critical section that made an operation invalid. This covers lock, IRQ, RCU, and preempt regions without adding any new verifier error reports. Category-specific commits decide where those recorded events should be rendered. Use context depth when selecting scoped history so nested regions anchor at the outer active region, and fall back to the earliest retained event when the matching entry was pruned. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-8-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
9ecd70304e |
bpf: Track verifier reference diagnostic events
Add reference acquire and release events to diagnostic history so Resource Lifetime Safety reports can show the lifetime of a specific reference id along the path. Record acquisitions after the verifier assigns the reference id. Record releases only after release_reference_nomark() succeeds, including the kptr_xchg RCU conversion path and owning-to-non-owning conversion path that consume an owning reference. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-7-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
af4ea6e20f |
bpf: Track verifier register diagnostic events
Record material register and outgoing stack argument changes so diagnostics can
explain how a value reached its current type, bounds, or unreadable state.
Store old and new register types, scalar ranges, tnum value and mask, map and
BTF type identity, and basic operand metadata in the environment-owned
diagnostic event stream.
Record invalidations when packet data moves, references are released, or
borrowed references leave their protected region. Register-scoped history
starts at the latest matching modification and then shows later branch
outcomes.
Also record fixed stack spills and overwrites, and tag register fills from
stack so register-scoped history can follow value flow through spilled stack
slots.
The type_is_map_ptr() helper previously lived as a static function in
kernel/bpf/log.c since commit
|
||
|
Kumar Kartikeya Dwivedi
|
a6debd5f25 |
bpf: Prune verifier diagnostics when switching paths
Save the diagnostic event-log position with each verifier stack entry and reset the environment-owned stream together with the normal verifier log when a queued state is popped. Also reset the diagnostic stream after successful subprogram verification even when level-2 logging preserves the normal verifier log. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-5-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
daf8248701 |
bpf: Add verifier diagnostic event log
Add an environment-owned diagnostic history for verifier reports. Event payloads keep the user-facing branch history shape, while storage lives in bpf_verifier_env and follows the active verifier path. Grow the event array geometrically up to a 64 MiB limit. Once storage reaches the limit, or an allocation fails, overwrite the oldest event so diagnostics retain the newest useful suffix without adding per-event metadata. Represent saved positions as absolute logical sequence numbers. A restore truncates to a retained position. If its prefix has already been evicted, clear the abandoned suffix and preserve the missing-history position. This keeps marks stable across rotation without increasing their size. Add the branch event renderer and branch recording. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-4-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
b9c5d822f6 |
bpf: Add source and instruction diagnostic context
Teach verifier diagnostics to annotate an instruction with BTF source line information and nearby BPF instructions. The renderer keeps source text in a fixed-width lane and prints instructions in a stable right-hand gutter. Wrap annotation text under the source line so long error labels remain readable while the source and instruction lanes keep their fixed layout. Keeping source and instruction context in one commit preserves the visual layout contract that later diagnostic reports rely on. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-3-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Kumar Kartikeya Dwivedi
|
5ad7461663 |
bpf: Add verifier diagnostics report helpers
Add the initial diagnostics renderer for verifier reports and wire it into the BPF build. The helper emits the common failure header through the verifier log. Later patches add prose wrapping, reusable report sections, and source and instruction context for category-specific diagnostics. Gate the helpers on normal verifier log output from the start, so BPF_LOG_STATS-only loads do not collect or render diagnostics. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Acked-by: Eduard Zingerman <eddyz87@gmail.com> Link: https://patch.msgid.link/20260815064612.378577-2-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Eduard Zingerman
|
d82ebfc685 |
Merge branch 'bpf-x86-fix-per-cpu-address-resolution-into-an-extended-register'
Vineet Gupta says: ==================== bpf, x86: fix per-CPU address resolution into an extended register The JIT resolves a per-CPU address with add <dst>, gs:[this_cpu_off] but builds the REX prefix with add_1mod(), which sets REX.B. The destination is encoded in ModRM.reg, which REX.R extends, and the memory operand is disp32 with no base, so REX.B does nothing and the high register bit is dropped. Every extended destination therefore resolves into whichever register shares the low three bits: R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI The address is left unadjusted and an unrelated register is clobbered. Patch 1 switches to add_2mod() so the bit goes through REX.R. Clang reloads the address into R1 before each per-CPU access, so the destination is never an extended register and the bug has been dormant since v6.10. GCC keeps several per-CPU addresses live at once, which is how it turned up: test_progs-bpf_gcc panics the kernel in global_percpu_data/init, with the address of a .percpu variable in R5. Patch 2 covers every register. A functional test only catches this if the address happens to land in an extended register, so the test matches the JITed add instead. Changes in v3: - Fold the five per-register programs into one that loads every register, and drop the comment explaining the register choice (Eduard Zingerman). - Move the percpu_data declaration inside the arch guard, so other targets no longer carry a .percpu section and an unused map (bpf-ci). - Match the movabsq of each address as well as the add, so the matchers stay on consecutive lines and the pair is checked to use the same register. - Restore the Reviewed-by on patch 1, dropped by mistake in v2. Changes in v2: - Add the selftest, patch 2/2 (Eduard Zingerman). It uses __jited() rather than __xlated(): the xlated stream is identical for every register, and the wrong prefix is only visible in the native encoding. - No functional change to patch 1. ==================== Link: https://patch.msgid.link/20260814220254.3797467-1-vineet.gupta@linux.dev Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Vineet Gupta
|
f61306e8c9 |
selftests/bpf: Check per-CPU address resolution per register
An ld_imm64 of a per-CPU map value is followed by a mov_percpu_addr that reuses the same register, so which register the address lands in decides how the JIT encodes the add. Getting the REX prefix wrong there is invisible to a functional test unless the address happens to land in an extended register, which is why this went unnoticed. Load a .percpu variable into every register in one program and match the JITed add against the register each one must resolve into. Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev> Link: https://patch.msgid.link/20260814220254.3797467-3-vineet.gupta@linux.dev Signed-off-by: Eduard Zingerman <eddyz87@gmail.com> |
||
|
Vineet Gupta
|
5bbbce02e5 |
bpf, x86: Fix per-CPU address resolution into an extended register
The destination of the per-CPU address MOV is encoded in ModRM.reg,
which is extended by REX.R, but the REX prefix is built with
add_1mod(), which sets REX.B. REX.B extends ModRM.rm and SIB.base, and
this instruction addresses memory as disp32 with no base, so the bit
has no effect at all and the high register bit is simply lost.
Every is_ereg() destination therefore resolves to the wrong register,
picking whichever one shares the low three bits:
R5 -> RAX R7 -> RBP R8 -> RSI R9 -> RDI
With BPF_REG_5, whose reg2hex is 0, the emitted
65 49 03 04 25 <off> add %gs:<off>,%rax
adds the per-CPU offset to RAX rather than R8. The destination keeps
the unadjusted address and RAX is clobbered, so the program goes on to
dereference a pointer that was never made per-CPU:
BUG: unable to handle page fault for address: 0000607e386a8894
RIP: bpf_prog_707837aafd2aa9ae_update_percpu_data+0x93/0xc9
Call Trace:
__bpf_prog_test_run_raw_tp+0x2dc/0x7d0
__flush_smp_call_function_queue+0x1e9/0xc80
Kernel panic - not syncing: Fatal exception in interrupt
R5 is the mildest of the four, aliasing a scratch register and faulting
at the store. R7 aliases RBP and would corrupt the frame pointer, R8
and R9 alias the argument registers.
Use add_2mod() so the register goes through REX.R, matching how
add_2reg() places it in ModRM.reg and how emit_priv_frame_ptr()
hardcodes 0x4c for the same instruction with R9. Encodings for the
non-extended registers are unchanged.
Problem showed up when trying to resurrect BPF_GCC CI (selftests built
with BPF_GCC).
This has gone unnoticed because clang reloads the address into R1
before each per-CPU access, so the destination is never an extended
register. GCC keeps several per-CPU addresses live at once, and
test_progs-bpf_gcc panics the kernel in global_percpu_data/init, where
the address of a .percpu variable ends up in R5.
Fixes:
|
||
|
Song Liu
|
f5b57e9e9c |
bpf: Populate mmap-able array map memory lazily
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory
vmalloc'ed up front at map creation time. array_map_mmap() then wired up
the whole mapping eagerly via remap_vmalloc_range(), which calls
vm_insert_page() for every page of the map. For large maps this makes
every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per
mmap() and tears them all down again on munmap(), even when user space
only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already
does. array_map_mmap() now only performs the bounds check and returns,
leaving the PTEs unpopulated; pages are inserted on demand by a new
array_map_mmap_fault() handler. Because the memory is already resident,
the fault handler simply resolves the vmalloc page and hands it to the
fault path. This makes mmap() O(1), and munmap() proportional to the
number of pages that were actually faulted in rather than to the size of
the map.
The handler is reached through a new optional ->map_mmap_fault callback.
Maps that provide it get a vm_operations_struct with a .fault handler;
maps that populate their mapping eagerly keep the one they had. Both
share the same open/close callbacks, so the existing VMA accounting
(VM_MAYWRITE write-active tracking, freeze handling) stays centralized
rather than each map installing its own vm_operations_struct.
Callers that want the pages populated up front can still request that
explicitly with MAP_POPULATE. Kernel-side access to the map (via the
vmalloc address) is unaffected.
Time for one mmap()+munmap() of an 8MiB mmap-able array map:
before after
no MAP_POPULATE, no access 226us 1.1us
no MAP_POPULATE, access all pages 236us 1341us
MAP_POPULATE, no access 312us 493us
MAP_POPULATE, access all pages 318us 519us
Mapping without touching the data, which is what this change targets,
gets ~160x cheaper. Faulting in the whole mapping one page at a time is
more expensive than the eager remap_vmalloc_range() loop, so users that
do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE
is not free before this change either: it adds ~85us (226us => 312us)
for no benefit, as the mapping is already fully populated.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814155623.111565-1-song@kernel.org
|
||
|
Israel Téllez García
|
fdd4fad0bb |
libbpf: Fix ring buffer consumer loop on 32-bit position wrap
ringbuf_process_ring() walks the records between the consumer and the
producer with an ordering comparison:
while (cons_pos < prod_pos) {
cons_pos and prod_pos mirror the kernel's ring positions and are
unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic.
When producer_pos has wrapped and consumer_pos has not, prod_pos is the
smaller of the two, the loop body never runs and no record is consumed.
Since consumer_pos only advances inside that loop, it never wraps either
and the consumer stops delivering samples for good, with no error
returned to the caller: ring_buffer__poll() keeps reporting zero
records while the kernel side fills up and starts dropping.
Compare the distance instead. The consumer never runs ahead of the
producer, so prod_pos - cons_pos is the amount of unconsumed data and
stays correct across the wrap.
64-bit hosts are unaffected in practice: the counters would need 16 EiB
to wrap. This is the userspace counterpart of the kernel-side walk fixed
in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit
consumer hits whichever of the two comes first.
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com
|
||
|
Israel Téllez García
|
3f611e9b82 |
bpf: Fix available-data accounting on 32-bit wrap in overwrite mode
In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer and overwrite positions before measuring how much data is available: return prod_pos - max(cons_pos, over_pos); max() is an ordering comparison, and consumer_pos, producer_pos and overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures, where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the two positions has wrapped and the other has not, max() returns the older one: the result is then a modular difference close to 2^32, so the function reports far more available data than the ring can hold. Pollers using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be woken with nothing to read. Compare distances rather than positions. prod_pos - X is the amount of data produced since X for either position, wrap or no wrap, so the newer position is simply the one with the smaller distance, which is also the value the function wants to return. 64-bit hosts are unaffected in practice: their counters would need 16 EiB to wrap. Found by review of the same class of bug fixed in "bpf: Fix pending_pos walk on 32-bit ring position wrap". Signed-off-by: Israel Téllez García <i.tellez@btesa.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com |
||
|
Israel Téllez García
|
6ff5b56a50 |
bpf: Fix pending_pos walk on 32-bit ring position wrap
The reservation path caches the position of the oldest not-yet-committed
record in rb->pending_pos and advances it past already committed records
on every reservation:
while (pend_pos < prod_pos) {
consumer_pos, producer_pos and pending_pos are unsigned long, i.e.
32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states
that these counters may wrap around there. Every other comparison in the
file is written as a difference, so modular arithmetic keeps them
correct across the wrap. This one is an ordering comparison, and it is
not wrap-safe.
Once producer_pos wraps past 2^32, prod_pos is small while pend_pos
still holds its pre-wrap value, so the loop condition is false and
pending_pos is never advanced again. Reservations keep succeeding for a
while, because bpf_ringbuf_has_space() uses differences, but
new_prod_pos - pend_pos grows as the producer advances, and once it
exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails:
the kernel believes a pending record spans the whole buffer. The ring
never recovers, bpf_ringbuf_output() drops every event from then on, and
nothing is logged.
Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a
tracepoint-based collector with a 512 KiB ring and 160-byte records.
Every one of them stopped delivering after exactly 26846821 records and
4295491360 bytes had passed through the ring, at event rates between 441
and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is
the byte count, not time or load. That figure is 2^32 plus 524064 bytes,
and the excess is one ring's worth of grace period, as expected while
new_prod_pos - pend_pos is still below rb->mask. The last reservation
that fits is the largest record boundary X with X + 160 <= 524287, and
since 2^32 mod 160 = 96 the boundaries after the wrap sit at
X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally
until the producer stopped, then read zero records for good. With this
patch applied, one of the four devices took 10 GiB through the same ring
with no stall, while the three unpatched ones kept wedging at the same
byte count.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap.
Compare the two positions as a difference instead. pending_pos never
runs ahead of producer_pos, so the unsigned difference is the real
distance between them and stays correct across the wrap.
Fixes:
|
||
|
Ihor Solodrai
|
a2b83a8c84 |
selftests/bpf: Fix selftest build after filter.h update
Upstream commit |
||
|
Leon Hwang
|
90bd0329ab |
selftests/bpf: Improve readability in iter test for percpu data
The original 'offsetof()' + offset is equal to the new 'offsetof()'. Use the new 'offsetof()' instead. Rename two variables btw: * offsetof_num -> num_off * percpu_data_sum -> sum Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260814173206.93082-6-leon.hwang@linux.dev |
||
|
Leon Hwang
|
3808171428 |
libbpf: Avoid unnecessary mmap resize for percpu data maps
Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz(). This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap() when the old and new value sizes occupy the same number of pages. Fix some typos btw: * mmapble -> mmapable * satisified -> satisfied * relocatin -> relocation * atach_btf_obj_fd -> attach_btf_obj_fd * len_secnd -> len_second * precendence -> precedence Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev |
||
|
Pu Lehui
|
f2aaa62159 |
riscv, bpf: Fix missing sign-ext for signed 1-byte and 2-byte kfunc args
On RV64, the ABI requires sign-extension for signed 1-byte and 2-byte kfunc args. However, the RV64 JIT currently does not perform sign-extension for such kfunc args. Before commit |
||
|
Mykyta Yatsenko
|
b0e872a31e |
bpf: Fix arm64 KASAN false positive after bpf_throw
arm64 passes zero as the stack pointer while walking BPF frames, so
bpf_throw() leaves stale KASAN stack poison after jumping to the
exception callback.
Use the frame pointer as the fallback stack watermark.
Fixes:
|
||
|
Andrii Nakryiko
|
c7e6175529 |
selftests/bpf: Make pyperf600 a success again
pyperf600 has been running into 8K BPF_COMPLEXITY_LIMIT_JMP_SEQ limitations for a long while now, after some internal compiler changes. Until BPF verifier is bestowed with scalar evolution logic, make that test actually work by doing what would anyone should do in such situations: by moving repeatable per-iteration work into independently verified global functions. `void *` argument is a problem for global funcs, but a static function wrapper doing necessary casts and a bit of __arg_nonnull magic dust is all it takes. Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Yonghong Song <yonghong.song@linux.dev> Tested-by: Pu Lehui <pulehui@huawei.com> # riscv Reviewed-by: Pu Lehui <pulehui@huawei.com> Link: https://lore.kernel.org/bpf/20260813232943.581283-1-andrii@kernel.org |
||
|
Andrii Nakryiko
|
073574da7a |
selftests/bpf: Use ping_command() for IPv6 pings in lwt_ip_encap
lwt_ip_encap hardcodes the ping6 binary for its IPv6 pings. iputils
merged ping6 into ping long ago and distros have started dropping the
compat symlink -- Arch's iputils 20250605 ships only arping, clockdiff,
ping and tracepath. There, every lwt_ip_encap subtest fails:
check_ping_ok:FAIL:ip netns exec ns-lwt-ip-encap-1-0101330 ping6 -c 1 \
-W1 -I veth1 fb04::1 > /dev/null unexpected error: 256 (errno 2)
#217/1 lwt_ip_encap_ipv4/egress:FAIL
The IPv4 subtests fail too, because check_ping_ok() pings both families.
SYS() runs the command through system(), so a missing binary is
indistinguishable from an unreachable peer.
network_helpers.c has had ping_command() for exactly this since commit
|
||
|
Kumar Kartikeya Dwivedi
|
409a9bda04 |
Merge branch 'bpf-arm64-__arena-kfunc-and-struct_ops-arguments'
Puranjay Mohan says:
====================
bpf, arm64: __arena kfunc and struct_ops arguments
The x86-64 JIT recently gained support for the __arena and
__arena__nullable argument suffixes on kfuncs and struct_ops stubs. This
adds the arm64 side and flips bpf_jit_supports_arena_args() on, so the
verifier stops rejecting these programs on arm64.
Patch 1 is an independent fix. save_args() reads stack-passed arguments
at FP + 32, which only holds when the trampoline is entered through the
fentry call and two frame records are pushed. A struct_ops trampoline is
entered via blr and pushes one frame fewer, so its stack arguments start
at FP + 16 and every one of them was read two slots off. No struct_ops
member passed arguments on the stack until the test added by commit
|
||
|
Puranjay Mohan
|
197d34b169 |
selftests/bpf: Test a multi-slot argument before a struct_ops arena argument
The trampoline reads the __arena flag from the btf_func_model per argument but stores the ctx one register slot at a time, so the two only line up if every preceding argument occupies exactly one slot. Every arena-bearing member of bpf_testmod_ops3 takes single-slot arguments, so nothing exercises the mapping and a mis-indexed arg_flags lookup would go unnoticed on any architecture. Add test_arena_multislot(), whose first argument is a 16-byte struct passed by value. It fills ctx[0] and ctx[1], putting the arena pointer at argument index one but slot two. The callback checks both halves of the struct before dereferencing ctx[2], so a JIT that walks registers instead of arguments converts the wrong slot and fails the test. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Acked-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-8-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Puranjay Mohan
|
05a3575f22 |
selftests/bpf: Enable __arena argument tests on arm64
The arena kfunc and struct_ops argument tests were restricted to x86-64 because it was the only JIT that implemented the conversions. arm64 does now, so let them run there too: tag every program in arena_kfunc.c with __arch_arm64 in addition to __arch_x86_64, and widen the __x86_64__ guards in the struct_ops arena test. Without this the tests report SKIP on arm64 rather than exercising the newly added JIT support. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Acked-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-7-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Puranjay Mohan
|
1c5bc60f95 |
selftests/bpf: Add arm64 JIT-sequence tests for __arena kfunc arguments
Pin the arm64 counterparts of the x86-64 rebase sequences: the single extended-register add for an unconditional argument, the nullable truncate-test-and-skip variant, and all five argument registers in one call. The nullable cases use a local label so the branch is pinned to the instruction right after the add, and the label line does not spell out the call because arm64 emits either a direct bl or a materialize- and-blr pair depending on the distance to the kfunc. Note that on arm64 an unconditional argument is one instruction with nothing to anchor it against, so arena_arg_jit_rebase alone cannot tell the two forms apart; it only requires that nothing is emitted between the rebase and the call. The args5 test is what pins the distinction, since its four consecutive adds leave no room for a nullable truncate-and-branch pair between them. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Acked-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-6-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Puranjay Mohan
|
bb5bad6a78 |
bpf, arm64: Convert struct_ops arena arguments in the trampoline
Implement the struct_ops arena argument conversion on arm64. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through x10 with the low half of the base materialized once into x11: sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */ str x10, [sp, #slot] A nullable argument tests the full 64-bit kernel pointer first: mov x10, xsrc cbz x10, 1f sub w10, w10, w11 1: str x10, [sp, #slot] The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through x10, so only the subtraction (and the NULL test) is inserted there. The register loop now walks arguments rather than registers so that the per-argument flags line up with the slots a multi-slot argument occupies; the sequence of stores is otherwise unchanged. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline, so a tracing trampoline emits exactly what it did before and never touches x11. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. Conversion must never reach the original function, which takes kernel addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin save_args() and assert the flag combination the same way x86 does, rather than leaving the invariant to a comment. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only qualifier from the kfunc documentation. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Reviewed-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-5-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Puranjay Mohan
|
760cb40cfd |
bpf, arm64: JIT __arena kfunc argument rebasing
Implement arena argument rebasing for kfunc calls on arm64. x28 already holds kern_vm_start whenever the prog has an arena, and the newly added extended-register add zero-extends the 32-bit arena offset in place, so an unconditional argument costs a single instruction emitted right before the call: add xN, x28, wN, uxtw A nullable argument first truncates into wN so that a zero offset leaves xN holding a real NULL, then tests it and jumps over the add: mov wN, wN cbz wN, 1f add xN, x28, wN, uxtw 1: The rebase is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees it and needs no special handling. The emitted count depends only on the kfunc model, so it is identical across JIT passes. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Reviewed-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-4-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Tejun Heo
|
f4adb983ef |
arm64: insn: Add encoder for ADD/SUB (extended register)
The insn library encodes the immediate and shifted-register forms of ADD/SUB but not the extended-register form. The BPF JIT wants it to rebase a 32-bit arena offset onto the arena kernel base in a single instruction, add xN, xBASE, wN, uxtw, instead of a separate zero-extend followed by a plain add. Add aarch64_insn_gen_add_sub_extended_reg(), modeled on the shifted-register generator. The option and imm3 fields occupy the same bits as the shifted form's shift amount, so they are encoded through the existing IMM_6 field type. The opt field in bits 23:22 is part of the opcode here rather than a shift type, and any value other than 00 is unallocated, so the decode masks cover it. Note that register 31 does not mean the same thing in the two forms: in the extended-register encoding it is SP for Rn, and for Rd unless the instruction sets the flags, while it stays XZR for Rm. Callers porting a shifted-register site that passes A64_ZR need to be aware of that, so say so above the function. Signed-off-by: Tejun Heo <tj@kernel.org> Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Reviewed-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/20260813190356.335181-3-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Puranjay Mohan
|
50de1c47a4 |
bpf, arm64: Fix stack-passed arguments for indirect trampolines
save_args() reads stack-passed arguments relative to FP assuming the trampoline is entered through the fentry call from a traced function, in which case both the parent frame (FP/x9) and the traced function frame (FP/LR) are saved before FP is set, so the arguments start at FP + 32. An indirect trampoline for a struct_ops callback is entered through a function pointer (blr), so only the FP/LR frame is pushed and the arguments start at FP + 16, not FP + 32. Every stack-passed argument of a struct_ops callback with more than eight argument slots is read two slots off. This went unnoticed because no struct_ops member passed arguments on the stack until bpf_testmod_ops3::test_arena_stack, added by commit |
||
|
Andrii Nakryiko
|
4d9551b39a |
Merge branch 'selftests-bpf-fix-for-veristat-file-prog-filters-processing'
Eduard Zingerman says: ==================== selftests/bpf: fix for veristat file/prog filters processing At the moment veristat filtering behaves unexpectedly for the following filter expression: -f !file/prog The expression rejects all programs with name 'prog', and all programs in a file with name 'file'. Fix the expression to exclude only a program 'prog' from a file 'file', also add a set of tests to exercise filtering logic. Changelog: v1 -> v2: - added fixes tag for patch #1 (bot+bpf-ci); - extended test cases for '!*foo*' and '*foo*' filters in patch #2 (bot+bpf-ci); - added patch #3, replacing direct read() calls with calls to read_output(), guaranteeing input buffer null termination (bot+bpf-ci). v1: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v1-0-b5b43c431550@gmail.com/ --- ==================== Link: https://patch.msgid.link/20260811-veristat-filter-fix-v2-0-6c234c4cd6ef@gmail.com Signed-off-by: Andrii Nakryiko <andrii@kernel.org> |