mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-30 14:33:24 -04:00
5b91f44f5b
Puranjay Mohan says:
====================
bpf: Inline the numeric open-coded iterator kfuncs
The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It
expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator
up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to
tear it down. The verifier emits these as ordinary kfunc calls, so a
bpf_for() loop pays function-call overhead on setup, teardown, and -- most
importantly -- on every single iteration via bpf_iter_num_next().
All three kfuncs are tiny and only touch the 8-byte on-stack iterator state
(struct bpf_iter_num_kern { int cur; int end; }). That makes them good
candidates for inlining, the same way several other special kfuncs are
already open-coded in bpf_fixup_kfunc_call(). This series replaces each of
the three calls with an equivalent inline BPF instruction sequence:
- bpf_iter_num_new(): the end - start range check is done with 32-bit
arithmetic (start <= end is checked first, so the distance fits in a
u32) and range-checked against BPF_MAX_LOOPS as unsigned. This avoids
the cpuv4 sign-extension insns that some JITs do not implement. Returns
the same -EINVAL / -E2BIG / 0 as the kfunc.
- bpf_iter_num_next(): the hot path. cur and end are int, so the kfunc's
s->cur + 1 >= s->end test is an ordinary signed 32-bit compare and the
inlined code needs no sign extension.
- bpf_iter_num_destroy(): the stack slot is no longer tracked as iterator
state once destroy() returns, so nothing needs to be written to it.
Both the kfunc and the inlined form become a no-op, which just drops the
call.
The emitted instructions are plain BPF and remain valid for the
interpreter, so interpreter fallback stays correct and no jit_required
marking is needed.
Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):
+--------+---------------------+---------------------+---------------------+
| arch | bpf_loop | bpf_for non-inlined | bpf_for inlined |
+--------+---------------------+---------------------+---------------------+
| x86-64 | 4252 M/s (0.24 ns) | 3608 M/s (0.28 ns) | 7417 M/s (0.13 ns) |
+--------+---------------------+---------------------+---------------------+
| arm64 | 649 M/s (1.54 ns) | 548 M/s (1.82 ns) | 546 M/s (1.83 ns) |
+--------+---------------------+---------------------+---------------------+
On x86-64 removing the per-iteration call roughly doubles bpf_for()
throughput. On arm64 it is neutral, and rather than guess why this was
checked with perf: inlining removes ~28% of the executed instructions (the
call) but leaves the cycle count unchanged -- IPC drops from ~4.2 to ~3.0
and backend stalls rise from ~50% to ~66%. The loop is bound by the latency
of the iterator's on-stack counter, not by call overhead:
bpf_iter_num_next() loads s->cur from the stack, increments it and stores it
back each iteration, and the next iteration's load depends on that store.
The removed call instructions were executing in the shadow of that
store->load stall and were never on the critical path.
A small userspace microbenchmark isolates the effect: a same-address
store->load->add round-trip (the shape of the on-stack counter) costs
~6 cycles/iteration on the tested arm64 core but ~1 cycle on x86-64, where
the core collapses the same-address round-trip into a register move (memory
renaming / store-to-load-forwarding elimination). So on x86-64 the loop is
not latency-bound and the per-iteration call dominates -- removing it is the
~2x win -- whereas on arm64 the call fits entirely inside the store->load
stall the loop already has, so adding or removing it changes nothing.
bpf_loop() is shown for reference only; it is a different construct (a
callback invoked per iteration) and this series does not change it. Its
counter lives in a register rather than on the stack, so on arm64 it avoids
the store->load latency above and is faster than bpf_for() there.
Changelog:
v4: https://lore.kernel.org/all/20260729203633.213973-1-puranjay@kernel.org/
Changes in v5:
- Inline the new()/next()/destroy() sequences directly in
bpf_fixup_kfunc_call() instead of via helper functions (Andrii Nakryiko)
- Trim the code comments; keep the explanation in the bpf_iter.c kfuncs and
leave only brief comments at the inline sites, and shorten the
bpf_iter_num_destroy() kfunc to /* no-op */ (Andrii Nakryiko)
- Reword the patch 1 comment so it no longer forward-references the inlined
bpf_iter_num_next(), which is only added later in the series (bpf-ci)
- Switch the bpf_iter_num_next() comment to the networking multi-line style
v3: https://lore.kernel.org/all/20260722132424.450230-1-puranjay@kernel.org/
Changes in v4:
- Drop the "elide range checks for constant bounds" patch (Andrii Nakryiko)
- bpf_iter_num_new(): range-check the distance against BPF_MAX_LOOPS with an
unsigned compare (Andrii Nakryiko)
- bpf_iter_num_destroy(): make it a no-op in both the kfunc and the inlined
form instead of zeroing the iterator state (Andrii Nakryiko)
- New patch: fix the misleading overflow comment in bpf_iter_num_next() and
drop the redundant (s64) cast; the int wraparound is intentional and
load-bearing (Andrii Nakryiko)
- bpf_for benchmark: nr_loops is int, matching what bpf_for() expects
(Andrii Nakryiko)
- Corroborate the arm64/x86 benchmark difference with perf counters and a
store-to-load-forwarding microbenchmark (Kumar Kartikeya Dwivedi,
Andrii Nakryiko)
v2: https://lore.kernel.org/bpf/20260717120215.2171057-1-puranjay@kernel.org/
Changes in v3:
- Elide the range checks in bpf_iter_num_new() when start and end are
constant, marking the registers precise so paths reaching the call with
different constants are not pruned (Eduard Zingerman)
- Add __xlated selftests pinning the inlined new()/next()/destroy() shapes
(Eduard Zingerman)
- Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
- Pick up Acked-by on patch 3
v1: https://lore.kernel.org/all/20260715130430.318421-1-puranjay@kernel.org/
Changes in v2:
- Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32,
sparc64) decode them as a plain move and would miscompile the range check
====================
Link: https://patch.msgid.link/20260804134601.2305303-1-puranjay@kernel.org
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Linux kernel ============ The Linux kernel is the core of any Linux operating system. It manages hardware, system resources, and provides the fundamental services for all other software. Quick Start ----------- * Report a bug: See Documentation/admin-guide/reporting-issues.rst * Get the latest kernel: https://kernel.org * Build the kernel: See Documentation/admin-guide/quickly-build-trimmed-linux.rst * Join the community: https://lore.kernel.org/ Essential Documentation ----------------------- All users should be familiar with: * Building requirements: Documentation/process/changes.rst * Code of Conduct: Documentation/process/code-of-conduct.rst * License: See COPYING Documentation can be built with make htmldocs or viewed online at: https://www.kernel.org/doc/html/latest/ Who Are You? ============ Find your role below: * New Kernel Developer - Getting started with kernel development * Academic Researcher - Studying kernel internals and architecture * Security Expert - Hardening and vulnerability analysis * Backport/Maintenance Engineer - Maintaining stable kernels * System Administrator - Configuring and troubleshooting * Maintainer - Leading subsystems and reviewing patches * Hardware Vendor - Writing drivers for new hardware * Distribution Maintainer - Packaging kernels for distros * AI Coding Assistant - LLMs and AI-powered development tools For Specific Users ================== New Kernel Developer -------------------- Welcome! Start your kernel development journey here: * Getting Started: Documentation/process/development-process.rst * Your First Patch: Documentation/process/submitting-patches.rst * Coding Style: Documentation/process/coding-style.rst * Build System: Documentation/kbuild/index.rst * Development Tools: Documentation/dev-tools/index.rst * Kernel Hacking Guide: Documentation/kernel-hacking/hacking.rst * Core APIs: Documentation/core-api/index.rst Academic Researcher ------------------- Explore the kernel's architecture and internals: * Researcher Guidelines: Documentation/process/researcher-guidelines.rst * Memory Management: Documentation/mm/index.rst * Scheduler: Documentation/scheduler/index.rst * Networking Stack: Documentation/networking/index.rst * Filesystems: Documentation/filesystems/index.rst * RCU (Read-Copy Update): Documentation/RCU/index.rst * Locking Primitives: Documentation/locking/index.rst * Power Management: Documentation/power/index.rst Security Expert --------------- Security documentation and hardening guides: * Security Documentation: Documentation/security/index.rst * LSM Development: Documentation/security/lsm-development.rst * Self Protection: Documentation/security/self-protection.rst * Reporting Vulnerabilities: Documentation/process/security-bugs.rst * CVE Procedures: Documentation/process/cve.rst * Embargoed Hardware Issues: Documentation/process/embargoed-hardware-issues.rst * Security Features: Documentation/userspace-api/seccomp_filter.rst Backport/Maintenance Engineer ----------------------------- Maintain and stabilize kernel versions: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * Backporting Guide: Documentation/process/backporting.rst * Applying Patches: Documentation/process/applying-patches.rst * Subsystem Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git for Maintainers: Documentation/maintainer/configure-git.rst System Administrator -------------------- Configure, tune, and troubleshoot Linux systems: * Admin Guide: Documentation/admin-guide/index.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Sysctl Tuning: Documentation/admin-guide/sysctl/index.rst * Tracing/Debugging: Documentation/trace/index.rst * Performance Security: Documentation/admin-guide/perf-security.rst * Hardware Monitoring: Documentation/hwmon/index.rst Maintainer ---------- Lead kernel subsystems and manage contributions: * Maintainer Handbook: Documentation/maintainer/index.rst * Pull Requests: Documentation/maintainer/pull-requests.rst * Managing Patches: Documentation/maintainer/modifying-patches.rst * Rebasing and Merging: Documentation/maintainer/rebasing-and-merging.rst * Development Process: Documentation/process/maintainer-handbooks.rst * Maintainer Entry Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git Configuration: Documentation/maintainer/configure-git.rst Hardware Vendor --------------- Write drivers and support new hardware: * Driver API Guide: Documentation/driver-api/index.rst * Driver Model: Documentation/driver-api/driver-model/driver.rst * Device Drivers: Documentation/driver-api/infrastructure.rst * Bus Types: Documentation/driver-api/driver-model/bus.rst * Device Tree Bindings: Documentation/devicetree/bindings/ * Power Management: Documentation/driver-api/pm/index.rst * DMA API: Documentation/core-api/dma-api.rst Distribution Maintainer ----------------------- Package and distribute the kernel: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * ABI Documentation: Documentation/ABI/README * Kernel Configuration: Documentation/kbuild/kconfig.rst * Module Signing: Documentation/admin-guide/module-signing.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Tainted Kernels: Documentation/admin-guide/tainted-kernels.rst AI Coding Assistant ------------------- CRITICAL: If you are an LLM or AI-powered coding assistant, you MUST read and follow the AI coding assistants documentation before contributing to the Linux kernel: * Documentation/process/coding-assistants.rst This documentation contains essential requirements about licensing, attribution, and the Developer Certificate of Origin that all AI tools must comply with. Communication and Support ========================= * Mailing Lists: https://lore.kernel.org/ * IRC: #kernelnewbies on irc.oftc.net * Bugzilla: https://bugzilla.kernel.org/ * MAINTAINERS file: Lists subsystem maintainers and mailing lists * Email Clients: Documentation/process/email-clients.rst