mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-31 08:15:07 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'bpf-next-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Browse Source 
Pull bpf updates from Daniel Borkmann:
 "Major changes:

   - Redesign the verifier error reporting: failures now carry source
     and instruction annotations along with the causal event history
     that led to them, making program rejections far easier to debug and
     repair (Kumar Kartikeya Dwivedi)

   - Add arena argument support to kfuncs and struct_ops through the new
     __arena and __arena__nullable suffixes (Tejun Heo, Puranjay Mohan,
     Kumar Kartikeya Dwivedi, Ihor Solodrai)

   - Signed BPF program loader rework to accommodate both BPF and
     security community needs where the kernel runs the signature
     verification at BPF_PROG_LOAD time before the LSM admission hook
     (Daniel Borkmann)

   - Add a set of ksock kfuncs which let BPF LSM and syscall programs
     create, connect and send on UDP sockets in order to emit telemetry
     data (Mahe Tardy)

   - Unify helper and kfunc call argument verification and classify
     kfunc arguments purely from BTF into a generated bpf_func_proto
     which is computed once at add-call time (Amery Hung)

  Other features and fixes:

   - Enable EXECMEM_ROX_CACHE for BPF allocations on x86 (Mike Rapoport)

   - Add bidirectional VLAN support to bpf_fib_lookup() through the new
     BPF_FIB_LOOKUP_VLAN and BPF_FIB_LOOKUP_VLAN_INPUT flags (Avinash
     Duduskar)

   - Infer zext_dst from static register liveness analysis to fix 32-bit
     zero-extension semantics, and remove the artificial limitations on
     pointer types eligible for spilling (Eduard Zingerman)

   - Inline the numeric open-coded iterator kfuncs so that bpf_for()
     loops no longer pay a kfunc call on every iteration (Puranjay
     Mohan)

   - Add an arena-based bitmap data structure to libarena along with
     serial and parallel selftests (Emil Tsalapatis)

   - Teach resolve_btfids to discover kfuncs from the kernel's BTF ID
     sets and to emit kfunc BTF decl tags, reducing the kernel build's
     dependency on pahole features (Ihor Solodrai)

   - Add BPF_F_ADJ_ROOM_DECAP_* flags to bpf_skb_adjust_room() so that
     tunnel decapsulation can update the GSO and encapsulation state of
     the skb (Nick Hudson)

   - Fix the ring buffer pending_pos walk and the available-data
     accounting on 32-bit position wrap (Israel Téllez García)

   - Add memory usage accounting for arena maps and fix an mmap_lock
     deadlock on arena lock failure (Jiayuan Chen)

   - Add tracing_multi link info support to the kernel UAPI and bpftool,
     and refactor the stack map code to run with preemption disabled
     (Jiri Olsa)

   - Support BPF_F_EGRESS in bpf_redirect_peer() to emit the skb in the
     egress direction of the target's peer device (Jordan Rife)

   - Add a KF_SPINLOCK_SAFE kfunc flag so that providers, in particular
     modules, can declare kfuncs safe to call under bpf_spin_lock
     instead of relying on the verifier's hard-coded allowlist (Kaitao
     Cheng)

   - Introduce global percpu data for BPF programs with libbpf probing
     and bpftool skeleton support, and stop exposing uninitialized
     kernel heap memory when copying per-CPU map values (Leon Hwang)

   - Add s390 JIT support for load-acquire and store-release
     instructions (Maxim Khmelevskii)

   - Fix a CFI mismatch in the task work callback and an arm64 KASAN
     false positive after bpf_throw() (Mykyta Yatsenko)

   - Reject writes through untrusted BTF pointers and bound the
     rdonly/rdwr_buf_size kfunc arguments (Nicholas Dudar)

   - Invalidate RCU pointers only after the final spin unlock and
     account for preempt and IRQ disabled regions as overlapping RCU
     protection (Ning Ding)

   - Support mixing bpf2bpf calls and tail calls on RV64, add signed
     operations and 32-bit atomics to the RV32 JIT, and add timed
     may_goto support (Pu Lehui, Kuan-Wei Chiu, Feng Jiang)

   - Fix a use-after-free on mm_struct in bpf_find_vma() for foreign
     tasks and an mmap_lock leak in the irq_work path (Sanghyun Park)

   - Populate mmap-able BPF array map memory lazily which makes mmap()
     O(1) instead of proportional to the map size (Song Liu)

   - Introduce a jit_required flag and reject programs with inlined
     helpers when no JIT is available, where the interpreter would
     otherwise jump into an invalid address (Tiezhu Yang)

   - Fix the x86 JIT per-CPU address resolution into an extended
     register where the REX prefix dropped the high destination register
     bit (Vineet Gupta)

   - Reject MEM_ALLOC BTF accesses past object bounds, arena frees below
     the arena base, and mixed arena and ordinary atomic paths (Yiyang
     Chen)

   - Fix the trampoline handling of 128-bit arguments and of return
     values larger than 8 bytes (Yonghong Song)

   - Ensure that any fault prone load is rewritten with exception table
     handling, and fix the arena load-acquire and atomic fetch handling
     in the x86, arm64, riscv and s390 JITs (Daniel Borkmann)

   - Many more fixes and cleanups across the verifier, arena,
     trampolines, sockmap, cgroup, ring buffer, x86/arm64/riscv/s390
     JITs, libbpf, bpftool, resolve_btfids and selftests"

* tag 'bpf-next-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (373 commits)
  selftests/bpf: Add tests for a store on a fault prone qdisc pointer
  selftests/bpf: Add tests for fault prone loads out of RCU pointers
  selftests/bpf: Add tests for pointer type merge at a shared load
  selftests/bpf: Remove duplicate copies of the arena spinlock qnodes
  selftests/bpf: Retry stat generation in cgroup_iter_memcg
  selftests/bpf: Test pseudo-function policy diagnostics
  bpf: Distinguish function references in policy diagnostics
  bpf: Preserve source attribution without source text
  selftests/bpf: Test kfunc argument diagnostics
  bpf: Correct kfunc argument diagnostics
  bpf: Use canonical stack argument names in diagnostics
  bpf: Preserve R0 lineage across helper calls
  selftests/bpf: Exercise negative optlen in cgroup getsockopt hook
  bpf: Reject negative optlen in cgroup getsockopt hook
  selftests/bpf: tc_tunnel - validate decap GSO and encapsulation state
  bpf: Clear decap state on skb_adjust_room shrink path
  bpf: Allow new DECAP flags and add guard rails
  bpf: Add BPF_F_ADJ_ROOM_DECAP_* flags for tunnel decapsulation
  bpf: Refactor masks for ADJ_ROOM flags and encap validation
  bpf: Name the enum for BPF_FUNC_skb_adjust_room flags
  ...
This commit is contained in:
Linus Torvalds
2026-08-20 07:36:20 -07:00
parent a4ff2be345 f79066c784
commit 5a8cd539ac
272 changed files with 19015 additions and 3464 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
Documentation/bpf/bpf_devel_QA.rst
View File

@@ -479,7 +479,10 @@ for details.
To maximize the number of tests passing, the .config of the kernel
under test should match the config file fragment in
tools/testing/selftests/bpf as closely as possible.
tools/testing/selftests/bpf as closely as possible. If not possible,
however, you can set ``BPF_STRICT_BUILD=0`` when invoking ``make``
to tolerate individual compilation failures and continue building
the remaining tests rather than treating each failure as fatal.
Finally to ensure support for latest BPF Type Format features -
discussed in Documentation/bpf/btf.rst - pahole version 1.16

1
Documentation/bpf/index.rst
View File

@@ -28,6 +28,7 @@ that goes into great technical depth about the BPF Architecture.
classic_vs_extended.rst
bpf_iterators
bpf_licensing
signing
test_debug
clang-notes
linux-notes

99
Documentation/bpf/kfuncs.rst
View File

@@ -250,6 +250,71 @@ Or::
...
}
2.3.7 __const_map and __map Annotations
---------------------------------------
These annotations are used for ``struct bpf_map *`` arguments and distinguish a
verifier-known map from an opaque one.
``__const_map`` indicates a map must be known at the verification time, i.e. a
concrete map fd the BPF program references directly.
An example is given below::
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map,
unsigned int flags)
{
...
}
``__map`` indicates an opaque ``struct bpf_map *`` that may be resolved
at run time. The argument may take either a map fd or a ``PTR_TO_BTF_ID``
``struct bpf_map`` pointer.
An example is given below::
__bpf_kfunc void *bpf_arena_alloc_pages(void *p__map, ...)
{
...
}
2.3.8 __arena and __arena__nullable Annotations
-----------------------------------------------
Both annotations indicate that the pointer argument points into the
calling program's arena. The JIT rebases the value at the call site so
the kfunc receives a directly dereferenceable kernel address, subject to
the access rules described in :ref:`BPF_kfunc_arena_access` (at most
``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access).
With ``__arena`` the rebase is unconditional and the argument is never
NULL: a value whose lower 32 bits are zero arrives as the arena base
address (arena offset 0). The kfunc must not check the argument for NULL.
With ``__arena__nullable`` such a value arrives as NULL instead and the
kfunc must check before dereferencing.
An example is given below::
__bpf_kfunc int bpf_process_item(struct item *item__arena)
{
...
}
Calling such a kfunc requires the program to use an arena map and a JIT with
arena argument support (currently x86-64 and arm64); verification fails
otherwise. The program can pass any value without compromising the kernel. A
value that does not point into the arena is a program bug.
The suffixes have the same meaning on the arguments of struct_ops stub
functions, with the conversion running in the opposite direction. The
kernel caller passes the kernel arena address and the trampoline converts
it while saving the arguments, so the callback receives an arena pointer
it can dereference directly. With ``__arena`` the kernel caller must not
pass NULL. With ``__arena__nullable`` a NULL kernel pointer arrives as NULL.
However, there is no obligation to prove to the verifier that such a pointer is
non-NULL before use, in-line with existing semantics of arena pointers used in
a program (or obtained from any other source).
.. _BPF_kfunc_nodef:
2.4 Using an existing kernel function
@@ -273,22 +338,29 @@ flags on a set of kfuncs as follows::
BTF_KFUNCS_END(bpf_task_set)
This set encodes the BTF ID of each kfunc listed above, and encodes the flags
along with it. Ofcourse, it is also allowed to specify no flags.
along with it. It is also allowed to specify no flags.
kfunc definitions should also always be annotated with the ``__bpf_kfunc``
macro. This prevents issues such as the compiler inlining the kfunc if it's a
static kernel function, or the function being elided in an LTO build as it's
not used in the rest of the kernel. Developers should not manually add
annotations to their kfunc to prevent these issues. If an annotation is
required to prevent such an issue with your kfunc, it is a bug and should be
added to the definition of the macro so that other kfuncs are similarly
protected. An example is given below::
macro. This prevents issues such as the compiler inlining the kfunc, or the
function being elided in an LTO build as it's not used in the rest of the
kernel. Developers should not manually add annotations to their kfunc to prevent
these issues. If an annotation is required to prevent such an issue with your
kfunc, it is a bug and should be added to the definition of the macro so that
other kfuncs are similarly protected. An example is given below::
__bpf_kfunc struct task_struct *bpf_get_task_pid(s32 pid)
{
...
}
Note that kfuncs must not be declared ``static``. A kfunc can be called from a
BPF program ``*.c`` file outside the compilation unit that defines it, so its
externally visible name must remain available for BTF ID lookup. ``static``
linkage allows the compiler to rename the function, which can break this
BTF-based kfunc resolution. Further note that sparse may warn that an otherwise
unreferenced kfunc should be static. Such warnings should be ignored for kfunc
definitions.
2.5.1 KF_ACQUIRE flag
---------------------
@@ -404,7 +476,7 @@ Example declaration:
.. code-block:: c
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
void *map__const_map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux) { ... }
Example usage in BPF program:
@@ -437,6 +509,13 @@ type. An example is shown below::
}
late_initcall(init_subsystem);
At kernel build time the ``resolve_btfids`` tool finds all kfuncs declared with
``BTF_KFUNCS_START()`` and emits their BTF annotations into the kernel's BTF.
For each kfunc it emits a ``bpf_kfunc`` BTF decl tag, a ``bpf_fastcall`` decl
tag when the kfunc is flagged ``KF_FASTCALL``, and the ``address_space(1)`` type
attribute on the return value and/or arguments that use arena pointers (see
sections 2.3.8 and 2.8).
2.7 Specifying no-cast aliases with ___init
--------------------------------------------
@@ -480,6 +559,8 @@ In order to accommodate such requirements, the verifier will enforce strict
PTR_TO_BTF_ID type matching if two types have the exact same name, with one
being suffixed with ``___init``.
.. _BPF_kfunc_arena_access:
2.8 Accessing arena memory through kfunc arguments
--------------------------------------------------

497
Documentation/bpf/signing.rst Normal file
View File

@@ -0,0 +1,497 @@
.. SPDX-License-Identifier: GPL-2.0
============
BPF signing
============
This document describes how BPF programs are cryptographically signed, how the
kernel verifies them at load time, and how Linux Security Modules (LSMs) -
including the BPF LSM - use the resulting verdict to enforce policy. It is
written for developers who want to produce signed BPF objects, understand what
the signature actually guarantees, or build a policy on top of it.
Motivation
==========
A signed BPF program lets the kernel establish that the bytecode being loaded
originates from a trusted producer and was not modified in transit. On its own
the kernel does not *require* signatures - an unsigned program loads exactly as
before - but it records a verdict (see `The verdict`_) that an LSM can gate on.
This is the building block for policies such as "only run BPF that was signed by
a key in the trusted keyring", as could in the future be enforced by an LSM
such as IPE.
Signing is orthogonal to the existing permission model: it does not replace the
capability checks or the verifier. A signed load still requires the usual
privileges (``CAP_BPF`` and any program-type-specific capability, subject to
``kernel.unprivileged_bpf_disabled``), and the loader's instructions are still
checked by the verifier like any other program. A valid signature establishes
*origin and integrity*, not safety - it lets a policy trust where the bytecode
came from, it does not let a load skip any check it would otherwise face.
The hard part is *what* gets signed. A naive scheme would sign a program's
instruction buffer at build time and verify that signature at
``BPF_PROG_LOAD``. That does not survive contact with real BPF objects, because
the bytes the kernel finally loads are not the bytes the developer built and
signed. Between the two, libbpf and the kernel rewrite the program:
- **map file descriptors** are patched into ``ld_imm64`` instructions
(``BPF_PSEUDO_MAP_FD``), and a map's fd is assigned at load time, so it
differs on every run;
- **CO-RE relocations** rewrite field offsets, sizes and existence flags against
the *running* kernel's BTF, so the result differs from one kernel to the next;
- **kfunc and ksym references** are resolved to ids/addresses in the running
kernel;
- **global data** (``.rodata``/``.data``/``.bss``) is created and seeded as maps
at load.
So a signature over the original instructions cannot match the relocated
instructions the verifier ends up checking, and the relocated form cannot be
produced ahead of time because it depends on the target kernel. There is no
fixed byte string that is both signable at build time and what the kernel
actually loads - which is why a program cannot simply be signed and loaded
directly.
The trusted loader
==================
The solution is to move that setup work *into* a small BPF program - the
**loader** - and sign the loader instead of the individual programs. libbpf's
``gen_loader`` machinery (``bpftool gen skeleton -L``, the "light skeleton")
emits a ``BPF_PROG_TYPE_SYSCALL`` program whose body performs the bpf() syscalls
that create maps, apply relocations, and load the real programs. The payload it
installs - the serialized programs, map descriptions, relocation data and
initial values - lives in a separate array map, the **metadata map**
(``__loader.map``).
So the unit of trust is the loader, and the signing contract is::
Sig(I_loader || D_meta)
where ``I_loader`` is the loader's instruction stream and ``D_meta`` is the
content of the metadata map. Verifying the loader's signature establishes that
both the loader *and* the payload it is about to install are authentic. The
loader is reproducible: ``gen_loader`` builds it from primitives so the same
object yields the same bytes on any build host.
Why the loader is signable when the program is not
--------------------------------------------------
The loader sidesteps every rewrite listed above, because the bytes that are
signed are *relocation-invariant*:
- The loader's own instructions are a fixed sequence of bpf() syscalls emitted
by ``gen_loader``; they carry no CO-RE relocations and resolve no ksyms, so
they are identical on every kernel. The metadata map is referenced by *index*
into ``fd_array`` (``BPF_PSEUDO_MAP_IDX_VALUE``), not by a baked-in file
descriptor, so even that reference does not change between build and load.
The loader instruction bytes the kernel verifies are exactly the bytes that
were signed.
- The metadata map is opaque, frozen data - the serialized target programs,
their relocation records, map descriptions and initial values. Its bytes are
identical at build time and at load time, so they are simply appended to the
instructions and covered by the same signature (there is no separate metadata
hash to compute or compare).
All the host-specific rewriting - creating maps, patching their fds into the
target programs, applying CO-RE, resolving ksyms, seeding global data - still
happens, but it happens *inside the loader at runtime*, on the verified
metadata, **after** the kernel has verified the ``insns || metadata`` signature.
The kernel never has to verify the relocated target programs: it verifies the
loader and its inputs once, and trust transfers to whatever that now-trusted,
deterministic loader installs. The relocation step is moved from "before the
signature can be checked" to "after a trusted program runs" - which is exactly
what makes it signable.
Because the metadata map is the loader's only untrusted input, two existing map
properties are reused to keep it trustworthy across the load:
Exclusive maps
A map created with ``excl_prog_hash`` (see ``BPF_MAP_CREATE``) may only be
accessed by a program whose digest matches that hash. The verifier enforces
``map->excl_prog_sha == prog->digest`` for every map a program uses, so the
metadata map is bound to exactly the signed loader and cannot be shared with
or mutated by another program.
Frozen maps
The metadata map is frozen (``BPF_MAP_FREEZE``) before the loader is loaded.
Freezing blocks further userspace writes, so the bytes folded into the
signature cannot change before the loader runs. (Freezing does not make the
map read-only to the loader program itself, which still writes created file
descriptors back into the blob's scratch area.)
Load-time verification
=======================
Rather than have the loader check its own metadata from within BPF, the kernel
verifies it directly at ``BPF_PROG_LOAD``, with no new UAPI. The mechanism
reuses the existing ``fd_array``:
#. Userspace creates the metadata map with ``excl_prog_hash`` set to the
loader's digest, populates it, and freezes it.
#. The loader is loaded with ``signature``/``signature_size``/``keyring_id``
set, the metadata map referenced through ``fd_array``, and ``fd_array_cnt``
set so the kernel knows the array's length.
#. Signature verification runs inside the verifier (``bpf_check()``), once it
has resolved the ``fd_array`` entries into the program's ``used_maps``. The
maps folded into the signature are therefore the very objects the program
binds - a single resolution of ``fd_array``, not a separate read, so the
verified bytes cannot be swapped for a different map after the check (no
time-of-check/time-of-use window). Each folded map must be exclusive (carry
``excl_prog_sha``) and a plain array map (``BPF_MAP_TYPE_ARRAY``); only an
array map exposes its value buffer through ``map_direct_value_addr()`` as a
kernel address spanning ``value_size`` bytes. A map that is not exclusive, not
frozen, or not a plain array is rejected, with a verifier log message naming
the offending map. The kernel appends each map's frozen
contents to the instruction buffer and verifies the PKCS#7 signature over the
concatenation ``insns || metadata_0 || metadata_1 || ...`` in ``used_maps``
order, before it rewrites the (signed) instructions.
A signed program therefore takes one of exactly two shapes, both fully
supported:
- **No bound maps** (``fd_array_cnt == 0``): there is nothing to append, so the
kernel verifies the signature over the instructions alone. A valid signature
yields ``BPF_SIG_VERIFIED`` and the program loads. This is the ordinary case
for a directly-loaded signed program with no separate payload; it is *not*
rejected for "missing" metadata, because it has none to cover.
- **Exclusive bound maps** (``fd_array_cnt > 0``): every entry is exclusive and
folded, so the signature covers ``insns || metadata``.
There is no third shape: a non-exclusive map in a signed program's ``fd_array``
is rejected rather than silently left out of the signature, so a signed loader
never binds a map its signature does not cover.
The digest binding (``excl_prog_sha == prog->digest``) is enforced by the
verifier as usual; because that check runs while ``fd_array`` is resolved -
before the verifier would otherwise compute the tag - ``prog->digest`` is
computed up front in the verifier, over the unmodified (signature-covered)
instructions, for any signed load.
Coverage is then enforced as the verifier resolves instructions, at the point
each object is bound rather than by a count taken afterwards. Once the signature
has been verified, binding any further map is refused: a map reached by a
directly-referenced fd, or a map swapped into an ``fd_array`` slot the loader
reads, is not among those already folded, so it is rejected the moment the
verifier tries to bind it. A BTF is refused outright for a signed program - a
ksym or a BTF fd in ``fd_array``, whether resolved up front or lazily for a
module kfunc, is rejected when it would be bound. Together with the fold rule
above this keeps the verdict binary: a signed program cannot use a map its
signature does not cover, and a different but equally digest-bound map cannot be
substituted at an ``fd_array`` slot. Non-exclusive maps are never folded, so a
signed program cannot use one at all.
The verdict
===========
A program is either unsigned or fully verified - there is no intermediate
state. The outcome is recorded in ``prog->aux->sig.verdict``:
.. code-block:: c
enum bpf_sig_verdict {
BPF_SIG_UNSIGNED = 0,
BPF_SIG_VERIFIED,
};
``BPF_SIG_VERIFIED`` means the signature is valid and covers the instructions
*and* the frozen contents of every exclusive map the program uses:
- For an ordinary, directly-loaded signed program the instructions are the whole
artifact and it uses no exclusive maps, so a valid instruction signature is
the complete verification.
- For a signed loader the metadata map is exclusive, so its contents are folded
in and the signature covers ``insns || metadata``.
There is deliberately no "instructions verified but metadata not" verdict: a
signed loader that fails to cover its metadata is *rejected* (see above), not
recorded with a weaker verdict. ``BPF_SIG_VERIFIED`` therefore always means the
program and everything the signature is responsible for are authentic, which is
what a policy can rely on.
Alongside the verdict the kernel records which keyring validated the signature;
see `Keyrings`_.
Enforcement via LSMs
====================
Signing only *records* a verdict; an LSM turns it into policy. The verdict and
keyring fields live in ``struct bpf_prog_aux``, so a BPF LSM program can read
them directly (see Documentation/bpf/prog_lsm.rst for writing and attaching BPF
LSM programs); the same fields are equally available to in-tree LSMs. Two hooks
are useful at different points of the load: the dedicated
``security_bpf_prog_load()`` gates admission before the main verification work,
and the existing ``security_bpf_prog()`` observes a program that has fully
loaded.
Admission: ``security_bpf_prog_load()``
---------------------------------------
This hook gates admission **for every load**, from a single call site inside the
verifier (``bpf_check()``), before the main verification work. It runs after the
optional signature verification, so the verdict and keyring fields are final - the
hook can see whether, and how strongly, the program was signed, which keyring
validated it, the load ``attr``, the BPF token and whether the load came from the
kernel. For a signed load the verdict is ``BPF_SIG_VERIFIED`` here (the signature
has just been checked); for an unsigned load it is ``BPF_SIG_UNSIGNED``.
This is the place for *coarse admission* that must also see unsigned and
not-yet-verified loads: require a signature at all, restrict the acceptable
keyring, restrict which token/credentials may load BPF, apply per-program-type
rules, or audit every load attempt that makes it past signature verification -
attempts failing the signature or the metadata binding abort before this hook
fires. It is the primary deny point.
One subtlety: this hook runs *before* the verifier finishes its work, so
``BPF_SIG_VERIFIED`` *here* means only "validly signed" - not "loaded". Allowing
a load at this point lets it *proceed*; it does not guarantee the program will
load. A validly signed program can still be rejected afterwards on two
independent grounds: the verifier may reject it like any other program (unsafe
memory access, bad control flow, resource limits, ...), and the kernel separately
refuses - as the verifier resolves instructions and binds each object - any map
the signature does not cover or any BTF at all, regardless of what this hook
returned. Only after the program has fully loaded, at the next hook
(``security_bpf_prog()``), does ``BPF_SIG_VERIFIED`` carry its full meaning:
validly signed *and* fully verified.
A more realistic admission policy than "is it signed at all": accept programs
signed by a system keyring, accept a user-keyring signature only if the
key/keyring it was verified against is on an explicit allowlist, and emit a
tamper-evident record of every decision so that even denied attempts are
auditable. (Illustrative - error checking elided.)
.. code-block:: c
/* Serials of user keys/keyrings we additionally trust. */
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, __s32); /* keyring_serial */
__type(value, __u8);
__uint(max_entries, 64);
} trusted_user_keys SEC(".maps");
/* Audit stream consumed by a userspace logger. */
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 1 << 16);
} audit SEC(".maps");
struct decision { __u32 prog_type, verdict, ktype; __s32 serial, ret; };
SEC("lsm/bpf_prog_load")
int BPF_PROG(admit, struct bpf_prog *prog, union bpf_attr *attr,
struct bpf_token *token, bool kernel)
{
__u32 verdict = prog->aux->sig.verdict;
__u32 ktype = prog->aux->sig.keyring_type;
__s32 serial = prog->aux->sig.keyring_serial;
struct decision *d;
int ret = 0;
if (kernel)
return 0; /* trust in-kernel loads */
if (verdict != BPF_SIG_VERIFIED)
ret = -EPERM; /* must be validly signed */
else if (ktype == BPF_SIG_KEYRING_USER &&
!bpf_map_lookup_elem(&trusted_user_keys, &serial))
ret = -EPERM; /* key/keyring not allowlisted */
d = bpf_ringbuf_reserve(&audit, sizeof(*d), 0);
if (d) {
d->prog_type = attr->prog_type;
d->verdict = verdict;
d->ktype = ktype;
d->serial = serial;
d->ret = ret;
bpf_ringbuf_submit(d, 0); /* record allow *and* deny */
}
return ret;
}
Observing a verified load: ``security_bpf_prog()``
--------------------------------------------------
There is deliberately no separate "metadata attested" hook. The coverage check
above is enforced by the kernel unconditionally, so a signed loader that fails
to cover its metadata never loads and an LSM never has to re-establish that
fact. To *act on* a program that has successfully and fully loaded, use the
existing ``security_bpf_prog()`` hook (``lsm/bpf_prog``), which fires from
``bpf_prog_new_fd()`` - after the verifier, after the coverage check, and after
``bpf_prog_alloc_id()``. Relative to the admission hook this point is strictly
later and stronger:
- the program has an id (``prog->aux->id``), so it can be recorded or correlated
with later events;
- ``verdict == BPF_SIG_VERIFIED`` *here* means **fully** verified - a program
that used a map the signature does not cover was already rejected, so it cannot
reach this point;
- it observes only programs that actually loaded; a failed load never mints an
fd, so it never reaches this hook.
It takes only the ``prog`` and a non-zero return still aborts (the fd is not
handed out), so it can veto as well as observe. One wrinkle: it also fires on
other paths that mint a new program fd - notably ``bpf_prog_get_fd_by_id()`` -
not just on a fresh load. Because the program already has its id here, an LSM
can tell the two apart with a small hash map: the *first* time an id is seen is
the load; a later sighting of the same id is just another fd to a program that
already exists.
To bound the map and let a reused id read as a fresh load, this can be paired
with ``security_bpf_prog_free()`` (``lsm/bpf_prog_free``), which deletes the
entry on teardown - keyed by the same ``prog`` pointer, since
``bpf_prog_free_id()`` has already cleared ``prog->aux->id`` to ``0`` by the time
that hook runs. (Illustrative - privileged LSM, error checking elided.)
.. code-block:: c
struct rec { __u32 id, ktype; __s32 serial; };
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, __u64); /* struct bpf_prog * -- stable id */
__type(value, struct rec);
__uint(max_entries, 4096);
} live SEC(".maps");
SEC("lsm/bpf_prog") /* fires after load and on every later fd */
int BPF_PROG(observe, struct bpf_prog *prog)
{
__u64 key = (__u64)(unsigned long)prog;
struct rec r;
if (prog->aux->sig.verdict != BPF_SIG_VERIFIED)
return 0;
if (bpf_map_lookup_elem(&live, &key))
return 0; /* seen before: a later fd, not a load */
/* First sighting == this program just loaded; id is valid here. */
r.id = prog->aux->id;
r.ktype = prog->aux->sig.keyring_type;
r.serial = prog->aux->sig.keyring_serial;
bpf_map_update_elem(&live, &key, &r, BPF_NOEXIST);
/* ... newly-loaded verified-program action, e.g. record r.id ... */
return 0;
}
Putting them together: to *require* verified BPF, deny at the admission hook
unless the verdict is ``BPF_SIG_VERIFIED`` (and, if desired, restrict the
keyring). The kernel then guarantees that any program which actually loads with
that verdict covered all of its exclusive maps, rejecting any that did not - so
a deny-by-default admission policy needs no second enforcement point. Use
``security_bpf_prog()`` to record or finally gate the verified programs once
they carry an id. The ``verdict``, ``keyring_type`` and ``keyring_serial`` fields
let a policy distinguish, for example, "verified and signed by a builtin key"
from "verified by a user key". A policy LSM such as IPE could consume the same
hooks to enforce system policy without writing any BPF, though none implements
this today.
Keyrings
========
``keyring_id`` selects the trusted keyring the PKCS#7 signature is verified
against. The well-known ids ``0`` (builtin), ``VERIFY_USE_SECONDARY_KEYRING``
and ``VERIFY_USE_PLATFORM_KEYRING`` select the corresponding system keyrings;
any other value is treated as the serial of a user/session key or keyring.
The keyring is looked up first, before the signature bytes are examined, so a
signature naming a non-existent keyring is rejected up front, and a failed
verification aborts the load - so a program that loads successfully with a
signature always has consistent keyring fields recorded.
Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect:
``keyring_type`` (``enum bpf_sig_keyring``)
Classified purely from ``keyring_id`` whenever the program is signed:
``BPF_SIG_KEYRING_BUILTIN``, ``_SECONDARY``, ``_PLATFORM`` for the system
keyrings, or ``_USER`` for a user/session keyring. It is
``BPF_SIG_KEYRING_NONE`` for an unsigned program.
``keyring_serial`` (``s32``)
Set **only** on a successful verification, to the serial of the
**user/session key or keyring** that ``keyring_id`` resolved to - the
object the signature was verified against, not the individual asymmetric
key inside it that matched the signer. Passing
``KEY_SPEC_SESSION_KEYRING``, for example, records the session keyring's
serial. The system keyrings are trusted as a whole and expose no serial
here, so the serial is ``0`` for builtin, secondary and platform
signatures, and ``0`` for unsigned programs. In other words, a non-zero
``keyring_serial`` is exactly "verified against the user key/keyring with
this serial".
.. list-table::
:header-rows: 1
* - ``keyring_id``
- ``keyring_type``
- ``keyring_serial``
* - (no signature)
- ``BPF_SIG_KEYRING_NONE``
- ``0``
* - ``0``
- ``BPF_SIG_KEYRING_BUILTIN``
- ``0``
* - ``VERIFY_USE_SECONDARY_KEYRING``
- ``BPF_SIG_KEYRING_SECONDARY``
- ``0``
* - ``VERIFY_USE_PLATFORM_KEYRING``
- ``BPF_SIG_KEYRING_PLATFORM``
- ``0``
* - other (a user/session key serial)
- ``BPF_SIG_KEYRING_USER``
- serial of the resolved key/keyring
Producing a signed object
==========================
``bpftool`` generates and signs a light skeleton in one step::
bpftool gen skeleton -L -S -k <private_key.pem> -i <certificate.x509> \
obj.bpf.o > obj.lskel.h
``-L`` selects the light-skeleton (``gen_loader``) backend and ``-S`` enables
signing; ``-k`` and ``-i`` supply the signing key and its X.509 certificate.
``bpftool`` signs ``insns || metadata`` - the exact bytes the kernel
reconstructs - and also computes ``excl_prog_hash`` as the digest of the loader
instructions so the metadata map can be bound to the loader. The signature and
hash are embedded in the generated header; the certificate is used only for
signing and is not included. Loading the skeleton performs the
create/populate/freeze/load sequence described above.
At runtime the trusted public key must be present in the chosen keyring (for
example added to the session keyring, or built into the kernel's builtin trusted
keyring) for verification to succeed.
UAPI reference
==============
``BPF_PROG_LOAD`` (``union bpf_attr``):
``signature``, ``signature_size``
Pointer to and length of the PKCS#7 signature blob.
``keyring_id``
Trusted keyring selector (see `Keyrings`_).
``fd_array``, ``fd_array_cnt``
Array of map (and module BTF) file descriptors bound to the program.
``fd_array_cnt`` must be set for the kernel to scan the array. When a
signature is present, a BTF entry is rejected outright, and every map must
be exclusive; its frozen contents are folded into the verified buffer, and
a non-exclusive entry is rejected.
``BPF_MAP_CREATE`` (``union bpf_attr``):
``excl_prog_hash``, ``excl_prog_hash_size``
SHA-256 digest of the program permitted to access this (exclusive) map. This
binds the metadata map to the loader; it is not a hash of the map *content*.
The map content is not hashed separately at all - it is covered, as bytes,
by the program signature.
Notes and limitations
======================
- The instructions plus folded metadata are verified as one ``bpf_dynptr``,
which bounds the combined size (currently ~16 MiB); very large objects can
exceed it.
- The metadata container is a single-element array map, accessed through
``map_direct_value_addr``.

5
Documentation/process/changes.rst
View File

@@ -147,11 +147,6 @@ Since Linux 5.2, if CONFIG_DEBUG_INFO_BTF is selected, the build system
generates BTF (BPF Type Format) from DWARF in vmlinux, a bit later from kernel
modules as well. This requires pahole v1.22 or later.
Since Linux 7.0, kfuncs annotated with KF_IMPLICIT_ARGS require pahole v1.26
or later. Without it, such kfuncs will have incorrect BTF prototypes in
vmlinux, causing BPF programs to fail to load with a "func_proto incompatible
with vmlinux" error. Many sched_ext kfuncs are affected.
It is found in the 'dwarves' or 'pahole' distro packages or from
https://fedorapeople.org/~acme/dwarves/.

6
MAINTAINERS
View File

@@ -4905,6 +4905,7 @@ R: Song Liu <song@kernel.org>
R: Yonghong Song <yonghong.song@linux.dev>
R: Jiri Olsa <jolsa@kernel.org>
R: Emil Tsalapatis <emil@etsalapatis.com>
R: Ihor Solodrai <ihor.solodrai@linux.dev>
L: bpf@vger.kernel.org
S: Supported
W: https://bpf.io/
@@ -4965,6 +4966,7 @@ F: net/unix/unix_bpf.c
BPF [LIBRARY] (libbpf)
M: Andrii Nakryiko <andrii@kernel.org>
M: Eduard Zingerman <eddyz87@gmail.com>
R: Ihor Solodrai <ihor.solodrai@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: tools/lib/bpf/
@@ -5017,7 +5019,7 @@ F: kernel/bpf/ringbuf.c
BPF [SECURITY & LSM] (Security Audit and Enforcement using BPF)
M: KP Singh <kpsingh@kernel.org>
M: Matt Bobrowski <mattbobrowski@google.com>
M: Matt Bobrowski <matt@bobrowski.net>
L: bpf@vger.kernel.org
S: Maintained
F: Documentation/bpf/prog_lsm.rst
@@ -5030,6 +5032,7 @@ F: security/bpf/
BPF [SELFTESTS] (Test Runners & Infrastructure)
M: Andrii Nakryiko <andrii@kernel.org>
M: Eduard Zingerman <eddyz87@gmail.com>
R: Ihor Solodrai <ihor.solodrai@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: tools/testing/selftests/bpf/
@@ -5044,6 +5047,7 @@ F: tools/bpf/bpftool/
BPF [TRACING]
M: Song Liu <song@kernel.org>
R: Jiri Olsa <jolsa@kernel.org>
R: Ihor Solodrai <ihor.solodrai@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: kernel/bpf/stackmap.c

23
arch/arm64/include/asm/insn.h
View File

@@ -205,6 +205,18 @@ enum aarch64_insn_adsb_type {
AARCH64_INSN_ADSB_SUB_SETFLAGS
};
/* option field of add/sub (extended register) */
enum aarch64_insn_extend_type {
AARCH64_INSN_EXTEND_UXTB,
AARCH64_INSN_EXTEND_UXTH,
AARCH64_INSN_EXTEND_UXTW,
AARCH64_INSN_EXTEND_UXTX,
AARCH64_INSN_EXTEND_SXTB,
AARCH64_INSN_EXTEND_SXTH,
AARCH64_INSN_EXTEND_SXTW,
AARCH64_INSN_EXTEND_SXTX,
};
enum aarch64_insn_movewide_type {
AARCH64_INSN_MOVEWIDE_ZERO,
AARCH64_INSN_MOVEWIDE_KEEP,
@@ -378,6 +390,10 @@ __AARCH64_INSN_FUNCS(add, 0x7F200000, 0x0B000000)
__AARCH64_INSN_FUNCS(adds, 0x7F200000, 0x2B000000)
__AARCH64_INSN_FUNCS(sub, 0x7F200000, 0x4B000000)
__AARCH64_INSN_FUNCS(subs, 0x7F200000, 0x6B000000)
__AARCH64_INSN_FUNCS(add_ext, 0x7FE00000, 0x0B200000)
__AARCH64_INSN_FUNCS(adds_ext, 0x7FE00000, 0x2B200000)
__AARCH64_INSN_FUNCS(sub_ext, 0x7FE00000, 0x4B200000)
__AARCH64_INSN_FUNCS(subs_ext, 0x7FE00000, 0x6B200000)
__AARCH64_INSN_FUNCS(madd, 0x7FE08000, 0x1B000000)
__AARCH64_INSN_FUNCS(msub, 0x7FE08000, 0x1B008000)
__AARCH64_INSN_FUNCS(udiv, 0x7FE0FC00, 0x1AC00800)
@@ -637,6 +653,13 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_adsb_type type);
u32 aarch64_insn_gen_add_sub_extended_reg(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
enum aarch64_insn_register reg,
enum aarch64_insn_extend_type extend,
int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_adsb_type type);
u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
enum aarch64_insn_variant variant,

60
arch/arm64/lib/insn.c
View File

@@ -986,6 +986,66 @@ u32 aarch64_insn_gen_add_sub_shifted_reg(enum aarch64_insn_register dst,
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn, shift);
}
/*
* Unlike the shifted-register form, register 31 is not XZR everywhere here:
* it encodes SP for @src, and for @dst too unless @type sets the flags. Only
* @reg keeps the XZR meaning.
*/
u32 aarch64_insn_gen_add_sub_extended_reg(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
enum aarch64_insn_register reg,
enum aarch64_insn_extend_type extend,
int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_adsb_type type)
{
u32 insn;
switch (type) {
case AARCH64_INSN_ADSB_ADD:
insn = aarch64_insn_get_add_ext_value();
break;
case AARCH64_INSN_ADSB_SUB:
insn = aarch64_insn_get_sub_ext_value();
break;
case AARCH64_INSN_ADSB_ADD_SETFLAGS:
insn = aarch64_insn_get_adds_ext_value();
break;
case AARCH64_INSN_ADSB_SUB_SETFLAGS:
insn = aarch64_insn_get_subs_ext_value();
break;
default:
pr_err("%s: unknown add/sub encoding %d\n", __func__, type);
return AARCH64_BREAK_FAULT;
}
switch (variant) {
case AARCH64_INSN_VARIANT_32BIT:
break;
case AARCH64_INSN_VARIANT_64BIT:
insn |= AARCH64_INSN_SF_BIT;
break;
default:
pr_err("%s: unknown variant encoding %d\n", __func__, variant);
return AARCH64_BREAK_FAULT;
}
if (shift < 0 || shift > 4) {
pr_err("%s: invalid shift encoding %d\n", __func__, shift);
return AARCH64_BREAK_FAULT;
}
insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, reg);
/* option in bits [15:13] and imm3 in [12:10] together fill IMM_6 */
return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_6, insn,
(extend << 3) | shift);
}
u32 aarch64_insn_gen_data1(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
enum aarch64_insn_variant variant,

11
arch/arm64/net/bpf_jit.h
View File

@@ -243,6 +243,17 @@
/* Rn - Rm; set condition flags */
#define A64_CMP(sf, Rn, Rm) A64_SUBS(sf, A64_ZR, Rn, Rm)
/* Add/subtract (extended register) */
#define A64_ADDSUB_EREG(sf, Rd, Rn, Rm, ext, shift, type) \
aarch64_insn_gen_add_sub_extended_reg(Rd, Rn, Rm, \
AARCH64_INSN_EXTEND_##ext, shift, A64_VARIANT(sf), \
AARCH64_INSN_ADSB_##type)
/* Rd = Rn + (EXT(Rm) << shift) */
#define A64_ADD_EXT(sf, Rd, Rn, Rm, ext, shift) \
A64_ADDSUB_EREG(sf, Rd, Rn, Rm, ext, shift, ADD)
/* Rd = Rn + (u32)Rm */
#define A64_ADD_UXTW(Rd, Rn, Rm) A64_ADD_EXT(1, Rd, Rn, Rm, UXTW, 0)
/* Data-processing (1 source) */
#define A64_DATA1(sf, Rd, Rn, type) aarch64_insn_gen_data1(Rd, Rn, \
A64_VARIANT(sf), AARCH64_INSN_DATA1_##type)

229
arch/arm64/net/bpf_jit_comp.c
View File

@@ -1082,23 +1082,27 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
*
* Bit layout of `fixup` (32-bit):
*
* +-----------+--------+-----------+-----------+----------+
* | 31-27 | 26-22 | 21 | 20-16 | 15-0 |
* | | | | | |
* | FIXUP_REG | Unused | ARENA_ACC | ARENA_REG | OFFSET |
* +-----------+--------+-----------+-----------+----------+
* +-----------+--------+-------------+-----------+-----------+----------+
* | 31-27 | 26-23 | 22 | 21 | 20-16 | 15-0 |
* | | | | | | |
* | FIXUP_REG | Unused | ARENA_WRITE | ARENA_ACC | ARENA_REG | OFFSET |
* +-----------+--------+-------------+-----------+-----------+----------+
*
* - OFFSET (16 bits): Offset used to compute address for Load/Store instruction.
* - ARENA_REG (5 bits): Register that is used to calculate the address for load/store when
* accessing the arena region.
* - ARENA_ACCESS (1 bit): This bit is set when the faulting instruction accessed the arena region.
* - ARENA_WRITE (1 bit): This bit is set when the faulting instruction wrote to the arena region.
* It is independent of FIXUP_REG, since a read-modify-write both writes to
* memory and reads the old value into a register.
* - FIXUP_REG (5 bits): Destination register for the load instruction (cleared on fault) or set to
* DONT_CLEAR if it is a store instruction.
* DONT_CLEAR if the instruction does not read into a register.
*/
#define BPF_FIXUP_OFFSET_MASK GENMASK(15, 0)
#define BPF_FIXUP_ARENA_REG_MASK GENMASK(20, 16)
#define BPF_ARENA_ACCESS BIT(21)
#define BPF_ARENA_WRITE BIT(22)
#define BPF_FIXUP_REG_MASK GENMASK(31, 27)
#define DONT_CLEAR 5 /* Unused ARM64 register from BPF's POV */
@@ -1109,7 +1113,7 @@ bool ex_handler_bpf(const struct exception_table_entry *ex,
s16 off = FIELD_GET(BPF_FIXUP_OFFSET_MASK, ex->fixup);
int arena_reg = FIELD_GET(BPF_FIXUP_ARENA_REG_MASK, ex->fixup);
bool is_arena = !!(ex->fixup & BPF_ARENA_ACCESS);
bool is_write = (dst_reg == DONT_CLEAR);
bool is_write = !!(ex->fixup & BPF_ARENA_WRITE);
unsigned long addr;
if (is_arena) {
@@ -1132,7 +1136,7 @@ static int add_exception_handler(const struct bpf_insn *insn,
{
off_t ins_offset;
s16 off = insn->off;
bool is_arena;
bool is_arena, is_write;
int arena_reg;
unsigned long pc;
struct exception_table_entry *ex;
@@ -1178,13 +1182,21 @@ static int add_exception_handler(const struct bpf_insn *insn,
ex->insn = ins_offset;
if (BPF_CLASS(insn->code) != BPF_LDX)
dst_reg = DONT_CLEAR;
/*
* A load-acquire is of BPF_STX class, but reads from src_reg into
* dst_reg like a BPF_LDX does, hence it must not be treated as a store
* here. A read-modify-write carrying BPF_FETCH is reported as a write
* even though it does have a register to clear, see the callers.
*/
is_write = BPF_CLASS(insn->code) != BPF_LDX &&
!bpf_atomic_is_load_acq(insn);
ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg);
if (is_arena) {
ex->fixup |= BPF_ARENA_ACCESS;
if (is_write)
ex->fixup |= BPF_ARENA_WRITE;
/*
* insn->src_reg/dst_reg holds the address in the arena region with upper 32-bits
* being zero because of a preceding addr_space_cast(r<n>, 0x0, 0x1) instruction.
@@ -1193,7 +1205,7 @@ static int add_exception_handler(const struct bpf_insn *insn,
* memory access. Pass the reg holding the unmodified 32-bit address to
* ex_handler_bpf.
*/
if (BPF_CLASS(insn->code) == BPF_LDX)
if (BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn))
arena_reg = bpf2a64[insn->src_reg];
else
arena_reg = bpf2a64[insn->dst_reg];
@@ -1244,6 +1256,43 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct
}
}
/*
* Rebase the __arena args of a kfunc call to arena kernel addresses,
* xN = kern_vm_start + (u32)xN, with the arena base register holding
* kern_vm_start. A nullable arg preserves NULL by skipping the add, tested
* on the truncated value as arena NULL is offset 0.
*/
static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn)
{
const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
const struct btf_func_model *fm;
int i;
fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
if (!fm)
return -EINVAL;
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
const u8 reg = bpf2a64[BPF_REG_1 + i];
u8 flags = fm->arg_flags[i];
if (!(flags & BTF_FMODEL_ARENA_ARG))
continue;
if (WARN_ON_ONCE(!ctx->arena_vm_start))
return -EINVAL;
if (flags & BTF_FMODEL_NULLABLE_ARG) {
/* 32-bit mov clears the upper 32 bits */
emit(A64_MOV(0, reg, reg), ctx);
/* skip the add so that NULL stays NULL */
emit(A64_CBZ(0, reg, 2), ctx);
}
emit(A64_ADD_UXTW(reg, arena_vm_base, reg), ctx);
}
return 0;
}
/* JITs an eBPF instruction.
* Returns:
* 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1284,12 +1333,25 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
case BPF_ALU | BPF_MOV | BPF_X:
case BPF_ALU64 | BPF_MOV | BPF_X:
if (insn_is_cast_user(insn)) {
emit(A64_MOV(0, tmp, src), ctx); // 32-bit mov clears the upper 32 bits
emit_a64_mov_i(0, dst, ctx->user_vm_start >> 32, ctx);
emit(A64_LSL(1, dst, dst, 32), ctx);
emit(A64_CBZ(1, tmp, 2), ctx);
emit(A64_ORR(1, tmp, dst, tmp), ctx);
emit(A64_MOV(1, dst, tmp), ctx);
u32 upper = ctx->user_vm_start >> 32;
u16 upper_low = upper & 0xffff;
u16 upper_high = upper >> 16;
int nr_movk = !!upper_low + !!upper_high;
/*
* Build the user address: the low 32 bits are the arena
* offset, the upper 32 bits come from user_vm_start. A
* zero offset must stay NULL, so branch over the MOVKs
* when it is zero.
*/
emit(A64_MOV(0, dst, src), ctx); /* 32-bit mov clears the upper 32 bits */
if (nr_movk) {
emit(A64_CBZ(0, dst, nr_movk + 1), ctx);
if (upper_low)
emit(A64_MOVK(1, dst, upper_low, 32), ctx);
if (upper_high)
emit(A64_MOVK(1, dst, upper_high, 48), ctx);
}
break;
} else if (insn_is_mov_percpu_addr(insn)) {
if (dst != src)
@@ -1653,6 +1715,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
&func_addr, &func_addr_fixed);
if (ret < 0)
return ret;
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
ret = emit_kfunc_arena_args(ctx, insn);
if (ret < 0)
return ret;
}
emit_call(func_addr, ctx);
/*
* Call to arch_bpf_timed_may_goto() is emitted by the
@@ -1871,7 +1938,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
break;
}
ret = add_exception_handler(insn, ctx, dst);
ret = add_exception_handler(insn, ctx, DONT_CLEAR);
if (ret)
return ret;
break;
@@ -1938,7 +2005,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
break;
}
ret = add_exception_handler(insn, ctx, dst);
ret = add_exception_handler(insn, ctx, DONT_CLEAR);
if (ret)
return ret;
break;
@@ -1961,7 +2028,16 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
return ret;
if (BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) {
ret = add_exception_handler(insn, ctx, dst);
/*
* A load-acquire reads into dst_reg, and a read-modify-write
* carrying BPF_FETCH reads the old value into src_reg, or into
* r0 for a BPF_CMPXCHG. Clear that register on fault, the
* remaining atomics have no destination register.
*/
int load_reg = bpf_atomic_load_reg(insn);
ret = add_exception_handler(insn, ctx, load_reg < 0 ?
DONT_CLEAR : bpf2a64[load_reg]);
if (ret)
return ret;
}
@@ -2317,6 +2393,11 @@ bool bpf_jit_supports_stack_args(void)
return true;
}
bool bpf_jit_supports_arena_args(void)
{
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (!aarch64_insn_copy(dst, src, len))
@@ -2490,34 +2571,76 @@ static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes)
}
}
static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
const struct btf_func_model *m,
const struct arg_aux *a,
bool for_call_origin)
/*
* Convert an arena kernel address into the arena pointer form on its way into
* the BPF ctx, dst = (u32)(src - kern_vm_start), with @base_lo holding the low
* 32 bits of kern_vm_start. A nullable arg preserves NULL, tested on the full
* 64-bit kernel pointer. The 32-bit subtraction both truncates and clears the
* upper half, so the stored value satisfies the JIT invariant for arena
* pointer registers.
*/
static void emit_arena_arg_conv(struct jit_ctx *ctx, u8 dst, u8 src, bool nullable, u8 base_lo)
{
if (nullable) {
if (dst != src)
emit(A64_MOV(1, dst, src), ctx);
/* skip the subtraction so that NULL stays NULL */
emit(A64_CBZ(1, dst, 2), ctx);
src = dst;
}
emit(A64_SUB(0, dst, src, base_lo), ctx);
}
static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
const struct btf_func_model *m, const struct arg_aux *a,
bool for_call_origin, bool is_struct_ops, u64 arena_base)
{
int i;
int reg;
int doff;
int soff;
int slots;
u8 tmp = bpf2a64[TMP_REG_1];
u8 base_lo = bpf2a64[TMP_REG_2];
int i, reg, doff, soff, slots;
/* only the low 32 bits of the base take part in the subtraction */
if (arena_base)
emit_a64_mov_i(0, base_lo, (s32)(u32)arena_base, ctx);
/* store arguments to the stack for the bpf program, or restore
* arguments from stack for the original function
*/
for (reg = 0; reg < a->regs_for_args; reg++) {
emit(for_call_origin ?
A64_LDR64I(reg, A64_SP, bargs_off) :
A64_STR64I(reg, A64_SP, bargs_off),
ctx);
bargs_off += 8;
for (i = 0, reg = 0; i < a->args_in_regs; i++) {
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
slots = (m->arg_size[i] + 7) / 8;
while (slots-- > 0) {
if (for_call_origin) {
emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);
} else if (arena_arg) {
emit_arena_arg_conv(ctx, tmp, reg, nullable, base_lo);
emit(A64_STR64I(tmp, A64_SP, bargs_off), ctx);
} else {
emit(A64_STR64I(reg, A64_SP, bargs_off), ctx);
}
reg++;
bargs_off += 8;
}
}
soff = 32; /* on stack arguments start from FP + 32 */
/*
* On-stack arguments start above the frame(s) pushed by the trampoline
* prologue. Entered through the fentry call from a traced function, the
* prologue saves both the parent (FP/x9) and the traced function
* (FP/LR) frames, so the arguments start at FP + 32. A struct_ops
* callback is called indirectly and only the FP/LR frame is saved, so
* they start at FP + 16.
*/
soff = is_struct_ops ? 16 : 32;
doff = (for_call_origin ? oargs_off : bargs_off);
/* save on stack arguments */
for (i = a->args_in_regs; i < m->nr_args; i++) {
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
slots = (m->arg_size[i] + 7) / 8;
/* verifier ensures arg_size <= 16, so slots equals 1 or 2 */
while (slots-- > 0) {
@@ -2527,6 +2650,15 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
*/
if (slots == 0 && !for_call_origin)
clear_garbage(ctx, tmp, m->arg_size[i] % 8);
/*
* No guard on for_call_origin here: only the indirect
* trampoline is given a base, and it never calls the
* original function, so arguments are never converted
* on their way back out to it. See the WARN_ON_ONCE()
* in prepare_trampoline().
*/
if (arena_arg)
emit_arena_arg_conv(ctx, tmp, tmp, nullable, base_lo);
emit(A64_STR64I(tmp, A64_SP, doff), ctx);
soff += 8;
doff += 8;
@@ -2544,12 +2676,6 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs)
}
}
static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
{
return fentry_nodes->nr_nodes == 1 &&
fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
}
static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)
{
emit_a64_mov_i64(A64_R(10), func_meta, ctx);
@@ -2592,8 +2718,21 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
bool is_struct_ops = is_struct_ops_tramp(fentry);
int cookie_off, cookie_cnt, cookie_bargs_off;
int fsession_cnt = bpf_fsession_cnt(tnodes);
u64 arena_base;
u64 func_meta;
/*
* F_INDIRECT is only compatible with F_RET_FENTRY_RET, it is explicitly
* incompatible with F_CALL_ORIG | F_SKIP_FRAME | F_IP_ARG because
* @func_addr. Arena conversion relies on this: bpf_tramp_arena_base()
* only returns a base for the indirect trampoline, which therefore
* never calls the original function with converted arguments.
*/
WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) &&
(flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET)));
arena_base = bpf_tramp_arena_base(m, tnodes, flags);
/* trampoline stack layout:
* [ parent ip ]
* [ FP ]
@@ -2709,7 +2848,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
store_func_meta(ctx, func_meta, func_meta_off);
/* save args for bpf */
save_args(ctx, bargs_off, oargs_off, m, a, false);
save_args(ctx, bargs_off, oargs_off, m, a, false, is_struct_ops, arena_base);
/* save callee saved registers */
emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx);
@@ -2757,8 +2896,8 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* save args for original func */
save_args(ctx, bargs_off, oargs_off, m, a, true);
/* the original func takes kernel addresses, never converted ones */
save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0);
/* call original func */
emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx);
emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx);

2
arch/riscv/net/Makefile
View File

@@ -3,7 +3,7 @@
obj-$(CONFIG_BPF_JIT) += bpf_jit_core.o
ifeq ($(CONFIG_ARCH_RV64I),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp64.o
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp64.o bpf_timed_may_goto.o
else
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
endif

1
arch/riscv/net/bpf_jit.h
View File

@@ -81,6 +81,7 @@ struct rv_jit_context {
int ex_jmp_off;
unsigned long flags;
int stack_size;
int tcc_offset;
u64 arena_vm_start;
u64 user_vm_start;
};

111
arch/riscv/net/bpf_jit_comp32.c
View File

@@ -509,12 +509,15 @@ static void emit_alu_r64(const s8 *dst, const s8 *src,
}
static void emit_alu_r32(const s8 *dst, const s8 *src,
struct rv_jit_context *ctx, const u8 op)
struct rv_jit_context *ctx,
const struct bpf_insn *insn)
{
const s8 *tmp1 = bpf2rv32[TMP_REG_1];
const s8 *tmp2 = bpf2rv32[TMP_REG_2];
const s8 *rd = bpf_get_reg32(dst, tmp1, ctx);
const s8 *rs = bpf_get_reg32(src, tmp2, ctx);
u8 op = BPF_OP(insn->code);
bool is_signed = insn->off == 1;
switch (op) {
case BPF_MOV:
@@ -539,10 +542,12 @@ static void emit_alu_r32(const s8 *dst, const s8 *src,
emit(rv_mul(lo(rd), lo(rd), lo(rs)), ctx);
break;
case BPF_DIV:
emit(rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
emit(is_signed ? rv_div(lo(rd), lo(rd), lo(rs)) :
rv_divu(lo(rd), lo(rd), lo(rs)), ctx);
break;
case BPF_MOD:
emit(rv_remu(lo(rd), lo(rd), lo(rs)), ctx);
emit(is_signed ? rv_rem(lo(rd), lo(rd), lo(rs)) :
rv_remu(lo(rd), lo(rd), lo(rs)), ctx);
break;
case BPF_LSH:
emit(rv_sll(lo(rd), lo(rd), lo(rs)), ctx);
@@ -869,14 +874,58 @@ static int emit_load_r64(const s8 *dst, const s8 *src, s16 off,
return 0;
}
static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
struct rv_jit_context *ctx, const u8 size,
const u8 mode)
static int emit_bpf_atomic(s8 dst, const s8 *src, const s8 *rs,
struct rv_jit_context *ctx,
const struct bpf_insn *insn)
{
s32 imm = insn->imm;
bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
s8 fetch_reg = is_fetch ? lo(rs) : RV_REG_ZERO;
int aq = is_fetch ? 1 : 0;
int rl = is_fetch ? 1 : 0;
switch (imm) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
emit(rv_amoadd_w(fetch_reg, lo(rs), dst, aq, rl), ctx);
break;
case BPF_AND:
case BPF_AND | BPF_FETCH:
emit(rv_amoand_w(fetch_reg, lo(rs), dst, aq, rl), ctx);
break;
case BPF_OR:
case BPF_OR | BPF_FETCH:
emit(rv_amoor_w(fetch_reg, lo(rs), dst, aq, rl), ctx);
break;
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
emit(rv_amoxor_w(fetch_reg, lo(rs), dst, aq, rl), ctx);
break;
case BPF_XCHG:
emit(rv_amoswap_w(fetch_reg, lo(rs), dst, aq, rl), ctx);
break;
default:
return -1;
}
if (is_fetch) {
emit(rv_addi(hi(rs), RV_REG_ZERO, 0), ctx);
bpf_put_reg64(src, rs, ctx);
}
return 0;
}
static int emit_store_r64(const s8 *dst, const s8 *src,
struct rv_jit_context *ctx,
const struct bpf_insn *insn)
{
const s8 *tmp1 = bpf2rv32[TMP_REG_1];
const s8 *tmp2 = bpf2rv32[TMP_REG_2];
const s8 *rd = bpf_get_reg64(dst, tmp1, ctx);
const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
u8 size = BPF_SIZE(insn->code);
u8 mode = BPF_MODE(insn->code);
s16 off = insn->off;
if (mode == BPF_ATOMIC && size != BPF_W)
return -1;
@@ -896,9 +945,9 @@ static int emit_store_r64(const s8 *dst, const s8 *src, s16 off,
case BPF_MEM:
emit(rv_sw(RV_REG_T0, 0, lo(rs)), ctx);
break;
case BPF_ATOMIC: /* Only BPF_ADD supported */
emit(rv_amoadd_w(RV_REG_ZERO, lo(rs), RV_REG_T0, 0, 0),
ctx);
case BPF_ATOMIC:
if (emit_bpf_atomic(RV_REG_T0, src, rs, ctx, insn))
return -1;
break;
}
break;
@@ -967,6 +1016,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
switch (code) {
case BPF_ALU64 | BPF_MOV | BPF_X:
if (insn->off != 0) {
const s8 *rd = bpf_get_reg64(dst, tmp1, ctx);
const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
if (insn->off == 8) {
emit(rv_slli(lo(rd), lo(rs), 24), ctx);
emit(rv_srai(lo(rd), lo(rd), 24), ctx);
} else if (insn->off == 16) {
emit(rv_slli(lo(rd), lo(rs), 16), ctx);
emit(rv_srai(lo(rd), lo(rd), 16), ctx);
} else {
emit(rv_addi(lo(rd), lo(rs), 0), ctx);
}
emit(rv_srai(hi(rd), lo(rd), 31), ctx);
bpf_put_reg64(dst, rd, ctx);
break;
}
fallthrough;
case BPF_ALU64 | BPF_ADD | BPF_X:
case BPF_ALU64 | BPF_ADD | BPF_K:
@@ -1017,6 +1084,20 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_zext64(dst, ctx);
break;
}
if (insn->off != 0) {
const s8 *rd = bpf_get_reg32(dst, tmp1, ctx);
const s8 *rs = bpf_get_reg32(src, tmp2, ctx);
if (insn->off == 8) {
emit(rv_slli(lo(rd), lo(rs), 24), ctx);
emit(rv_srai(lo(rd), lo(rd), 24), ctx);
} else if (insn->off == 16) {
emit(rv_slli(lo(rd), lo(rs), 16), ctx);
emit(rv_srai(lo(rd), lo(rd), 16), ctx);
}
bpf_put_reg32(dst, rd, ctx);
break;
}
fallthrough;
case BPF_ALU | BPF_ADD | BPF_X:
@@ -1041,7 +1122,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_imm32(tmp2, imm, ctx);
src = tmp2;
}
emit_alu_r32(dst, src, ctx, BPF_OP(code));
emit_alu_r32(dst, src, ctx, insn);
break;
case BPF_ALU | BPF_MOV | BPF_K:
@@ -1065,7 +1146,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
* src is ignored---choose tmp2 as a dummy register since it
* is not on the stack.
*/
emit_alu_r32(dst, tmp2, ctx, BPF_OP(code));
emit_alu_r32(dst, tmp2, ctx, insn);
break;
case BPF_ALU | BPF_END | BPF_FROM_LE:
@@ -1266,21 +1347,19 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
src = tmp2;
}
if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
BPF_MODE(code)))
if (emit_store_r64(dst, src, ctx, insn))
return -1;
break;
case BPF_STX | BPF_ATOMIC | BPF_W:
if (insn->imm != BPF_ADD) {
if (insn->imm == BPF_CMPXCHG) {
pr_info_once(
"bpf-jit: not supported: atomic operation %02x ***\n",
insn->imm);
return -EFAULT;
}
if (emit_store_r64(dst, src, off, ctx, BPF_SIZE(code),
BPF_MODE(code)))
if (emit_store_r64(dst, src, ctx, insn))
return -1;
break;

158
arch/riscv/net/bpf_jit_comp64.c
View File

@@ -18,12 +18,12 @@
#define RV_MAX_REG_ARGS 8
#define RV_FENTRY_NINSNS 2
#define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4)
#define RV_KCFI_NINSNS (IS_ENABLED(CONFIG_CFI) ? 1 : 0)
/* imm that allows emit_imm to emit max count insns */
#define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF
/* fentry and TCC init insns will be skipped on tailcall */
#define RV_TAILCALL_OFFSET ((RV_FENTRY_NINSNS + 1) * 4)
#define RV_REG_TCC RV_REG_A6
#define RV_REG_TCC_SAVED RV_REG_S6 /* Store A6 in S6 if program do calls */
#define RV_REG_ARENA RV_REG_S7 /* For storing arena_vm_start */
static const int regmap[] = {
@@ -57,14 +57,12 @@ static const int pt_regmap[] = {
};
enum {
RV_CTX_F_SEEN_TAIL_CALL = 0,
RV_CTX_F_SEEN_CALL = RV_REG_RA,
RV_CTX_F_SEEN_S1 = RV_REG_S1,
RV_CTX_F_SEEN_S2 = RV_REG_S2,
RV_CTX_F_SEEN_S3 = RV_REG_S3,
RV_CTX_F_SEEN_S4 = RV_REG_S4,
RV_CTX_F_SEEN_S5 = RV_REG_S5,
RV_CTX_F_SEEN_S6 = RV_REG_S6,
};
static u8 bpf_to_rv_reg(int bpf_reg, struct rv_jit_context *ctx)
@@ -77,7 +75,6 @@ static u8 bpf_to_rv_reg(int bpf_reg, struct rv_jit_context *ctx)
case RV_CTX_F_SEEN_S3:
case RV_CTX_F_SEEN_S4:
case RV_CTX_F_SEEN_S5:
case RV_CTX_F_SEEN_S6:
__set_bit(reg, &ctx->flags);
}
return reg;
@@ -92,7 +89,6 @@ static bool seen_reg(int reg, struct rv_jit_context *ctx)
case RV_CTX_F_SEEN_S3:
case RV_CTX_F_SEEN_S4:
case RV_CTX_F_SEEN_S5:
case RV_CTX_F_SEEN_S6:
return test_bit(reg, &ctx->flags);
}
return false;
@@ -108,32 +104,6 @@ static void mark_call(struct rv_jit_context *ctx)
__set_bit(RV_CTX_F_SEEN_CALL, &ctx->flags);
}
static bool seen_call(struct rv_jit_context *ctx)
{
return test_bit(RV_CTX_F_SEEN_CALL, &ctx->flags);
}
static void mark_tail_call(struct rv_jit_context *ctx)
{
__set_bit(RV_CTX_F_SEEN_TAIL_CALL, &ctx->flags);
}
static bool seen_tail_call(struct rv_jit_context *ctx)
{
return test_bit(RV_CTX_F_SEEN_TAIL_CALL, &ctx->flags);
}
static u8 rv_tail_call_reg(struct rv_jit_context *ctx)
{
mark_tail_call(ctx);
if (seen_call(ctx)) {
__set_bit(RV_CTX_F_SEEN_S6, &ctx->flags);
return RV_REG_S6;
}
return RV_REG_A6;
}
static bool is_32b_int(s64 val)
{
return -(1L << 31) <= val && val < (1L << 31);
@@ -258,23 +228,20 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
emit_ld(RV_REG_S5, store_offset, RV_REG_SP, ctx);
store_offset -= 8;
}
if (seen_reg(RV_REG_S6, ctx)) {
emit_ld(RV_REG_S6, store_offset, RV_REG_SP, ctx);
store_offset -= 8;
}
if (ctx->arena_vm_start) {
emit_ld(RV_REG_ARENA, store_offset, RV_REG_SP, ctx);
store_offset -= 8;
}
/* restore TCC from stack to RV_REG_TCC */
emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
/* Set return value. */
if (!is_tail_call)
emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx);
emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA,
/* kcfi, fentry and TCC init insns will be skipped on tailcall */
is_tail_call ? (RV_KCFI_NINSNS + RV_FENTRY_NINSNS + 1) * 4 : 0,
ctx);
is_tail_call ? RV_TAILCALL_OFFSET : 0, ctx);
}
static void emit_bcc(u8 cond, u8 rd, u8 rs, int rvoff,
@@ -355,7 +322,6 @@ static void emit_branch(u8 cond, u8 rd, u8 rs, int rvoff,
static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx)
{
int tc_ninsn, off, start_insn = ctx->ninsns;
u8 tcc = rv_tail_call_reg(ctx);
/* a0: &ctx
* a1: &array
@@ -378,7 +344,8 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx)
/* if (--TCC < 0)
* goto out;
*/
emit_addi(RV_REG_TCC, tcc, -1, ctx);
emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
emit_addi(RV_REG_TCC, RV_REG_TCC, -1, ctx);
off = ninsns_rvoff(tc_ninsn - (ctx->ninsns - start_insn));
emit_branch(BPF_JSLT, RV_REG_TCC, RV_REG_ZERO, off, ctx);
@@ -394,7 +361,10 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx)
off = ninsns_rvoff(tc_ninsn - (ctx->ninsns - start_insn));
emit_branch(BPF_JEQ, RV_REG_T2, RV_REG_ZERO, off, ctx);
/* goto *(prog->bpf_func + 4); */
/* store updated TCC back to stack */
emit_sd(RV_REG_SP, ctx->tcc_offset, RV_REG_TCC, ctx);
/* goto *(prog->bpf_func + RV_TAILCALL_OFFSET); */
off = offsetof(struct bpf_prog, bpf_func);
if (is_12b_check(off, insn))
return -1;
@@ -1028,12 +998,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
int i, ret, offset;
int *branches_off = NULL;
int stack_size = 0, nr_arg_slots = 0;
int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
int retval_off, args_off, func_meta_off, ip_off;
int run_ctx_off, sreg_off, stk_arg_off, tcc_off;
int cookie_off, cookie_cnt;
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
bool is_struct_ops = is_struct_ops_tramp(fentry);
void *orig_call = func_addr;
bool save_ret;
u64 func_meta;
@@ -1079,6 +1050,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
*
* FP - sreg_off [ callee saved reg ]
*
* FP - tcc_off [ tail call count ] BPF_TRAMP_F_TAIL_CALL_CTX
*
* [ pads ] pads for 16 bytes alignment
*
* [ stack_argN ]
@@ -1126,6 +1099,11 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
stack_size += 8;
sreg_off = stack_size;
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
stack_size += 8;
tcc_off = stack_size;
}
if ((flags & BPF_TRAMP_F_CALL_ORIG) && (nr_arg_slots - RV_MAX_REG_ARGS > 0))
stack_size += (nr_arg_slots - RV_MAX_REG_ARGS) * 8;
@@ -1160,6 +1138,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_addi(RV_REG_FP, RV_REG_SP, stack_size, ctx);
}
/* store tail call count */
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx);
/* callee saved register S1 to pass start time */
emit_sd(RV_REG_FP, -sreg_off, RV_REG_S1, ctx);
@@ -1195,7 +1177,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
if (fmod_ret->nr_nodes) {
branches_off = kzalloc_objs(int, fmod_ret->nr_nodes);
branches_off = kvzalloc_objs(int, fmod_ret->nr_nodes);
if (!branches_off)
return -ENOMEM;
@@ -1218,9 +1200,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
orig_call += RV_FENTRY_NINSNS * 4;
restore_args(min_t(int, nr_arg_slots, RV_MAX_REG_ARGS), args_off, ctx);
restore_stack_args(nr_arg_slots - RV_MAX_REG_ARGS, args_off, stk_arg_off, ctx);
/* restore TCC to RV_REG_TCC before calling the orig bpf func */
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
emit_ld(RV_REG_TCC, -tcc_off, RV_REG_FP, ctx);
ret = emit_call((const u64)orig_call, true, ctx);
if (ret)
goto out;
/* store updated TCC back to stack after calling the orig bpf func */
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
emit_sd(RV_REG_FP, -tcc_off, RV_REG_TCC, ctx);
emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx);
emit_sd(RV_REG_FP, -(retval_off - 8), regmap[BPF_REG_0], ctx);
im->ip_after_call = ctx->ro_insns + ctx->ninsns;
@@ -1273,6 +1261,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_ld(RV_REG_S1, -sreg_off, RV_REG_FP, ctx);
/* restore TCC from stack to RV_REG_TCC */
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
emit_ld(RV_REG_TCC, -tcc_off, RV_REG_FP, ctx);
if (!is_struct_ops) {
/* trampoline called from function entry */
emit_ld(RV_REG_T0, stack_size - 8, RV_REG_SP, ctx);
@@ -1300,7 +1292,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
ret = ctx->ninsns;
out:
kfree(branches_off);
kvfree(branches_off);
return ret;
}
@@ -1831,17 +1823,31 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
for (idx = 0; idx < fm->nr_args; idx++) {
u8 reg = bpf_to_rv_reg(BPF_REG_1 + idx, ctx);
bool sign = fm->arg_flags[idx] & BTF_FMODEL_SIGNED_ARG;
if (fm->arg_size[idx] == sizeof(int))
emit_sextw(reg, reg, ctx);
if (sign_extend(reg, reg, fm->arg_size[idx], sign, ctx))
return -EINVAL;
}
}
/* restore TCC to RV_REG_TCC before bpf2bpf call */
if (aux->tail_call_reachable && insn->src_reg == BPF_PSEUDO_CALL)
emit_ld(RV_REG_TCC, ctx->tcc_offset, RV_REG_SP, ctx);
ret = emit_call(addr, fixed_addr, ctx);
if (ret)
return ret;
if (insn->src_reg != BPF_PSEUDO_CALL)
/* store updated TCC back to stack after bpf2bpf call */
if (aux->tail_call_reachable && insn->src_reg == BPF_PSEUDO_CALL)
emit_sd(RV_REG_SP, ctx->tcc_offset, RV_REG_TCC, ctx);
/*
* arch_bpf_timed_may_goto() is emitted by the verifier and
* returns its result in BPF_REG_AX instead of BPF_REG_0, so
* skip the normal "move return register into R0".
*/
if (insn->src_reg != BPF_PSEUDO_CALL && addr != (u64)arch_bpf_timed_may_goto)
emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_A0, ctx);
break;
}
@@ -1986,7 +1992,21 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
else
ret = emit_atomic_rmw(rd, rs, insn, ctx);
ret = ret ?: add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx);
/* ret can be 1 (skip-zext); extable entry still needs to be added */
if (ret >= 0) {
/*
* A load-acquire reads into dst_reg, and a read-modify-write
* carrying BPF_FETCH reads the old value into src_reg, or into
* r0 for a BPF_CMPXCHG. Clear that register on fault, the
* remaining atomics have no destination register.
*/
int load_reg = bpf_atomic_load_reg(insn);
ret = add_exception_handler(insn, load_reg < 0 ?
REG_DONT_CLEAR_MARKER : regmap[load_reg],
ctx) ?: ret;
}
if (ret)
return ret;
break;
@@ -2020,10 +2040,9 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
stack_adjust += 8;
if (seen_reg(RV_REG_S5, ctx))
stack_adjust += 8;
if (seen_reg(RV_REG_S6, ctx))
stack_adjust += 8;
if (ctx->arena_vm_start)
stack_adjust += 8;
stack_adjust += 8; /* RV_REG_TCC */
stack_adjust = round_up(stack_adjust, STACK_ALIGN);
stack_adjust += bpf_stack_adjust;
@@ -2033,15 +2052,16 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
/* emit kcfi type preamble immediately before the first insn */
emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx);
/* bpf prog starts here as kcfi skipped during prog->bpf_func setup */
/* nops reserved for auipc+jalr pair */
for (i = 0; i < RV_FENTRY_NINSNS; i++)
emit(rv_nop(), ctx);
/* First instruction is always setting the tail-call-counter
* (TCC) register. This instruction is skipped for tail calls.
* Force using a 4-byte (non-compressed) instruction.
*/
emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx);
if (!is_subprog)
emit(rv_addi(RV_REG_TCC, RV_REG_ZERO, MAX_TAIL_CALL_CNT), ctx);
/* tailcall starts here, emit insn before it must be fixed */
emit_addi(RV_REG_SP, RV_REG_SP, -stack_adjust, ctx);
@@ -2071,26 +2091,20 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
emit_sd(RV_REG_SP, store_offset, RV_REG_S5, ctx);
store_offset -= 8;
}
if (seen_reg(RV_REG_S6, ctx)) {
emit_sd(RV_REG_SP, store_offset, RV_REG_S6, ctx);
store_offset -= 8;
}
if (ctx->arena_vm_start) {
emit_sd(RV_REG_SP, store_offset, RV_REG_ARENA, ctx);
store_offset -= 8;
}
/* store TCC from RV_REG_TCC to stack */
emit_sd(RV_REG_SP, store_offset, RV_REG_TCC, ctx);
ctx->tcc_offset = store_offset;
emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
if (bpf_stack_adjust)
emit_addi(RV_REG_S5, RV_REG_SP, bpf_stack_adjust, ctx);
/* Program contains calls and tail calls, so RV_REG_TCC need
* to be saved across calls.
*/
if (seen_tail_call(ctx) && seen_call(ctx))
emit_mv(RV_REG_TCC_SAVED, RV_REG_TCC, ctx);
ctx->stack_size = stack_adjust;
if (ctx->arena_vm_start)
@@ -2157,3 +2171,13 @@ bool bpf_jit_supports_fsession(void)
{
return true;
}
bool bpf_jit_supports_subprog_tailcalls(void)
{
return true;
}
bool bpf_jit_supports_timed_may_goto(void)
{
return true;
}

5
arch/riscv/net/bpf_jit_core.c
View File

@@ -72,7 +72,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
ctx->prog = prog;
ctx->offset = kzalloc_objs(int, prog->len);
ctx->offset = kvzalloc_objs(int, prog->len);
if (!ctx->offset)
goto out_offset;
@@ -170,7 +170,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
ctx->offset[i] = ninsns_rvoff(ctx->offset[i]);
bpf_prog_fill_jited_linfo(prog, ctx->offset);
out_offset:
kfree(ctx->offset);
kvfree(ctx->offset);
kfree(jit_data);
prog->aux->jit_data = NULL;
}
@@ -234,6 +234,7 @@ void bpf_jit_free(struct bpf_prog *prog)
*/
if (jit_data) {
bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header);
kvfree(jit_data->ctx.offset);
kfree(jit_data);
}
hdr = bpf_jit_binary_pack_hdr(prog);

47
arch/riscv/net/bpf_timed_may_goto.S Normal file
View File

@@ -0,0 +1,47 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright (c) 2026 Feng Jiang <jiangfeng@kylinos.cn> */
#include <linux/linkage.h>
#include <asm/asm.h>
/*
* Trampoline for the BPF timed may_goto loop bound. Custom calling convention:
* - input: stack offset in BPF_REG_AX (t0)
* - output: updated count in BPF_REG_AX (t0)
*
* Calls bpf_check_timed_may_goto(ptr) with the standard RISC-V ABI, where
* ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (a5, a0-a4) are saved
* across the call; BPF_REG_FP (s5) is callee-saved and needs no saving.
*/
SYM_FUNC_START(arch_bpf_timed_may_goto)
addi sp, sp, -(8*SZREG)
REG_S ra, 7*SZREG(sp)
REG_S s0, 6*SZREG(sp)
addi s0, sp, 8*SZREG
/* Save BPF registers R0-R5 (a5, a0-a4) */
REG_S a5, 5*SZREG(sp)
REG_S a0, 4*SZREG(sp)
REG_S a1, 3*SZREG(sp)
REG_S a2, 2*SZREG(sp)
REG_S a3, 1*SZREG(sp)
REG_S a4, 0*SZREG(sp)
add a0, t0, s5
call bpf_check_timed_may_goto
mv t0, a0
/* Restore BPF registers R0-R5 */
REG_L a4, 0*SZREG(sp)
REG_L a3, 1*SZREG(sp)
REG_L a2, 2*SZREG(sp)
REG_L a1, 3*SZREG(sp)
REG_L a0, 4*SZREG(sp)
REG_L a5, 5*SZREG(sp)
REG_L s0, 6*SZREG(sp)
REG_L ra, 7*SZREG(sp)
addi sp, sp, 8*SZREG
ret
SYM_FUNC_END(arch_bpf_timed_may_goto)

213
arch/s390/net/bpf_jit_comp.c
View File

@@ -743,10 +743,12 @@ static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn,
{
if (BPF_MODE(insn->code) != BPF_PROBE_MEM &&
BPF_MODE(insn->code) != BPF_PROBE_MEMSX &&
BPF_MODE(insn->code) != BPF_PROBE_MEM32)
BPF_MODE(insn->code) != BPF_PROBE_MEM32 &&
BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return;
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32) {
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32 ||
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) {
/* lgrl %r1,kern_arena */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, jit->kern_arena);
probe->arena_reg = REG_W1;
@@ -758,7 +760,8 @@ static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn,
static void bpf_jit_probe_store_pre(struct bpf_jit *jit, struct bpf_insn *insn,
struct bpf_jit_probe *probe)
{
if (BPF_MODE(insn->code) != BPF_PROBE_MEM32)
if (BPF_MODE(insn->code) != BPF_PROBE_MEM32 &&
BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return;
/* lgrl %r1,kern_arena */
@@ -771,6 +774,8 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit,
struct bpf_insn *insn,
struct bpf_jit_probe *probe)
{
int load_reg;
if (BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return;
@@ -780,6 +785,14 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit,
EMIT4(0xb9080000, REG_W1, insn->dst_reg);
probe->arena_reg = REG_W1;
probe->prg = jit->prg;
/*
* A read-modify-write carrying BPF_FETCH reads the old value into
* src_reg, or into r0 for a BPF_CMPXCHG. Clear that register on
* fault, the remaining atomics only write memory.
*/
load_reg = bpf_atomic_load_reg(insn);
if (load_reg >= 0)
probe->reg = reg2hex[load_reg];
}
static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp,
@@ -830,6 +843,72 @@ static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp,
return 0;
}
static int emit_ldx(struct bpf_jit *jit, struct bpf_prog *fp, struct bpf_insn *insn)
{
struct bpf_jit_probe probe;
bpf_jit_probe_init(&probe);
bpf_jit_probe_load_pre(jit, insn, &probe);
switch (BPF_SIZE(insn->code)) {
case BPF_B: /* dst = *(u8 *)(ul) (src + off) */
/* llgc %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0090, insn->dst_reg, insn->src_reg,
probe.arena_reg, insn->off);
break;
case BPF_H: /* dst = *(u16 *)(ul) (src + off) */
/* llgh %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0091, insn->dst_reg, insn->src_reg,
probe.arena_reg, insn->off);
break;
case BPF_W: /* dst = *(u32 *)(ul) (src + off) */
/* llgf %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0016, insn->dst_reg, insn->src_reg,
probe.arena_reg, insn->off);
break;
case BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
/* lg %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0004, insn->dst_reg, insn->src_reg,
probe.arena_reg, insn->off);
break;
}
return bpf_jit_probe_post(jit, fp, &probe);
}
static int emit_stx(struct bpf_jit *jit, struct bpf_prog *fp, struct bpf_insn *insn)
{
struct bpf_jit_probe probe;
bpf_jit_probe_init(&probe);
bpf_jit_probe_store_pre(jit, insn, &probe);
switch (BPF_SIZE(insn->code)) {
case BPF_B: /* *(u8 *)(dst + off) = src_reg */
/* stcy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0072, insn->src_reg, insn->dst_reg,
probe.arena_reg, insn->off);
break;
case BPF_H: /* (u16 *)(dst + off) = src */
/* sthy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0070, insn->src_reg, insn->dst_reg,
probe.arena_reg, insn->off);
break;
case BPF_W: /* *(u32 *)(dst + off) = src */
/* sty %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0050, insn->src_reg, insn->dst_reg,
probe.arena_reg, insn->off);
break;
case BPF_DW: /* (u64 *)(dst + off) = src */
/* stg %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0024, insn->src_reg, insn->dst_reg,
probe.arena_reg, insn->off);
break;
}
return bpf_jit_probe_post(jit, fp, &probe);
}
/*
* Sign- or zero-extend the register if necessary
*/
@@ -1477,44 +1556,13 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
*/
case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src_reg */
case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stcy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0072, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sthy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0070, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sty %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0050, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
case BPF_STX | BPF_PROBE_MEM32 | BPF_DW:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stg %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0024, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
err = emit_stx(jit, fp, insn);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
@@ -1574,19 +1622,23 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
/*
* BPF_ATOMIC
*/
case BPF_STX | BPF_ATOMIC | BPF_B:
case BPF_STX | BPF_ATOMIC | BPF_H:
case BPF_STX | BPF_ATOMIC | BPF_DW:
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_B:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_H:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_W:
{
bool is32 = BPF_SIZE(insn->code) == BPF_W;
/*
* Unlike loads and stores, atomics have only a base register,
* but no index register. For the non-arena case, simply use
* %dst as a base. For the arena case, use the work register
* %r1: first, load the arena base into it, and then add %dst
* to it.
* Unlike loads and stores, s390 atomics have only a base
* register, but no index register. For the non-arena case,
* simply use %dst as a base. For the arena case, use the
* work register %r1: first, load the arena base into it,
* and then add %dst to it.
*/
probe.arena_reg = dst_reg;
@@ -1642,6 +1694,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
if (load_probe.prg != -1) {
probe.prg = jit->prg;
probe.arena_reg = load_probe.arena_reg;
probe.reg = load_probe.reg;
}
loop_start = jit->prg;
/* 0: {csy|csg} %w0,%src,off(%arena) */
@@ -1673,6 +1726,18 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
if (err < 0)
return err;
break;
case BPF_LOAD_ACQ:
/* s390 has strong ordering, just use load */
err = emit_ldx(jit, fp, insn);
if (err < 0)
return err;
break;
case BPF_STORE_REL:
/* s390 has strong ordering, just use store */
err = emit_stx(jit, fp, insn);
if (err < 0)
return err;
break;
default:
pr_err("Unknown atomic operation %02x\n", insn->imm);
return -1;
@@ -1687,15 +1752,20 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_B:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_B:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgc %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_W:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:
err = emit_ldx(jit, fp, insn);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
if (insn_is_zext(&insn[1]))
if (BPF_SIZE(insn->code) != BPF_DW && insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_B: /* dst = *(s8 *)(ul) (src + off) */
@@ -1708,20 +1778,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgh %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0091, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
if (insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_H: /* dst = *(s16 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
bpf_jit_probe_load_pre(jit, insn, &probe);
@@ -1732,20 +1788,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_W:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgf %dst,off(%src) */
jit->seen |= SEEN_MEM;
EMIT6_DISP_LH(0xe3000000, 0x0016, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
if (insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_W: /* dst = *(s32 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
bpf_jit_probe_load_pre(jit, insn, &probe);
@@ -1756,18 +1798,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
if (err < 0)
return err;
break;
case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* lg %dst,off(%src,%arena) */
jit->seen |= SEEN_MEM;
EMIT6_DISP_LH(0xe3000000, 0x0004, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
break;
/*
* BPF_JMP / CALL
*/
@@ -1783,8 +1813,8 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
insn->imm == BPF_FUNC_get_smp_processor_id) {
const u32 *cpu_nr = &get_lowcore()->cpu_nr;
/* ly %b0, cpu_nr */
EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0,
/* llgf %b0, cpu_nr */
EMIT6_DISP_LH(0xe3000000, 0x0016, BPF_REG_0, REG_0, REG_0,
(unsigned long)cpu_nr);
break;
}
@@ -3028,13 +3058,6 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
if (!in_arena)
return true;
switch (insn->code) {
case BPF_STX | BPF_ATOMIC | BPF_B:
case BPF_STX | BPF_ATOMIC | BPF_H:
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_ATOMIC | BPF_DW:
if (bpf_atomic_is_load_store(insn))
return false;
break;
case BPF_LDX | BPF_MEMSX | BPF_B:
case BPF_LDX | BPF_MEMSX | BPF_H:
case BPF_LDX | BPF_MEMSX | BPF_W:

4
arch/x86/mm/init.c
View File

@@ -1107,10 +1107,10 @@ struct execmem_info __init *execmem_arch_setup(void)
.alignment = MODULE_ALIGN,
},
[EXECMEM_BPF] = {
.flags = EXECMEM_KASAN_SHADOW,
.flags = flags,
.start = start,
.end = MODULES_END,
.pgprot = PAGE_KERNEL,
.pgprot = pgprot,
.alignment = MODULE_ALIGN,
},
[EXECMEM_MODULE_DATA] = {

178
arch/x86/net/bpf_jit_comp.c
View File

@@ -1473,17 +1473,20 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
*
* Bit layout of `fixup` (32-bit):
*
* +-----------+--------+-----------+---------+----------+
* | 31 | 30-24 | 23-16 | 15-8 | 7-0 |
* | | | | | |
* | ARENA_ACC | Unused | ARENA_REG | DST_REG | INSN_LEN |
* +-----------+--------+-----------+---------+----------+
* +-----------+-------------+--------+-----------+---------+----------+
* | 31 | 30 | 29-24 | 23-16 | 15-8 | 7-0 |
* | | | | | | |
* | ARENA_ACC | ARENA_WRITE | Unused | ARENA_REG | DST_REG | INSN_LEN |
* +-----------+-------------+--------+-----------+---------+----------+
*
* - INSN_LEN (8 bits): Length of faulting insn (max x86 insn = 15 bytes (fits in 8 bits)).
* - DST_REG (8 bits): Offset of dst_reg from reg2pt_regs[] (max offset = 112 (fits in 8 bits)).
* This is set to DONT_CLEAR if the insn is a store.
* This is set to DONT_CLEAR if the insn does not read into a register.
* - ARENA_REG (8 bits): Offset of the register that is used to calculate the
* address for load/store when accessing the arena region.
* - ARENA_WRITE (1 bit): This bit is set when the faulting instruction wrote to the arena region.
* It is independent of DST_REG, since a read-modify-write both writes to
* memory and reads the old value into a register.
* - ARENA_ACCESS (1 bit): This bit is set when the faulting instruction accessed the arena region.
*
* Bit layout of `data` (32-bit):
@@ -1502,6 +1505,7 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
#define FIXUP_INSN_LEN_MASK GENMASK(7, 0)
#define FIXUP_REG_MASK GENMASK(15, 8)
#define FIXUP_ARENA_REG_MASK GENMASK(23, 16)
#define FIXUP_ARENA_WRITE BIT(30)
#define FIXUP_ARENA_ACCESS BIT(31)
#define DATA_ARENA_OFFSET_MASK GENMASK(31, 16)
@@ -1510,7 +1514,7 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
u32 reg = FIELD_GET(FIXUP_REG_MASK, x->fixup);
u32 insn_len = FIELD_GET(FIXUP_INSN_LEN_MASK, x->fixup);
bool is_arena = !!(x->fixup & FIXUP_ARENA_ACCESS);
bool is_write = (reg == DONT_CLEAR);
bool is_write = !!(x->fixup & FIXUP_ARENA_WRITE);
unsigned long addr;
s16 off;
u32 arena_reg;
@@ -1678,6 +1682,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
return 0;
}
/*
* Rebase the __arena args of a kfunc call to arena kernel addresses,
* rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
* arg preserves NULL by skipping the add, tested on the truncated value as
* arena NULL is offset 0. Return the number of emitted bytes.
*/
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
const struct bpf_insn *insn, u8 **pprog)
{
const struct btf_func_model *fm;
u8 *prog = *pprog;
u8 *start = prog;
int i;
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
return -EINVAL;
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
u8 flags = fm->arg_flags[i];
u32 reg = BPF_REG_1 + i;
if (!(flags & BTF_FMODEL_ARENA_ARG))
continue;
if (WARN_ON_ONCE(!bpf_prog->aux->arena))
return -EINVAL;
/* mov eN, eN: truncate and clear the upper 32 bits */
emit_mov_reg(&prog, false, reg, reg);
if (flags & BTF_FMODEL_NULLABLE_ARG) {
/* test eN, eN; jz over the 3-byte add */
maybe_emit_mod(&prog, reg, reg, false);
EMIT2(0x85, add_2reg(0xC0, reg, reg));
EMIT2(X86_JE, 3);
}
/* add rN, r12 */
maybe_emit_mod(&prog, reg, X86_REG_R12, true);
EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
}
*pprog = prog;
return prog - start;
}
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -1887,7 +1935,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
EMIT_mov(dst_reg, src_reg);
#ifdef CONFIG_SMP
/* add <dst>, gs:[<off>] */
EMIT2(0x65, add_1mod(0x48, dst_reg));
EMIT2(0x65, add_2mod(0x48, 0, dst_reg));
EMIT3(0x03, add_2reg(0x04, 0, dst_reg), 0x25);
EMIT((u32)(unsigned long)&this_cpu_off, 4);
#endif
@@ -2304,6 +2352,7 @@ st: insn_off = insn->off;
struct exception_table_entry *ex;
u8 *_insn = image + proglen + (start_of_ldx - temp);
u32 arena_reg, fixup_reg;
bool is_write;
s64 delta;
if (!bpf_prog->aux->extable)
@@ -2331,19 +2380,38 @@ st: insn_off = insn->off;
* BPF_PROBE_ATOMIC) before being used for the memory access. Pass
* the reg holding the unmodified 32-bit address to
* ex_handler_bpf().
*
* A load-acquire is of BPF_STX class, but reads from src_reg
* into dst_reg like a BPF_LDX does, hence it must not be
* treated as a store here.
*/
if (BPF_CLASS(insn->code) == BPF_LDX) {
if (BPF_CLASS(insn->code) == BPF_LDX ||
bpf_atomic_is_load_acq(insn)) {
arena_reg = reg2pt_regs[src_reg];
fixup_reg = reg2pt_regs[dst_reg];
is_write = false;
} else {
/*
* A store has no destination register to clear,
* except for a read-modify-write with BPF_FETCH,
* which also reads the old value into src_reg, or
* into r0 for a BPF_CMPXCHG. Either way the access
* is still reported as a write.
*/
int load_reg = bpf_atomic_load_reg(insn);
arena_reg = reg2pt_regs[dst_reg];
fixup_reg = DONT_CLEAR;
fixup_reg = load_reg < 0 ? DONT_CLEAR :
reg2pt_regs[load_reg];
is_write = true;
}
ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |
FIELD_PREP(FIXUP_REG_MASK, fixup_reg);
ex->fixup |= FIXUP_ARENA_ACCESS;
if (is_write)
ex->fixup |= FIXUP_ARENA_WRITE;
ex->data |= FIELD_PREP(DATA_ARENA_OFFSET_MASK, insn->off);
}
@@ -2583,6 +2651,12 @@ st: insn_off = insn->off;
}
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
if (err < 0)
return err;
ip += err;
}
if (priv_frame_ptr) {
push_r9(&prog);
ip += 2;
@@ -2993,11 +3067,39 @@ static int get_nr_used_regs(const struct btf_func_model *m)
return nr_used_regs;
}
/*
* Convert an arena kernel address into the arena pointer form on its way
* into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg
* preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit
* subtraction both truncates and clears the upper half, so the stored
* value satisfies the JIT invariant for arena pointer registers.
*/
static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
{
u8 *prog = *pprog;
if (nullable) {
if (src_reg != BPF_REG_0)
emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
/* test rax, rax; jz over the 5-byte sub */
EMIT3(0x48, 0x85, 0xC0);
EMIT2(X86_JE, 5);
} else if (src_reg != BPF_REG_0) {
emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
}
/* sub eax, base_lo */
EMIT1_off32(0x2D, base_lo);
*pprog = prog;
}
static void save_args(const struct btf_func_model *m, u8 **prog,
int stack_size, bool for_call_origin, u32 flags)
int stack_size, bool for_call_origin, u32 flags,
u64 arena_base)
{
int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
bool use_jmp = bpf_trampoline_use_jmp(flags);
int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24;
int i, j;
/* Store function arguments to stack.
@@ -3006,6 +3108,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
* mov QWORD PTR [rbp-0x8],rsi
*/
for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) {
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
arg_regs = (m->arg_size[i] + 7) / 8;
/* According to the research of Yonghong, struct members
@@ -3029,16 +3134,19 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
/* copy function arguments from origin stack frame
* into current stack frame.
*
* The starting address of the arguments on-stack
* is:
* rbp + 8(push rbp) +
* 8(return addr of origin call) +
* 8(return addr of the caller)
* which means: rbp + 24
* The arguments on-stack start above the saved rbp
* and the return addresses: two return addresses
* (origin call and caller) when the trampoline is
* entered through the fentry call, so rbp + 24, and
* a single one when it is entered with a jmp or
* called indirectly, so rbp + 16.
*/
for (j = 0; j < arg_regs; j++) {
emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
nr_stack_slots * 8 + 16 + (!use_jmp) * 8);
nr_stack_slots * 8 + stack_args_off);
if (arena_arg)
emit_arena_arg_conv(prog, BPF_REG_0, nullable,
(u32)arena_base);
emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
-stack_size);
@@ -3059,9 +3167,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
/* copy the arguments from regs into stack */
for (j = 0; j < arg_regs; j++) {
emit_stx(prog, BPF_DW, BPF_REG_FP,
nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
-stack_size);
u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
if (arena_arg) {
emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
src = BPF_REG_0;
}
emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
stack_size -= 8;
nr_regs++;
}
@@ -3357,6 +3469,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
void *orig_call = func_addr;
int cookie_off, cookie_cnt;
u8 **branches = NULL;
u64 arena_base;
u64 func_meta;
u8 *prog;
bool save_ret;
@@ -3369,11 +3482,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) &&
(flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET)));
/* extra registers for struct arguments */
for (i = 0; i < m->nr_args; i++) {
if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)
nr_regs += (m->arg_size[i] + 7) / 8 - 1;
}
arena_base = bpf_tramp_arena_base(m, tnodes, flags);
for (i = 0; i < m->nr_args; i++)
nr_regs += (m->arg_size[i] + 7) / 8 - 1;
/* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6
* are passed through regs, the remains are through stack.
@@ -3506,7 +3618,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
}
save_args(m, &prog, regs_off, false, flags);
save_args(m, &prog, regs_off, false, flags, arena_base);
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* arg1: mov rdi, im */
@@ -3548,7 +3660,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
if (flags & BPF_TRAMP_F_CALL_ORIG) {
restore_regs(m, &prog, regs_off);
save_args(m, &prog, arg_stack_off, true, flags);
save_args(m, &prog, arg_stack_off, true, flags, 0);
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
/* Before calling the original function, load the
@@ -3703,13 +3815,12 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
int ret;
/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
* This will NOT cause fragmentation in direct map, as we do not
* call set_memory_*() on this buffer.
*
* We cannot use kvmalloc here, because we need image to be in
* module memory range.
* Since it must be writable use bpf_jit_alloc_exec_rw().
*/
image = bpf_jit_alloc_exec(PAGE_SIZE);
image = bpf_jit_alloc_exec_rw(PAGE_SIZE);
if (!image)
return -ENOMEM;
@@ -4050,6 +4161,11 @@ bool bpf_jit_supports_stack_args(void)
return true;
}
bool bpf_jit_supports_arena_args(void)
{
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (text_poke_copy(dst, src, len) == NULL)

10
drivers/hid/bpf/hid_bpf_dispatch.c
View File

@@ -590,11 +590,11 @@ static const struct btf_kfunc_id_set hid_bpf_kfunc_set = {
/* for syscall HID-BPF */
BTF_KFUNCS_START(hid_bpf_syscall_kfunc_ids)
BTF_ID_FLAGS(func, hid_bpf_allocate_context, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, hid_bpf_release_context, KF_RELEASE)
BTF_ID_FLAGS(func, hid_bpf_hw_request)
BTF_ID_FLAGS(func, hid_bpf_hw_output_report)
BTF_ID_FLAGS(func, hid_bpf_input_report)
BTF_ID_FLAGS(func, hid_bpf_allocate_context, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, hid_bpf_release_context, KF_RELEASE | KF_SLEEPABLE)
BTF_ID_FLAGS(func, hid_bpf_hw_request, KF_SLEEPABLE)
BTF_ID_FLAGS(func, hid_bpf_hw_output_report, KF_SLEEPABLE)
BTF_ID_FLAGS(func, hid_bpf_input_report, KF_SLEEPABLE)
BTF_KFUNCS_END(hid_bpf_syscall_kfunc_ids)
static const struct btf_kfunc_id_set hid_bpf_syscall_kfunc_set = {

123
include/linux/bpf.h
View File

@@ -145,6 +145,7 @@ struct bpf_map_ops {
int (*map_direct_value_meta)(const struct bpf_map *map,
u64 imm, u32 *off);
int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
struct poll_table_struct *pts);
unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
@@ -299,7 +300,6 @@ struct bpf_map_owner {
struct bpf_map {
u8 sha[SHA256_DIGEST_SIZE];
u32 excl;
const struct bpf_map_ops *ops;
struct bpf_map *inner_map_meta;
#ifdef CONFIG_SECURITY
@@ -570,7 +570,7 @@ static inline void bpf_obj_memcpy(struct btf_record *rec,
if (IS_ERR_OR_NULL(rec)) {
if (long_memcpy)
bpf_long_memcpy(dst, src, round_up(size, 8));
bpf_long_memcpy(dst, src, size);
else
memcpy(dst, src, size);
return;
@@ -593,7 +593,7 @@ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, true);
bpf_obj_memcpy(map->record, dst, src, round_up(map->value_size, 8), true);
}
static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src)
@@ -889,8 +889,8 @@ enum bpf_arg_type {
ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */
ARG_PTR_TO_ARENA,
ARG_CONST_SIZE, /* number of bytes accessed from memory */
ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */
ARG_MEM_SIZE, /* number of bytes accessed from memory */
ARG_MEM_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */
ARG_PTR_TO_CTX, /* pointer to context */
ARG_ANYTHING, /* any (initialized) argument is ok */
@@ -961,6 +961,21 @@ enum bpf_return_type {
};
static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
* Also reuse this macro for maximum number of arguments a BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots. The JIT may map some stack arg slots to registers based
* on the native calling convention (e.g., arg 6 to R9 on x86-64).
*/
#define MAX_BPF_FUNC_ARGS 12
/* The maximum number of arguments passed through registers
* a single function may have.
*/
#define MAX_BPF_FUNC_REG_ARGS 5
/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
* to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
* instructions after verifying
@@ -985,7 +1000,7 @@ struct bpf_func_proto {
enum bpf_arg_type arg4_type;
enum bpf_arg_type arg5_type;
};
enum bpf_arg_type arg_type[5];
enum bpf_arg_type arg_type[MAX_BPF_FUNC_ARGS];
};
union {
struct {
@@ -995,7 +1010,7 @@ struct bpf_func_proto {
u32 *arg4_btf_id;
u32 *arg5_btf_id;
};
u32 *arg_btf_id[5];
u32 *arg_btf_id[MAX_BPF_FUNC_ARGS];
struct {
size_t arg1_size;
size_t arg2_size;
@@ -1003,7 +1018,7 @@ struct bpf_func_proto {
size_t arg4_size;
size_t arg5_size;
};
size_t arg_size[5];
size_t arg_size[MAX_BPF_FUNC_ARGS];
};
int *ret_btf_id; /* return value btf_id */
bool (*allowed)(const struct bpf_prog *prog);
@@ -1118,21 +1133,6 @@ static inline bool bpf_pseudo_func(const struct bpf_insn *insn)
return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC;
}
/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
* atomic load or store, and false if it is a read-modify-write instruction.
*/
static inline bool
bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
{
switch (atomic_insn->imm) {
case BPF_LOAD_ACQ:
case BPF_STORE_REL:
return true;
default:
return false;
}
}
struct bpf_prog_ops {
int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
@@ -1193,27 +1193,15 @@ struct bpf_prog_offload {
u32 jited_len;
};
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
* Also reuse this macro for maximum number of arguments a BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots. The JIT may map some stack arg slots to registers based
* on the native calling convention (e.g., arg 6 to R9 on x86-64).
*/
#define MAX_BPF_FUNC_ARGS 12
/* The maximum number of arguments passed through registers
* a single function may have.
*/
#define MAX_BPF_FUNC_REG_ARGS 5
/* The argument is a structure or a union. */
#define BTF_FMODEL_STRUCT_ARG BIT(0)
/* The argument is signed. */
#define BTF_FMODEL_SIGNED_ARG BIT(1)
/* The argument is an arena pointer. */
#define BTF_FMODEL_ARENA_ARG BIT(2)
/* The argument is nullable. */
#define BTF_FMODEL_NULLABLE_ARG BIT(3)
struct btf_func_model {
u8 ret_size;
u8 ret_flags;
@@ -1287,6 +1275,15 @@ struct bpf_tramp_nodes {
int nr_nodes;
};
/*
* The arena base against which a struct_ops trampoline converts the
* arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF
* ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline
* converts nothing.
*/
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
struct bpf_tramp_nodes *tnodes, u32 flags);
struct bpf_tramp_run_ctx;
/* Different use cases for BPF trampoline:
@@ -1522,8 +1519,9 @@ int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_func
int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link);
int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link);
void bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link);
void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags);
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
@@ -1642,11 +1640,11 @@ static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link)
static inline void bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link)
{
return -ENOTSUPP;
}
static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {}
#endif
struct bpf_func_info_aux {
@@ -1865,8 +1863,9 @@ struct bpf_prog_aux {
struct bpf_prog {
u16 pages; /* Number of allocated pages */
u16 jited:1, /* Is our filter JIT'ed? */
u32 jited:1, /* Is our filter JIT'ed? */
jit_requested:1,/* archs need to JIT the prog */
jit_required:1, /* program strictly requires JIT compiler */
gpl_compatible:1, /* Is filter GPL compatible? */
cb_access:1, /* Is control block accessed? */
dst_needed:1, /* Do we need dst entry? */
@@ -2196,6 +2195,12 @@ static inline bool is_tracing_multi(enum bpf_attach_type type)
type == BPF_TRACE_FSESSION_MULTI;
}
static inline bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
{
return fentry_nodes->nr_nodes == 1 &&
fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
}
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
/* This macro helps developer to register a struct_ops type and generate
* type information correctly. Developers should use this macro to register
@@ -3146,7 +3151,7 @@ int btf_struct_access(struct bpf_verifier_log *log,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
bool strict);
bool strict, bool walk_flex_arrays);
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
@@ -3170,7 +3175,6 @@ const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id,
const struct bpf_prog *prog);
void bpf_task_storage_free(struct task_struct *task);
void bpf_cgrp_storage_free(struct cgroup *cgroup);
bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);
const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn);
@@ -3509,11 +3513,6 @@ static inline void bpf_task_storage_free(struct task_struct *task)
{
}
static inline bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog)
{
return false;
}
static inline const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn)
@@ -4148,8 +4147,16 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog)
}
const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off);
void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo,
const char **filep, const char **linep, int *nump);
struct bpf_linfo_source {
const char *file;
const char *line;
u32 file_name_off;
int line_num;
int line_col;
};
void bpf_get_linfo_source(struct btf *btf, const struct bpf_line_info *linfo,
struct bpf_linfo_source *src);
int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep,
const char **linep, int *nump);
struct bpf_prog *bpf_prog_find_from_stack(void);
@@ -4169,7 +4176,7 @@ bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
}
#endif
static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)
static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type)
{
switch (map_type) {
case BPF_MAP_TYPE_PERCPU_ARRAY:
@@ -4196,7 +4203,7 @@ static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 all
return -EINVAL;
if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
if (!bpf_map_supports_cpu_flags(map->map_type))
if (!bpf_map_is_percpu_map(map->map_type))
return -EINVAL;
if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
return -EINVAL;

36
include/linux/bpf_ksock.h Normal file
View File

@@ -0,0 +1,36 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2026 Isovalent */
#ifndef _BPF_KSOCK_H
#define _BPF_KSOCK_H
#include <linux/types.h>
#include <linux/in.h>
#include <linux/in6.h>
/**
* struct bpf_ksock_create_opts - BPF kernel socket creation parameters
* @family: Address family: AF_INET or AF_INET6.
* @type: Socket type: only SOCK_DGRAM supported for now.
* @protocol: Protocol number (e.g. IPPROTO_UDP), or 0 for the default protocol
* of the given type.
* @reserved: Must be zero. Reserved for future use.
*/
struct bpf_ksock_create_opts {
__u8 family;
__u8 type;
__u8 protocol;
__u8 reserved;
};
/**
* union bpf_ksock_addr - IPv4 or IPv6 socket address
* @sin: IPv4 socket address.
* @sin6: IPv6 socket address.
*/
union bpf_ksock_addr {
struct sockaddr_in sin;
struct sockaddr_in6 sin6;
};
#endif /* _BPF_KSOCK_H */

173
include/linux/bpf_verifier.h
View File

@@ -15,7 +15,7 @@
* ensures that umax_value + (int)off + (int)size cannot overflow a u64.
*/
#define BPF_MAX_VAR_OFF (1 << 29)
/* Maximum variable size permitted for ARG_CONST_SIZE[_OR_ZERO]. This ensures
/* Maximum variable size permitted for ARG_MEM_SIZE[_OR_ZERO]. This ensures
* that converting umax_value to int cannot overflow.
*/
#define BPF_MAX_VAR_SIZ (1 << 29)
@@ -162,11 +162,6 @@ struct bpf_reg_state {
* pointing to bpf_func_state.
*/
u32 frameno;
/* Tracks subreg definition. The stored value is the insn_idx of the
* writing insn. This is safe because subreg_def is used before any insn
* patching which only happens after main verification finished.
*/
s32 subreg_def;
/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
bool precise;
};
@@ -359,6 +354,11 @@ struct bpf_func_state {
* 0 = main function, 1 = first callee.
*/
u32 frameno;
/*
* Unique diagnostic identity for this function invocation. Frame depth is
* reused after returns, while this ID is preserved across state clones.
*/
u32 diag_frame_id;
/* subprog number == index within subprog_info
* zero == main subprog
*/
@@ -385,6 +385,8 @@ struct bpf_func_state {
* | number of simulations is tracked in frame N
*/
u32 callback_depth;
/* Instructions processed in this frame and callees on the current path. */
u32 insns_subtotal;
/* The following fields should be last. See copy_func_state() */
/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -803,7 +805,8 @@ struct bpf_subprog_info {
u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
u32 insn_processed;
u32 insns_total;
u32 insns_self;
/* offsets in range [stack_depth .. fastcall_stack_off)
* are used for bpf_fastcall spills and fills.
*/
@@ -835,6 +838,7 @@ static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
return 0;
}
struct bpf_diag;
struct bpf_verifier_env;
struct backtrack_state {
@@ -898,6 +902,14 @@ struct bpf_scc_info {
struct bpf_liveness;
struct bpf_fd_array {
union {
struct bpf_map *map;
struct btf *btf;
unsigned long val;
};
};
/* single container for all structs
* one verifier_env per bpf_check() call
*/
@@ -939,9 +951,12 @@ struct bpf_verifier_env {
bool bypass_spec_v4;
bool seen_direct_write;
bool seen_exception;
bool signature;
u32 insn_aux_data_len;
struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */
const struct bpf_line_info *prev_linfo;
struct bpf_verifier_log log;
struct bpf_diag *diag;
struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */
/* subprog indices sorted in topological order: leaves first, callers last */
int subprog_topo_order[BPF_MAX_SUBPROGS + 2];
@@ -989,7 +1004,19 @@ struct bpf_verifier_env {
u32 free_list_size;
u32 explored_states_size;
u32 num_backedges;
bpfptr_t fd_array;
/*
* The program's fd_array comes in two shapes, told apart by whether
* the caller passed fd_array_cnt. They are mutually exclusive:
* - continuous (fd_array_cnt given): ->fd_array holds every entry
* resolved to its object up front, indexed by fd_array position,
* with ->fd_array_cnt slots; ->fd_array_raw is unused.
* - sparse (no fd_array_cnt): ->fd_array is NULL, and entries are
* read from ->fd_array_raw (the caller's fd_array) and resolved
* on the spot at each reference.
*/
struct bpf_fd_array *fd_array;
u32 fd_array_cnt;
bpfptr_t fd_array_raw;
/* bit mask to keep track of whether a register has been accessed
* since the last time the function state was printed
@@ -1156,8 +1183,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
*btf_id = key & 0x7FFFFFFF;
}
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
int bpf_check_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
@@ -1243,6 +1270,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame,
bt->stack_masks[frame] |= 1ull << slot;
}
static inline void bpf_bt_set_frame_slot_mask(struct backtrack_state *bt, u32 frame, u64 mask)
{
bt->stack_masks[frame] |= mask;
}
static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
bt->stack_arg_masks[frame] |= 1 << slot;
@@ -1276,7 +1308,37 @@ static inline u32 type_flag(u32 type)
return type & ~BPF_BASE_TYPE_MASK;
}
/* only use after check_attach_btf_id() */
static inline bool bpf_is_ptr_to_mem_or_btf_id(enum bpf_reg_type type)
{
switch (base_type(type)) {
case PTR_TO_MEM:
case PTR_TO_BTF_ID:
return true;
default:
return false;
}
}
static inline bool bpf_may_fault_on_deref(enum bpf_reg_type type)
{
/*
* The pointer types which must not be dereferenced without fault
* protection, that is, the ones bpf_convert_ctx_accesses() has to
* turn a BPF_LDX into a BPF_PROBE_MEM one for.
*/
return type == PTR_TO_BTF_ID || (type_flag(type) & PTR_UNTRUSTED);
}
static inline bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
{
int i;
for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
return true;
return false;
}
static inline enum bpf_prog_type resolve_prog_type(const struct bpf_prog *prog)
{
return (prog->type == BPF_PROG_TYPE_EXT && prog->aux->saved_dst_prog_type) ?
@@ -1315,6 +1377,18 @@ static inline bool type_is_non_owning_ref(u32 type)
return type_is_ptr_alloc_obj(type) && type_flag(type) & NON_OWN_REF;
}
static inline bool type_is_map_ptr(enum bpf_reg_type type)
{
switch (base_type(type)) {
case CONST_PTR_TO_MAP:
case PTR_TO_MAP_KEY:
case PTR_TO_MAP_VALUE:
return true;
default:
return false;
}
}
static inline bool type_is_pkt_pointer(enum bpf_reg_type type)
{
type = base_type(type);
@@ -1399,8 +1473,10 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate,
u32 frameno);
u32 bpf_vlog_alignment(u32 pos);
const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn);
struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
@@ -1438,23 +1514,46 @@ struct ref_obj_desc {
u8 cnt;
};
struct bpf_kfunc_call_arg_meta {
/* In parameters */
/*
* A memory argument a call fills in. The verifier allows the stack to be uninitialized if
* the range is a known constant. Stack slots are marked as STACK_MISC by check_mem_access().
*/
struct arg_raw_mem_desc {
u8 regno;
int size;
};
/* Size of PTR_TO_MEM returned, taken from a constant allocation-size argument */
struct ret_mem_desc {
u32 size;
bool found;
};
/* A constant scalar argument; Populated by process_const_arg() */
struct arg_constant_desc {
u64 value;
bool found;
};
struct bpf_call_arg_meta {
/* Common */
struct btf *btf;
u32 func_id;
const struct bpf_func_proto *fn;
u8 release_regno;
u32 ret_btf_id;
u32 subprogno;
struct bpf_map_desc map;
struct bpf_dynptr_desc dynptr;
struct ref_obj_desc ref_obj;
struct ret_mem_desc ret_mem;
/* Only set by kfunc */
bool r0_rdonly;
u32 kfunc_flags;
const struct btf_type *func_proto;
const char *func_name;
/* Out parameters */
u8 release_regno;
bool r0_rdonly;
u32 ret_btf_id;
u64 r0_size;
u32 subprogno;
struct {
u64 value;
bool found;
} arg_constant;
struct arg_constant_desc arg_constant;
/* arg_{btf,btf_id,owning_ref} are used by kfunc-specific handling,
* generally to pass info about user-defined local kptr types to later
@@ -1481,28 +1580,31 @@ struct bpf_kfunc_call_arg_meta {
u8 spi;
u8 frameno;
} iter;
struct bpf_map_desc map;
struct bpf_dynptr_desc dynptr;
struct ref_obj_desc ref_obj;
u64 mem_size;
/* Only set by helper */
u64 msize_max_value;
s64 const_map_key;
struct btf *ret_btf;
struct btf_field *kptr_field;
struct arg_raw_mem_desc arg_raw_mem;
};
int bpf_get_helper_proto(struct bpf_verifier_env *env, int func_id,
const struct bpf_func_proto **ptr);
int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env, s32 func_id,
s16 offset, struct bpf_kfunc_call_arg_meta *meta);
s16 offset, struct bpf_call_arg_meta *meta);
bool bpf_is_async_callback_calling_insn(struct bpf_insn *insn);
bool bpf_is_sync_callback_calling_insn(struct bpf_insn *insn);
static inline bool bpf_is_iter_next_kfunc(struct bpf_kfunc_call_arg_meta *meta)
static inline bool bpf_is_iter_next_kfunc(struct bpf_call_arg_meta *meta)
{
return meta->kfunc_flags & KF_ITER_NEXT;
}
static inline bool bpf_is_kfunc_sleepable(struct bpf_kfunc_call_arg_meta *meta)
static inline bool bpf_is_kfunc_sleepable(struct bpf_call_arg_meta *meta)
{
return meta->kfunc_flags & KF_SLEEPABLE;
}
bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta);
bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta);
struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem);
int bpf_copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off);
bool bpf_insn_is_cond_jump(u8 code);
@@ -1566,6 +1668,7 @@ enum bpf_reg_arg_type {
struct bpf_kfunc_desc {
struct btf_func_model func_model;
struct bpf_func_proto proto;
u32 func_id;
s32 imm;
u16 offset;
@@ -1573,17 +1676,18 @@ struct bpf_kfunc_desc {
};
struct bpf_kfunc_desc_tab {
u32 nr_descs;
/* Sorted by func_id (BTF ID) and offset (fd_array offset) during
* verification. JITs do lookups by bpf_insn, where func_id may not be
* available, therefore at the end of verification do_misc_fixups()
* sorts this by imm and offset.
*
* Grown one entry at a time by bpf_add_kfunc_call().
*/
struct bpf_kfunc_desc descs[MAX_KFUNC_DESCS];
u32 nr_descs;
struct bpf_kfunc_desc descs[];
};
/* Functions exported from verifier.c, used by fixups.c */
bool bpf_is_reg64(struct bpf_insn *insn, u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t);
void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len);
void bpf_mark_subprog_exc_cb(struct bpf_verifier_env *env, int subprog);
bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env);
@@ -1607,5 +1711,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
int bpf_jit_subprogs(struct bpf_verifier_env *env);
int bpf_fixup_call_args(struct bpf_verifier_env *env);
int bpf_do_misc_fixups(struct bpf_verifier_env *env);
int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn);
#endif /* _LINUX_BPF_VERIFIER_H */

2
include/linux/btf.h
View File

@@ -79,6 +79,7 @@
#define KF_ARENA_ARG1 (1 << 14) /* kfunc takes an arena pointer as its first argument */
#define KF_ARENA_ARG2 (1 << 15) /* kfunc takes an arena pointer as its second argument */
#define KF_IMPLICIT_ARGS (1 << 16) /* kfunc has implicit arguments supplied by the verifier */
#define KF_SPINLOCK_SAFE (1 << 17) /* kfunc is allowed inside bpf_spin_lock-ed region */
/*
* Tag marking a kernel function as a kfunc. This is meant to minimize the
@@ -213,6 +214,7 @@ int btf_type_seq_show_flags(const struct btf *btf, u32 type_id, void *obj,
*/
int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
char *buf, int len, u64 flags);
int btf_type_name_to_buf(const struct btf *btf, u32 type_id, char *buf, int len);
int btf_get_fd_by_id(u32 id);
u32 btf_obj_id(const struct btf *btf);

70
include/linux/filter.h
View File

@@ -383,6 +383,61 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn)
/* Legacy alias */
#define BPF_STX_XADD(SIZE, DST, SRC, OFF) BPF_ATOMIC_OP(SIZE, BPF_ADD, DST, SRC, OFF)
/*
* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
* atomic load or store, and false if it is a read-modify-write instruction.
*/
static inline bool
bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
{
switch (atomic_insn->imm) {
case BPF_LOAD_ACQ:
case BPF_STORE_REL:
return true;
default:
return false;
}
}
/*
* A load-acquire is the only BPF_STX class instruction that reads into
* dst_reg from src_reg + off16, i.e. it has the operand roles of a BPF_LDX.
* Unlike bpf_atomic_is_load_store(), @insn is not assumed to be a BPF_ATOMIC
* instruction here, so that callers which walk all instruction classes can
* use this directly.
*/
static inline bool bpf_atomic_is_load_acq(const struct bpf_insn *insn)
{
return BPF_CLASS(insn->code) == BPF_STX &&
(BPF_MODE(insn->code) == BPF_ATOMIC ||
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
insn->imm == BPF_LOAD_ACQ;
}
/*
* Given an instruction @insn, return the number of the BPF register that a
* BPF_ATOMIC reads the value at its memory operand into, or -1 if there is
* no such register. That is the register a BPF_PROBE_ATOMIC has to clear when
* the access faults. Like bpf_atomic_is_load_acq(), @insn is not assumed to
* be a BPF_ATOMIC here.
*/
static inline int bpf_atomic_load_reg(const struct bpf_insn *insn)
{
if (BPF_CLASS(insn->code) != BPF_STX ||
(BPF_MODE(insn->code) != BPF_ATOMIC &&
BPF_MODE(insn->code) != BPF_PROBE_ATOMIC))
return -1;
switch (insn->imm) {
case BPF_LOAD_ACQ:
return insn->dst_reg;
case BPF_CMPXCHG:
return BPF_REG_0;
default:
return (insn->imm & BPF_FETCH) ? insn->src_reg : -1;
}
}
/* Memory store, *(uint *) (dst_reg + off16) = imm32 */
#define BPF_ST_MEM(SIZE, DST, OFF, IMM) \
@@ -1183,6 +1238,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);
@@ -1211,25 +1267,12 @@ struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off,
#ifdef CONFIG_BPF_SYSCALL
struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
const struct bpf_insn *patch, u32 len);
struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env);
void bpf_restore_insn_aux_data(struct bpf_verifier_env *env,
struct bpf_insn_aux_data *orig_insn_aux);
#else
static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
const struct bpf_insn *patch, u32 len)
{
return ERR_PTR(-ENOTSUPP);
}
static inline struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env)
{
return NULL;
}
static inline void bpf_restore_insn_aux_data(struct bpf_verifier_env *env,
struct bpf_insn_aux_data *orig_insn_aux)
{
}
#endif /* CONFIG_BPF_SYSCALL */
int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt);
@@ -1333,6 +1376,7 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr,
void bpf_jit_binary_free(struct bpf_binary_header *hdr);
u64 bpf_jit_alloc_exec_limit(void);
void *bpf_jit_alloc_exec(unsigned long size);
void *bpf_jit_alloc_exec_rw(unsigned long size);
void bpf_jit_free_exec(void *addr);
void bpf_jit_free(struct bpf_prog *fp);
struct bpf_binary_header *

2
include/linux/socket.h
View File

@@ -461,6 +461,8 @@ extern struct file *__sys_socket_file(int family, int type, int protocol);
extern int __sys_bind(int fd, struct sockaddr __user *umyaddr, int addrlen);
extern int __sys_bind_socket(struct socket *sock, struct sockaddr_storage *address,
int addrlen);
int connect_socket(struct socket *sock, struct sockaddr_storage *addr,
int addrlen, int flags);
extern int __sys_connect_file(struct file *file, struct sockaddr_storage *addr,
int addrlen, int file_flags);
extern int __sys_connect(int fd, struct sockaddr __user *uservaddr,

117
include/uapi/linux/bpf.h
View File

@@ -3038,8 +3038,34 @@ union bpf_attr {
*
* * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**,
* **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**:
* Indicate the new IP header version after decapsulating the outer
* IP header. Used when the inner and outer IP versions are different.
* Indicate the new IP header version after decapsulating the
* outer IP header. Used when the inner and outer IP versions
* are different. These flags only trigger a protocol change
* without clearing any tunnel-specific GSO flags.
*
* * **BPF_F_ADJ_ROOM_DECAP_L4_GRE**:
* Clear GRE tunnel GSO flags (SKB_GSO_GRE and SKB_GSO_GRE_CSUM)
* when decapsulating a GRE tunnel.
*
* * **BPF_F_ADJ_ROOM_DECAP_L4_UDP**:
* Clear UDP tunnel GSO flags (SKB_GSO_UDP_TUNNEL and
* SKB_GSO_UDP_TUNNEL_CSUM) when decapsulating a UDP tunnel.
*
* * **BPF_F_ADJ_ROOM_DECAP_IPXIP4**:
* Clear IPIP/SIT tunnel GSO flag (SKB_GSO_IPXIP4) when decapsulating
* a tunnel with an outer IPv4 header (IPv4-in-IPv4 or IPv6-in-IPv4).
*
* * **BPF_F_ADJ_ROOM_DECAP_IPXIP6**:
* Clear IPv6 encapsulation tunnel GSO flag (SKB_GSO_IPXIP6) when
* decapsulating a tunnel with an outer IPv6 header (IPv6-in-IPv6
* or IPv4-in-IPv6).
*
* When using the decapsulation flags above, the skb->encapsulation
* flag is automatically cleared if all tunnel-specific GSO flags
* (SKB_GSO_UDP_TUNNEL, SKB_GSO_UDP_TUNNEL_CSUM, SKB_GSO_GRE,
* SKB_GSO_GRE_CSUM, SKB_GSO_IPXIP4, SKB_GSO_IPXIP6) have been
* removed from the packet. This handles cases where all tunnel
* layers have been decapsulated.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
@@ -3532,6 +3558,47 @@ union bpf_attr {
* Use the mark present in *params*->mark for the fib lookup.
* This option should not be used with BPF_FIB_LOOKUP_DIRECT,
* as it only has meaning for full lookups.
* **BPF_FIB_LOOKUP_VLAN**
* If the fib lookup resolves to a VLAN device whose
* parent is a real (non-VLAN) device, set
* *params*->h_vlan_proto and *params*->h_vlan_TCI from
* the VLAN device and replace *params*->ifindex with the
* parent's ifindex. *params*->h_vlan_TCI carries the VID
* only, with PCP and DEI bits zero; a consumer wanting to
* set egress priority writes PCP itself. *params*->smac is
* the VLAN device's own address, which can differ from the
* parent's. Only the immediate parent is resolved; if it
* is itself a VLAN device (QinQ) or in another namespace,
* the egress cannot be reduced to a physical device plus
* one tag and the lookup returns
* **BPF_FIB_LKUP_RET_VLAN_FAILURE** with *params*->ifindex
* left at the input. To obtain the VLAN device's own
* ifindex, repeat the lookup without
* **BPF_FIB_LOOKUP_VLAN**, re-initializing *params*
* first: output fields overwrite the inputs they share
* storage with. The swap and the vlan fields
* are written only on success; other output fields keep
* the helper's existing behaviour, so a frag-needed result
* still reports the route mtu in *params*->mtu_result.
* This flag is only valid for XDP programs; tc programs
* receive -EINVAL since they can redirect to the VLAN
* device directly.
* **BPF_FIB_LOOKUP_VLAN_INPUT**
* Treat *params*->h_vlan_proto and *params*->h_vlan_TCI
* as an input VLAN tag and run the lookup as if ingress
* had happened on the VLAN subinterface carrying that tag
* on *params*->ifindex. The VID is the low 12 bits of
* *params*->h_vlan_TCI; *params*->h_vlan_proto must be
* ETH_P_8021Q or ETH_P_8021AD in network byte order, else
* **-EINVAL**. If *params*->ifindex is itself a VLAN
* device, its inner (QinQ) subinterface is matched; for a
* bond or team, pass the master's ifindex. An unmatched
* tag, a down device, or one in another namespace returns
* **BPF_FIB_LKUP_RET_NOT_FWDED**, mirroring real ingress.
* A VID of 0 is looked up literally, so do not set this
* flag for priority-tagged frames. Cannot be combined with
* **BPF_FIB_LOOKUP_TBID** or **BPF_FIB_LOOKUP_OUTPUT**
* (returns **-EINVAL**).
*
* *ctx* is either **struct xdp_md** for XDP programs or
* **struct sk_buff** tc cls_act programs.
@@ -4694,6 +4761,7 @@ union bpf_attr {
* * **BPF_RB_RING_SIZE**: The size of ring buffer.
* * **BPF_RB_CONS_POS**: Consumer position (can wrap around).
* * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around).
* * **BPF_RB_OVERWRITE_POS**: Overwrite position (can wrap around).
*
* Data returned is just a momentary snapshot of actual values
* and could be inaccurate, so this facility should be used to
@@ -5079,17 +5147,19 @@ union bpf_attr {
* Description
* Redirect the packet to another net device of index *ifindex*.
* This helper is somewhat similar to **bpf_redirect**\ (), except
* that the redirection happens to the *ifindex*' peer device and
* the netns switch takes place from ingress to ingress without
* going through the CPU's backlog queue.
* that the redirection happens to the *ifindex*' peer device. If
* *flags* is 0, the netns switch takes place from ingress to
* ingress without going through the CPU's backlog queue. If the
* **BPF_F_EGRESS** flag is provided then redirection happens in
* the egress direction of the peer device.
*
* *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during
* the netns switch.
*
* The *flags* argument is reserved and must be 0. The helper is
* currently only supported for tc BPF program types at the
* ingress hook and for veth and netkit target device types. The
* peer device must reside in a different network namespace.
* If the *flags* argument is 0, the helper is currently only
* supported for tc BPF program types at the ingress hook and for
* veth and netkit target device types. The peer device must reside
* in a different network namespace.
* Return
* The helper returns **TC_ACT_REDIRECT** on success or
* **TC_ACT_SHOT** on error.
@@ -6239,7 +6309,7 @@ enum {
};
/* BPF_FUNC_skb_adjust_room flags. */
enum {
enum bpf_adj_room_flags {
BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2),
@@ -6249,6 +6319,10 @@ enum {
BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6),
BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7),
BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8),
BPF_F_ADJ_ROOM_DECAP_L4_GRE = (1ULL << 9),
BPF_F_ADJ_ROOM_DECAP_L4_UDP = (1ULL << 10),
BPF_F_ADJ_ROOM_DECAP_IPXIP4 = (1ULL << 11),
BPF_F_ADJ_ROOM_DECAP_IPXIP6 = (1ULL << 12),
};
enum {
@@ -6336,9 +6410,10 @@ enum {
/* Flags for bpf_redirect and bpf_redirect_map helpers */
enum {
BPF_F_INGRESS = (1ULL << 0), /* used for skb path */
BPF_F_EGRESS = (1ULL << 1), /* used for skb path */
BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */
BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */
#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS)
#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_EGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS)
};
#define __bpf_md_ptr(type, name) \
@@ -6839,6 +6914,15 @@ struct bpf_link_info {
__u32 flags;
__u32 pid;
} uprobe_multi;
struct {
__u32 attach_type;
__u32 count; /* in/out: tracing_multi target count */
__u32 btf_obj_id;
__u32 :32;
__aligned_u64 ids;
__aligned_u64 addrs;
__aligned_u64 cookies;
} tracing_multi;
struct {
__u32 type; /* enum bpf_perf_event_type */
__u32 :32;
@@ -7327,6 +7411,8 @@ enum {
BPF_FIB_LOOKUP_TBID = (1U << 3),
BPF_FIB_LOOKUP_SRC = (1U << 4),
BPF_FIB_LOOKUP_MARK = (1U << 5),
BPF_FIB_LOOKUP_VLAN = (1U << 6),
BPF_FIB_LOOKUP_VLAN_INPUT = (1U << 7),
};
enum {
@@ -7340,6 +7426,7 @@ enum {
BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */
BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */
BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */
BPF_FIB_LKUP_RET_VLAN_FAILURE, /* VLAN egress, parent unresolvable */
};
struct bpf_fib_lookup {
@@ -7393,7 +7480,13 @@ struct bpf_fib_lookup {
union {
struct {
/* output */
/*
* output with BPF_FIB_LOOKUP_VLAN: set from the
* resolved egress VLAN device (see the flag); zeroed
* on other successful lookups. input with
* BPF_FIB_LOOKUP_VLAN_INPUT: the VLAN tag to scope
* the lookup by.
*/
__be16 h_vlan_proto;
__be16 h_vlan_TCI;
};

2
kernel/bpf/Makefile
View File

@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
endif
CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o

43
kernel/bpf/arena.c
View File

@@ -55,8 +55,10 @@ struct bpf_arena {
struct vm_struct *kern_vm;
struct page *scratch_page;
struct range_tree rt;
/* protects rt */
/* protects rt and nr_pages */
rqspinlock_t spinlock;
/* number of pages currently populated in the arena */
u64 nr_pages;
struct list_head vma_list;
/* protects vma_list */
struct mutex lock;
@@ -143,14 +145,14 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)
}
struct apply_range_data {
struct bpf_arena *arena;
struct page **pages;
struct page *scratch_page;
int i;
};
struct clear_range_data {
struct bpf_arena *arena;
struct llist_head *free_pages;
struct page *scratch_page;
};
static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
@@ -180,7 +182,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
if (pte_none(old))
continue;
if (WARN_ON_ONCE(pte_page(old) != d->scratch_page))
if (WARN_ON_ONCE(pte_page(old) != d->arena->scratch_page))
return -EBUSY;
ptep_get_and_clear(&init_mm, addr, pte);
flush_tlb_before_set(addr);
@@ -196,6 +198,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
set_pte_at(&init_mm, addr, pte, pteval);
#endif
d->i++;
WRITE_ONCE(d->arena->nr_pages, d->arena->nr_pages + 1);
return 0;
}
@@ -227,10 +230,11 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
* scratches its PTE. A later bpf_arena_free_pages() over that range walks
* here. Without the skip, scratch_page would be freed.
*/
if (page == d->scratch_page)
if (page == d->arena->scratch_page)
return 0;
__llist_add(&page->pcp_llist, d->free_pages);
WRITE_ONCE(d->arena->nr_pages, d->arena->nr_pages - 1);
return 0;
}
@@ -413,7 +417,9 @@ static int arena_map_check_btf(struct bpf_map *map, const struct btf *btf,
static u64 arena_map_mem_usage(const struct bpf_map *map)
{
return 0;
struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
return (u64)READ_ONCE(arena->nr_pages) << PAGE_SHIFT;
}
struct vma_list {
@@ -484,8 +490,12 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
kaddr = kbase + (u32)(vmf->address);
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
/* Make a reasonable effort to address impossible case */
return VM_FAULT_RETRY;
/*
* A failed lock means a possible deadlock was detected. Don't
* return VM_FAULT_RETRY: this handler never took mmap_lock, but
* the fault path would re-take it on retry and deadlock. Fail.
*/
return VM_FAULT_SIGBUS;
page = vmalloc_to_page((void *)kaddr);
if (page) {
@@ -506,8 +516,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
if (ret)
goto out_sigsegv_memcg;
struct apply_range_data data = { .pages = &page, .i = 0,
.scratch_page = arena->scratch_page };
struct apply_range_data data = { .arena = arena, .pages = &page, .i = 0 };
/* Account into memcg of the process that created bpf_arena */
ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
if (ret) {
@@ -696,8 +705,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
bpf_map_memcg_exit(old_memcg, new_memcg);
return 0;
}
data.arena = arena;
data.pages = pages;
data.scratch_page = arena->scratch_page;
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
goto out_free_pages;
@@ -853,6 +862,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
uaddr &= PAGE_MASK;
kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
if (full_uaddr < arena->user_vm_start)
return;
uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));
if (full_uaddr >= uaddr_end)
return;
@@ -873,8 +884,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
range_tree_set(&arena->rt, pgoff, page_cnt);
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
cdata.scratch_page = arena->scratch_page;
/* clear ptes and collect struct pages */
apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
apply_range_clear_cb, &cdata);
@@ -981,8 +992,8 @@ static void arena_free_worker(struct work_struct *work)
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
cdata.scratch_page = arena->scratch_page;
arena_vm_start = bpf_arena_get_kern_vm_start(arena);
user_vm_start = bpf_arena_get_user_vm_start(arena);
@@ -1107,9 +1118,9 @@ __bpf_kfunc int bpf_arena_reserve_pages(void *p__map, void *ptr__ign, u32 page_c
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(arena_kfuncs)
BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_ARENA_RET | KF_ARENA_ARG2)
BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2)
BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2)
BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_ARENA_RET | KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_KFUNCS_END(arena_kfuncs)
static const struct btf_kfunc_id_set common_kfunc_set = {

72
kernel/bpf/arraymap.c
View File

@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
return this_cpu_ptr(array->pptrs[index & array->index_mask]);
}
static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
if (!bpf_jit_supports_percpu_insn())
return -EOPNOTSUPP;
if (map->max_entries != 1)
return -EOPNOTSUPP;
if (off >= map->value_size)
return -EINVAL;
*imm = (u64)(__force unsigned long) array->pptrs[0];
return 0;
}
static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
u64 base = (u64)(__force unsigned long) array->pptrs[0];
if (!bpf_jit_supports_percpu_insn())
return -EOPNOTSUPP;
if (map->max_entries != 1)
return -EOPNOTSUPP;
if (imm < base || imm >= base + array->elem_size)
return -ENOENT;
*off = imm - base;
return 0;
}
/* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
{
@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,
const struct btf_type *key_type,
const struct btf_type *value_type)
{
/* One exception for keyless BTF: .bss/.data/.rodata map */
/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
if (btf_type_is_void(key_type)) {
if (map->map_type != BPF_MAP_TYPE_ARRAY ||
if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
map->max_entries != 1)
return -EINVAL;
@@ -576,17 +608,42 @@ static int array_map_check_btf(struct bpf_map *map,
static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
if (!(map->map_flags & BPF_F_MMAPABLE))
return -EINVAL;
if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
/* use u64 math so the offset cannot overflow on 32-bit archs */
if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
return -EINVAL;
return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
vma->vm_pgoff + pgoff);
/*
* Pages are faulted in on demand by array_map_mmap_fault(). Set the
* same flags that the eager remap_vmalloc_range() path used to set
* via vm_insert_page(), so that e.g. NUMA balancing keeps skipping
* these VMAs.
*/
vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP);
return 0;
}
static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
struct vm_fault *vmf)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct page *page;
page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT));
if (!page)
return VM_FAULT_SIGBUS;
/* the eager remap_vmalloc_range() flushed via vm_insert_page() */
flush_dcache_folio(page_folio(page));
get_page(page);
vmf->page = page;
return 0;
}
static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -812,6 +869,7 @@ const struct bpf_map_ops array_map_ops = {
.map_direct_value_addr = array_map_direct_value_addr,
.map_direct_value_meta = array_map_direct_value_meta,
.map_mmap = array_map_mmap,
.map_mmap_fault = array_map_mmap_fault,
.map_seq_show_elem = array_map_seq_show_elem,
.map_check_btf = array_map_check_btf,
.map_lookup_batch = generic_map_lookup_batch,
@@ -832,6 +890,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
.map_get_next_key = bpf_array_get_next_key,
.map_lookup_elem = percpu_array_map_lookup_elem,
.map_gen_lookup = percpu_array_map_gen_lookup,
.map_direct_value_addr = percpu_array_map_direct_value_addr,
.map_direct_value_meta = percpu_array_map_direct_value_meta,
.map_update_elem = array_map_update_elem,
.map_delete_elem = array_map_delete_elem,
.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,

5
kernel/bpf/backtrack.c
View File

@@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
return bt->reg_masks[bt->frame] & (1 << reg);
}
/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
{
@@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
}
}
/* For given verifier state backtrack_insn() is called from the last insn to
* the first insn. Its purpose is to compute a bitmask of registers and
* stack slots that needs precision in the parent verifier state.
@@ -285,6 +283,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
verbose(env, "stack=%s before ", env->tmp_str_buf);
verbose(env, "%d: ", idx);
bpf_verbose_insn(env, insn);
verbose(env, "\n");
}
/* If there is a history record that some registers gained range at this insn,
@@ -636,7 +635,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* r5 += 1
* ...
* call bpf_perf_event_output#25
* where .arg5_type = ARG_CONST_SIZE_OR_ZERO
* where .arg5_type = ARG_MEM_SIZE_OR_ZERO
*
* and this case:
* r6 = 1

19
kernel/bpf/bloom_filter.c
View File

@@ -41,7 +41,7 @@ static long bloom_map_peek_elem(struct bpf_map *map, void *value)
for (i = 0; i < bloom->nr_hash_funcs; i++) {
h = hash(bloom, value, map->value_size, i);
if (!test_bit(h, bloom->bitset))
if (!test_bit(h % BITS_PER_LONG, bloom->bitset + BIT_WORD(h)))
return -ENOENT;
}
@@ -57,9 +57,13 @@ static long bloom_map_push_elem(struct bpf_map *map, void *value, u64 flags)
if (flags != BPF_ANY)
return -EINVAL;
/*
* On 32-bit architectures, hashes larger than INT_MAX would be
* treated as negative by set_bit().
*/
for (i = 0; i < bloom->nr_hash_funcs; i++) {
h = hash(bloom, value, map->value_size, i);
set_bit(h, bloom->bitset);
set_bit(h % BITS_PER_LONG, bloom->bitset + BIT_WORD(h));
}
return 0;
@@ -94,9 +98,10 @@ static int bloom_map_alloc_check(union bpf_attr *attr)
static struct bpf_map *bloom_map_alloc(union bpf_attr *attr)
{
u32 bitset_bytes, bitset_mask, nr_hash_funcs, nr_bits;
u32 bitset_mask, nr_hash_funcs, nr_bits;
int numa_node = bpf_map_attr_numa_node(attr);
struct bpf_bloom_filter *bloom;
u64 bitset_bytes;
if (attr->key_size != 0 || attr->value_size == 0 ||
attr->max_entries == 0 ||
@@ -127,22 +132,16 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr)
if (check_mul_overflow(attr->max_entries, nr_hash_funcs, &nr_bits) ||
check_mul_overflow(nr_bits / 5, (u32)7, &nr_bits) ||
nr_bits > (1UL << 31)) {
/* The bit array size is 2^32 bits but to avoid overflowing the
* u32, we use U32_MAX, which will round up to the equivalent
* number of bytes
*/
bitset_bytes = BITS_TO_BYTES(U32_MAX);
bitset_mask = U32_MAX;
} else {
if (nr_bits <= BITS_PER_LONG)
nr_bits = BITS_PER_LONG;
else
nr_bits = roundup_pow_of_two(nr_bits);
bitset_bytes = BITS_TO_BYTES(nr_bits);
bitset_mask = nr_bits - 1;
}
bitset_bytes = roundup(bitset_bytes, sizeof(unsigned long));
bitset_bytes = BITS_TO_LONGS((u64)bitset_mask + 1) * sizeof(unsigned long);
bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node);
if (!bloom)

17
kernel/bpf/bpf_iter.c
View File

@@ -782,8 +782,8 @@ __bpf_kfunc int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end)
return -EINVAL;
}
/* avoid overflows, e.g., if start == INT_MIN and end == INT_MAX */
if ((s64)end - (s64)start > BPF_MAX_LOOPS) {
/* start <= end here, so end - start fits in a u32 without overflow */
if ((u32)(end - start) > BPF_MAX_LOOPS) {
s->cur = s->end = 0;
return -E2BIG;
}
@@ -802,12 +802,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
{
struct bpf_iter_num_kern *s = (void *)it;
/* check failed initialization or if we are done (same behavior);
* need to be careful about overflow, so convert to s64 for checks,
* e.g., if s->cur == s->end == INT_MAX, we can't just do
* s->cur + 1 >= s->end
/*
* s->cur < s->end while iterating, else s->cur == s->end == 0; the signed
* s->cur + 1 >= s->end holds even when s->cur + 1 wraps (start == INT_MIN).
*/
if ((s64)(s->cur + 1) >= s->end) {
if (s->cur + 1 >= s->end) {
s->cur = s->end = 0;
return NULL;
}
@@ -819,9 +818,7 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
__bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it)
{
struct bpf_iter_num_kern *s = (void *)it;
s->cur = s->end = 0;
/* no-op */
}
__bpf_kfunc_end_defs();

5
kernel/bpf/bpf_lsm.c
View File

@@ -186,7 +186,7 @@ static const struct bpf_func_proto bpf_ima_inode_hash_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_ima_inode_hash_btf_ids[0],
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
.allowed = bpf_ima_inode_hash_allowed,
};
@@ -205,7 +205,7 @@ static const struct bpf_func_proto bpf_ima_file_hash_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_ima_file_hash_btf_ids[0],
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
.allowed = bpf_ima_inode_hash_allowed,
};
@@ -295,7 +295,6 @@ BTF_ID(func, bpf_lsm_bpf_map_create)
BTF_ID(func, bpf_lsm_bpf_map_free)
BTF_ID(func, bpf_lsm_bpf_prog)
BTF_ID(func, bpf_lsm_bpf_prog_load)
BTF_ID(func, bpf_lsm_bpf_prog_free)
BTF_ID(func, bpf_lsm_bpf_token_create)
BTF_ID(func, bpf_lsm_bpf_token_free)
BTF_ID(func, bpf_lsm_bpf_token_cmd)

68
kernel/bpf/bpf_struct_ops.c
View File

@@ -147,6 +147,8 @@ void bpf_struct_ops_image_free(void *image)
#define MAYBE_NULL_SUFFIX "__nullable"
#define REFCOUNTED_SUFFIX "__ref"
#define ARENA_SUFFIX "__arena"
#define ARENA_MAYBE_NULL_SUFFIX "__arena__nullable"
/* Prepare argument info for every nullable argument of a member of a
* struct_ops type.
@@ -159,7 +161,7 @@ void bpf_struct_ops_image_free(void *image)
* to provide an array of struct bpf_ctx_arg_aux, which in turn provides
* the information that used by the verifier to check the arguments of the
* BPF struct_ops program assigned to the member. Here, we only care about
* the arguments that are marked as __nullable.
* the arguments that are marked as __nullable, __ref or __arena.
*
* The array of struct bpf_ctx_arg_aux is eventually assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
@@ -172,10 +174,12 @@ static int prepare_arg_info(struct btf *btf,
const char *st_ops_name,
const char *member_name,
const struct btf_type *func_proto, void *stub_func_addr,
struct btf_func_model *model,
struct bpf_struct_ops_arg_info *arg_info)
{
const struct btf_type *stub_func_proto, *pointed_type;
bool is_nullable = false, is_refcounted = false;
bool is_nullable = false, is_refcounted = false, is_arena = false;
bool is_arena_nullable = false;
const struct btf_param *stub_args, *args;
struct bpf_ctx_arg_aux *info, *info_buf;
u32 nargs, arg_no, info_cnt = 0;
@@ -225,27 +229,39 @@ static int prepare_arg_info(struct btf *btf,
/* Prepare info for every nullable argument */
info = info_buf;
for (arg_no = 0; arg_no < nargs; arg_no++) {
/* Skip arguments that is not suffixed with
* "__nullable or __ref".
bool ptr_to_arena, ptr_to_struct;
/*
* Skip arguments that are not suffixed with "__arena__nullable",
* "__arena", "__nullable", or "__ref".
*/
is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
MAYBE_NULL_SUFFIX);
is_arena_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
ARENA_MAYBE_NULL_SUFFIX);
is_arena = btf_param_match_suffix(btf, &stub_args[arg_no], ARENA_SUFFIX);
is_nullable = !is_arena_nullable &&
btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX);
is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no],
REFCOUNTED_SUFFIX);
if (is_nullable)
if (is_arena_nullable)
suffix = ARENA_MAYBE_NULL_SUFFIX;
else if (is_arena)
suffix = ARENA_SUFFIX;
else if (is_nullable)
suffix = MAYBE_NULL_SUFFIX;
else if (is_refcounted)
suffix = REFCOUNTED_SUFFIX;
else
continue;
/* Should be a pointer to struct */
pointed_type = btf_type_resolve_ptr(btf,
args[arg_no].type,
&arg_btf_id);
if (!pointed_type ||
!btf_type_is_struct(pointed_type)) {
/*
* Should be a pointer to struct, or any pointer for __arena or
* __arena__nullable.
*/
pointed_type = btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id);
ptr_to_arena = pointed_type && (is_arena || is_arena_nullable);
ptr_to_struct = pointed_type && btf_type_is_struct(pointed_type);
if (!ptr_to_arena && !ptr_to_struct) {
pr_warn("stub function %s has %s tagging to an unsupported type\n",
stub_fname, suffix);
goto err_out;
@@ -268,7 +284,18 @@ static int prepare_arg_info(struct btf *btf,
info->btf_id = arg_btf_id;
info->btf = btf;
info->offset = offset;
if (is_nullable) {
if (is_arena || is_arena_nullable) {
/*
* Both types get PTR_TO_ARENA. In verifier state,
* PTR_TO_ARENA encompasses potential NULL values, but
* we do not force the program to check it, or maintain
* precision around it, since it has no safety implication.
*/
info->reg_type = PTR_TO_ARENA;
model->arg_flags[arg_no] |= BTF_FMODEL_ARENA_ARG;
if (is_arena_nullable)
model->arg_flags[arg_no] |= BTF_FMODEL_NULLABLE_ARG;
} else if (is_nullable) {
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID | PTR_MAYBE_NULL;
} else if (is_refcounted) {
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID;
@@ -445,9 +472,22 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
goto errout;
}
/*
* A >8 byte return value is passed back in a register pair,
* which the struct_ops trampoline does not preserve (only
* 8 bytes of the return value are saved and restored).
*/
if (st_ops->func_models[i].ret_size > 8) {
pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n",
mname, st_ops->name);
err = -EOPNOTSUPP;
goto errout;
}
stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
err = prepare_arg_info(btf, st_ops->name, mname,
func_proto, stub_func_addr,
&st_ops->func_models[i],
arg_info + i);
if (err)
goto errout;

100
kernel/bpf/btf.c
View File

@@ -1169,19 +1169,19 @@ static const char *btf_show_name(struct btf_show *show)
id = t->type;
break;
default:
id = 0;
break;
goto resolved;
}
t = btf_type_skip_qualifiers(show->btf, id);
if (!id)
break;
t = btf_type_skip_qualifiers(show->btf, id);
}
/* We may not be able to represent this type; bail to be safe */
if (i == BTF_SHOW_MAX_ITER)
return "";
resolved:
if (!name)
name = btf_name_by_offset(show->btf, t->name_off);
name = btf_type_is_void(t) ? "void" : btf_name_by_offset(show->btf, t->name_off);
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_STRUCT:
@@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void *data, u8 bits_offset,
btf_int128_print(show, print_num);
}
static void btf_int_bits_show(const struct btf *btf,
const struct btf_type *t,
void *data, u8 bits_offset,
@@ -3669,7 +3668,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_
{ BPF_LIST_NODE, "bpf_list_node", false },
{ BPF_RB_ROOT, "bpf_rb_root", false },
{ BPF_RB_NODE, "bpf_rb_node", false },
{ BPF_REFCOUNT, "bpf_refcount", false },
{ BPF_REFCOUNT, "bpf_refcount", true },
};
int type = 0, i;
const char *name = __btf_name_by_offset(btf, var_type->name_off);
@@ -3751,7 +3750,7 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
static int btf_find_struct_field(const struct btf *btf,
const struct btf_type *t, u32 field_mask,
struct btf_field_info *info, int info_cnt,
u32 level);
u32 level, u32 *seen_mask);
/* Find special fields in the struct type of a field.
*
@@ -3762,7 +3761,7 @@ static int btf_find_struct_field(const struct btf *btf,
static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
u32 off, u32 nelems,
u32 field_mask, struct btf_field_info *info,
int info_cnt, u32 level)
int info_cnt, u32 level, u32 *seen_mask)
{
int ret, err, i;
@@ -3770,7 +3769,7 @@ static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *
if (level >= MAX_RESOLVE_DEPTH)
return -E2BIG;
ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level, seen_mask);
if (ret <= 0)
return ret;
@@ -3827,7 +3826,7 @@ static int btf_find_field_one(const struct btf *btf,
if (expected_size && expected_size != sz * nelems)
return 0;
ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
&info[0], info_cnt, level);
&info[0], info_cnt, level, seen_mask);
return ret;
}
@@ -3892,11 +3891,11 @@ static int btf_find_field_one(const struct btf *btf,
static int btf_find_struct_field(const struct btf *btf,
const struct btf_type *t, u32 field_mask,
struct btf_field_info *info, int info_cnt,
u32 level)
u32 level, u32 *seen_mask)
{
int ret, idx = 0;
const struct btf_member *member;
u32 i, off, seen_mask = 0;
u32 i, off;
for_each_member(i, t, member) {
const struct btf_type *member_type = btf_type_by_id(btf,
@@ -3910,7 +3909,7 @@ static int btf_find_struct_field(const struct btf *btf,
ret = btf_find_field_one(btf, t, member_type, i,
off, 0,
field_mask, &seen_mask,
field_mask, seen_mask,
&info[idx], info_cnt - idx, level);
if (ret < 0)
return ret;
@@ -3921,11 +3920,11 @@ static int btf_find_struct_field(const struct btf *btf,
static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
u32 field_mask, struct btf_field_info *info,
int info_cnt, u32 level)
int info_cnt, u32 level, u32 *seen_mask)
{
int ret, idx = 0;
const struct btf_var_secinfo *vsi;
u32 i, off, seen_mask = 0;
u32 i, off;
for_each_vsi(i, t, vsi) {
const struct btf_type *var = btf_type_by_id(btf, vsi->type);
@@ -3933,7 +3932,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
off = vsi->offset;
ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
field_mask, &seen_mask,
field_mask, seen_mask,
&info[idx], info_cnt - idx,
level);
if (ret < 0)
@@ -3947,10 +3946,12 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t,
u32 field_mask, struct btf_field_info *info,
int info_cnt)
{
u32 seen_mask = 0;
if (__btf_type_is_struct(t))
return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0, &seen_mask);
else if (btf_type_is_datasec(t))
return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0, &seen_mask);
return -EINVAL;
}
@@ -4168,7 +4169,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
rec->spin_lock_off = rec->fields[i].offset;
break;
case BPF_RES_SPIN_LOCK:
WARN_ON_ONCE(rec->spin_lock_off >= 0);
WARN_ON_ONCE(rec->res_spin_lock_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->res_spin_lock_off = rec->fields[i].offset;
break;
@@ -6451,7 +6452,7 @@ struct btf *btf_parse_vmlinux(void)
if (IS_ERR(btf))
goto err_out;
/* btf_parse_vmlinux() runs under bpf_verifier_lock */
/* btf_parse_vmlinux() runs under btf_vmlinux_lock */
bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]);
err = btf_alloc_id(btf);
if (err) {
@@ -6954,15 +6955,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
return false;
}
/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */
/*
* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or
* PTR_TO_ARENA (both nullable and non-nullable cases).
*/
for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
u32 type, flag;
type = base_type(ctx_arg_info->reg_type);
flag = type_flag(ctx_arg_info->reg_type);
if (ctx_arg_info->offset == off && type == PTR_TO_BUF &&
(flag & PTR_MAYBE_NULL)) {
if (ctx_arg_info->offset == off &&
(type == PTR_TO_ARENA ||
(type == PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) {
info->reg_type = ctx_arg_info->reg_type;
return true;
}
@@ -7101,7 +7106,7 @@ enum bpf_struct_walk_result {
static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, int off, int size,
u32 *next_btf_id, enum bpf_type_flag *flag,
const char **field_name)
const char **field_name, bool walk_flex_arrays)
{
u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
const struct btf_type *mtype, *elem_type = NULL;
@@ -7128,11 +7133,14 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
*flag |= PTR_UNTRUSTED;
if (off + size > t->size) {
struct btf_array *array_elem;
if (!walk_flex_arrays)
goto error;
/* If the last element is a variable size array, we may
* need to relax the rule.
*/
struct btf_array *array_elem;
if (vlen == 0)
goto error;
@@ -7397,7 +7405,8 @@ int btf_struct_access(struct bpf_verifier_log *log,
t = btf_type_by_id(btf, id);
do {
err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name);
err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag,
field_name, !type_is_alloc(reg->type));
switch (err) {
case WALK_PTR:
@@ -7456,7 +7465,7 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
bool strict)
bool strict, bool walk_flex_arrays)
{
const struct btf_type *type;
enum bpf_type_flag flag = 0;
@@ -7475,7 +7484,8 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log,
type = btf_type_by_id(btf, id);
if (!type)
return false;
err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL);
err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL,
walk_flex_arrays);
if (err != WALK_STRUCT)
return false;
@@ -7519,14 +7529,28 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)
{
u8 flags = 0;
if (btf_type_is_struct(t))
flags |= BTF_FMODEL_STRUCT_ARG;
if (btf_type_is_signed_int(t))
flags |= BTF_FMODEL_SIGNED_ARG;
return flags;
}
static u8 __get_arg_fmodel_flags(const struct btf *btf,
const struct btf_param *arg,
const struct btf_type *t)
{
u8 flags = __get_type_fmodel_flags(t);
if (btf_param_match_suffix(btf, arg, "__arena__nullable"))
flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG;
else if (btf_param_match_suffix(btf, arg, "__arena"))
flags |= BTF_FMODEL_ARENA_ARG;
else if (btf_param_match_suffix(btf, arg, "__nullable"))
flags |= BTF_FMODEL_NULLABLE_ARG;
return flags;
}
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *func,
@@ -7592,7 +7616,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
return -EINVAL;
}
m->arg_size[i] = ret;
m->arg_flags[i] = __get_type_fmodel_flags(t);
m->arg_flags[i] = __get_arg_fmodel_flags(btf, &args[i], t);
}
m->nr_args = nargs;
return 0;
@@ -8285,6 +8309,16 @@ int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
return ssnprintf.len;
}
int btf_type_name_to_buf(const struct btf *btf, u32 type_id, char *buf, int len)
{
struct btf_show show = {
.btf = btf,
.state.type_id = type_id,
};
return snprintf(buf, len, "%s", btf_show_name(&show));
}
#ifdef CONFIG_PROC_FS
static void bpf_btf_show_fdinfo(struct seq_file *m, struct file *filp)
{
@@ -8695,7 +8729,7 @@ const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_ANYTHING,
};

38
kernel/bpf/cfg.c
View File

@@ -5,6 +5,8 @@
#include <linux/filter.h>
#include <linux/sort.h>
#include "diagnostics.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
/* non-recursive DFS pseudo code
@@ -47,7 +49,6 @@ enum {
BRANCH = 2,
};
static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)
{
struct bpf_subprog_info *subprog;
@@ -113,6 +114,10 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
if (w < 0 || w >= env->prog->len) {
verbose_linfo(env, t, "%d: ", t);
verbose(env, "jump out of range from insn %d to %d\n", t, w);
bpf_diag_program_structure(
env, t, "jump out of range", "Keep branch targets inside the program.",
"Instruction %d jumps to instruction %d, but the program only contains instructions 0 through %d.",
t, w, env->prog->len - 1);
return -EINVAL;
}
@@ -136,6 +141,11 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
verbose_linfo(env, t, "%d: ", t);
verbose_linfo(env, w, "%d: ", w);
verbose(env, "back-edge from insn %d to %d\n", t, w);
bpf_diag_program_structure(
env, t, "back-edge is not allowed",
"Load with privileges that allow this back-edge, or rewrite the control flow so it does not branch backward.",
"Instruction %d branches back to instruction %d. This program is being rejected without the privilege needed for this back-edge.",
t, w);
return -EINVAL;
} else if (insn_state[w] == EXPLORED) {
/* forward- or cross-edge */
@@ -316,6 +326,11 @@ static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env,
if (!jt) {
verbose(env, "no jump tables found for subprog starting at %u\n", subprog_start);
bpf_diag_program_structure(
env, subprog_start, "missing jump table",
"Make sure subprograms containing gotox instructions are accompanied by jump tables referencing these subprograms.",
"No jump table was found for the subprogram that starts at instruction %u.",
subprog_start);
return ERR_PTR(-EINVAL);
}
@@ -343,6 +358,11 @@ create_jt(int t, struct bpf_verifier_env *env)
if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]\n",
t, subprog_start, subprog_end);
bpf_diag_program_structure(
env, t, "jump table target out of range",
"Keep every jump-table target inside the same subprogram.",
"The jump table for instruction %d points outside subprogram range [%u,%u).",
t, subprog_start, subprog_end);
kvfree(jt);
return ERR_PTR(-EINVAL);
}
@@ -374,6 +394,11 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
w = jt->items[i];
if (w < 0 || w >= env->prog->len) {
verbose(env, "indirect jump out of range from insn %d to %d\n", t, w);
bpf_diag_program_structure(
env, t, "indirect jump out of range",
"Keep indirect jump targets inside the program.",
"Instruction %d can jump indirectly to instruction %d, but the program only contains instructions 0 through %d.",
t, w, env->prog->len - 1);
return -EINVAL;
}
@@ -491,7 +516,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
return ret;
}
} else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
struct bpf_kfunc_call_arg_meta meta;
struct bpf_call_arg_meta meta;
ret = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta);
if (ret == 0 && bpf_is_iter_next_kfunc(&meta)) {
@@ -624,12 +649,21 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
if (insn_state[i] != EXPLORED) {
verbose(env, "unreachable insn %d\n", i);
bpf_diag_program_structure(
env, i, "unreachable instruction",
"Remove the unreachable instruction or add valid control flow that reaches it.",
"Instruction %d is not reachable from the program entry point.", i);
ret = -EINVAL;
goto err_free;
}
if (bpf_is_ldimm64(insn)) {
if (insn_state[i + 1] != 0) {
verbose(env, "jump into the middle of ldimm64 insn %d\n", i);
bpf_diag_program_structure(
env, i, "jump into ldimm64 immediate",
"Target the first instruction of the ldimm64 pair, or restructure the jump target.",
"Control flow reaches the second half of the ldimm64 instruction pair that starts at instruction %d.",
i);
ret = -EINVAL;
goto err_free;
}

35
kernel/bpf/cgroup.c
View File

@@ -813,8 +813,10 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *old_prog = NULL;
struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_cgroup_storage *new_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_cgroup_storage *old_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_prog *new_prog = prog ? : link->link.prog;
enum cgroup_bpf_attach_type atype;
u32 old_flags, old_pl_flags;
struct bpf_prog_list *pl;
struct hlist_head *progs;
int err;
@@ -865,6 +867,8 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
if (pl) {
old_prog = pl->prog;
old_pl_flags = pl->flags;
bpf_cgroup_storages_assign(old_storage, pl->storage);
} else {
pl = kmalloc_obj(*pl);
if (!pl) {
@@ -884,6 +888,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
pl->link = link;
pl->flags = flags;
bpf_cgroup_storages_assign(pl->storage, storage);
old_flags = cgrp->bpf.flags[atype];
cgrp->bpf.flags[atype] = saved_flags;
if (type == BPF_LSM_CGROUP) {
@@ -915,12 +920,15 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
if (old_prog) {
pl->prog = old_prog;
pl->link = NULL;
pl->flags = old_pl_flags;
bpf_cgroup_storages_assign(pl->storage, old_storage);
}
bpf_cgroup_storages_free(new_storage);
if (!old_prog) {
hlist_del(&pl->node);
kfree(pl);
}
cgrp->bpf.flags[atype] = old_flags;
return err;
}
@@ -1018,6 +1026,20 @@ static void replace_effective_prog(struct cgroup *cgrp,
}
}
static bool cgroup_bpf_storages_compatible(struct bpf_prog *old_prog,
struct bpf_prog *new_prog)
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype) {
if (old_prog->aux->cgroup_storage[stype] !=
new_prog->aux->cgroup_storage[stype])
return false;
}
return true;
}
/**
* __cgroup_bpf_replace() - Replace link's program and propagate the change
* to descendants
@@ -1056,6 +1078,9 @@ static int __cgroup_bpf_replace(struct cgroup *cgrp,
if (!found)
return -ENOENT;
if (!cgroup_bpf_storages_compatible(link->link.prog, new_prog))
return -EINVAL;
cgrp->bpf.revisions[atype] += 1;
old_prog = xchg(&link->link.prog, new_prog);
replace_effective_prog(cgrp, atype, pl);
@@ -2235,7 +2260,7 @@ int __cgroup_bpf_run_filter_getsockopt_kern(struct sock *sk, int level,
if (ret < 0)
return ret;
if (ctx.optlen > *optlen)
if (ctx.optlen > *optlen || ctx.optlen < 0)
return -EFAULT;
/* BPF programs can shrink the buffer, export the modifications.
@@ -2305,7 +2330,7 @@ static const struct bpf_func_proto bpf_sysctl_get_name_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
.arg4_type = ARG_ANYTHING,
};
@@ -2347,7 +2372,7 @@ static const struct bpf_func_proto bpf_sysctl_get_current_value_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
};
BPF_CALL_3(bpf_sysctl_get_new_value, struct bpf_sysctl_kern *, ctx, char *, buf,
@@ -2367,7 +2392,7 @@ static const struct bpf_func_proto bpf_sysctl_get_new_value_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
};
BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx,
@@ -2393,7 +2418,7 @@ static const struct bpf_func_proto bpf_sysctl_set_new_value_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
};
static const struct bpf_func_proto *

14
kernel/bpf/check_btf.c
View File

@@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
#define MIN_BPF_FUNCINFO_SIZE 8
#define MAX_FUNCINFO_REC_SIZE 252
static int check_btf_func_early(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
static int prepare_btf_func(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
{
u32 krec_size = sizeof(struct bpf_func_info);
const struct btf_type *type, *func_proto;
@@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,
return err;
}
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
{
struct btf *btf;
int err;
@@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env,
}
env->prog->aux->btf = btf;
err = check_btf_func_early(env, attr, uattr);
err = prepare_btf_func(env, attr, uattr);
if (err)
return err;
return 0;

12
kernel/bpf/const_fold.c
View File

@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
u64 val = 0;
if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
off < 0 || off + size > map->value_size ||
bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
*dst = unknown;
@@ -199,14 +198,9 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
ci_out[r] = unknown;
break;
case BPF_STX:
if (mode != BPF_ATOMIC)
break;
if (insn->imm == BPF_CMPXCHG)
ci_out[BPF_REG_0] = unknown;
else if (insn->imm == BPF_LOAD_ACQ)
*dst = unknown;
else if (insn->imm & BPF_FETCH)
*src = unknown;
r = bpf_atomic_load_reg(insn);
if (r >= 0)
ci_out[r] = unknown;
break;
}
}

95
kernel/bpf/core.c
View File

@@ -126,6 +126,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
fp->aux->main_prog_aux = aux;
fp->aux->prog = fp;
fp->jit_requested = ebpf_jit_enabled();
fp->jit_required = IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON);
fp->blinding_requested = bpf_jit_blinding_enabled(fp);
#ifdef CONFIG_CGROUP_BPF
aux->cgroup_atype = CGROUP_BPF_ATTACH_TYPE_INVALID;
@@ -305,7 +306,7 @@ int bpf_prog_calc_tag(struct bpf_prog *fp)
bool was_ld_map;
u32 i;
dst = vmalloc(size);
dst = __vmalloc(size, GFP_KERNEL_ACCOUNT);
if (!dst)
return -ENOMEM;
@@ -915,6 +916,11 @@ static LIST_HEAD(pack_list);
#define BPF_PROG_CHUNK_COUNT (BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE)
static bool bpf_jit_mem_is_rox(void)
{
return execmem_is_rox(EXECMEM_BPF);
}
static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_insns)
{
struct bpf_prog_pack *pack;
@@ -926,16 +932,18 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
pack->ptr = bpf_jit_alloc_exec(BPF_PROG_PACK_SIZE);
if (!pack->ptr)
goto out;
bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE);
bitmap_zero(pack->bitmap, BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE);
if (static_branch_unlikely(&bpf_pred_flush_enabled))
pack->arch_flush_needed = true;
set_vm_flush_reset_perms(pack->ptr);
err = set_memory_rox((unsigned long)pack->ptr,
BPF_PROG_PACK_SIZE / PAGE_SIZE);
if (err)
goto out;
if (!bpf_jit_mem_is_rox()) {
bpf_fill_ill_insns(pack->ptr, BPF_PROG_PACK_SIZE);
set_vm_flush_reset_perms(pack->ptr);
err = set_memory_rox((unsigned long)pack->ptr,
BPF_PROG_PACK_SIZE / PAGE_SIZE);
if (err)
goto out;
}
list_add_tail(&pack->list, &pack_list);
return pack;
@@ -964,7 +972,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
pr_warn_once("BPF: Predictors not flushed for allocations greater than BPF_PROG_PACK_SIZE\n");
size = round_up(size, PAGE_SIZE);
ptr = bpf_jit_alloc_exec(size);
if (ptr) {
if (ptr && !bpf_jit_mem_is_rox()) {
int err;
bpf_fill_ill_insns(ptr, size);
@@ -1115,12 +1123,17 @@ void bpf_jit_uncharge_modmem(u32 size)
atomic_long_sub(size, &bpf_jit_current);
}
void *__weak bpf_jit_alloc_exec(unsigned long size)
void *bpf_jit_alloc_exec(unsigned long size)
{
return execmem_alloc(EXECMEM_BPF, size);
}
void __weak bpf_jit_free_exec(void *addr)
void *bpf_jit_alloc_exec_rw(unsigned long size)
{
return execmem_alloc_rw(EXECMEM_BPF, size);
}
void bpf_jit_free_exec(void *addr)
{
execmem_free(addr);
}
@@ -2621,22 +2634,10 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
{
#ifdef CONFIG_BPF_JIT
struct bpf_prog *orig_prog;
struct bpf_insn_aux_data *orig_insn_aux;
if (!bpf_prog_need_blind(prog))
return bpf_int_jit_compile(env, prog);
if (env) {
/*
* If env is not NULL, we are called from the end of bpf_check(), at this
* point, only insn_aux_data is used after failure, so it should be restored
* on failure.
*/
orig_insn_aux = bpf_dup_insn_aux_data(env);
if (!orig_insn_aux)
return prog;
}
orig_prog = prog;
prog = bpf_jit_blind_constants(env, prog);
/*
@@ -2649,8 +2650,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
prog = bpf_int_jit_compile(env, prog);
if (prog->jited) {
bpf_jit_prog_release_other(prog, orig_prog);
if (env)
vfree(orig_insn_aux);
return prog;
}
@@ -2658,8 +2657,6 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
out_restore:
prog = orig_prog;
if (env)
bpf_restore_insn_aux_data(env, orig_insn_aux);
#endif
return prog;
}
@@ -2670,15 +2667,11 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
/* In case of BPF to BPF calls, verifier did all the prep
* work with regards to JITing, etc.
*/
bool jit_needed = false;
bool jit_needed = fp->jit_required;
if (fp->bpf_func)
goto finalize;
if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
bpf_prog_has_kfunc_call(fp))
jit_needed = true;
if (!bpf_prog_select_interpreter(fp))
jit_needed = true;
@@ -3299,6 +3292,11 @@ bool __weak bpf_jit_supports_stack_args(void)
return false;
}
bool __weak bpf_jit_supports_arena_args(void)
{
return false;
}
bool __weak bpf_jit_supports_far_kfunc_call(void)
{
return false;
@@ -3463,24 +3461,14 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(xdp_bulk_tx);
#ifdef CONFIG_BPF_SYSCALL
void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo,
const char **filep, const char **linep, int *nump)
void bpf_get_linfo_source(struct btf *btf, const struct bpf_line_info *linfo,
struct bpf_linfo_source *src)
{
/* Get base component of the file path. */
if (filep) {
*filep = btf_name_by_offset(btf, linfo->file_name_off);
*filep = kbasename(*filep);
}
/* Obtain the source line, and strip whitespace in prefix. */
if (linep) {
*linep = btf_name_by_offset(btf, linfo->line_off);
while (isspace(**linep))
*linep += 1;
}
if (nump)
*nump = BPF_LINE_INFO_LINE_NUM(linfo->line_col);
src->file = kbasename(btf_name_by_offset(btf, linfo->file_name_off));
src->line = btf_name_by_offset(btf, linfo->line_off);
src->file_name_off = linfo->file_name_off;
src->line_num = BPF_LINE_INFO_LINE_NUM(linfo->line_col);
src->line_col = BPF_LINE_INFO_LINE_COL(linfo->line_col);
}
const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off)
@@ -3523,6 +3511,7 @@ const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn
int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep,
const char **linep, int *nump)
{
struct bpf_linfo_source src;
int idx = -1, insn_start, insn_end, len;
struct bpf_line_info *linfo;
void **jited_linfo;
@@ -3554,7 +3543,15 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
if (idx == -1)
return -ENOENT;
bpf_get_linfo_file_line(btf, &linfo[idx], filep, linep, nump);
bpf_get_linfo_source(btf, &linfo[idx], &src);
while (isspace(*src.line))
src.line++;
if (filep)
*filep = src.file;
if (linep)
*linep = src.line;
if (nump)
*nump = src.line_num;
return 0;
}

6
kernel/bpf/cpumask.c
View File

@@ -449,12 +449,12 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask)
* @src__sz: Length of the BPF memory region in bytes.
*
* Return:
* * 0 if the struct cpumask * instance was populated successfully.
* * 0 if the struct bpf_cpumask * instance was populated successfully.
* * -EACCES if the memory region is too small to populate the cpumask.
* * -EINVAL if the memory region is not aligned to the size of a long
* and the architecture does not support efficient unaligned accesses.
*/
__bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t src__sz)
__bpf_kfunc int bpf_cpumask_populate(struct bpf_cpumask *cpumask, void *src, size_t src__sz)
{
unsigned long source = (unsigned long)src;
@@ -467,7 +467,7 @@ __bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t
!IS_ALIGNED(source, sizeof(long)))
return -EINVAL;
bitmap_copy(cpumask_bits(cpumask), src, nr_cpu_ids);
bitmap_copy(cpumask_bits(&cpumask->cpumask), src, nr_cpu_ids);
return 0;
}

2359
kernel/bpf/diagnostics.c Normal file
View File

File diff suppressed because it is too large Load Diff

111
kernel/bpf/diagnostics.h Normal file
View File

@@ -0,0 +1,111 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef __BPF_DIAGNOSTICS_H
#define __BPF_DIAGNOSTICS_H
#include <linux/bpf.h>
#include <linux/compiler_attributes.h>
#include <linux/stdarg.h>
#include <linux/types.h>
struct bpf_reference_state;
struct bpf_func_state;
struct bpf_reg_state;
struct bpf_verifier_env;
struct bpf_verifier_state;
struct btf;
const char *bpf_diag_fmt_s64_sum(struct bpf_verifier_env *env, s64 value, int addend);
enum bpf_diag_mod_reason {
BPF_DIAG_MOD_WRITE,
BPF_DIAG_MOD_SPILL,
BPF_DIAG_MOD_VAR_WRITE,
BPF_DIAG_MOD_REF_RELEASE,
BPF_DIAG_MOD_PKT_DATA_CHANGE,
BPF_DIAG_MOD_NON_OWN_REF,
BPF_DIAG_MOD_CALLER_SAVED,
};
enum bpf_diag_context_kind {
BPF_DIAG_CONTEXT_NONE,
BPF_DIAG_CONTEXT_RCU,
BPF_DIAG_CONTEXT_PREEMPT,
BPF_DIAG_CONTEXT_IRQ,
BPF_DIAG_CONTEXT_LOCK,
};
enum bpf_diag_invalid_deref_kind {
BPF_DIAG_DEREF_SCALAR,
BPF_DIAG_DEREF_NULLABLE_PTR,
BPF_DIAG_DEREF_MODIFIED_PTR,
BPF_DIAG_DEREF_INVALID_PTR,
};
bool bpf_diag_enabled(const struct bpf_verifier_env *env);
int bpf_diag_init(struct bpf_verifier_env *env);
void bpf_diag_init_frame(struct bpf_verifier_env *env, struct bpf_func_state *state);
char *bpf_diag_fmt_buf(struct bpf_verifier_env *env, size_t size);
const char *bpf_diag_vfmt(struct bpf_verifier_env *env, const char *fmt, va_list args)
__printf(2, 0);
const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __printf(2, 3);
const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id);
const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type);
u64 bpf_diag_event_log_save(struct bpf_verifier_env *env);
void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos);
u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state);
void bpf_diag_free(struct bpf_verifier_env *env);
void bpf_diag_register_type(struct bpf_verifier_env *env, u32 insn_idx, int regno,
const char *problem, const char *reason, const char *suggestion);
void bpf_diag_invalid_deref(struct bpf_verifier_env *env, u32 insn_idx, int regno,
const char *reg_name, const struct bpf_reg_state *reg,
enum bpf_diag_invalid_deref_kind kind, s64 offset);
void bpf_diag_unreadable_reg(struct bpf_verifier_env *env, u32 insn_idx, int regno);
void bpf_diag_stack_arg_uninit(struct bpf_verifier_env *env, u32 insn_idx, int nargs,
int stack_arg_slot, const char *callee_name,
const char *arg_name);
void bpf_diag_memory(struct bpf_verifier_env *env, u32 insn_idx, const char *problem,
const char *reason, const char *suggestion);
void bpf_diag_mem_bounds(struct bpf_verifier_env *env, u32 insn_idx, int regno,
const char *reg_name, const char *type_name, const char *proof,
int off, int size, u32 mem_size, const struct bpf_reg_state *reg);
void bpf_diag_res(struct bpf_verifier_env *env, u32 insn_idx, const char *problem,
const char *reason, const char *suggestion);
void bpf_diag_lock(struct bpf_verifier_env *env, u32 insn_idx, const char *problem,
const char *reason, const char *suggestion,
const struct bpf_reference_state *active_lock);
void bpf_diag_irq(struct bpf_verifier_env *env, u32 insn_idx, const char *problem,
const char *reason, const char *suggestion, u32 depth);
void bpf_diag_leak(struct bpf_verifier_env *env, u32 ref_id, u32 alloc_insn, u32 fail_insn);
void bpf_diag_call_type(struct bpf_verifier_env *env, u32 insn_idx, int argno, int regno,
int stack_arg_slot, const char *call_name, const char *arg_name,
const char *reason, const char *suggestion);
void bpf_diag_ctx_forbidden(struct bpf_verifier_env *env, u32 insn_idx,
const char *operation, const char *suggestion);
void bpf_diag_ctx_active(struct bpf_verifier_env *env, u32 insn_idx, const char *operation,
enum bpf_diag_context_kind ctx_kind, const char *suggestion);
void bpf_diag_ctx_required(struct bpf_verifier_env *env, u32 insn_idx, const char *operation,
enum bpf_diag_context_kind ctx_kind, const char *suggestion);
void bpf_diag_ctx_underflow(struct bpf_verifier_env *env, u32 insn_idx,
const char *operation, enum bpf_diag_context_kind ctx_kind,
const char *suggestion);
void bpf_diag_program_structure(struct bpf_verifier_env *env, u32 insn_idx,
const char *problem, const char *suggestion,
const char *reason_fmt, ...) __printf(5, 6);
void bpf_diag_policy(struct bpf_verifier_env *env, u32 insn_idx, const char *operation,
const char *reason, const char *suggestion);
void bpf_diag_record_branch(struct bpf_verifier_env *env, u32 insn_idx, bool cond_true);
void bpf_diag_mod_begin(struct bpf_verifier_env *env, const struct bpf_reg_state *reg,
const struct bpf_reg_state *origin, enum bpf_diag_mod_reason reason);
void bpf_diag_mod_end(struct bpf_verifier_env *env);
void bpf_diag_record_scrub(struct bpf_verifier_env *env, const struct bpf_reg_state *reg,
enum bpf_diag_mod_reason reason);
void bpf_diag_record_scrub_stack(struct bpf_verifier_env *env,
const struct bpf_func_state *state, s16 min_off, s16 max_off,
enum bpf_diag_mod_reason reason);
void bpf_diag_record_ref_acquire(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id);
void bpf_diag_record_ref_release(struct bpf_verifier_env *env, u32 insn_idx, u32 ref_id);
void bpf_diag_record_context(struct bpf_verifier_env *env, u32 insn_idx,
enum bpf_diag_context_kind ctx_kind, bool enter, u32 depth);
#endif /* __BPF_DIAGNOSTICS_H */

68
kernel/bpf/disasm.c
View File

@@ -139,7 +139,7 @@ static void print_bpf_end_insn(bpf_insn_print_t verbose,
void *private_data,
const struct bpf_insn *insn)
{
verbose(private_data, "(%02x) r%d = %s%d r%d\n",
verbose(private_data, "(%02x) r%d = %s%d r%d",
insn->code, insn->dst_reg,
BPF_SRC(insn->code) == BPF_TO_BE ? "be" : "le",
insn->imm, insn->dst_reg);
@@ -149,7 +149,7 @@ static void print_bpf_bswap_insn(bpf_insn_print_t verbose,
void *private_data,
const struct bpf_insn *insn)
{
verbose(private_data, "(%02x) r%d = bswap%d r%d\n",
verbose(private_data, "(%02x) r%d = bswap%d r%d",
insn->code, insn->dst_reg,
insn->imm, insn->dst_reg);
}
@@ -197,19 +197,19 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
else
print_bpf_end_insn(verbose, cbs->private_data, insn);
} else if (BPF_OP(insn->code) == BPF_NEG) {
verbose(cbs->private_data, "(%02x) %c%d = -%c%d\n",
verbose(cbs->private_data, "(%02x) %c%d = -%c%d",
insn->code, class == BPF_ALU ? 'w' : 'r',
insn->dst_reg, class == BPF_ALU ? 'w' : 'r',
insn->dst_reg);
} else if (is_addr_space_cast(insn)) {
verbose(cbs->private_data, "(%02x) r%d = addr_space_cast(r%d, %u, %u)\n",
verbose(cbs->private_data, "(%02x) r%d = addr_space_cast(r%d, %u, %u)",
insn->code, insn->dst_reg,
insn->src_reg, ((u32)insn->imm) >> 16, (u16)insn->imm);
} else if (is_mov_percpu_addr(insn)) {
verbose(cbs->private_data, "(%02x) r%d = &(void __percpu *)(r%d)\n",
verbose(cbs->private_data, "(%02x) r%d = &(void __percpu *)(r%d)",
insn->code, insn->dst_reg, insn->src_reg);
} else if (BPF_SRC(insn->code) == BPF_X) {
verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d\n",
verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d",
insn->code, class == BPF_ALU ? 'w' : 'r',
insn->dst_reg,
is_sdiv_smod(insn) ? bpf_alu_sign_string[BPF_OP(insn->code) >> 4]
@@ -218,7 +218,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
class == BPF_ALU ? 'w' : 'r',
insn->src_reg);
} else {
verbose(cbs->private_data, "(%02x) %c%d %s %d\n",
verbose(cbs->private_data, "(%02x) %c%d %s %d",
insn->code, class == BPF_ALU ? 'w' : 'r',
insn->dst_reg,
is_sdiv_smod(insn) ? bpf_alu_sign_string[BPF_OP(insn->code) >> 4]
@@ -227,7 +227,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
}
} else if (class == BPF_STX) {
if (BPF_MODE(insn->code) == BPF_MEM)
verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = r%d\n",
verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = r%d",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg,
@@ -235,7 +235,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
else if (BPF_MODE(insn->code) == BPF_ATOMIC &&
(insn->imm == BPF_ADD || insn->imm == BPF_AND ||
insn->imm == BPF_OR || insn->imm == BPF_XOR)) {
verbose(cbs->private_data, "(%02x) lock *(%s *)(r%d %+d) %s r%d\n",
verbose(cbs->private_data, "(%02x) lock *(%s *)(r%d %+d) %s r%d",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg, insn->off,
@@ -246,7 +246,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->imm == (BPF_AND | BPF_FETCH) ||
insn->imm == (BPF_OR | BPF_FETCH) ||
insn->imm == (BPF_XOR | BPF_FETCH))) {
verbose(cbs->private_data, "(%02x) r%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)\n",
verbose(cbs->private_data, "(%02x) r%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)",
insn->code, insn->src_reg,
BPF_SIZE(insn->code) == BPF_DW ? "64" : "",
bpf_atomic_alu_string[BPF_OP(insn->imm) >> 4],
@@ -254,7 +254,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->dst_reg, insn->off, insn->src_reg);
} else if (BPF_MODE(insn->code) == BPF_ATOMIC &&
insn->imm == BPF_CMPXCHG) {
verbose(cbs->private_data, "(%02x) r0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)\n",
verbose(cbs->private_data, "(%02x) r0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)",
insn->code,
BPF_SIZE(insn->code) == BPF_DW ? "64" : "",
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
@@ -262,44 +262,44 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->src_reg);
} else if (BPF_MODE(insn->code) == BPF_ATOMIC &&
insn->imm == BPF_XCHG) {
verbose(cbs->private_data, "(%02x) r%d = atomic%s_xchg((%s *)(r%d %+d), r%d)\n",
verbose(cbs->private_data, "(%02x) r%d = atomic%s_xchg((%s *)(r%d %+d), r%d)",
insn->code, insn->src_reg,
BPF_SIZE(insn->code) == BPF_DW ? "64" : "",
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg, insn->off, insn->src_reg);
} else if (BPF_MODE(insn->code) == BPF_ATOMIC &&
insn->imm == BPF_LOAD_ACQ) {
verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))\n",
verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))",
insn->code, insn->dst_reg,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->src_reg, insn->off);
} else if (BPF_MODE(insn->code) == BPF_ATOMIC &&
insn->imm == BPF_STORE_REL) {
verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)\n",
verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg, insn->off, insn->src_reg);
} else {
verbose(cbs->private_data, "BUG_%02x\n", insn->code);
verbose(cbs->private_data, "BUG_%02x", insn->code);
}
} else if (class == BPF_ST) {
if (BPF_MODE(insn->code) == BPF_MEM) {
verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d\n",
verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg,
insn->off, insn->imm);
} else if (BPF_MODE(insn->code) == 0xc0 /* BPF_NOSPEC, no UAPI */) {
verbose(cbs->private_data, "(%02x) nospec\n", insn->code);
verbose(cbs->private_data, "(%02x) nospec", insn->code);
} else {
verbose(cbs->private_data, "BUG_st_%02x\n", insn->code);
verbose(cbs->private_data, "BUG_st_%02x", insn->code);
}
} else if (class == BPF_LDX) {
if (BPF_MODE(insn->code) != BPF_MEM && BPF_MODE(insn->code) != BPF_MEMSX) {
verbose(cbs->private_data, "BUG_ldx_%02x\n", insn->code);
verbose(cbs->private_data, "BUG_ldx_%02x", insn->code);
return;
}
verbose(cbs->private_data, "(%02x) r%d = *(%s *)(r%d %+d)\n",
verbose(cbs->private_data, "(%02x) r%d = *(%s *)(r%d %+d)",
insn->code, insn->dst_reg,
BPF_MODE(insn->code) == BPF_MEM ?
bpf_ldst_string[BPF_SIZE(insn->code) >> 3] :
@@ -307,12 +307,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->src_reg, insn->off);
} else if (class == BPF_LD) {
if (BPF_MODE(insn->code) == BPF_ABS) {
verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[%d]\n",
verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[%d]",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->imm);
} else if (BPF_MODE(insn->code) == BPF_IND) {
verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[r%d + %d]\n",
verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[r%d + %d]",
insn->code,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->src_reg, insn->imm);
@@ -332,12 +332,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
if (is_ptr && !allow_ptr_leaks)
imm = 0;
verbose(cbs->private_data, "(%02x) r%d = %s\n",
verbose(cbs->private_data, "(%02x) r%d = %s",
insn->code, insn->dst_reg,
__func_imm_name(cbs, insn, imm,
tmp, sizeof(tmp)));
} else {
verbose(cbs->private_data, "BUG_ld_%02x\n", insn->code);
verbose(cbs->private_data, "BUG_ld_%02x", insn->code);
return;
}
} else if (class == BPF_JMP32 || class == BPF_JMP) {
@@ -347,35 +347,35 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
char tmp[64];
if (insn->src_reg == BPF_PSEUDO_CALL) {
verbose(cbs->private_data, "(%02x) call pc%s\n",
verbose(cbs->private_data, "(%02x) call pc%s",
insn->code,
__func_get_name(cbs, insn,
tmp, sizeof(tmp)));
} else {
strcpy(tmp, "unknown");
verbose(cbs->private_data, "(%02x) call %s#%d\n", insn->code,
verbose(cbs->private_data, "(%02x) call %s#%d", insn->code,
__func_get_name(cbs, insn,
tmp, sizeof(tmp)),
insn->imm);
}
} else if (insn->code == (BPF_JMP | BPF_JA)) {
verbose(cbs->private_data, "(%02x) goto pc%+d\n",
verbose(cbs->private_data, "(%02x) goto pc%+d",
insn->code, insn->off);
} else if (insn->code == (BPF_JMP | BPF_JA | BPF_X)) {
verbose(cbs->private_data, "(%02x) gotox r%d\n",
verbose(cbs->private_data, "(%02x) gotox r%d",
insn->code, insn->dst_reg);
} else if (insn->code == (BPF_JMP | BPF_JCOND) &&
insn->src_reg == BPF_MAY_GOTO) {
verbose(cbs->private_data, "(%02x) may_goto pc%+d\n",
verbose(cbs->private_data, "(%02x) may_goto pc%+d",
insn->code, insn->off);
} else if (insn->code == (BPF_JMP32 | BPF_JA)) {
verbose(cbs->private_data, "(%02x) gotol pc%+d\n",
verbose(cbs->private_data, "(%02x) gotol pc%+d",
insn->code, insn->imm);
} else if (insn->code == (BPF_JMP | BPF_EXIT)) {
verbose(cbs->private_data, "(%02x) exit\n", insn->code);
verbose(cbs->private_data, "(%02x) exit", insn->code);
} else if (BPF_SRC(insn->code) == BPF_X) {
verbose(cbs->private_data,
"(%02x) if %c%d %s %c%d goto pc%+d\n",
"(%02x) if %c%d %s %c%d goto pc%+d",
insn->code, class == BPF_JMP32 ? 'w' : 'r',
insn->dst_reg,
bpf_jmp_string[BPF_OP(insn->code) >> 4],
@@ -383,14 +383,14 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
insn->src_reg, insn->off);
} else {
verbose(cbs->private_data,
"(%02x) if %c%d %s 0x%x goto pc%+d\n",
"(%02x) if %c%d %s 0x%x goto pc%+d",
insn->code, class == BPF_JMP32 ? 'w' : 'r',
insn->dst_reg,
bpf_jmp_string[BPF_OP(insn->code) >> 4],
(u32)insn->imm, insn->off);
}
} else {
verbose(cbs->private_data, "(%02x) %s\n",
verbose(cbs->private_data, "(%02x) %s",
insn->code, bpf_class_string[class]);
}
}

5
kernel/bpf/dispatcher.c
View File

@@ -148,7 +148,10 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
d->image = bpf_prog_pack_alloc(PAGE_SIZE, bpf_jit_fill_hole_with_zero, false);
if (!d->image)
goto out;
d->rw_image = bpf_jit_alloc_exec(PAGE_SIZE);
/* d->rw_image doesn't need to be in module memory range, so we
* can use vzalloc.
*/
d->rw_image = vzalloc(PAGE_SIZE);
if (!d->rw_image) {
bpf_prog_pack_free(d->image, PAGE_SIZE);
d->image = NULL;

254
kernel/bpf/fixups.c
View File

@@ -20,6 +20,26 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn)
insn->imm == BPF_CMPXCHG;
}
/* Returns true if 'insn' is an address space cast instruction translated as BPF_ALU op */
static bool is_addr_space_cast32(struct bpf_prog *prog, const struct bpf_insn *insn)
{
struct bpf_map *arena = (struct bpf_map *)prog->aux->arena;
if (insn->code != (BPF_ALU64 | BPF_MOV | BPF_X) || insn->off != BPF_ADDR_SPACE_CAST)
return false;
/* cast from as(1) to as(0) */
if (insn->imm == 1)
return true;
/* cast from as(0) to as(1) */
if (insn->imm == 1 << 16)
return arena && arena->map_flags & BPF_F_NO_USER_CONV;
/* non-BPF_F_NO_USER_CONV cast from as(0) to as(1) should be handled by JIT */
return false;
}
/* Return the regno defined by the insn, or -1. */
static int insn_def_regno(const struct bpf_insn *insn)
{
@@ -29,30 +49,66 @@ static int insn_def_regno(const struct bpf_insn *insn)
case BPF_ST:
return -1;
case BPF_STX:
if (BPF_MODE(insn->code) == BPF_ATOMIC ||
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) {
if (insn->imm == BPF_CMPXCHG)
return BPF_REG_0;
else if (insn->imm == BPF_LOAD_ACQ)
return insn->dst_reg;
else if (insn->imm & BPF_FETCH)
return insn->src_reg;
}
return -1;
return bpf_atomic_load_reg(insn);
default:
return insn->dst_reg;
}
}
/* Return TRUE if INSN has defined any 32-bit value explicitly. */
static bool insn_has_def32(struct bpf_insn *insn)
/*
* For use only in combination with insn_def_regno() >= 0.
* Returns TRUE if the destination register operates on 64-bit,
* otherwise return FALSE.
*/
static bool bpf_is_reg64(struct bpf_prog *prog, struct bpf_insn *insn)
{
u8 class = BPF_CLASS(insn->code);
u8 mode = BPF_MODE(insn->code);
u8 size = BPF_SIZE(insn->code);
u8 op = BPF_OP(insn->code);
bool mode_mem;
/* subregister endiness swap */
if ((class == BPF_ALU || class == BPF_ALU64) && op == BPF_END && insn->imm != 64)
return false;
/* w0 += 1 */
if (class == BPF_ALU && op != BPF_END)
return false;
/* address space casts converted to BPF_ALU, see bpf_do_misc_fixups() */
if (is_addr_space_cast32(prog, insn))
return false;
/* non 64-bit, non signed extended loads */
mode_mem = mode == BPF_MEM || mode == BPF_PROBE_MEM || mode == BPF_PROBE_MEM32;
if (class == BPF_LDX && mode_mem && size != BPF_DW)
return false;
/* atomics, see insn_def_regno() */
if (class == BPF_STX && size != BPF_DW)
return false;
/* both LD_IND and LD_ABS return 32-bit data. */
if (class == BPF_LD && (mode == BPF_IND || mode == BPF_ABS))
return false;
/* Conservatively return true at default. */
return true;
}
/*
* Return the 32-bit subregister defined by INSN, or -1 if INSN does not
* explicitly define a 32-bit value.
*/
int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn)
{
int dst_reg = insn_def_regno(insn);
if (dst_reg == -1)
return false;
if (dst_reg < 0 || bpf_is_reg64(prog, insn))
return -1;
return !bpf_is_reg64(insn, dst_reg, NULL, DST_OP);
return dst_reg;
}
static int kfunc_desc_cmp_by_imm_off(const void *a, const void *b)
@@ -169,11 +225,12 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
* (cnt == 1) is taken or not. There is no guarantee INSN at OFF is the
* original insn at old prog.
*/
data[off].zext_dst = insn_has_def32(insn + off + cnt - 1);
data[off].zext_dst = bpf_insn_def32(new_prog, insn + off + cnt - 1) >= 0;
if (cnt == 1)
return;
prog_len = new_prog->len;
env->insn_aux_data_len = prog_len;
memmove(data + off + cnt - 1, data + off,
sizeof(struct bpf_insn_aux_data) * (prog_len - off - cnt + 1));
@@ -181,7 +238,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
for (i = off; i < off + cnt - 1; i++) {
/* Expand insni[off]'s seen count to the patched range. */
data[i].seen = old_seen;
data[i].zext_dst = insn_has_def32(insn + i);
data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
}
/*
@@ -345,13 +402,17 @@ static int adjust_subprog_starts_after_remove(struct bpf_verifier_env *env,
sizeof(*env->subprog_info) * move);
env->subprog_cnt -= j - i;
/* remove func_info */
/* remove func_info and its aux */
if (aux->func_info) {
move = aux->func_info_cnt - j;
memmove(aux->func_info + i,
aux->func_info + j,
sizeof(*aux->func_info) * move);
if (aux->func_info_aux)
memmove(aux->func_info_aux + i,
aux->func_info_aux + j,
sizeof(*aux->func_info_aux) * move);
aux->func_info_cnt -= j - i;
/* func_info->insn_off is set after all code rewrites,
* in adjust_btf_func() - no need to adjust
@@ -440,7 +501,6 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off,
void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
{
struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
struct bpf_insn *insns = env->prog->insnsi;
int end = start + len;
int i;
@@ -449,9 +509,6 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
kvfree(aux_data[i].jt);
aux_data[i].jt = NULL;
}
if (bpf_is_ldimm64(&insns[i]))
i++;
}
}
@@ -464,7 +521,6 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
if (bpf_prog_is_offloaded(env->prog->aux))
bpf_prog_offload_remove_insns(env, off, cnt);
/* Should be called before bpf_remove_insns, as it uses prog->insnsi */
bpf_clear_insn_aux_data(env, off, cnt);
err = bpf_remove_insns(env->prog, off, cnt);
@@ -483,6 +539,7 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
memmove(aux_data + off, aux_data + off + cnt,
sizeof(*aux_data) * (orig_prog_len - off - cnt));
env->insn_aux_data_len -= cnt;
return 0;
}
@@ -616,11 +673,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,
if (load_reg == -1)
continue;
/* NOTE: arg "reg" (the fourth one) is only used for
* BPF_STX + SRC_OP, so it is safe to pass NULL
* here.
*/
if (bpf_is_reg64(&insn, load_reg, NULL, DST_OP)) {
if (bpf_is_reg64(env->prog, &insn)) {
if (class == BPF_LD &&
BPF_MODE(code) == BPF_IMM)
i++;
@@ -759,6 +812,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
for (i = 0; i < insn_cnt; i++, insn++) {
bpf_convert_ctx_access_t convert_ctx_access;
enum bpf_reg_type ptr_type;
u8 mode;
if (env->insn_aux_data[i + delta].nospec) {
@@ -851,7 +905,8 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
continue;
}
switch ((int)env->insn_aux_data[i + delta].ptr_type) {
ptr_type = env->insn_aux_data[i + delta].ptr_type;
switch ((int)ptr_type) {
case PTR_TO_CTX:
if (!ops->convert_ctx_access)
continue;
@@ -867,26 +922,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
case PTR_TO_XDP_SOCK:
convert_ctx_access = bpf_xdp_sock_convert_ctx_access;
break;
case PTR_TO_BTF_ID:
case PTR_TO_BTF_ID | PTR_UNTRUSTED:
/* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike
* PTR_TO_BTF_ID, and an active referenced id, but the same cannot
* be said once it is marked PTR_UNTRUSTED, hence we must handle
* any faults for loads into such types. BPF_WRITE is disallowed
* for this case.
*/
case PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED:
case PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED:
if (type == BPF_READ) {
if (BPF_MODE(insn->code) == BPF_MEM)
insn->code = BPF_LDX | BPF_PROBE_MEM |
BPF_SIZE((insn)->code);
else
insn->code = BPF_LDX | BPF_PROBE_MEMSX |
BPF_SIZE((insn)->code);
env->prog->aux->num_exentries++;
}
continue;
case PTR_TO_ARENA:
if (BPF_MODE(insn->code) == BPF_MEMSX) {
if (!bpf_jit_supports_insn(insn, true)) {
@@ -900,6 +935,29 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
env->prog->aux->num_exentries++;
continue;
default:
/*
* A pointer which may fault on a dereference must not
* be loaded from without fault protection, hence turn
* the BPF_LDX into a BPF_PROBE_MEM one so that a bad
* address is handled rather than panicking the kernel.
* A store through one is rejected earlier, there is no
* probed counterpart to rewrite it into.
*/
if (bpf_is_ptr_to_mem_or_btf_id(ptr_type) &&
bpf_may_fault_on_deref(ptr_type) &&
type == BPF_READ) {
if (BPF_MODE(insn->code) == BPF_MEM)
insn->code = BPF_LDX | BPF_PROBE_MEM |
BPF_SIZE(insn->code);
else
insn->code = BPF_LDX | BPF_PROBE_MEMSX |
BPF_SIZE(insn->code);
env->prog->aux->num_exentries++;
continue;
}
if (verifier_bug_if(bpf_may_fault_on_deref(ptr_type), env,
"access to a fault prone pointer is not rewritten as a probed one"))
return -EFAULT;
continue;
}
@@ -1005,26 +1063,6 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s
env->subprog_info[env->subprog_cnt].start = env->prog->len;
}
struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env)
{
size_t size;
void *new_aux;
size = array_size(sizeof(struct bpf_insn_aux_data), env->prog->len);
new_aux = __vmalloc(size, GFP_KERNEL_ACCOUNT);
if (new_aux)
memcpy(new_aux, env->insn_aux_data, size);
return new_aux;
}
void bpf_restore_insn_aux_data(struct bpf_verifier_env *env,
struct bpf_insn_aux_data *orig_insn_aux)
{
/* the expanded elements are zero-filled, so no special handling is required */
vfree(env->insn_aux_data);
env->insn_aux_data = orig_insn_aux;
}
static int jit_subprogs(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog, **func, *tmp;
@@ -1299,7 +1337,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)
bool blinded = false;
struct bpf_insn *insn;
struct bpf_prog *prog, *orig_prog;
struct bpf_insn_aux_data *orig_insn_aux;
u32 *orig_subprog_starts;
if (env->subprog_cnt <= 1)
@@ -1307,14 +1344,8 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)
prog = orig_prog = env->prog;
if (bpf_prog_need_blind(prog)) {
orig_insn_aux = bpf_dup_insn_aux_data(env);
if (!orig_insn_aux) {
err = -ENOMEM;
goto out_cleanup;
}
orig_subprog_starts = bpf_dup_subprog_starts(env);
if (!orig_subprog_starts) {
vfree(orig_insn_aux);
err = -ENOMEM;
goto out_cleanup;
}
@@ -1334,7 +1365,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)
if (blinded) {
bpf_jit_prog_release_other(prog, orig_prog);
kvfree(orig_subprog_starts);
vfree(orig_insn_aux);
}
return 0;
@@ -1364,7 +1394,6 @@ int bpf_jit_subprogs(struct bpf_verifier_env *env)
out_restore:
bpf_restore_subprog_starts(env, orig_subprog_starts);
bpf_restore_insn_aux_data(env, orig_insn_aux);
kvfree(orig_subprog_starts);
out_cleanup:
/* cleanup main prog to be interpreted */
@@ -1378,7 +1407,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
struct bpf_prog *prog = env->prog;
struct bpf_insn *insn = prog->insnsi;
bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);
int depth;
#endif
int i, err = 0;
@@ -1404,8 +1432,8 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
return err;
}
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
if (has_kfunc_call) {
verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
if (prog->jit_required) {
verbose(env, "program requires BPF JIT compiler but it is not available\n");
return -EINVAL;
}
for (i = 0; i < env->subprog_cnt; i++) {
@@ -1447,7 +1475,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
return err;
}
/* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */
static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)
{
@@ -1514,15 +1541,12 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
for (i = 0; i < insn_cnt;) {
if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && insn->imm) {
if ((insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1) ||
(((struct bpf_map *)env->prog->aux->arena)->map_flags & BPF_F_NO_USER_CONV)) {
/* convert to 32-bit mov that clears upper 32-bit */
insn->code = BPF_ALU | BPF_MOV | BPF_X;
/* clear off and imm, so it's a normal 'wX = wY' from JIT pov */
insn->off = 0;
insn->imm = 0;
} /* cast from as(0) to as(1) should be handled by JIT */
if (is_addr_space_cast32(env->prog, insn)) {
/* convert to 32-bit mov that clears upper 32-bit */
insn->code = BPF_ALU | BPF_MOV | BPF_X;
/* clear off and imm, so it's a normal 'wX = wY' from JIT pov */
insn->off = 0;
insn->imm = 0;
goto next_insn;
}
@@ -1819,6 +1843,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
goto next_insn;
}
if (bpf_jit_supports_percpu_insn() &&
insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
(insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
struct bpf_map *map;
aux = &env->insn_aux_data[i + delta];
map = env->used_maps[aux->map_index];
if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
goto next_insn;
prog->jit_required = true;
/*
* We are *skipping* first half of ld_imm64 insn
* with 'i++;', patching over second half of it
* with that same half + mov64_percpu_reg insn.
* All because bpf_patch_insn_data() can only
* replace one 8-byte insn, which does not work
* well for ld_imm64 insn.
*/
insn_buf[0] = insn[1];
insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
cnt = 2;
i++;
new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
if (!new_prog)
return -ENOMEM;
delta += cnt - 1;
env->prog = prog = new_prog;
insn = new_prog->insnsi + i + delta;
goto next_insn;
}
if (insn->code != (BPF_JMP | BPF_CALL))
goto next_insn;
if (insn->src_reg == BPF_PSEUDO_CALL)
@@ -1841,8 +1902,10 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
/* Skip inlining the helper call if the JIT does it. */
if (bpf_jit_inlines_helper_call(insn->imm))
if (bpf_jit_inlines_helper_call(insn->imm)) {
prog->jit_required = 1;
goto next_insn;
}
if (insn->imm == BPF_FUNC_get_route_realm)
prog->dst_needed = 1;
@@ -2007,6 +2070,9 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
return -EFAULT;
}
if (bpf_map_is_percpu_map(map_ptr->map_type))
prog->jit_required = true;
new_prog = bpf_patch_insn_data(env, i + delta,
insn_buf, cnt);
if (!new_prog)
@@ -2111,6 +2177,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
* way, it's fine to back out this inlining logic
*/
#ifdef CONFIG_SMP
prog->jit_required = true;
insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_number);
insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0);
@@ -2132,6 +2199,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
/* Implement bpf_get_current_task() and bpf_get_current_task_btf() inline. */
if ((insn->imm == BPF_FUNC_get_current_task || insn->imm == BPF_FUNC_get_current_task_btf) &&
bpf_verifier_inlines_helper_call(env, insn->imm)) {
prog->jit_required = true;
insn_buf[0] = BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&current_task);
insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
insn_buf[2] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0);
@@ -2338,7 +2406,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
func_id_name(insn->imm), insn->imm);
return -EFAULT;
}
insn->imm = fn->func - __bpf_call_base;
insn->imm = BPF_CALL_IMM(fn->func);
next_insn:
if (subprogs[cur_subprog + 1].start == i + delta + 1) {
subprogs[cur_subprog].stack_depth += stack_depth_extra;

2
kernel/bpf/hashtab.c
View File

@@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab *htab)
atomic_dec(&htab->count);
}
static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
{
htab_put_fd_value(htab, l);
@@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v
return 0;
}
static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);

116
kernel/bpf/helpers.c
View File

@@ -278,7 +278,7 @@ const struct bpf_func_proto bpf_get_current_comm_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
@@ -539,7 +539,7 @@ const struct bpf_func_proto bpf_strtol_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
@@ -567,7 +567,7 @@ const struct bpf_func_proto bpf_strtoul_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
@@ -583,7 +583,7 @@ static const struct bpf_func_proto bpf_strncmp_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
@@ -627,7 +627,7 @@ const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
.arg4_type = ARG_MEM_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
@@ -653,7 +653,7 @@ const struct bpf_func_proto bpf_event_output_data_proto = {
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
@@ -675,7 +675,7 @@ const struct bpf_func_proto bpf_copy_from_user_proto = {
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -706,7 +706,7 @@ const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
@@ -1093,10 +1093,10 @@ const struct bpf_func_proto bpf_snprintf_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
@@ -1888,7 +1888,7 @@ static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
@@ -1943,7 +1943,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
@@ -2004,7 +2004,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = {
.arg1_type = ARG_PTR_TO_DYNPTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_MEM_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
@@ -3395,19 +3395,21 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
/*
* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
* which skips compiler generated instrumentation to do the same. Some
* architectures cannot recover sp while unwinding, so fall back to bp.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
kasan_unpoison_task_stack_below((void *)(long)(ctx.sp ?: ctx.bp));
ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
struct bpf_map *map = p__const_map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
@@ -4388,7 +4390,7 @@ struct bpf_task_work_ctx {
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
bpf_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
@@ -4471,7 +4473,8 @@ static void bpf_task_work_callback(struct callback_head *cb)
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
ctx->callback_fn((u64)(long)ctx->map, (u64)(long)key,
(u64)(long)ctx->map_val, 0, 0);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
@@ -4594,7 +4597,7 @@ static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_map *map, void *callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
@@ -4619,7 +4622,7 @@ static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->callback_fn = (bpf_callback_t)callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
@@ -4642,17 +4645,17 @@ static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @map__const_map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
void *map__const_map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
return bpf_task_work_schedule(task, tw, map__const_map, callback, aux, TWA_SIGNAL);
}
/**
@@ -4660,17 +4663,17 @@ __bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct b
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @map__const_map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
void *map__const_map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
return bpf_task_work_schedule(task, tw, map__const_map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
@@ -4811,30 +4814,32 @@ BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_is_first)
BTF_ID_FLAGS(func, bpf_list_is_last)
BTF_ID_FLAGS(func, bpf_list_empty)
BTF_ID_FLAGS(func, bpf_refcount_acquire,
KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl,
KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_push_front_impl, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_push_back_impl, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_is_first, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_is_last, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_list_empty, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL | KF_SPINLOCK_SAFE)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
@@ -4868,7 +4873,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
@@ -4884,9 +4888,9 @@ BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
@@ -4961,8 +4965,8 @@ BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)

120
kernel/bpf/liveness.c
View File

@@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
__diag_pop();
static inline bool update_insn(struct bpf_verifier_env *env,
struct func_instance *instance, u32 frame, u32 insn_idx)
{
@@ -497,7 +496,6 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
pos = env->log.end_pos;
verbose(env, "%3d: ", insn_idx);
bpf_verbose_insn(env, &insns[insn_idx]);
bpf_vlog_reset(&env->log, env->log.end_pos - 1); /* remove \n */
insn_pos = env->log.end_pos;
verbose(env, "%*c;", bpf_vlog_alignment(insn_pos - pos), ' ');
pos = env->log.end_pos;
@@ -1043,7 +1041,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
if (!printed) {
verbose(env, "%3d: ", idx);
bpf_verbose_insn(env, insn);
bpf_vlog_reset(&env->log, env->log.end_pos - 1);
printed = true;
}
verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);
@@ -1058,7 +1055,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
if (!printed) {
verbose(env, "%3d: ", idx);
bpf_verbose_insn(env, insn);
bpf_vlog_reset(&env->log, env->log.end_pos - 1);
printed = true;
}
verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]);
@@ -1070,7 +1066,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
if (!printed) {
verbose(env, "%3d: ", idx);
bpf_verbose_insn(env, insn);
bpf_vlog_reset(&env->log, env->log.end_pos - 1);
printed = true;
}
verbose(env, "\tfp%+d: ", -(i + 1) * 8); verbose_arg_track(env, &at_stack_in[i]);
@@ -1213,12 +1208,9 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
clear_stack_for_all_offs(insn, at_out, insn->dst_reg,
at_stack_out, sz);
if (insn->imm == BPF_CMPXCHG)
at_out[BPF_REG_0] = none;
else if (insn->imm == BPF_LOAD_ACQ)
*dst = none;
else if (insn->imm & BPF_FETCH)
*src = none;
r = bpf_atomic_load_reg(insn);
if (r >= 0)
at_out[r] = none;
}
} else if (class == BPF_ST && BPF_MODE(insn->code) == BPF_MEM) {
u32 sz = bpf_size_to_bytes(BPF_SIZE(insn->code));
@@ -1545,6 +1537,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
verbose(env, "%3d: ", idx);
bpf_verbose_insn(env, &insns[idx]);
verbose(env, "\n");
/* Collect what needs printing */
if (is_ldx_stx_call &&
@@ -1868,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env *env,
if (need_resched())
cond_resched();
/*
* When an instance is reused (must_write_initialized == true),
* record into a fresh instance and merge afterward. This avoids
@@ -2050,29 +2042,38 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
/* Each field is a register bitmask */
struct insn_live_regs {
u16 use; /* registers read by instruction */
u16 def; /* registers written by instruction */
u16 in; /* registers that may be alive before instruction */
u16 out; /* registers that may be alive after instruction */
u32 use; /* registers read by instruction */
u32 def; /* registers written by instruction */
u32 in; /* registers that may be alive before instruction */
u32 out; /* registers that may be alive after instruction */
};
/* Bitmask with 1s for all caller saved registers */
#define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
static inline u32 reg32_mask(u32 n) { return BIT(n); }
static inline u32 reg64_mask(u32 n) { return BIT(n) | BIT(n + 16); }
static inline u32 mask_widen(u32 m) { return m | (m << 16); }
static inline u16 mask_lo(u32 m) { return (u16)m; }
static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
/* Compute info->{use,def} fields for the instruction */
static void compute_insn_live_regs(struct bpf_verifier_env *env,
struct bpf_insn *insn,
struct insn_live_regs *info)
{
struct bpf_call_summary cs;
u8 class = BPF_CLASS(insn->code);
u8 code = BPF_OP(insn->code);
u8 mode = BPF_MODE(insn->code);
u16 src = BIT(insn->src_reg);
u16 dst = BIT(insn->dst_reg);
u16 r0 = BIT(0);
u16 def = 0;
u16 use = 0xffff;
const u8 class = BPF_CLASS(insn->code);
const u8 code = BPF_OP(insn->code);
const u8 mode = BPF_MODE(insn->code);
const u8 size = BPF_SIZE(insn->code);
const u32 src = reg64_mask(insn->src_reg);
const u32 dst = reg64_mask(insn->dst_reg);
const u32 src32 = mask_lo(src);
const u32 dst32 = mask_lo(dst);
const u32 r0 = reg64_mask(0);
u32 def = 0;
u32 use = U32_MAX;
switch (class) {
case BPF_LD:
@@ -2083,8 +2084,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
use = 0;
}
break;
case BPF_LD | BPF_ABS:
case BPF_LD | BPF_IND:
case BPF_ABS:
case BPF_IND:
/* stick with defaults */
break;
}
@@ -2092,7 +2093,15 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
case BPF_LDX:
switch (mode) {
case BPF_MEM:
/* a narrow load still redefines the whole register */
def = dst;
use = src;
break;
case BPF_MEMSX:
/*
* sign extension defines the whole register;
* src holds a pointer, hence is used as 64-bit.
*/
def = dst;
use = src;
break;
@@ -2110,12 +2119,19 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
switch (mode) {
case BPF_MEM:
def = 0;
use = dst | src;
use = dst | (size == BPF_DW ? src : src32);
break;
case BPF_ATOMIC:
case BPF_ATOMIC: {
/*
* dst holds a pointer and is always used as 64-bit;
* the value operand and r0 are read as 32-bit for BPF_W atomics.
*/
u32 srcv = size == BPF_DW ? src : src32;
u32 r0v = size == BPF_DW ? r0 : mask_lo(r0);
switch (insn->imm) {
case BPF_CMPXCHG:
use = r0 | dst | src;
use = r0v | dst | srcv;
def = r0;
break;
case BPF_LOAD_ACQ:
@@ -2124,10 +2140,10 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
break;
case BPF_STORE_REL:
def = 0;
use = dst | src;
use = dst | srcv;
break;
default:
use = dst | src;
use = dst | srcv;
if (insn->imm & BPF_FETCH)
def = src;
else
@@ -2135,6 +2151,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
}
break;
}
}
break;
case BPF_ALU:
case BPF_ALU64:
@@ -2148,14 +2165,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
if (BPF_SRC(insn->code) == BPF_K)
use = 0;
else
use = src;
use = class == BPF_ALU64 ? src : src32;
break;
default:
def = dst;
if (BPF_SRC(insn->code) == BPF_K)
use = dst;
use = class == BPF_ALU64 ? dst : dst32;
else
use = dst | src;
use = class == BPF_ALU64 ? (dst | src) : (dst32 | src32);
}
break;
case BPF_JMP:
@@ -2181,13 +2198,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
use = def & ~BIT(BPF_REG_0);
if (bpf_get_call_summary(env, insn, &cs))
use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
def = mask_widen(def);
use = mask_widen(use);
break;
default:
def = 0;
if (BPF_SRC(insn->code) == BPF_K)
use = dst;
else
use = dst | src;
use = class == BPF_JMP ? dst : dst32;
if (BPF_SRC(insn->code) == BPF_X)
use |= class == BPF_JMP ? src : src32;
}
break;
}
@@ -2209,6 +2227,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
struct bpf_insn *insns = env->prog->insnsi;
struct insn_live_regs *state;
int insn_cnt = env->prog->len;
u64 pos, insn_pos;
int err = 0, i, j;
bool changed;
@@ -2252,8 +2271,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
int insn_idx = env->cfg.insn_postorder[i];
struct insn_live_regs *live = &state[insn_idx];
struct bpf_iarray *succ;
u16 new_out = 0;
u16 new_in = 0;
u32 new_out = 0;
u32 new_in = 0;
succ = bpf_insn_successors(env, insn_idx);
for (int s = 0; s < succ->cnt; ++s)
@@ -2267,8 +2286,20 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
}
}
for (i = 0; i < insn_cnt; ++i)
insn_aux[i].live_regs_before = state[i].in;
for (i = 0; i < insn_cnt; ++i) {
int def32 = bpf_insn_def32(env->prog, &insns[i]);
u32 out = state[i].out;
u32 in = state[i].in;
insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in);
/*
* On architectures where 32-bit operations do not reset upper halves
* of the registers, the verifier needs to zero extend a destination
* register if an instruction defines a 32-bit subregister and the
* upper half of that register is alive after the instruction.
*/
insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
}
if (env->log.level & BPF_LOG_LEVEL2) {
verbose(env, "Live regs before insn:\n");
@@ -2284,7 +2315,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
else
verbose(env, ".");
verbose(env, " ");
pos = env->log.end_pos;
bpf_verbose_insn(env, &insns[i]);
insn_pos = env->log.end_pos;
if (insn_aux[i].zext_dst)
verbose(env, "%*c; zext", bpf_vlog_alignment(insn_pos - pos), ' ');
verbose(env, "\n");
if (bpf_is_ldimm64(&insns[i]))
i++;
}

11
kernel/bpf/log.c
View File

@@ -615,17 +615,6 @@ static void print_scalar_ranges(struct bpf_verifier_env *env,
}
}
static bool type_is_map_ptr(enum bpf_reg_type t) {
switch (base_type(t)) {
case CONST_PTR_TO_MAP:
case PTR_TO_MAP_KEY:
case PTR_TO_MAP_VALUE:
return true;
default:
return false;
}
}
/*
* _a stands for append, was shortened to avoid multiline statements below.
* This macro is used to output a comma separated list of attributes.

51
kernel/bpf/mmap_unlock_work.h
View File

@@ -4,12 +4,15 @@
#ifndef __MMAP_UNLOCK_WORK_H__
#define __MMAP_UNLOCK_WORK_H__
#include <linux/atomic.h>
#include <linux/err.h>
#include <linux/irq_work.h>
/* irq_work to run mmap_read_unlock() in irq_work */
struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
atomic_t active;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
* We cannot do mmap_read_unlock() when the irq is disabled, because of
* risk to deadlock with rq_lock. To look up vma when the irqs are
* disabled, we need to run mmap_read_unlock() in irq_work. We use a
* percpu variable to do the irq_work. If the irq_work is already used
* by another lookup, we fall over.
* percpu variable to do the irq_work. The active flag reserves the slot
* before mmap_read_trylock() and until the irq_work callback consumes mm.
*/
static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr)
static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
struct mmap_unlock_irq_work *work = NULL;
bool irq_work_busy = false;
struct mmap_unlock_irq_work *work;
if (irqs_disabled()) {
if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
work = this_cpu_ptr(&mmap_unlock_work);
if (irq_work_is_busy(&work->irq_work)) {
/* cannot queue more up_read, fallback */
irq_work_busy = true;
}
} else {
/*
* PREEMPT_RT does not allow to trylock mmap sem in
* interrupt disabled context. Force the fallback code.
*/
irq_work_busy = true;
}
}
if (!irqs_disabled())
return NULL;
*work_ptr = work;
return irq_work_busy;
/*
* PREEMPT_RT does not allow to trylock mmap sem in interrupt
* disabled context. Force the fallback code.
*/
if (IS_ENABLED(CONFIG_PREEMPT_RT))
return ERR_PTR(-EBUSY);
work = this_cpu_ptr(&mmap_unlock_work);
if (irq_work_is_busy(&work->irq_work) ||
atomic_cmpxchg_acquire(&work->active, 0, 1))
return ERR_PTR(-EBUSY);
return work;
}
static inline void
bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
{
if (work)
atomic_set_release(&work->active, 0);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)

17
kernel/bpf/net_namespace.c
View File

@@ -171,33 +171,28 @@ static int bpf_netns_link_update_prog(struct bpf_link *link,
struct net *net;
int idx, ret;
guard(mutex)(&netns_bpf_mutex);
if (old_prog && old_prog != link->prog)
return -EPERM;
if (new_prog->type != link->prog->type)
return -EINVAL;
mutex_lock(&netns_bpf_mutex);
net = net_link->net;
if (!net || !check_net(net)) {
if (!net || !check_net(net))
/* Link auto-detached or netns dying */
ret = -ENOLINK;
goto out_unlock;
}
return -ENOLINK;
run_array = rcu_dereference_protected(net->bpf.run_array[type],
lockdep_is_held(&netns_bpf_mutex));
idx = link_index(net, type, net_link);
ret = bpf_prog_array_update_at(run_array, idx, new_prog);
if (ret)
goto out_unlock;
return ret;
old_prog = xchg(&link->prog, new_prog);
bpf_prog_put(old_prog);
out_unlock:
mutex_unlock(&netns_bpf_mutex);
return ret;
return 0;
}
static int bpf_netns_link_fill_info(const struct bpf_link *link,

9
kernel/bpf/queue_stack_maps.c
View File

@@ -99,8 +99,10 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)
int err = 0;
void *ptr;
if (raw_res_spin_lock_irqsave(&qs->lock, flags))
if (raw_res_spin_lock_irqsave(&qs->lock, flags)) {
memset(value, 0, qs->map.value_size);
return -EBUSY;
}
if (queue_stack_map_is_empty(qs)) {
memset(value, 0, qs->map.value_size);
@@ -121,7 +123,6 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)
return err;
}
static long __stack_map_get(struct bpf_map *map, void *value, bool delete)
{
struct bpf_queue_stack *qs = bpf_queue_stack(map);
@@ -130,8 +131,10 @@ static long __stack_map_get(struct bpf_map *map, void *value, bool delete)
void *ptr;
u32 index;
if (raw_res_spin_lock_irqsave(&qs->lock, flags))
if (raw_res_spin_lock_irqsave(&qs->lock, flags)) {
memset(value, 0, qs->map.value_size);
return -EBUSY;
}
if (queue_stack_map_is_empty(qs)) {
memset(value, 0, qs->map.value_size);

6
kernel/bpf/ringbuf.c
View File

@@ -321,7 +321,7 @@ static unsigned long ringbuf_avail_data_sz(struct bpf_ringbuf *rb)
if (unlikely(rb->overwrite_mode)) {
over_pos = smp_load_acquire(&rb->overwrite_pos);
prod_pos = smp_load_acquire(&rb->producer_pos);
return prod_pos - max(cons_pos, over_pos);
return min(prod_pos - cons_pos, prod_pos - over_pos);
} else {
prod_pos = smp_load_acquire(&rb->producer_pos);
return prod_pos - cons_pos;
@@ -482,7 +482,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size)
prod_pos = rb->producer_pos;
new_prod_pos = prod_pos + len;
while (pend_pos < prod_pos) {
while (prod_pos - pend_pos > 0) {
hdr = (void *)rb->data + (pend_pos & rb->mask);
hdr_len = READ_ONCE(hdr->len);
if (hdr_len & BPF_RINGBUF_BUSY_BIT)
@@ -634,7 +634,7 @@ const struct bpf_func_proto bpf_ringbuf_output_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};

8
kernel/bpf/rqspinlock.c
View File

@@ -745,10 +745,10 @@ __bpf_kfunc void bpf_res_spin_unlock_irqrestore(struct bpf_res_spin_lock *lock,
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(rqspinlock_kfunc_ids)
BTF_ID_FLAGS(func, bpf_res_spin_lock, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_res_spin_unlock)
BTF_ID_FLAGS(func, bpf_res_spin_lock_irqsave, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_res_spin_unlock_irqrestore)
BTF_ID_FLAGS(func, bpf_res_spin_lock, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_res_spin_unlock, KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_res_spin_lock_irqsave, KF_RET_NULL | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_res_spin_unlock_irqrestore, KF_SPINLOCK_SAFE)
BTF_KFUNCS_END(rqspinlock_kfunc_ids)
static const struct btf_kfunc_id_set rqspinlock_kfunc_set = {

351
kernel/bpf/stackmap.c
View File

@@ -414,8 +414,7 @@ static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *i
static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
u32 trace_nr, bool user, bool may_fault)
{
struct mmap_unlock_irq_work *work = NULL;
bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
struct mmap_unlock_irq_work *work;
bool has_user_ctx = user && current && current->mm;
struct stack_map_build_id_cache cache = {};
struct vm_area_struct *vma;
@@ -426,15 +425,16 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
return;
}
/* If the irq_work is in use, fall back to report ips. Same
* fallback is used for kernel stack (!user) on a stackmap with
* build_id.
*/
if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) {
/* cannot access current->mm, fall back to ips */
for (i = 0; i < trace_nr; i++)
stack_map_build_id_set_ip(&id_offs[i]);
return;
if (!has_user_ctx)
goto fallback;
work = bpf_mmap_unlock_guard_get();
if (IS_ERR(work))
goto fallback;
if (!mmap_read_trylock(current->mm)) {
bpf_mmap_unlock_guard_put(work);
goto fallback;
}
for (i = 0; i < trace_nr; i++) {
@@ -465,6 +465,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
vma->vm_pgoff);
}
bpf_mmap_unlock_mm(work, current->mm);
return;
fallback:
/* cannot access current->mm, fall back to ips */
for (i = 0; i < trace_nr; i++)
stack_map_build_id_set_ip(&id_offs[i]);
}
static struct perf_callchain_entry *
@@ -504,78 +510,116 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth)
#endif
}
static long __bpf_get_stackid(struct bpf_map *map,
struct perf_callchain_entry *trace, u64 flags)
struct stackid {
struct stack_map_bucket *bucket;
const u64 *ips;
u32 nr;
u32 len;
u32 hash;
u32 id;
bool hash_matches;
};
static int stackid_init(struct stackid *stackid, struct bpf_map *map,
const struct perf_callchain_entry *trace, u32 trace_nr, u64 flags)
{
struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
struct stack_map_bucket *bucket, *new_bucket, *old_bucket;
u32 hash, id, trace_nr, trace_len, i, max_depth;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
bool user = flags & BPF_F_USER_STACK;
u64 *ips;
bool hash_matches;
u32 max_depth;
if (trace->nr <= skip)
if (trace_nr <= skip)
/* skipping more than usable stack trace */
return -EFAULT;
max_depth = stack_map_calculate_max_depth(map->value_size, stack_map_data_size(map), flags);
trace_nr = min_t(u32, trace->nr - skip, max_depth - skip);
trace_len = trace_nr * sizeof(u64);
ips = trace->ip + skip;
hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0);
id = hash & (smap->n_buckets - 1);
bucket = READ_ONCE(smap->buckets[id]);
stackid->nr = min_t(u32, trace_nr - skip, max_depth - skip);
stackid->len = stackid->nr * sizeof(u64);
stackid->ips = trace->ip + skip;
stackid->hash = jhash2((const u32 *)stackid->ips, stackid->len / sizeof(u32), 0);
stackid->id = stackid->hash & (smap->n_buckets - 1);
stackid->bucket = READ_ONCE(smap->buckets[stackid->id]);
stackid->hash_matches = stackid->bucket && stackid->bucket->hash == stackid->hash;
return 0;
}
static int stackid_fastpath(struct stackid *stackid, struct bpf_map *map,
const struct perf_callchain_entry *trace, u32 trace_nr,
u64 flags)
{
int err;
err = stackid_init(stackid, map, trace, trace_nr, flags);
if (err)
return err;
hash_matches = bucket && bucket->hash == hash;
/* fast cmp */
if (hash_matches && flags & BPF_F_FAST_STACK_CMP)
return id;
if (stackid->hash_matches && flags & BPF_F_FAST_STACK_CMP)
return stackid->id;
if (stack_map_use_build_id(map))
return -ENOENT;
if (stackid->hash_matches && stackid->bucket->nr == stackid->nr &&
memcmp(stackid->bucket->data, stackid->ips, stackid->len) == 0)
return stackid->id;
if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID))
return -EEXIST;
return -ENOENT;
}
static struct stack_map_bucket *
stackid_new_bucket(struct stackid *stackid, struct bpf_map *map)
{
struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
struct bpf_stack_build_id *id_offs;
struct stack_map_bucket *bucket;
u32 i;
bucket = (struct stack_map_bucket *) pcpu_freelist_pop(&smap->freelist);
if (unlikely(!bucket))
return NULL;
if (stack_map_use_build_id(map)) {
id_offs = (struct bpf_stack_build_id *)bucket->data;
for (i = 0; i < stackid->nr; i++)
id_offs[i].ip = stackid->ips[i];
} else {
memcpy(bucket->data, stackid->ips, stackid->len);
}
bucket->hash = stackid->hash;
bucket->nr = stackid->nr;
return bucket;
}
static long stackid_install(struct stackid *stackid, struct bpf_map *map,
struct stack_map_bucket *new_bucket, u64 flags)
{
struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
bool user = flags & BPF_F_USER_STACK;
struct stack_map_bucket *old_bucket;
u32 trace_len;
if (stack_map_use_build_id(map)) {
struct bpf_stack_build_id *id_offs;
/* for build_id+offset, pop a bucket before slow cmp */
new_bucket = (struct stack_map_bucket *)
pcpu_freelist_pop(&smap->freelist);
if (unlikely(!new_bucket))
return -ENOMEM;
new_bucket->nr = trace_nr;
id_offs = (struct bpf_stack_build_id *)new_bucket->data;
for (i = 0; i < trace_nr; i++)
id_offs[i].ip = ips[i];
stack_map_get_build_id_offset(id_offs, trace_nr, user, false /* !may_fault */);
trace_len = trace_nr * sizeof(struct bpf_stack_build_id);
if (hash_matches && bucket->nr == trace_nr &&
memcmp(bucket->data, new_bucket->data, trace_len) == 0) {
stack_map_get_build_id_offset(id_offs, stackid->nr, user, false /* !may_fault */);
trace_len = stackid->nr * sizeof(struct bpf_stack_build_id);
if (stackid->hash_matches && stackid->bucket->nr == stackid->nr &&
memcmp(stackid->bucket->data, new_bucket->data, trace_len) == 0) {
pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
return id;
return stackid->id;
}
if (bucket && !(flags & BPF_F_REUSE_STACKID)) {
if (stackid->bucket && !(flags & BPF_F_REUSE_STACKID)) {
pcpu_freelist_push(&smap->freelist, &new_bucket->fnode);
return -EEXIST;
}
} else {
if (hash_matches && bucket->nr == trace_nr &&
memcmp(bucket->data, ips, trace_len) == 0)
return id;
if (bucket && !(flags & BPF_F_REUSE_STACKID))
return -EEXIST;
new_bucket = (struct stack_map_bucket *)
pcpu_freelist_pop(&smap->freelist);
if (unlikely(!new_bucket))
return -ENOMEM;
memcpy(new_bucket->data, ips, trace_len);
}
new_bucket->hash = hash;
new_bucket->nr = trace_nr;
old_bucket = xchg(&smap->buckets[id], new_bucket);
old_bucket = xchg(&smap->buckets[stackid->id], new_bucket);
if (old_bucket)
pcpu_freelist_push(&smap->freelist, &old_bucket->fnode);
return id;
return stackid->id;
}
BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
@@ -583,23 +627,36 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
{
u32 elem_size = stack_map_data_size(map);
bool user = flags & BPF_F_USER_STACK;
struct stack_map_bucket *new_bucket;
struct perf_callchain_entry *trace;
struct stackid stackid;
bool kernel = !user;
u32 max_depth;
int err;
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID)))
return -EINVAL;
max_depth = stack_map_calculate_max_depth(map->value_size, elem_size, flags);
trace = get_perf_callchain(regs, kernel, user, max_depth,
false, false, 0);
if (unlikely(!trace))
/* couldn't fetch the stack trace */
return -EFAULT;
scoped_guard(preempt) {
trace = get_perf_callchain(regs, kernel, user, max_depth,
false, false, 0);
if (unlikely(!trace))
/* couldn't fetch the stack trace */
return -EFAULT;
return __bpf_get_stackid(map, trace, flags);
err = stackid_fastpath(&stackid, map, trace, trace->nr, flags);
if (err != -ENOENT)
return err;
new_bucket = stackid_new_bucket(&stackid, map);
if (!new_bucket)
return -ENOMEM;
}
return stackid_install(&stackid, map, new_bucket, flags);
}
const struct bpf_func_proto bpf_get_stackid_proto = {
@@ -611,7 +668,7 @@ const struct bpf_func_proto bpf_get_stackid_proto = {
.arg3_type = ARG_ANYTHING,
};
static __u64 count_kernel_ip(struct perf_callchain_entry *trace)
static __u64 count_kernel_ip(const struct perf_callchain_entry *trace)
{
__u64 nr_kernel = 0;
@@ -626,10 +683,13 @@ static __u64 count_kernel_ip(struct perf_callchain_entry *trace)
BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx,
struct bpf_map *, map, u64, flags)
{
const struct perf_callchain_entry *trace;
struct perf_event *event = ctx->event;
struct perf_callchain_entry *trace;
struct stack_map_bucket *new_bucket;
struct stackid stackid;
bool kernel, user;
__u64 nr_kernel;
u32 trace_nr;
int ret;
/* perf_sample_data doesn't have callchain, use bpf_get_stackid */
@@ -649,26 +709,28 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx,
return -EFAULT;
nr_kernel = count_kernel_ip(trace);
__u64 nr = trace->nr; /* save original */
if (kernel) {
trace->nr = nr_kernel;
ret = __bpf_get_stackid(map, trace, flags);
trace_nr = nr_kernel;
} else { /* user */
u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
trace_nr = trace->nr;
skip += nr_kernel;
if (skip > BPF_F_SKIP_FIELD_MASK)
return -EFAULT;
flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip;
ret = __bpf_get_stackid(map, trace, flags);
}
/* restore nr */
trace->nr = nr;
ret = stackid_fastpath(&stackid, map, trace, trace_nr, flags);
if (ret != -ENOENT)
return ret;
return ret;
new_bucket = stackid_new_bucket(&stackid, map);
if (new_bucket)
return stackid_install(&stackid, map, new_bucket, flags);
return -ENOMEM;
}
const struct bpf_func_proto bpf_get_stackid_proto_pe = {
@@ -680,19 +742,55 @@ const struct bpf_func_proto bpf_get_stackid_proto_pe = {
.arg3_type = ARG_ANYTHING,
};
static u32 callchain_store(const struct perf_callchain_entry *trace, u32 trace_nr,
void *buf, u32 elem_size, u64 flags)
{
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
const u64 *ips;
u32 copy_len;
trace_nr = trace_nr - skip;
copy_len = trace_nr * elem_size;
ips = trace->ip + skip;
if (user_build_id) {
struct bpf_stack_build_id *id_offs = buf;
for (u32 i = 0; i < trace_nr; i++)
id_offs[i].ip = ips[i];
} else {
memcpy(buf, ips, copy_len);
}
return trace_nr;
}
static long callchain_finalize(void *buf, u32 size, u32 trace_nr, u32 elem_size,
u64 flags, bool may_fault)
{
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
bool user = flags & BPF_F_USER_STACK;
u32 copy_len = trace_nr * elem_size;
if (user_build_id)
stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
if (size > copy_len)
memset(buf + copy_len, 0, size - copy_len);
return copy_len;
}
static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
struct perf_callchain_entry *trace_in,
void *buf, u32 size, u64 flags, bool may_fault)
{
u32 trace_nr, copy_len, elem_size, max_depth;
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
bool crosstask = task && task != current;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
bool user = flags & BPF_F_USER_STACK;
struct perf_callchain_entry *trace;
u32 trace_nr, elem_size, max_depth;
bool kernel = !user;
int err = -EINVAL;
u64 *ips;
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -718,13 +816,11 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
max_depth = stack_map_calculate_max_depth(size, elem_size, flags);
preempt_disable();
if (may_fault)
rcu_read_lock(); /* need RCU for perf's callchain below */
if (trace_in) {
trace = trace_in;
trace->nr = min_t(u32, trace->nr, max_depth);
} else if (kernel && task) {
if (kernel && task) {
trace = get_callchain_entry_for_task(task, max_depth);
} else {
trace = get_perf_callchain(regs, kernel, user, max_depth,
@@ -734,33 +830,18 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
if (unlikely(!trace) || trace->nr < skip) {
if (may_fault)
rcu_read_unlock();
preempt_enable();
goto err_fault;
}
trace_nr = trace->nr - skip;
copy_len = trace_nr * elem_size;
trace_nr = callchain_store(trace, trace->nr, buf, elem_size, flags);
ips = trace->ip + skip;
if (user_build_id) {
struct bpf_stack_build_id *id_offs = buf;
u32 i;
for (i = 0; i < trace_nr; i++)
id_offs[i].ip = ips[i];
} else {
memcpy(buf, ips, copy_len);
}
/* trace/ips should not be dereferenced after this point */
/* trace should not be dereferenced after this point */
if (may_fault)
rcu_read_unlock();
preempt_enable();
if (user_build_id)
stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
if (size > copy_len)
memset(buf + copy_len, 0, size - copy_len);
return copy_len;
return callchain_finalize(buf, size, trace_nr, elem_size, flags, may_fault);
err_fault:
err = -EFAULT;
@@ -772,7 +853,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
}
const struct bpf_func_proto bpf_get_stack_proto = {
@@ -781,14 +862,14 @@ const struct bpf_func_proto bpf_get_stack_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_get_stack_sleepable, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, true /* may_fault */);
return __bpf_get_stack(regs, NULL, buf, size, flags, true /* may_fault */);
}
const struct bpf_func_proto bpf_get_stack_sleepable_proto = {
@@ -797,7 +878,7 @@ const struct bpf_func_proto bpf_get_stack_sleepable_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -807,14 +888,17 @@ static long __bpf_get_task_stack(struct task_struct *task, void *buf, u32 size,
struct pt_regs *regs;
long res = -EINVAL;
if (!try_get_task_stack(task))
if (!try_get_task_stack(task)) {
memset(buf, 0, size);
return -EFAULT;
}
regs = task_pt_regs(task);
if (regs)
res = __bpf_get_stack(regs, task, NULL, buf, size, flags, may_fault);
res = __bpf_get_stack(regs, task, buf, size, flags, may_fault);
else
memset(buf, 0, size);
put_task_stack(task);
return res;
}
@@ -831,7 +915,7 @@ const struct bpf_func_proto bpf_get_task_stack_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -848,22 +932,48 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static int __bpf_get_stack_pe(const struct perf_callchain_entry *trace, u32 trace_nr,
void *buf, u32 size, u64 flags)
{
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
bool user = flags & BPF_F_USER_STACK;
u32 elem_size, max_depth, nr_trace;
bool kernel = !user;
if (kernel && user_build_id)
return -EINVAL;
elem_size = user_build_id ? sizeof(struct bpf_stack_build_id) : sizeof(u64);
if (unlikely(size % elem_size))
return -EINVAL;
max_depth = stack_map_calculate_max_depth(size, elem_size, flags);
trace_nr = min_t(u32, trace_nr, max_depth);
if (trace_nr < skip)
return -EFAULT;
nr_trace = callchain_store(trace, trace_nr, buf, elem_size, flags);
return callchain_finalize(buf, size, nr_trace, elem_size, flags, false /* !may_fault */);
}
BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
void *, buf, u32, size, u64, flags)
{
struct pt_regs *regs = (struct pt_regs *)(ctx->regs);
const struct perf_callchain_entry *trace;
struct perf_event *event = ctx->event;
struct perf_callchain_entry *trace;
bool kernel, user;
int err = -EINVAL;
__u64 nr_kernel;
if (!(event->attr.sample_type & PERF_SAMPLE_CALLCHAIN))
return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -880,27 +990,20 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
nr_kernel = count_kernel_ip(trace);
if (kernel) {
__u64 nr = trace->nr;
trace->nr = nr_kernel;
err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
/* restore nr */
trace->nr = nr;
err = __bpf_get_stack_pe(trace, nr_kernel, buf, size, flags);
} else { /* user */
u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
skip += nr_kernel;
if (skip > BPF_F_SKIP_FIELD_MASK)
goto clear;
flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip;
err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
err = __bpf_get_stack_pe(trace, trace->nr, buf, size, flags);
}
return err;
clear:
memset(buf, 0, size);
if (err < 0)
memset(buf, 0, size);
return err;
}
@@ -911,7 +1014,7 @@ const struct bpf_func_proto bpf_get_stack_proto_pe = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};

3
kernel/bpf/states.c
View File

@@ -812,7 +812,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* infinite loop check triggering, see
* iter_active_depths_differ()
*/
if (old_reg->iter.btf != cur_reg->iter.btf ||
if (old_reg->type != cur_reg->type ||
old_reg->iter.btf != cur_reg->iter.btf ||
old_reg->iter.btf_id != cur_reg->iter.btf_id ||
old_reg->iter.state != cur_reg->iter.state ||
/* ignore {old_reg,cur_reg}->iter.depth, see above */

142
kernel/bpf/syscall.c
View File

@@ -40,7 +40,6 @@
#include <linux/tracepoint.h>
#include <linux/overflow.h>
#include <linux/cookie.h>
#include <linux/verification.h>
#include <linux/btf_ids.h>
#include <net/netfilter/nf_bpf_link.h>
@@ -637,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
return ret;
}
static int btf_field_cmp(const void *a, const void *b)
{
const struct btf_field *f1 = a, *f2 = b;
@@ -1078,11 +1076,24 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
bpf_map_write_active_dec(map);
}
static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
{
struct bpf_map *map = vmf->vma->vm_private_data;
return map->ops->map_mmap_fault(map, vmf);
}
static const struct vm_operations_struct bpf_map_default_vmops = {
.open = bpf_map_mmap_open,
.close = bpf_map_mmap_close,
};
static const struct vm_operations_struct bpf_map_lazy_vmops = {
.open = bpf_map_mmap_open,
.close = bpf_map_mmap_close,
.fault = bpf_map_mmap_fault,
};
static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
{
struct bpf_map *map = filp->private_data;
@@ -1118,7 +1129,7 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
return err;
/* set default open/close callbacks */
vma->vm_ops = &bpf_map_default_vmops;
vma->vm_ops = map->ops->map_mmap_fault ? &bpf_map_lazy_vmops : &bpf_map_default_vmops;
vma->vm_private_data = map;
vm_flags_clear(vma, VM_MAYEXEC);
/* If mapping is read-only, then disallow potentially re-mapping with
@@ -1599,13 +1610,6 @@ static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_ver
err = -EFAULT;
goto free_map;
}
/* See libbpf: emit_signature_match() */
BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE);
BUILD_BUG_ON(!__same_type(map->excl, u32));
BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0);
BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE]));
map->excl = 1;
} else if (attr->excl_prog_hash_size) {
bpf_log(log, "Invalid excl_prog_hash_size.\n");
err = -EINVAL;
@@ -1657,7 +1661,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_at
err = security_bpf_map_create(map, attr, token, uattr.is_kernel);
if (err)
goto free_map_sec;
goto free_map;
err = bpf_map_alloc_id(map);
if (err)
@@ -1838,7 +1842,6 @@ static int map_lookup_elem(union bpf_attr *attr)
return err;
}
#define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags
static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
@@ -2886,64 +2889,6 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
}
}
static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
{
switch (keyring_id) {
case 0:
return BPF_SIG_KEYRING_BUILTIN;
case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING:
return BPF_SIG_KEYRING_SECONDARY;
case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING:
return BPF_SIG_KEYRING_PLATFORM;
default:
return BPF_SIG_KEYRING_USER;
}
}
static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr,
bool is_kernel, s32 *keyring_serial)
{
bpfptr_t usig = make_bpfptr(attr->signature, is_kernel);
struct bpf_dynptr_kern sig_ptr, insns_ptr;
struct bpf_key *key = NULL;
void *sig;
int err = 0;
/*
* Don't attempt to use kmalloc_large or vmalloc for signatures.
* Practical signature for BPF program should be below this limit.
*/
if (attr->signature_size > KMALLOC_MAX_CACHE_SIZE)
return -EINVAL;
if (system_keyring_id_check(attr->keyring_id) == 0)
key = bpf_lookup_system_key(attr->keyring_id);
else
key = bpf_lookup_user_key(attr->keyring_id, 0);
if (!key)
return -EINVAL;
sig = kvmemdup_bpfptr(usig, attr->signature_size);
if (IS_ERR(sig)) {
bpf_key_put(key);
return PTR_ERR(sig);
}
bpf_dynptr_init(&sig_ptr, sig, BPF_DYNPTR_TYPE_LOCAL, 0,
attr->signature_size);
bpf_dynptr_init(&insns_ptr, prog->insnsi, BPF_DYNPTR_TYPE_LOCAL, 0,
prog->len * sizeof(struct bpf_insn));
err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr,
(struct bpf_dynptr *)&sig_ptr, key);
if (!err)
*keyring_serial = bpf_key_serial(key);
bpf_key_put(key);
kvfree(sig);
return err;
}
static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog)
{
int err;
@@ -3109,6 +3054,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
prog->aux->attach_btf = attach_btf;
prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
prog->aux->dst_prog = dst_prog;
if (dst_prog) {
prog->aux->saved_dst_prog_type = dst_prog->type;
prog->aux->saved_dst_attach_type = dst_prog->expected_attach_type;
}
prog->aux->dev_bound = !!attr->prog_ifindex;
prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
@@ -3133,17 +3082,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
/* eBPF programs must be GPL compatible to use GPL-ed functions */
prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;
if (attr->signature) {
err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel,
&prog->aux->sig.keyring_serial);
if (err)
goto free_prog;
prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id);
prog->aux->sig.verdict = BPF_SIG_VERIFIED;
} else {
prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE;
prog->aux->sig.verdict = BPF_SIG_UNSIGNED;
}
prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE;
prog->aux->sig.verdict = BPF_SIG_UNSIGNED;
prog->orig_prog = NULL;
prog->jited = 0;
@@ -3189,10 +3129,6 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
if (err < 0)
goto free_prog;
err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);
if (err)
goto free_prog;
/* run eBPF verifier */
err = bpf_check(&prog, attr, uattr, attr_log);
if (err < 0)
@@ -3471,9 +3407,10 @@ static const char *bpf_link_type_strs[] = {
static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
{
const struct bpf_link *link = filp->private_data;
const struct bpf_prog *prog = link->prog;
const struct bpf_prog *prog;
enum bpf_link_type type = link->type;
char prog_tag[sizeof(prog->tag) * 2 + 1] = { };
u32 prog_id = 0;
if (type < ARRAY_SIZE(bpf_link_type_strs) && bpf_link_type_strs[type]) {
if (link->type == BPF_LINK_TYPE_KPROBE_MULTI)
@@ -3490,13 +3427,20 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
}
seq_printf(m, "link_id:\t%u\n", link->id);
rcu_read_lock();
prog = READ_ONCE(link->prog);
if (prog) {
bin2hex(prog_tag, prog->tag, sizeof(prog->tag));
prog_id = prog->aux->id;
}
rcu_read_unlock();
if (prog) {
seq_printf(m,
"prog_tag:\t%s\n"
"prog_id:\t%u\n",
prog_tag,
prog->aux->id);
prog_id);
}
if (link->ops->show_fdinfo)
link->ops->show_fdinfo(link, m);
@@ -3564,7 +3508,6 @@ int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)
if (fd < 0)
return fd;
id = bpf_link_alloc_id(link);
if (id < 0) {
put_unused_fd(fd);
@@ -3626,10 +3569,12 @@ static void bpf_tracing_link_release(struct bpf_link *link)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
int err;
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node,
tr_link->trampoline,
tr_link->tgt_prog));
err = bpf_trampoline_unlink_prog(&tr_link->link.node,
tr_link->trampoline,
tr_link->tgt_prog);
WARN_ONCE(err, "bpf_trampoline_unlink_prog failed: %d\n", err);
bpf_trampoline_put(tr_link->trampoline);
@@ -5535,6 +5480,7 @@ static int bpf_link_get_info_by_fd(struct file *file,
{
struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info);
struct bpf_link_info info;
const struct bpf_prog *prog;
u32 info_len = attr->info.info_len;
int err;
@@ -5549,8 +5495,12 @@ static int bpf_link_get_info_by_fd(struct file *file,
info.type = link->type;
info.id = link->id;
if (link->prog)
info.prog_id = link->prog->aux->id;
rcu_read_lock();
prog = READ_ONCE(link->prog);
if (prog)
info.prog_id = prog->aux->id;
rcu_read_unlock();
if (link->ops->fill_link_info) {
err = link->ops->fill_link_info(link, &info);
@@ -5565,7 +5515,6 @@ static int bpf_link_get_info_by_fd(struct file *file,
return 0;
}
static int token_get_info_by_fd(struct file *file,
struct bpf_token *token,
const union bpf_attr *attr,
@@ -6567,7 +6516,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size)
return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
}
/* To shut up -Wmissing-prototypes.
* This function is used by the kernel light skeleton
* to load bpf programs when modules are loaded or during kernel boot.
@@ -6623,7 +6571,7 @@ static const struct bpf_func_proto bpf_sys_bpf_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
};
const struct bpf_func_proto * __weak
@@ -6670,7 +6618,7 @@ static const struct bpf_func_proto bpf_kallsyms_lookup_name_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),

48
kernel/bpf/task_iter.c
View File

@@ -753,9 +753,9 @@ static struct bpf_iter_reg task_vma_reg_info = {
BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags)
{
struct mmap_unlock_irq_work *work = NULL;
struct mmap_unlock_irq_work *work;
struct vm_area_struct *vma;
bool irq_work_busy = false;
bool __maybe_unused mmput_needed = false;
struct mm_struct *mm;
int ret = -ENOENT;
@@ -765,14 +765,43 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
if (!task)
return -ENOENT;
mm = task->mm;
if (task == current) {
mm = task->mm;
} else {
/*
* Foreign task: pin task->mm against a concurrent exit_mm().
* Use trylock on alloc_lock instead of get_task_mm()'s
* blocking task_lock() to avoid deadlocking the target task.
*/
if (!IS_ENABLED(CONFIG_MMU))
return -EOPNOTSUPP;
if (irqs_disabled())
return -EBUSY;
if (!spin_trylock(&task->alloc_lock))
return -EBUSY;
mm = task->mm;
if (mm && !(task->flags & PF_KTHREAD)) {
mmget(mm);
mmput_needed = true;
} else {
mm = NULL;
}
spin_unlock(&task->alloc_lock);
}
if (!mm)
return -ENOENT;
irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
work = bpf_mmap_unlock_guard_get();
if (IS_ERR(work)) {
ret = PTR_ERR(work);
goto out;
}
if (irq_work_busy || !mmap_read_trylock(mm))
return -EBUSY;
if (!mmap_read_trylock(mm)) {
bpf_mmap_unlock_guard_put(work);
ret = -EBUSY;
goto out;
}
vma = find_vma(mm, start);
@@ -782,6 +811,11 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
ret = 0;
}
bpf_mmap_unlock_mm(work, mm);
out:
#ifdef CONFIG_MMU
if (mmput_needed)
mmput_async(mm);
#endif
return ret;
}
@@ -1161,6 +1195,8 @@ static void do_mmap_read_unlock(struct irq_work *entry)
work = container_of(entry, struct mmap_unlock_irq_work, irq_work);
mmap_read_unlock_non_owner(work->mm);
work->mm = NULL;
bpf_mmap_unlock_guard_put(work);
}
static int __init task_iter_init(void)

73
kernel/bpf/trampoline.c
View File

@@ -529,6 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
return tnodes;
}
/*
* The arena base against which save_args() converts the arguments marked
* with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline
* converts: it dispatches to a single prog whose arena is known at
* generation time. Return 0 when there is nothing to convert.
*/
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
struct bpf_tramp_nodes *tnodes, u32 flags)
{
const struct bpf_prog *prog;
int i;
if (!(flags & BPF_TRAMP_F_INDIRECT) ||
tnodes[BPF_TRAMP_FENTRY].nr_nodes != 1)
return 0;
for (i = 0; i < m->nr_args; i++)
if (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG)
break;
if (i == m->nr_args)
return 0;
/* Verification rejects an arena argument without an arena. */
prog = tnodes[BPF_TRAMP_FENTRY].nodes[0]->link->prog;
if (WARN_ON_ONCE(!prog->aux->arena))
return 0;
return bpf_arena_get_kern_vm_start(prog->aux->arena);
}
static void bpf_tramp_image_free(struct bpf_tramp_image *im)
{
bpf_image_ksym_del(&im->ksym);
@@ -670,6 +700,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)
return ERR_PTR(err);
}
void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags)
{
trampoline_lock(tr);
tr->flags |= flags;
trampoline_unlock(tr);
}
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
const struct bpf_trampoline_ops *ops, void *data)
{
@@ -913,6 +950,13 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
int cnt = 0, i;
kind = bpf_attach_type_to_tramp(node->link->prog);
/*
* Arena ctx args are converted only by struct_ops indirect
* trampolines. They must never be attached to a generic trampoline.
*/
if (WARN_ON_ONCE(bpf_prog_has_arena_ctx_arg(node->link->prog)))
return -ENOTSUPP;
if (tr->extension_prog)
/* cannot attach fentry/fexit if extension prog is attached.
* cannot overwrite extension prog either.
@@ -997,12 +1041,15 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link)
{
struct bpf_shim_tramp_link *shim_link =
container_of(link, struct bpf_shim_tramp_link, link.link);
int err;
/* paired with 'shim_link->trampoline = tr' in bpf_trampoline_link_cgroup_shim */
if (!shim_link->trampoline)
return;
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL));
err = bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL);
WARN_ONCE(err, "bpf_trampoline_unlink_prog failed: %d\n", err);
bpf_trampoline_put(shim_link->trampoline);
}
@@ -1536,6 +1583,7 @@ static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_ima
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
tr->func.ftrace_managed = true;
ftrace_hash_add(data->reg, data->entry, ip, addr);
tr->cur_image = im;
return 0;
@@ -1584,7 +1632,17 @@ static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr)
static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr)
{
if (tr->multi_attach.old_image)
/*
* Only free old_image if it is no longer the active image.
* When bpf_trampoline_update() fails before modify_fentry_multi()/
* unregister_fentry_multi() is called, cur_image is unchanged
* (cur_image == old_image) and ftrace still points to it. Freeing
* it would cause a UAF when ftrace calls into the freed memory.
* On success, cur_image is either a new image or NULL, so
* old_image != cur_image means the image is stale.
*/
if (tr->multi_attach.old_image &&
tr->multi_attach.old_image != tr->cur_image)
bpf_tramp_image_put(tr->multi_attach.old_image);
tr->multi_attach.old_image = NULL;
@@ -1708,19 +1766,21 @@ int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
return err;
}
int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link)
void bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link)
{
struct bpf_tracing_multi_data *data = &link->data;
struct bpf_tracing_multi_node *mnode;
int i;
int i, err;
trampoline_lock_all();
for_each_mnode(mnode, link) {
data->entry = &mnode->entry;
bpf_trampoline_multi_attach_init(mnode->trampoline);
WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline,
NULL, &trampoline_multi_ops, data));
err = __bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline, NULL,
&trampoline_multi_ops, data);
WARN_ONCE(err, "__bpf_trampoline_unlink_prog failed: %d\n", err);
}
if (ftrace_hash_count(data->unreg))
@@ -1737,7 +1797,6 @@ int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_
bpf_trampoline_put(mnode->trampoline);
clear_tracing_multi_data(data);
return 0;
}
#undef for_each_mnode_cnt

3551
kernel/bpf/verifier.c
View File

File diff suppressed because it is too large Load Diff

119
kernel/trace/bpf_trace.c
View File

@@ -221,7 +221,7 @@ const struct bpf_func_proto bpf_probe_read_user_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -258,7 +258,7 @@ const struct bpf_func_proto bpf_probe_read_user_str_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -273,7 +273,7 @@ const struct bpf_func_proto bpf_probe_read_kernel_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -308,7 +308,7 @@ const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -328,7 +328,7 @@ static const struct bpf_func_proto bpf_probe_read_compat_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
@@ -347,7 +347,7 @@ static const struct bpf_func_proto bpf_probe_read_compat_str_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
#endif /* CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE */
@@ -383,7 +383,7 @@ static const struct bpf_func_proto bpf_probe_write_user_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
};
#define MAX_TRACE_PRINTK_VARARGS 3
@@ -418,7 +418,7 @@ static const struct bpf_func_proto bpf_trace_printk_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
};
static void __set_printk_clr_event(struct work_struct *work)
@@ -474,9 +474,9 @@ static const struct bpf_func_proto bpf_trace_vprintk_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_MEM_SIZE_OR_ZERO,
};
const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void)
@@ -518,9 +518,9 @@ static const struct bpf_func_proto bpf_seq_printf_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_MEM_SIZE,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
@@ -535,7 +535,7 @@ static const struct bpf_func_proto bpf_seq_write_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
};
BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr,
@@ -559,7 +559,7 @@ static const struct bpf_func_proto bpf_seq_printf_btf_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -633,7 +633,7 @@ static const struct bpf_func_proto bpf_perf_event_read_value_proto = {
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
.arg4_type = ARG_MEM_SIZE,
};
const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void)
@@ -730,7 +730,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto = {
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
static DEFINE_PER_CPU(int, bpf_event_output_nest_level);
@@ -996,7 +996,7 @@ static const struct bpf_func_proto bpf_d_path_proto = {
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_d_path_btf_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.allowed = bpf_d_path_allowed,
};
@@ -1053,9 +1053,9 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE,
.arg2_type = ARG_MEM_SIZE,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE,
.arg4_type = ARG_MEM_SIZE,
.arg5_type = ARG_ANYTHING,
};
@@ -1218,7 +1218,7 @@ const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg2_type = ARG_MEM_SIZE_OR_ZERO,
};
BPF_CALL_3(get_func_arg, void *, ctx, u32, n, u64 *, value)
@@ -1421,7 +1421,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_tp = {
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_get_stackid_tp, void *, tp_buff, struct bpf_map *, map,
@@ -1462,7 +1462,7 @@ static const struct bpf_func_proto bpf_get_stack_proto_tp = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -1524,12 +1524,12 @@ BPF_CALL_3(bpf_perf_prog_read_value, struct bpf_perf_event_data_kern *, ctx,
}
static const struct bpf_func_proto bpf_perf_prog_read_value_proto = {
.func = bpf_perf_prog_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
.func = bpf_perf_prog_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_MEM_SIZE,
};
BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,
@@ -1566,7 +1566,7 @@ static const struct bpf_func_proto bpf_read_branch_records_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -1646,7 +1646,7 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_MEM_SIZE_OR_ZERO,
};
extern const struct bpf_func_proto bpf_skb_output_proto;
@@ -1701,7 +1701,7 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_MEM_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
@@ -3687,7 +3687,7 @@ static void bpf_tracing_multi_link_release(struct bpf_link *link)
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link));
bpf_trampoline_multi_detach(link->prog, tr_link);
}
static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
@@ -3700,6 +3700,60 @@ static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
kvfree(tr_link);
}
static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
u64 __user *ucookies = u64_to_user_ptr(info->tracing_multi.cookies);
u64 __user *uaddrs = u64_to_user_ptr(info->tracing_multi.addrs);
u32 __user *uids = u64_to_user_ptr(info->tracing_multi.ids);
struct bpf_tracing_multi_link *tr_link;
u32 ucount = info->tracing_multi.count;
bool has_cookies, show_addrs;
int err = 0;
if ((uids || ucookies || uaddrs) && !ucount)
return -EINVAL;
tr_link = container_of(link, struct bpf_tracing_multi_link, link);
info->tracing_multi.attach_type = tr_link->link.attach_type;
info->tracing_multi.count = tr_link->nodes_cnt;
info->tracing_multi.btf_obj_id = btf_obj_id(tr_link->link.prog->aux->attach_btf);
if (!uids && !ucookies && !uaddrs)
return 0;
if (ucount < tr_link->nodes_cnt)
err = -ENOSPC;
else
ucount = tr_link->nodes_cnt;
has_cookies = !!tr_link->cookies;
show_addrs = kallsyms_show_value(current_cred());
for (int i = 0; i < ucount; i++) {
struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i];
u64 addr, cookie;
u32 id;
bpf_trampoline_unpack_key(mnode->trampoline->key, NULL, &id);
addr = show_addrs ? mnode->trampoline->ip : 0;
cookie = has_cookies ? tr_link->cookies[i] : 0;
if (uids && put_user(id, uids + i))
return -EFAULT;
if (uaddrs && put_user(addr, uaddrs + i))
return -EFAULT;
if (ucookies && put_user(cookie, ucookies + i))
return -EFAULT;
cond_resched();
}
return err;
}
#ifdef CONFIG_PROC_FS
static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
@@ -3730,6 +3784,7 @@ static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
.release = bpf_tracing_multi_link_release,
.dealloc_deferred = bpf_tracing_multi_link_dealloc,
.fill_link_info = bpf_tracing_multi_link_fill_link_info,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_tracing_multi_show_fdinfo,
#endif

3
net/core/Makefile
View File

@@ -44,6 +44,9 @@ obj-$(CONFIG_FAILOVER) += failover.o
obj-$(CONFIG_NET_SOCK_MSG) += skmsg.o
obj-$(CONFIG_BPF_SYSCALL) += sock_map.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o
ifdef CONFIG_INET
obj-$(CONFIG_BPF_SYSCALL) += bpf_ksock.o
endif
obj-$(CONFIG_OF) += of_net.o
obj-$(CONFIG_NET_TEST) += net_test.o
obj-$(CONFIG_NET_DEVMEM) += devmem.o

328
net/core/bpf_ksock.c Normal file
View File

@@ -0,0 +1,328 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2026 Isovalent */
#include <linux/bpf.h>
#include <linux/bpf_ksock.h>
#include <linux/btf.h>
#include <linux/btf_ids.h>
#include <linux/in.h>
#include <linux/in6.h>
#include <linux/net.h>
#include <linux/refcount.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/socket.h>
#include <linux/unaligned.h>
#include <linux/workqueue.h>
#include <net/sock.h>
/**
* struct bpf_ksock - refcounted BPF kernel socket context
* @sock: The underlying kernel socket.
* @usage: Reference counter.
* @rwork: RCU work for deferred cleanup (sock_release may sleep).
*/
struct bpf_ksock {
struct socket *sock;
refcount_t usage;
struct rcu_work rwork;
};
static void ksock_release_work_fn(struct work_struct *work)
{
struct bpf_ksock *ks;
ks = container_of(to_rcu_work(work), struct bpf_ksock, rwork);
sock_release(ks->sock);
kfree(ks);
}
static bool bpf_ksock_has_user_task_context(void)
{
/*
* Task work can run from do_exit() after exit_nsproxy_namespaces()
* cleared current->nsproxy, while current is still not a kthread.
*/
return !(current->flags & PF_KTHREAD) && current->nsproxy;
}
__bpf_kfunc_start_defs();
/**
* bpf_ksock_create() - Create a BPF kernel socket.
*
* Allocates and creates a kernel socket.
*
* The returned context must either be stored in a map as a kptr, or
* freed with bpf_ksock_release().
*
* This function may sleep (sock_create), so it can only be used
* in sleepable BPF programs (SYSCALL).
* It cannot be called from a BPF workqueue callback because that callback
* does not retain the invoking task's namespace or security context.
*
* @opts: Pointer to struct bpf_ksock_create_opts with socket parameters.
* @opts__sz: Size of the opts struct.
* @err__uninit: Integer to store error code when NULL is returned.
*/
__bpf_kfunc struct bpf_ksock *
bpf_ksock_create(const struct bpf_ksock_create_opts *opts, u32 opts__sz,
int *err__uninit)
{
struct bpf_ksock_create_opts opts_copy;
struct bpf_ksock *ks;
int err;
/*
* sock_create() derives the network namespace, credentials, and cgroup
* from current. Kernel threads, including BPF workqueue callbacks, do
* not carry the context of the task that invoked the BPF program.
*/
if (!bpf_ksock_has_user_task_context()) {
err = -EOPNOTSUPP;
goto err_out;
}
if (!opts || opts__sz != sizeof(struct bpf_ksock_create_opts)) {
err = -EINVAL;
goto err_out;
}
opts_copy = (struct bpf_ksock_create_opts){
.family = READ_ONCE(opts->family),
.type = READ_ONCE(opts->type),
.protocol = READ_ONCE(opts->protocol),
.reserved = READ_ONCE(opts->reserved),
};
if (opts_copy.reserved) {
err = -EINVAL;
goto err_out;
}
if (opts_copy.family != AF_INET && opts_copy.family != AF_INET6) {
err = -EAFNOSUPPORT;
goto err_out;
}
if (opts_copy.type != SOCK_DGRAM) {
err = -EPROTONOSUPPORT;
goto err_out;
}
if (opts_copy.protocol != IPPROTO_UDP && opts_copy.protocol != 0) {
err = -EPROTONOSUPPORT;
goto err_out;
}
ks = kzalloc_obj(*ks);
if (!ks) {
err = -ENOMEM;
goto err_out;
}
/*
* Use the normal current-task socket path so LSM/cgroup policy,
* socket labels, and the active netns reference match a socket(2)
* created by the BPF program's caller.
*/
err = sock_create(opts_copy.family, opts_copy.type, opts_copy.protocol,
&ks->sock);
if (err)
goto err_free;
ks->sock->sk->sk_rcvbuf = SOCK_MIN_RCVBUF;
ks->sock->sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
refcount_set(&ks->usage, 1);
put_unaligned(0, err__uninit);
return ks;
err_free:
kfree(ks);
err_out:
put_unaligned(err, err__uninit);
return NULL;
}
/**
* bpf_ksock_connect() - Connect a BPF kernel socket to a remote address.
* @ks: The BPF kernel socket context.
* @addr: Pointer to an IPv4 or IPv6 socket address.
* @addr__sz: Size of the address union.
*
* Connects the socket to the specified remote address and port.
*
* This function may sleep while connecting the socket, so it can only be used
* in sleepable BPF programs (SYSCALL).
*
* Return: 0 on success, negative errno on error.
*/
__bpf_kfunc int bpf_ksock_connect(struct bpf_ksock *ks,
const union bpf_ksock_addr *addr,
u32 addr__sz)
{
struct sockaddr_storage sa;
int addrlen;
if (!bpf_ksock_has_user_task_context())
return -EOPNOTSUPP;
if (!addr || addr__sz != sizeof(*addr))
return -EINVAL;
/* Kfunc memory arguments may be unaligned. */
memcpy(&sa, addr, sizeof(*addr));
switch (sa.ss_family) {
case AF_INET:
addrlen = sizeof(struct sockaddr_in);
break;
case AF_INET6:
addrlen = sizeof(struct sockaddr_in6);
break;
default:
return -EAFNOSUPPORT;
}
return connect_socket(ks->sock, &sa, addrlen, 0);
}
/**
* bpf_ksock_acquire() - Acquire a reference to a BPF kernel socket.
* @ks: The BPF kernel socket context to acquire. Must be a
* trusted pointer (e.g. RCU-protected kptr from a map).
*
* The acquired context must either be stored in a map as a kptr, or
* freed with bpf_ksock_release().
*/
__bpf_kfunc struct bpf_ksock *bpf_ksock_acquire(struct bpf_ksock *ks)
{
if (!refcount_inc_not_zero(&ks->usage))
return NULL;
return ks;
}
/**
* bpf_ksock_release() - Release a BPF kernel socket.
* @ks: The BPF kernel socket context to release.
*
* When the final reference is released, the socket is cleaned up via
* queue_rcu_work() (since sock_release may sleep).
*/
__bpf_kfunc void bpf_ksock_release(struct bpf_ksock *ks)
{
if (refcount_dec_and_test(&ks->usage)) {
INIT_RCU_WORK(&ks->rwork, ksock_release_work_fn);
queue_rcu_work(system_dfl_wq, &ks->rwork);
}
}
__bpf_kfunc void bpf_ksock_release_dtor(void *ks)
{
bpf_ksock_release(ks);
}
CFI_NOSEAL(bpf_ksock_release_dtor);
/**
* bpf_ksock_send() - Send data through a BPF kernel socket.
* @ks: The BPF kernel socket context. Must be an acquired reference.
* @data: Pointer to the data to send.
* @data__sz: Size of the data to send.
*
* Sends data on a connected socket, best-effort and nonblocking. This may sleep
* (kernel_sendmsg), so it can only be called from sleepable BPF programs.
*
* Return: Number of bytes sent on success, negative errno on error.
*/
__bpf_kfunc int bpf_ksock_send(struct bpf_ksock *ks, const void *data,
u32 data__sz)
{
struct msghdr msg = {
.msg_flags = MSG_DONTWAIT,
};
struct kvec iov = {
.iov_base = (void *)data,
.iov_len = data__sz,
};
int ret;
if (!bpf_ksock_has_user_task_context())
return -EOPNOTSUPP;
ret = kernel_sendmsg(ks->sock, &msg, &iov, 1, data__sz);
return ret;
}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(ksock_init_kfunc_btf_ids)
BTF_ID_FLAGS(func, bpf_ksock_create, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_ksock_connect, KF_SLEEPABLE)
BTF_KFUNCS_END(ksock_init_kfunc_btf_ids)
static const struct btf_kfunc_id_set ksock_init_kfunc_set = {
.owner = THIS_MODULE,
.set = &ksock_init_kfunc_btf_ids,
};
BTF_KFUNCS_START(ksock_kfunc_btf_ids)
BTF_ID_FLAGS(func, bpf_ksock_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_ksock_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_ksock_send, KF_SLEEPABLE)
BTF_KFUNCS_END(ksock_kfunc_btf_ids)
#ifdef CONFIG_BPF_LSM
BTF_ID_LIST_SINGLE(bpf_lsm_socket_sendmsg_id, func, bpf_lsm_socket_sendmsg)
#endif
static int bpf_ksock_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
{
if (!btf_id_set8_contains(&ksock_kfunc_btf_ids, kfunc_id))
return 0;
if (prog->type == BPF_PROG_TYPE_SYSCALL)
return 0;
#ifdef CONFIG_BPF_LSM
if (prog->type == BPF_PROG_TYPE_LSM &&
prog->aux->attach_btf_id != bpf_lsm_socket_sendmsg_id[0])
return 0;
#endif
return -EACCES;
}
static const struct btf_kfunc_id_set ksock_kfunc_set = {
.owner = THIS_MODULE,
.set = &ksock_kfunc_btf_ids,
.filter = bpf_ksock_kfunc_filter,
};
BTF_ID_LIST(bpf_ksock_dtor_ids)
BTF_ID(struct, bpf_ksock)
BTF_ID(func, bpf_ksock_release_dtor)
static int __init bpf_ksock_kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc bpf_ksock_dtors[] = {
{
.btf_id = bpf_ksock_dtor_ids[0],
.kfunc_btf_id = bpf_ksock_dtor_ids[1],
},
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
&ksock_init_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL,
&ksock_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_LSM,
&ksock_kfunc_set);
return ret ?: register_btf_id_dtor_kfuncs(bpf_ksock_dtors,
ARRAY_SIZE(bpf_ksock_dtors),
THIS_MODULE);
}
late_initcall(bpf_ksock_kfunc_init);

508
net/core/filter.c
View File

File diff suppressed because it is too large Load Diff

15
net/core/lwt_bpf.c
View File

@@ -167,10 +167,10 @@ static int bpf_output(struct net *net, struct sock *sk, struct sk_buff *skb)
return dst->lwtstate->orig_output(net, sk, skb);
}
static int xmit_check_hhlen(struct sk_buff *skb, int hh_len)
static int xmit_check_headroom(struct sk_buff *skb, int hroom)
{
if (skb_headroom(skb) < hh_len) {
int nhead = HH_DATA_ALIGN(hh_len - skb_headroom(skb));
if (skb_headroom(skb) < hroom) {
int nhead = hroom - skb_headroom(skb);
if (pskb_expand_head(skb, nhead, 0, GFP_ATOMIC))
return -ENOMEM;
@@ -282,7 +282,7 @@ static int bpf_xmit(struct sk_buff *skb)
bpf = bpf_lwt_lwtunnel(dst->lwtstate);
if (bpf->xmit.prog) {
int hh_len = dst->dev->hard_header_len;
int hroom = LL_RESERVED_SPACE(dst->dev);
__be16 proto = skb->protocol;
int ret;
@@ -298,9 +298,12 @@ static int bpf_xmit(struct sk_buff *skb)
return -EINVAL;
}
/* If the header was expanded, headroom might be too
* small for L2 header to come, expand as needed.
* small for the L2 header to come, expand as needed.
* neigh_hh_output() copies the cached header in
* HH_DATA_MOD aligned chunks, so match the reservation
* made before LWT xmit.
*/
ret = xmit_check_hhlen(skb, hh_len);
ret = xmit_check_headroom(skb, hroom);
if (unlikely(ret))
return ret;

8
net/core/sock_map.c
View File

@@ -392,8 +392,8 @@ static void *sock_map_lookup(struct bpf_map *map, void *key)
sk = __sock_map_lookup_elem(map, *(u32 *)key);
if (!sk)
return NULL;
if (sk_is_refcounted(sk) && !refcount_inc_not_zero(&sk->sk_refcnt))
return NULL;
if (sk_is_refcounted(sk))
sock_hold(sk);
return sk;
}
@@ -1218,8 +1218,8 @@ static void *sock_hash_lookup(struct bpf_map *map, void *key)
sk = __sock_hash_lookup_elem(map, key);
if (!sk)
return NULL;
if (sk_is_refcounted(sk) && !refcount_inc_not_zero(&sk->sk_refcnt))
return NULL;
if (sk_is_refcounted(sk))
sock_hold(sk);
return sk;
}

4
net/ipv4/tcp_ipv4.c
View File

@@ -2931,8 +2931,8 @@ static int bpf_iter_tcp_realloc_batch(struct bpf_tcp_iter_state *iter,
{
union bpf_tcp_iter_batch_item *new_batch;
new_batch = kvmalloc(sizeof(*new_batch) * new_batch_sz,
flags | __GFP_NOWARN);
new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch),
flags | __GFP_NOWARN);
if (!new_batch)
return -ENOMEM;

32
net/socket.c
View File

@@ -2128,6 +2128,20 @@ SYSCALL_DEFINE3(accept, int, fd, struct sockaddr __user *, upeer_sockaddr,
return __sys_accept4(fd, upeer_sockaddr, upeer_addrlen, 0);
}
int connect_socket(struct socket *sock, struct sockaddr_storage *address,
int addrlen, int flags)
{
int err;
err = security_socket_connect(sock, (struct sockaddr *)address,
addrlen);
if (err)
return err;
return READ_ONCE(sock->ops)->connect(sock, (struct sockaddr_unsized *)address,
addrlen, flags);
}
/*
* Attempt to connect to a socket with the server address. The address
* is in user space so we verify it is OK and move it to kernel space.
@@ -2144,23 +2158,13 @@ int __sys_connect_file(struct file *file, struct sockaddr_storage *address,
int addrlen, int file_flags)
{
struct socket *sock;
int err;
sock = sock_from_file(file);
if (!sock) {
err = -ENOTSOCK;
goto out;
}
if (!sock)
return -ENOTSOCK;
err =
security_socket_connect(sock, (struct sockaddr *)address, addrlen);
if (err)
goto out;
err = READ_ONCE(sock->ops)->connect(sock, (struct sockaddr_unsized *)address,
addrlen, sock->file->f_flags | file_flags);
out:
return err;
return connect_socket(sock, address, addrlen,
sock->file->f_flags | file_flags);
}
int __sys_connect(int fd, struct sockaddr __user *uservaddr, int addrlen)

4
net/unix/af_unix.c
View File

@@ -3655,8 +3655,8 @@ static int bpf_iter_unix_realloc_batch(struct bpf_unix_iter_state *iter,
{
struct sock **new_batch;
new_batch = kvmalloc(sizeof(*new_batch) * new_batch_sz,
GFP_USER | __GFP_NOWARN);
new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch),
GFP_USER | __GFP_NOWARN);
if (!new_batch)
return -ENOMEM;

4
scripts/Makefile.btf
View File

@@ -14,9 +14,7 @@ pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsis
else
# Switch to using --btf_features for v1.26 and later.
pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
pahole-flags-$(call test-ge, $(pahole-ver), 130) += --btf_features=attributes
pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j$(JOBS) --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func
pahole-flags-$(call test-ge, $(pahole-ver), 131) += --btf_features=layout

2
tools/bpf/bpftool/Makefile
View File

@@ -99,7 +99,7 @@ endif
HOST_LDFLAGS := $(LDFLAGS)
# Remove warnings for libbpf bootstrap build
LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS))
LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat%,$(HOST_CFLAGS))
INSTALL ?= install
RM ?= rm -f

13
tools/bpf/bpftool/btf.c
View File

@@ -179,7 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
const struct btf_member *m = (const void *)(t + 1);
__u32 i, vlen = BTF_INFO_VLEN(t->info);
__u32 i, vlen = btf_vlen(t);
if (json_output) {
jsonw_uint_field(w, "size", t->size);
@@ -193,7 +193,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
const char *name = btf_str(btf, m->name_off);
__u32 bit_off, bit_sz;
if (BTF_INFO_KFLAG(t->info)) {
if (btf_kflag(t)) {
bit_off = BTF_MEMBER_BIT_OFFSET(m->offset);
bit_sz = BTF_MEMBER_BITFIELD_SIZE(m->offset);
} else {
@@ -224,7 +224,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
}
case BTF_KIND_ENUM: {
const struct btf_enum *v = (const void *)(t + 1);
__u32 i, vlen = BTF_INFO_VLEN(t->info);
__u32 i, vlen = btf_vlen(t);
const char *encoding;
encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
@@ -300,8 +300,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
break;
}
case BTF_KIND_FWD: {
const char *fwd_kind = BTF_INFO_KFLAG(t->info) ? "union"
: "struct";
const char *fwd_kind = btf_kflag(t) ? "union" : "struct";
if (json_output)
jsonw_string_field(w, "fwd_kind", fwd_kind);
@@ -322,7 +321,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
}
case BTF_KIND_FUNC_PROTO: {
const struct btf_param *p = (const void *)(t + 1);
__u32 i, vlen = BTF_INFO_VLEN(t->info);
__u32 i, vlen = btf_vlen(t);
if (json_output) {
jsonw_uint_field(w, "ret_type_id", t->type);
@@ -365,7 +364,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
case BTF_KIND_DATASEC: {
const struct btf_var_secinfo *v = (const void *)(t + 1);
const struct btf_type *vt;
__u32 i, vlen = BTF_INFO_VLEN(t->info);
__u32 i, vlen = btf_vlen(t);
if (json_output) {
jsonw_uint_field(w, "size", t->size);

14
tools/bpf/bpftool/btf_dumper.c
View File

@@ -476,8 +476,8 @@ static int btf_dumper_struct(const struct btf_dumper *d, __u32 type_id,
if (!t)
return -EINVAL;
kind_flag = BTF_INFO_KFLAG(t->info);
vlen = BTF_INFO_VLEN(t->info);
kind_flag = btf_kflag(t);
vlen = btf_vlen(t);
jsonw_start_object(d->jw);
m = (struct btf_member *)(t + 1);
@@ -535,7 +535,7 @@ static int btf_dumper_datasec(const struct btf_dumper *d, __u32 type_id,
if (!t)
return -EINVAL;
vlen = BTF_INFO_VLEN(t->info);
vlen = btf_vlen(t);
vsi = (struct btf_var_secinfo *)(t + 1);
jsonw_start_object(d->jw);
@@ -557,7 +557,7 @@ static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id,
{
const struct btf_type *t = btf__type_by_id(d->btf, type_id);
switch (BTF_INFO_KIND(t->info)) {
switch (btf_kind(t)) {
case BTF_KIND_INT:
return btf_dumper_int(t, bit_offset, data, d->jw,
d->is_plain_text);
@@ -631,7 +631,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
t = btf__type_by_id(btf, type_id);
switch (BTF_INFO_KIND(t->info)) {
switch (btf_kind(t)) {
case BTF_KIND_INT:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FLOAT:
@@ -661,7 +661,7 @@ static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
break;
case BTF_KIND_FWD:
BTF_PRINT_ARG("%s %s ",
BTF_INFO_KFLAG(t->info) ? "union" : "struct",
btf_kflag(t) ? "union" : "struct",
btf__name_by_offset(btf, t->name_off));
break;
case BTF_KIND_VOLATILE:
@@ -718,7 +718,7 @@ static int btf_dump_func(const struct btf *btf, char *func_sig,
BTF_PRINT_ARG("%s(", btf__name_by_offset(btf, func->name_off));
else
BTF_PRINT_ARG("(");
vlen = BTF_INFO_VLEN(func_proto->info);
vlen = btf_vlen(func_proto);
for (i = 0; i < vlen; i++) {
struct btf_param *arg = &((struct btf_param *)(func_proto + 1))[i];

4
tools/bpf/bpftool/common.c
View File

@@ -832,6 +832,8 @@ static int prog_fd_by_nametag(void *nametag, int **fds, bool tag)
fd = bpf_prog_get_fd_by_id(id);
if (fd < 0) {
if (errno == ENOENT)
continue;
p_err("can't get prog by id (%u): %s",
id, strerror(errno));
goto err_close_fds;
@@ -996,6 +998,8 @@ static int map_fd_by_name(char *name, int **fds,
opts_ro.open_flags = BPF_F_RDONLY;
fd = bpf_map_get_fd_by_id_opts(id, &opts_ro);
if (fd < 0) {
if (errno == ENOENT)
continue;
p_err("can't get map by id (%u): %s",
id, strerror(errno));
goto err_close_fds;

55
tools/bpf/bpftool/gen.c
View File

@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
return true;
}
if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
snprintf(buf, buf_sz, "%s", name + 1);
sanitize_identifier(buf);
return true;
}
for (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) {
const char *sfx = sfxs[i], *p;
@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
{
static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
int i, n;
/* recognize hard coded LLVM section name */
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
return NULL;
}
static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)
{
size_t tmp_sz;
@@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
return true;
}
if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
if (!bpf_map__is_internal(map))
return false;
if (!get_map_ident(map, buf, sz))
return false;
return true;
if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
return true;
if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
return bpf_map__btf_value_type_id(map) != 0;
return false;
}
static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
{
return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY;
}
static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
bpf_object__for_each_map(map, obj) {
/* only generate definitions for memory-mapped internal maps */
if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
if (!is_skel_data(map, map_ident, sizeof(map_ident)))
continue;
sec = find_type_for_map(btf, map_ident);
@@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
", obj_name);
bpf_object__for_each_map(map, obj) {
if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
if (!is_skel_data(map, map_ident, sizeof(map_ident)))
continue;
sec = find_type_for_map(btf, map_ident);
@@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
bpf_object__for_each_map(map, obj) {
if (!get_map_ident(map, ident, sizeof(ident)))
continue;
if (bpf_map__is_internal(map) &&
(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
if (is_skel_data(map, ident, sizeof(ident)))
printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
ident, bpf_map_mmap_sz(map));
codegen("\
@@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
const void *mmap_data = NULL;
size_t mmap_size = 0;
if (!is_mmapable_map(map, ident, sizeof(ident)))
if (!is_skel_data(map, ident, sizeof(ident)))
continue;
codegen("\
@@ -793,6 +809,8 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
if (sign_progs) {
sopts.insns = opts.insns;
sopts.insns_sz = opts.insns_sz;
sopts.data = opts.data;
sopts.data_sz = opts.data_sz;
sopts.excl_prog_hash = prog_sha;
sopts.excl_prog_hash_sz = sizeof(prog_sha);
sopts.signature = sig_buf;
@@ -847,9 +865,23 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
bpf_object__for_each_map(map, obj) {
const char *mmap_flags;
if (!is_mmapable_map(map, ident, sizeof(ident)))
if (!is_skel_data(map, ident, sizeof(ident)))
continue;
if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
codegen("\
\n\
err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
if (err) \n\
return err; \n\
#ifdef __KERNEL__ \n\
skel->%1$s = NULL; \n\
#endif \n\
",
ident, bpf_map_mmap_sz(map));
continue;
}
if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
mmap_flags = "PROT_READ";
else
@@ -953,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
map->map = &obj->maps.%s; \n\
",
i, bpf_map__name(map), ident);
/* memory-mapped internal maps */
if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
}

133
tools/bpf/bpftool/link.c
View File

@@ -377,6 +377,25 @@ static __u64 *u64_to_arr(__u64 val)
return (__u64 *) u64_to_ptr(val);
}
static __u32 *u64_to_u32_arr(__u64 val)
{
return (__u32 *)u64_to_ptr(val);
}
static struct kernel_sym *find_kernel_sym_by_addr(__u64 addr, bool is_ibt_enabled)
{
struct kernel_sym *sym;
if (!addr)
return NULL;
sym = kernel_syms_search(&dd, addr);
if (!sym && is_ibt_enabled && addr >= 4)
sym = kernel_syms_search(&dd, addr - 4);
return sym;
}
static void
show_uprobe_multi_json(struct bpf_link_info *info, json_writer_t *wtr)
{
@@ -403,6 +422,52 @@ show_uprobe_multi_json(struct bpf_link_info *info, json_writer_t *wtr)
jsonw_end_array(json_wtr);
}
static void
show_tracing_multi_json(struct bpf_link_info *info, json_writer_t *wtr)
{
bool is_ibt_enabled = is_x86_ibt_enabled(), show_symbol;
__u64 *addrs, *cookies;
__u32 i, *ids;
if (!dd.sym_count)
kernel_syms_load(&dd);
show_symbol = !!dd.sym_count;
show_link_attach_type_json(info->tracing_multi.attach_type, wtr);
jsonw_uint_field(wtr, "func_cnt", info->tracing_multi.count);
jsonw_uint_field(wtr, "btf_obj_id", info->tracing_multi.btf_obj_id);
jsonw_name(wtr, "funcs");
jsonw_start_array(wtr);
ids = u64_to_u32_arr(info->tracing_multi.ids);
addrs = u64_to_arr(info->tracing_multi.addrs);
cookies = u64_to_arr(info->tracing_multi.cookies);
for (i = 0; i < info->tracing_multi.count; i++) {
struct kernel_sym *sym;
__u64 addr = addrs[i];
sym = show_symbol ? find_kernel_sym_by_addr(addr, is_ibt_enabled) : NULL;
jsonw_start_object(wtr);
jsonw_uint_field(wtr, "id", ids[i]);
jsonw_uint_field(wtr, "addr", addr);
if (sym) {
jsonw_string_field(wtr, "func", sym->name);
if (sym->module[0] == '\0') {
jsonw_name(wtr, "module");
jsonw_null(wtr);
} else {
jsonw_string_field(wtr, "module", sym->module);
}
}
jsonw_uint_field(wtr, "cookie", cookies[i]);
jsonw_end_object(wtr);
}
jsonw_end_array(wtr);
}
static void
show_perf_event_kprobe_json(struct bpf_link_info *info, json_writer_t *wtr)
{
@@ -589,6 +654,9 @@ static int show_link_close_json(int fd, struct bpf_link_info *info)
case BPF_LINK_TYPE_UPROBE_MULTI:
show_uprobe_multi_json(info, json_wtr);
break;
case BPF_LINK_TYPE_TRACING_MULTI:
show_tracing_multi_json(info, json_wtr);
break;
case BPF_LINK_TYPE_PERF_EVENT:
switch (info->perf_event.type) {
case BPF_PERF_EVENT_EVENT:
@@ -833,6 +901,46 @@ static void show_uprobe_multi_plain(struct bpf_link_info *info)
}
}
static void show_tracing_multi_plain(struct bpf_link_info *info)
{
bool is_ibt_enabled = is_x86_ibt_enabled(), show_symbol;
__u64 *addrs, *cookies;
__u32 i, *ids;
if (!info->tracing_multi.count)
return;
if (!dd.sym_count)
kernel_syms_load(&dd);
show_symbol = !!dd.sym_count;
printf("\n\t");
show_link_attach_type_plain(info->tracing_multi.attach_type);
printf("btf_obj_id %u ", info->tracing_multi.btf_obj_id);
printf("count %u ", info->tracing_multi.count);
printf("\n\t%-16s %-16s %-16s %s",
"btf_id", "addr", "cookie", "func [module]");
ids = u64_to_u32_arr(info->tracing_multi.ids);
addrs = u64_to_arr(info->tracing_multi.addrs);
cookies = u64_to_arr(info->tracing_multi.cookies);
for (i = 0; i < info->tracing_multi.count; i++) {
__u64 addr = addrs[i];
struct kernel_sym *sym;
sym = show_symbol ? find_kernel_sym_by_addr(addr, is_ibt_enabled) : NULL;
printf("\n\t%-16u %016llx %-16llu", ids[i], addr, cookies[i]);
if (sym) {
printf(" %s", sym->name);
if (sym->module[0] != '\0')
printf(" [%s]", sym->module);
}
}
}
static void show_perf_event_kprobe_plain(struct bpf_link_info *info)
{
const char *buf;
@@ -989,6 +1097,9 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info)
case BPF_LINK_TYPE_UPROBE_MULTI:
show_uprobe_multi_plain(info);
break;
case BPF_LINK_TYPE_TRACING_MULTI:
show_tracing_multi_plain(info);
break;
case BPF_LINK_TYPE_PERF_EVENT:
switch (info->perf_event.type) {
case BPF_PERF_EVENT_EVENT:
@@ -1029,6 +1140,7 @@ static int show_link_close_plain(int fd, struct bpf_link_info *info)
static int do_show_link(int fd)
{
__u64 *ref_ctr_offsets = NULL, *offsets = NULL, *cookies = NULL;
__u32 *ids = NULL;
struct bpf_link_info info;
__u32 len = sizeof(info);
char path_buf[PATH_MAX];
@@ -1114,6 +1226,26 @@ static int do_show_link(int fd)
goto again;
}
}
if (info.type == BPF_LINK_TYPE_TRACING_MULTI && !info.tracing_multi.ids) {
count = info.tracing_multi.count;
if (count) {
ids = calloc(count, sizeof(__u32));
addrs = calloc(count, sizeof(__u64));
cookies = calloc(count, sizeof(__u64));
if (!ids || !addrs || !cookies) {
p_err("mem alloc failed");
close(fd);
free(cookies);
free(addrs);
free(ids);
return -ENOMEM;
}
info.tracing_multi.ids = ptr_to_u64(ids);
info.tracing_multi.addrs = ptr_to_u64(addrs);
info.tracing_multi.cookies = ptr_to_u64(cookies);
goto again;
}
}
if (info.type == BPF_LINK_TYPE_PERF_EVENT) {
switch (info.perf_event.type) {
case BPF_PERF_EVENT_TRACEPOINT:
@@ -1153,6 +1285,7 @@ static int do_show_link(int fd)
free(cookies);
free(offsets);
free(addrs);
free(ids);
close(fd);
return 0;
}

16
tools/bpf/bpftool/map.c
View File

@@ -659,8 +659,6 @@ static int do_show_subset(int argc, char **argv)
show_map_close_json(fds[i], &info);
else
show_map_close_plain(fds[i], &info);
close(fds[i]);
}
if (json_output && nb_fds > 1)
jsonw_end_array(json_wtr); /* root array */
@@ -790,6 +788,12 @@ static int maps_have_btf(int *fds, int nb_fds)
static struct btf *btf_vmlinux;
static void free_btf_vmlinux(void)
{
btf__free(btf_vmlinux);
btf_vmlinux = NULL;
}
static int get_map_kv_btf(const struct bpf_map_info *info, struct btf **btf)
{
int err = 0;
@@ -889,7 +893,6 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr,
exit_free:
free(key);
free(value);
close(fd);
free_map_kv_btf(btf);
return err;
@@ -938,6 +941,7 @@ static int do_dump(int argc, char **argv)
for (i = 0; i < nb_fds; i++) {
if (bpf_map_get_info_by_fd(fds[i], &info, &len)) {
p_err("can't get map info: %s", strerror(errno));
err = -1;
break;
}
err = map_dump(fds[i], &info, wtr, nb_fds > 1);
@@ -958,7 +962,7 @@ static int do_dump(int argc, char **argv)
close(fds[i]);
exit_free:
free(fds);
btf__free(btf_vmlinux);
free_btf_vmlinux();
return err;
}
@@ -1049,7 +1053,7 @@ static void print_key_value(struct bpf_map_info *info, void *key,
btf_wtr = get_btf_writer();
if (!btf_wtr) {
p_info("failed to create json writer for btf. falling back to plain output");
btf__free(btf);
free_map_kv_btf(btf);
btf = NULL;
print_entry_plain(info, key, value);
} else {
@@ -1065,7 +1069,7 @@ static void print_key_value(struct bpf_map_info *info, void *key,
} else {
print_entry_plain(info, key, value);
}
btf__free(btf);
free_map_kv_btf(btf);
}
static int do_lookup(int argc, char **argv)

24
tools/bpf/bpftool/sign.c
View File

@@ -135,9 +135,21 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
CMS_ContentInfo *cms = NULL;
long actual_sig_len = 0;
X509 *x509 = NULL;
void *data = NULL;
size_t data_sz;
int err = 0;
bd_in = BIO_new_mem_buf(opts->insns, opts->insns_sz);
data_sz = (size_t)opts->insns_sz + opts->data_sz;
data = malloc(data_sz);
if (!data) {
err = -ENOMEM;
goto cleanup;
}
memcpy(data, opts->insns, opts->insns_sz);
if (opts->data_sz)
memcpy((char *)data + opts->insns_sz, opts->data, opts->data_sz);
bd_in = BIO_new_mem_buf(data, data_sz);
if (!bd_in) {
err = -ENOMEM;
goto cleanup;
@@ -175,10 +187,13 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
goto cleanup;
}
EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash,
&opts->excl_prog_hash_sz, EVP_sha256(), NULL);
if (EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash,
&opts->excl_prog_hash_sz, EVP_sha256(), NULL) != 1) {
err = -EIO;
goto cleanup;
}
bd_out = BIO_new(BIO_s_mem());
bd_out = BIO_new(BIO_s_mem());
if (!bd_out) {
err = -ENOMEM;
goto cleanup;
@@ -212,6 +227,7 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts)
X509_free(x509);
EVP_PKEY_free(private_key);
BIO_free(bd_in);
free(data);
DISPLAY_OSSL_ERR(err < 0);
return err;
}

4
tools/bpf/bpftool/struct_ops.c
View File

@@ -643,6 +643,10 @@ int do_struct_ops(int argc, char **argv)
err = cmd_select(cmds, argc, argv, do_help);
btf__free(btf_vmlinux);
btf_vmlinux = NULL;
map_info_type = NULL;
map_info_alloc_len = 0;
map_info_type_id = 0;
return err;
}

19
tools/bpf/bpftool/xlated_dumper.c
View File

@@ -107,14 +107,7 @@ print_insn_for_graph(void *private_data, const char *fmt, ...)
p = buf;
while (*p != '\0') {
if (*p == '\n') {
memmove(p + 3, p, strlen(buf) + 1 - (p - buf));
/* Align each instruction dump row left. */
*p++ = '\\';
*p++ = 'l';
/* Output multiline concatenation. */
*p++ = '\\';
} else if (*p == '<' || *p == '>' || *p == '|' || *p == '&') {
if (*p == '<' || *p == '>' || *p == '|' || *p == '&') {
memmove(p + 1, p, strlen(buf) + 1 - (p - buf));
/* Escape special character. */
*p++ = '\\';
@@ -129,16 +122,10 @@ print_insn_for_graph(void *private_data, const char *fmt, ...)
static void __printf(2, 3)
print_insn_json(void *private_data, const char *fmt, ...)
{
unsigned int l = strlen(fmt);
char chomped_fmt[l];
va_list args;
va_start(args, fmt);
if (l > 0) {
strncpy(chomped_fmt, fmt, l - 1);
chomped_fmt[l - 1] = '\0';
}
jsonw_vprintf_enquote(json_wtr, chomped_fmt, args);
jsonw_vprintf_enquote(json_wtr, fmt, args);
va_end(args);
}
@@ -351,6 +338,7 @@ void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len,
printf("%4u: ", i);
print_bpf_insn(&cbs, insn + i, true);
printf("\n");
if (opcodes) {
printf(" ");
@@ -417,6 +405,7 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
printf("%u: ", insn_off);
print_bpf_insn(&cbs, cur, true);
printf("\\l\\\n");
if (opcodes) {
printf("\\ \\ \\ \\ ");

5
tools/bpf/resolve_btfids/Makefile
View File

@@ -40,7 +40,6 @@ LIBBPF_DESTDIR := $(LIBBPF_OUT)
LIBBPF_INCLUDE := $(LIBBPF_DESTDIR)include
SUBCMD_DESTDIR := $(SUBCMD_OUT)
SUBCMD_INCLUDE := $(SUBCMD_DESTDIR)include
BINARY := $(OUTPUT)/resolve_btfids
BINARY_IN := $(BINARY)-in.o
@@ -56,7 +55,7 @@ $(OUTPUT) $(OUTPUT)/libsubcmd $(LIBBPF_OUT):
$(SUBCMDOBJ): fixdep FORCE | $(OUTPUT)/libsubcmd
$(Q)$(MAKE) -C $(SUBCMD_SRC) OUTPUT=$(SUBCMD_OUT) \
DESTDIR=$(SUBCMD_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \
$(abspath $@) install_headers
$(abspath $@)
$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(LIBBPF_OUT)
$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) OUTPUT=$(LIBBPF_OUT) \
@@ -77,7 +76,7 @@ HOSTCFLAGS_resolve_btfids += -g \
-I$(srctree)/tools/include \
-I$(srctree)/tools/include/uapi \
-I$(LIBBPF_INCLUDE) \
-I$(SUBCMD_INCLUDE) \
-I$(srctree)/tools/lib \
$(LIBELF_FLAGS) \
-Wall -Werror

478
tools/bpf/resolve_btfids/main.c
View File

@@ -58,6 +58,17 @@
* __BTF_ID__func__vfs_fallocate__5:
* .zero 4
* .word (1 << 3) | (1 << 1) | (1 << 2)
*
* In addition to resolving BTF IDs, resolve_btfids performs kernel-specific
* BTF-to-BTF transformations for kfuncs found in BTF_SET8_KFUNCS sets. For
* each such kfunc it:
*
* - emits a "bpf_kfunc" decl tag, and "bpf_fastcall" when KF_FASTCALL is set;
* - wraps the return value and/or arguments that use arena pointers
* with the "address_space(1)" type attribute;
* - rewrites the prototype of KF_IMPLICIT_ARGS kfuncs.
*
* These kfunc annotations were historically produced by pahole.
*/
#define _GNU_SOURCE
@@ -119,6 +130,11 @@ struct btf_id {
Elf64_Addr addr[ADDR_CNT];
};
struct addr_sym {
Elf64_Addr addr;
const char *name;
};
struct object {
const char *path;
const char *btf_path;
@@ -150,12 +166,27 @@ struct object {
int nr_structs;
int nr_unions;
int nr_typedefs;
struct addr_sym *addr_syms;
u32 addr_syms_cnt;
u32 addr_syms_cap;
};
#define DECL_TAG_FASTCALL "bpf_fastcall"
#define DECL_TAG_KFUNC "bpf_kfunc"
#define KF_FASTCALL (1 << 12)
#define KF_ARENA_RET (1 << 13)
#define KF_ARENA_ARG1 (1 << 14)
#define KF_ARENA_ARG2 (1 << 15)
#define KF_IMPLICIT_ARGS (1 << 16)
#define KF_IMPL_SUFFIX "_impl"
#define TYPE_ATTR_ARENA "address_space(1)"
#define PARAM_SUFFIX_ARENA "__arena"
#define PARAM_SUFFIX_ARENA_NULLABLE "__arena__nullable"
struct kfunc {
struct rb_node rb_node;
const char *name;
u32 btf_id;
u32 flags;
@@ -166,9 +197,7 @@ struct btf2btf_context {
u32 *decl_tags;
u32 nr_decl_tags;
u32 max_decl_tags;
struct kfunc *kfuncs;
u32 nr_kfuncs;
u32 max_kfuncs;
struct rb_root kfuncs;
};
static int verbose;
@@ -201,6 +230,35 @@ static int eprintf(int level, int var, const char *fmt, ...)
#define pr_info(fmt, ...) \
eprintf(0, verbose, pr_fmt(fmt), ##__VA_ARGS__)
/*
* Grow *data so it can hold at least cnt elements of elem_sz bytes each.
* *cap is the capacity in elements and is updated on growth.
*/
static int __ensure_mem(void **data, u32 *cap, u32 cnt, size_t elem_sz)
{
u32 new_cap, old_cap = *cap;
void *arr;
if (cnt <= old_cap)
return 0;
new_cap = max(old_cap + 256, old_cap * 2);
if (new_cap < cnt)
new_cap = cnt;
arr = realloc(*data, elem_sz * new_cap);
if (!arr)
return -ENOMEM;
*data = arr;
*cap = new_cap;
return 0;
}
#define ensure_mem(arr_ptr, cap_ptr, cnt) \
__ensure_mem((void **)(arr_ptr), (cap_ptr), (cnt), sizeof(**(arr_ptr)))
static bool is_btf_id(const char *name)
{
return name && !strncmp(name, BTF_ID_PREFIX, sizeof(BTF_ID_PREFIX) - 1);
@@ -480,6 +538,40 @@ static int elf_collect(struct object *obj)
return 0;
}
static int push_addr_sym(struct object *obj, Elf64_Addr addr, const char *name)
{
if (ensure_mem(&obj->addr_syms, &obj->addr_syms_cap, obj->addr_syms_cnt + 1))
return -ENOMEM;
obj->addr_syms[obj->addr_syms_cnt++] = (struct addr_sym){
.addr = addr,
.name = name,
};
return 0;
}
static int cmp_addr_sym(const void *a, const void *b)
{
Elf64_Addr aa = ((const struct addr_sym *)a)->addr;
Elf64_Addr ab = ((const struct addr_sym *)b)->addr;
return (aa > ab) - (aa < ab);
}
static const char *find_name_by_addr(struct object *obj, Elf64_Addr addr)
{
struct addr_sym key = { .addr = addr };
struct addr_sym *res;
if (!obj->addr_syms_cnt)
return NULL;
res = bsearch(&key, obj->addr_syms, obj->addr_syms_cnt,
sizeof(*obj->addr_syms), cmp_addr_sym);
return res ? res->name : NULL;
}
static int symbols_collect(struct object *obj)
{
Elf_Scn *scn = NULL;
@@ -573,8 +665,15 @@ static int symbols_collect(struct object *obj)
return -1;
}
id->addr[id->addr_cnt++] = sym.st_value;
if (push_addr_sym(obj, sym.st_value, id->name))
return -1;
}
if (obj->addr_syms_cnt)
qsort(obj->addr_syms, obj->addr_syms_cnt,
sizeof(*obj->addr_syms), cmp_addr_sym);
return 0;
}
@@ -890,17 +989,8 @@ static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf, s3
static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id)
{
u32 *arr = ctx->decl_tags;
u32 cap = ctx->max_decl_tags;
if (ctx->nr_decl_tags + 1 > cap) {
cap = max(cap + 256, cap * 2);
arr = realloc(arr, sizeof(u32) * cap);
if (!arr)
return -ENOMEM;
ctx->max_decl_tags = cap;
ctx->decl_tags = arr;
}
if (ensure_mem(&ctx->decl_tags, &ctx->max_decl_tags, ctx->nr_decl_tags + 1))
return -ENOMEM;
ctx->decl_tags[ctx->nr_decl_tags++] = decl_tag_id;
@@ -909,23 +999,57 @@ static int push_decl_tag_id(struct btf2btf_context *ctx, u32 decl_tag_id)
static int push_kfunc(struct btf2btf_context *ctx, struct kfunc *kfunc)
{
struct kfunc *arr = ctx->kfuncs;
u32 cap = ctx->max_kfuncs;
struct rb_node **p = &ctx->kfuncs.rb_node;
struct rb_node *parent = NULL;
struct kfunc *k;
if (ctx->nr_kfuncs + 1 > cap) {
cap = max(cap + 256, cap * 2);
arr = realloc(arr, sizeof(struct kfunc) * cap);
if (!arr)
return -ENOMEM;
ctx->max_kfuncs = cap;
ctx->kfuncs = arr;
/*
* Dedup by BTF ID: collecting the same kfunc twice is a no-op,
* UNLESS the kfunc flags are inconsistent, in which case we
* fail hard because it indicates a bug in a kfunc set declaration.
*/
while (*p) {
parent = *p;
k = rb_entry(parent, struct kfunc, rb_node);
if (kfunc->btf_id < k->btf_id) {
p = &(*p)->rb_left;
} else if (kfunc->btf_id > k->btf_id) {
p = &(*p)->rb_right;
} else if (k->flags == kfunc->flags) {
return 0;
} else {
pr_err("ERROR: resolve_btfids: kfunc %s has inconsistent flags across BTF ID sets: 0x%x != 0x%x\n",
kfunc->name, k->flags, kfunc->flags);
return -EINVAL;
}
}
ctx->kfuncs[ctx->nr_kfuncs++] = *kfunc;
k = zalloc(sizeof(*k));
if (!k)
return -ENOMEM;
*k = *kfunc;
rb_link_node(&k->rb_node, parent, p);
rb_insert_color(&k->rb_node, &ctx->kfuncs);
return 0;
}
static void free_kfuncs(struct rb_root *root)
{
struct rb_node *next;
struct kfunc *kfunc;
next = rb_first(root);
while (next) {
kfunc = rb_entry(next, struct kfunc, rb_node);
next = rb_next(&kfunc->rb_node);
rb_erase(&kfunc->rb_node, root);
free(kfunc);
}
}
static int collect_decl_tags(struct btf2btf_context *ctx)
{
const u32 type_cnt = btf__type_cnt(ctx->btf);
@@ -945,94 +1069,76 @@ static int collect_decl_tags(struct btf2btf_context *ctx)
return 0;
}
/*
* To find the kfunc flags having its struct btf_id (with ELF addresses)
* we need to find the address that is in range of a set8.
* If a set8 is found, then the flags are located at addr + 4 bytes.
* Return 0 (no flags!) if not found.
*/
static u32 find_kfunc_flags(struct object *obj, struct btf_id *kfunc_id)
static bool param_name_has_suffix(const char *name, const char *suffix)
{
const u32 *elf_data_ptr = obj->efile.idlist->d_buf;
u64 set_lower_addr, set_upper_addr, addr;
struct btf_id *set_id;
struct rb_node *next;
u32 flags;
u64 idx;
size_t name_len = strlen(name);
size_t suffix_len = strlen(suffix);
for (next = rb_first(&obj->sets); next; next = rb_next(next)) {
set_id = rb_entry(next, struct btf_id, rb_node);
if (set_id->kind != BTF_ID_KIND_SET8 || set_id->addr_cnt != 1)
continue;
return name_len >= suffix_len && !strcmp(name + name_len - suffix_len, suffix);
}
set_lower_addr = set_id->addr[0];
set_upper_addr = set_lower_addr + set_id->cnt * sizeof(u64);
static bool is_arena_param(const struct btf *btf, const struct btf_param *param)
{
const char *name = btf__name_by_offset(btf, param->name_off);
for (u32 i = 0; i < kfunc_id->addr_cnt; i++) {
addr = kfunc_id->addr[i];
/*
* Lower bound is exclusive to skip the 8-byte header of the set.
* Upper bound is inclusive to capture the last entry at offset 8*cnt.
*/
if (set_lower_addr < addr && addr <= set_upper_addr) {
pr_debug("found kfunc %s in BTF_ID_FLAGS %s\n",
kfunc_id->name, set_id->name);
idx = addr - obj->efile.idlist_addr;
idx = idx / sizeof(u32) + 1;
flags = elf_data_ptr[idx];
return flags;
}
}
}
return 0;
return param_name_has_suffix(name, PARAM_SUFFIX_ARENA) ||
param_name_has_suffix(name, PARAM_SUFFIX_ARENA_NULLABLE);
}
static int collect_kfuncs(struct object *obj, struct btf2btf_context *ctx)
{
const char *tag_name, *func_name;
Elf_Data *idlist = obj->efile.idlist;
struct btf *btf = ctx->btf;
const struct btf_type *t;
u32 flags, func_id;
struct kfunc kfunc;
struct btf_id *id;
int err;
struct rb_node *next;
if (ctx->nr_decl_tags == 0)
if (!idlist || !idlist->d_buf)
return 0;
for (u32 i = 0; i < ctx->nr_decl_tags; i++) {
t = btf__type_by_id(btf, ctx->decl_tags[i]);
if (btf_kflag(t) || btf_decl_tag(t)->component_idx != -1)
for (next = rb_first(&obj->sets); next; next = rb_next(next)) {
struct btf_id_set8 *set8;
struct btf_id *set_id;
u64 set_addr;
set_id = rb_entry(next, struct btf_id, rb_node);
if (set_id->kind != BTF_ID_KIND_SET8 || set_id->addr_cnt != 1)
continue;
tag_name = btf__name_by_offset(btf, t->name_off);
if (strcmp(tag_name, "bpf_kfunc") != 0)
set_addr = set_id->addr[0];
set8 = idlist->d_buf + (set_addr - obj->efile.idlist_addr);
if (!(set8->flags & BTF_SET8_KFUNCS))
continue;
func_id = t->type;
t = btf__type_by_id(btf, func_id);
if (!btf_is_func(t))
continue;
for (u32 i = 0; i < set_id->cnt; i++) {
size_t off = (char *)&set8->pairs[i] - (char *)set8;
const char *name = find_name_by_addr(obj, set_addr + off);
struct kfunc kfunc;
s32 func_id;
int err;
func_name = btf__name_by_offset(btf, t->name_off);
if (!func_name)
continue;
if (!name) {
pr_err("WARN: resolve_btfids: no BTF ID symbol for %s entry %u\n",
set_id->name, i);
warnings++;
continue;
}
id = btf_id__find(&obj->funcs, func_name);
if (!id || id->kind != BTF_ID_KIND_SYM)
continue;
func_id = btf__find_by_name_kind_own(btf, name, BTF_KIND_FUNC);
if (func_id < 0) {
pr_err("WARN: resolve_btfids: no BTF func for kfunc %s in %s\n",
name, set_id->name);
warnings++;
continue;
}
flags = find_kfunc_flags(obj, id);
pr_debug("found kfunc %s in %s\n", name, set_id->name);
kfunc.name = id->name;
kfunc.btf_id = func_id;
kfunc.flags = flags;
err = push_kfunc(ctx, &kfunc);
if (err)
return err;
kfunc.name = name;
kfunc.btf_id = func_id;
kfunc.flags = set8->pairs[i].flags;
err = push_kfunc(ctx, &kfunc);
if (err)
return err;
}
}
return 0;
@@ -1141,7 +1247,7 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct
return -E2BIG;
}
if (btf__find_by_name_kind(btf, tmp_name, BTF_KIND_FUNC) > 0) {
if (btf__find_by_name_kind_own(btf, tmp_name, BTF_KIND_FUNC) > 0) {
pr_debug("resolve_btfids: function %s already exists in BTF\n", tmp_name);
goto add_new_proto;
}
@@ -1160,7 +1266,7 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct
continue;
tag_name = btf__name_by_offset(btf, t->name_off);
if (strcmp(tag_name, "bpf_kfunc") == 0)
if (strcmp(tag_name, DECL_TAG_KFUNC) == 0)
continue;
idx = btf_decl_tag(t)->component_idx;
@@ -1211,22 +1317,187 @@ static int process_kfunc_with_implicit_args(struct btf2btf_context *ctx, struct
return 0;
}
static bool is_arena_arg(const struct btf *btf, const struct kfunc *kfunc,
const struct btf_param *param, u32 idx)
{
if (is_arena_param(btf, param))
return true;
switch (idx) {
case 0:
return kfunc->flags & KF_ARENA_ARG1;
case 1:
return kfunc->flags & KF_ARENA_ARG2;
default:
return false;
}
}
static s32 arena_tag_ptr(struct btf *btf, u32 ptr_id, struct kfunc *kfunc)
{
const struct btf_type *ptr = btf__type_by_id(btf, ptr_id);
s32 tag_id, new_ptr_id;
if (!btf_is_ptr(ptr)) {
pr_err("ERROR: resolve_btfids: kfunc %s: arena type is not a pointer\n",
kfunc->name);
return -EINVAL;
}
tag_id = btf__add_type_attr(btf, TYPE_ATTR_ARENA, ptr->type);
if (tag_id < 0) {
pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a type attr to BTF: %d\n",
kfunc->name, tag_id);
return tag_id;
}
new_ptr_id = btf__add_ptr(btf, tag_id);
if (new_ptr_id < 0) {
pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a pointer to BTF: %d\n",
kfunc->name, new_ptr_id);
}
return new_ptr_id;
}
/*
* Add a FUNC_PROTO for @kfunc with each arena pointer tagged with an
* "address_space(1)" attribute. The original proto may be shared with
* other FUNCs, so it is never modified in place. Returns the original
* proto id when @kfunc has no arena return value or arguments.
*/
static s32 add_arena_tagged_proto(struct btf *btf, struct kfunc *kfunc)
{
const struct btf_type *func = btf__type_by_id(btf, kfunc->btf_id);
u32 proto_id = func->type;
const struct btf_type *proto = btf__type_by_id(btf, proto_id);
const struct btf_param *params = btf_params(proto);
u32 nr_params = btf_vlen(proto);
s32 ret_type_id = proto->type;
const struct btf_type *t;
struct btf_param *tag_params;
s32 new_proto_id, id;
const char *name;
bool has_arena_arg = false;
int err, i;
for (i = 0; i < nr_params; i++) {
if (is_arena_arg(btf, kfunc, &params[i], i)) {
has_arena_arg = true;
break;
}
}
if (!(kfunc->flags & KF_ARENA_RET) && !has_arena_arg)
return proto_id;
if (kfunc->flags & KF_ARENA_RET) {
ret_type_id = arena_tag_ptr(btf, ret_type_id, kfunc);
if (ret_type_id < 0)
return ret_type_id;
}
new_proto_id = btf__add_func_proto(btf, ret_type_id);
if (new_proto_id < 0) {
pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a func proto to BTF: %d\n",
kfunc->name, new_proto_id);
return new_proto_id;
}
for (i = 0; i < nr_params; i++) {
/* btf__add_func_param() below may move the proto, re-fetch */
proto = btf__type_by_id(btf, proto_id);
name = btf__name_by_offset(btf, btf_params(proto)[i].name_off);
err = btf__add_func_param(btf, name ?: "", btf_params(proto)[i].type);
if (err < 0) {
pr_err("ERROR: resolve_btfids: kfunc %s: failed to add a proto param to BTF: %d\n",
kfunc->name, err);
return err;
}
}
for (i = 0; i < nr_params; i++) {
t = btf__type_by_id(btf, new_proto_id);
tag_params = btf_params(t);
if (!is_arena_arg(btf, kfunc, &tag_params[i], i))
continue;
id = arena_tag_ptr(btf, tag_params[i].type, kfunc);
if (id < 0)
return id;
t = btf__type_by_id(btf, new_proto_id);
tag_params = btf_params(t);
tag_params[i].type = id;
}
pr_debug("added arena-tagged proto for kfunc %s: %d\n", kfunc->name, new_proto_id);
return new_proto_id;
}
static int process_kfunc_with_arena_attrs(struct btf2btf_context *ctx,
struct kfunc *kfunc)
{
struct btf_type *t;
s32 proto_id;
proto_id = add_arena_tagged_proto(ctx->btf, kfunc);
if (proto_id < 0)
return proto_id;
t = (struct btf_type *)btf__type_by_id(ctx->btf, kfunc->btf_id);
t->type = proto_id;
return 0;
}
static int add_decl_tag(struct btf2btf_context *ctx, const char *tag_name,
u32 target_btf_id, int component_idx)
{
s32 new_id;
new_id = btf__add_decl_tag(ctx->btf, tag_name, target_btf_id, component_idx);
if (new_id < 0) {
pr_err("ERROR: resolve_btfids: failed to add '%s' decl tag for BTF id %u: %d\n",
tag_name, target_btf_id, new_id);
return new_id;
}
return push_decl_tag_id(ctx, new_id);
}
static int btf2btf(struct object *obj)
{
struct btf2btf_context ctx = {};
struct rb_node *next;
int err;
err = build_btf2btf_context(obj, &ctx);
if (err)
goto out;
for (u32 i = 0; i < ctx.nr_kfuncs; i++) {
struct kfunc *kfunc = &ctx.kfuncs[i];
for (next = rb_first(&ctx.kfuncs); next; next = rb_next(next)) {
struct kfunc *kfunc = rb_entry(next, struct kfunc, rb_node);
if (!(kfunc->flags & KF_IMPLICIT_ARGS))
continue;
err = add_decl_tag(&ctx, DECL_TAG_KFUNC, kfunc->btf_id, -1);
if (err)
goto out;
err = process_kfunc_with_implicit_args(&ctx, kfunc);
if (kfunc->flags & KF_FASTCALL) {
err = add_decl_tag(&ctx, DECL_TAG_FASTCALL, kfunc->btf_id, -1);
if (err)
goto out;
}
if (kfunc->flags & KF_IMPLICIT_ARGS) {
err = process_kfunc_with_implicit_args(&ctx, kfunc);
if (err)
goto out;
}
err = process_kfunc_with_arena_attrs(&ctx, kfunc);
if (err)
goto out;
}
@@ -1234,7 +1505,7 @@ static int btf2btf(struct object *obj)
err = 0;
out:
free(ctx.decl_tags);
free(ctx.kfuncs);
free_kfuncs(&ctx.kfuncs);
return err;
}
@@ -1309,6 +1580,12 @@ static int finalize_btf(struct object *obj)
struct btf *base_btf = obj->base_btf, *btf = obj->btf;
int err;
err = btf__dedup(obj->btf, NULL);
if (err) {
pr_err("FAILED to dedup BTF: %s\n", strerror(errno));
goto out_err;
}
if (obj->base_btf && obj->distill_base) {
err = btf__distill_base(obj->btf, &base_btf, &btf);
if (err) {
@@ -1575,6 +1852,7 @@ int main(int argc, const char **argv)
btf_id__free_all(&obj.typedefs);
btf_id__free_all(&obj.funcs);
btf_id__free_all(&obj.sets);
free(obj.addr_syms);
if (obj.efile.elf) {
elf_end(obj.efile.elf);
close(obj.efile.fd);

78
tools/include/linux/btf_ids.h
View File

@@ -10,6 +10,9 @@ struct btf_id_set {
u32 ids[];
};
/* This flag implies BTF_SET8 holds kfunc(s) */
#define BTF_SET8_KFUNCS (1 << 0)
struct btf_id_set8 {
u32 cnt;
u32 flags;
@@ -22,6 +25,7 @@ struct btf_id_set8 {
#ifdef CONFIG_DEBUG_INFO_BTF
#include <linux/compiler.h> /* for __PASTE */
#include <linux/stringify.h>
/*
* Following macros help to define lists of BTF IDs placed
@@ -35,7 +39,7 @@ struct btf_id_set8 {
#define BTF_IDS_SECTION ".BTF_ids"
#define ____BTF_ID(symbol) \
#define ____BTF_ID(symbol, word) \
asm( \
".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
".local " #symbol " ; \n" \
@@ -43,10 +47,11 @@ asm( \
".size " #symbol ", 4; \n" \
#symbol ": \n" \
".zero 4 \n" \
word \
".popsection; \n");
#define __BTF_ID(symbol) \
____BTF_ID(symbol)
#define __BTF_ID(symbol, word) \
____BTF_ID(symbol, word)
#define __ID(prefix) \
__PASTE(__PASTE(prefix, __COUNTER__), __LINE__)
@@ -56,7 +61,14 @@ asm( \
* to 4 zero bytes.
*/
#define BTF_ID(prefix, name) \
__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__))
__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), "")
#define ____BTF_ID_FLAGS(prefix, name, flags) \
__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), ".long " #flags "\n")
#define __BTF_ID_FLAGS(prefix, name, flags, ...) \
____BTF_ID_FLAGS(prefix, name, flags)
#define BTF_ID_FLAGS(prefix, name, ...) \
__BTF_ID_FLAGS(prefix, name, ##__VA_ARGS__, 0)
/*
* The BTF_ID_LIST macro defines pure (unsorted) list
@@ -155,10 +167,58 @@ asm( \
".popsection; \n"); \
extern struct btf_id_set name;
/*
* The BTF_SET8_START/END macros pair defines sorted list of
* BTF IDs and their flags plus its members count, with the
* following layout:
*
* BTF_SET8_START(list)
* BTF_ID_FLAGS(type1, name1, flags)
* BTF_ID_FLAGS(type2, name2, flags)
* BTF_SET8_END(list)
*
* __BTF_ID__set8__list:
* .zero 8
* list:
* __BTF_ID__type1__name1__3:
* .zero 4
* .word (1 << 0) | (1 << 2)
* __BTF_ID__type2__name2__5:
* .zero 4
* .word (1 << 3) | (1 << 1) | (1 << 2)
*
*/
#define __BTF_SET8_START(name, scope, flags) \
__BTF_ID_LIST(name, local) \
asm( \
".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
"." #scope " __BTF_ID__set8__" #name "; \n" \
"__BTF_ID__set8__" #name ":; \n" \
".zero 4 \n" \
".long " __stringify(flags) "\n" \
".popsection; \n");
#define BTF_SET8_START(name) \
__BTF_SET8_START(name, local, 0)
#define BTF_SET8_END(name) \
asm( \
".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \
".size __BTF_ID__set8__" #name ", .-" #name " \n" \
".popsection; \n"); \
extern struct btf_id_set8 name;
#define BTF_KFUNCS_START(name) \
__BTF_SET8_START(name, local, BTF_SET8_KFUNCS)
#define BTF_KFUNCS_END(name) \
BTF_SET8_END(name)
#else
#define BTF_ID_LIST(name) static u32 __maybe_unused name[5];
#define BTF_ID_LIST(name) static u32 __maybe_unused name[128];
#define BTF_ID(prefix, name)
#define BTF_ID_FLAGS(prefix, name, ...)
#define BTF_ID_UNUSED
#define BTF_ID_LIST_GLOBAL(name, n) u32 __maybe_unused name[n];
#define BTF_ID_LIST_SINGLE(name, prefix, typename) static u32 __maybe_unused name[1];
@@ -166,6 +226,10 @@ extern struct btf_id_set name;
#define BTF_SET_START(name) static struct btf_id_set __maybe_unused name = { 0 };
#define BTF_SET_START_GLOBAL(name) static struct btf_id_set __maybe_unused name = { 0 };
#define BTF_SET_END(name)
#define BTF_SET8_START(name) static struct btf_id_set8 __maybe_unused name = { 0 };
#define BTF_SET8_END(name)
#define BTF_KFUNCS_START(name) static struct btf_id_set8 __maybe_unused name = { .flags = BTF_SET8_KFUNCS };
#define BTF_KFUNCS_END(name)
#endif /* CONFIG_DEBUG_INFO_BTF */
@@ -215,5 +279,9 @@ MAX_BTF_TRACING_TYPE,
};
extern u32 btf_tracing_ids[];
extern u32 bpf_cgroup_btf_id[];
extern u32 bpf_local_storage_map_btf_id[];
extern u32 btf_bpf_map_id[];
extern u32 bpf_kmem_cache_btf_id[];
#endif

117
tools/include/uapi/linux/bpf.h
View File

@@ -3038,8 +3038,34 @@ union bpf_attr {
*
* * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**,
* **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**:
* Indicate the new IP header version after decapsulating the outer
* IP header. Used when the inner and outer IP versions are different.
* Indicate the new IP header version after decapsulating the
* outer IP header. Used when the inner and outer IP versions
* are different. These flags only trigger a protocol change
* without clearing any tunnel-specific GSO flags.
*
* * **BPF_F_ADJ_ROOM_DECAP_L4_GRE**:
* Clear GRE tunnel GSO flags (SKB_GSO_GRE and SKB_GSO_GRE_CSUM)
* when decapsulating a GRE tunnel.
*
* * **BPF_F_ADJ_ROOM_DECAP_L4_UDP**:
* Clear UDP tunnel GSO flags (SKB_GSO_UDP_TUNNEL and
* SKB_GSO_UDP_TUNNEL_CSUM) when decapsulating a UDP tunnel.
*
* * **BPF_F_ADJ_ROOM_DECAP_IPXIP4**:
* Clear IPIP/SIT tunnel GSO flag (SKB_GSO_IPXIP4) when decapsulating
* a tunnel with an outer IPv4 header (IPv4-in-IPv4 or IPv6-in-IPv4).
*
* * **BPF_F_ADJ_ROOM_DECAP_IPXIP6**:
* Clear IPv6 encapsulation tunnel GSO flag (SKB_GSO_IPXIP6) when
* decapsulating a tunnel with an outer IPv6 header (IPv6-in-IPv6
* or IPv4-in-IPv6).
*
* When using the decapsulation flags above, the skb->encapsulation
* flag is automatically cleared if all tunnel-specific GSO flags
* (SKB_GSO_UDP_TUNNEL, SKB_GSO_UDP_TUNNEL_CSUM, SKB_GSO_GRE,
* SKB_GSO_GRE_CSUM, SKB_GSO_IPXIP4, SKB_GSO_IPXIP6) have been
* removed from the packet. This handles cases where all tunnel
* layers have been decapsulated.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
@@ -3532,6 +3558,47 @@ union bpf_attr {
* Use the mark present in *params*->mark for the fib lookup.
* This option should not be used with BPF_FIB_LOOKUP_DIRECT,
* as it only has meaning for full lookups.
* **BPF_FIB_LOOKUP_VLAN**
* If the fib lookup resolves to a VLAN device whose
* parent is a real (non-VLAN) device, set
* *params*->h_vlan_proto and *params*->h_vlan_TCI from
* the VLAN device and replace *params*->ifindex with the
* parent's ifindex. *params*->h_vlan_TCI carries the VID
* only, with PCP and DEI bits zero; a consumer wanting to
* set egress priority writes PCP itself. *params*->smac is
* the VLAN device's own address, which can differ from the
* parent's. Only the immediate parent is resolved; if it
* is itself a VLAN device (QinQ) or in another namespace,
* the egress cannot be reduced to a physical device plus
* one tag and the lookup returns
* **BPF_FIB_LKUP_RET_VLAN_FAILURE** with *params*->ifindex
* left at the input. To obtain the VLAN device's own
* ifindex, repeat the lookup without
* **BPF_FIB_LOOKUP_VLAN**, re-initializing *params*
* first: output fields overwrite the inputs they share
* storage with. The swap and the vlan fields
* are written only on success; other output fields keep
* the helper's existing behaviour, so a frag-needed result
* still reports the route mtu in *params*->mtu_result.
* This flag is only valid for XDP programs; tc programs
* receive -EINVAL since they can redirect to the VLAN
* device directly.
* **BPF_FIB_LOOKUP_VLAN_INPUT**
* Treat *params*->h_vlan_proto and *params*->h_vlan_TCI
* as an input VLAN tag and run the lookup as if ingress
* had happened on the VLAN subinterface carrying that tag
* on *params*->ifindex. The VID is the low 12 bits of
* *params*->h_vlan_TCI; *params*->h_vlan_proto must be
* ETH_P_8021Q or ETH_P_8021AD in network byte order, else
* **-EINVAL**. If *params*->ifindex is itself a VLAN
* device, its inner (QinQ) subinterface is matched; for a
* bond or team, pass the master's ifindex. An unmatched
* tag, a down device, or one in another namespace returns
* **BPF_FIB_LKUP_RET_NOT_FWDED**, mirroring real ingress.
* A VID of 0 is looked up literally, so do not set this
* flag for priority-tagged frames. Cannot be combined with
* **BPF_FIB_LOOKUP_TBID** or **BPF_FIB_LOOKUP_OUTPUT**
* (returns **-EINVAL**).
*
* *ctx* is either **struct xdp_md** for XDP programs or
* **struct sk_buff** tc cls_act programs.
@@ -4694,6 +4761,7 @@ union bpf_attr {
* * **BPF_RB_RING_SIZE**: The size of ring buffer.
* * **BPF_RB_CONS_POS**: Consumer position (can wrap around).
* * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around).
* * **BPF_RB_OVERWRITE_POS**: Overwrite position (can wrap around).
*
* Data returned is just a momentary snapshot of actual values
* and could be inaccurate, so this facility should be used to
@@ -5079,17 +5147,19 @@ union bpf_attr {
* Description
* Redirect the packet to another net device of index *ifindex*.
* This helper is somewhat similar to **bpf_redirect**\ (), except
* that the redirection happens to the *ifindex*' peer device and
* the netns switch takes place from ingress to ingress without
* going through the CPU's backlog queue.
* that the redirection happens to the *ifindex*' peer device. If
* *flags* is 0, the netns switch takes place from ingress to
* ingress without going through the CPU's backlog queue. If the
* **BPF_F_EGRESS** flag is provided then redirection happens in
* the egress direction of the peer device.
*
* *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during
* the netns switch.
*
* The *flags* argument is reserved and must be 0. The helper is
* currently only supported for tc BPF program types at the
* ingress hook and for veth and netkit target device types. The
* peer device must reside in a different network namespace.
* If the *flags* argument is 0, the helper is currently only
* supported for tc BPF program types at the ingress hook and for
* veth and netkit target device types. The peer device must reside
* in a different network namespace.
* Return
* The helper returns **TC_ACT_REDIRECT** on success or
* **TC_ACT_SHOT** on error.
@@ -6239,7 +6309,7 @@ enum {
};
/* BPF_FUNC_skb_adjust_room flags. */
enum {
enum bpf_adj_room_flags {
BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2),
@@ -6249,6 +6319,10 @@ enum {
BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6),
BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7),
BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8),
BPF_F_ADJ_ROOM_DECAP_L4_GRE = (1ULL << 9),
BPF_F_ADJ_ROOM_DECAP_L4_UDP = (1ULL << 10),
BPF_F_ADJ_ROOM_DECAP_IPXIP4 = (1ULL << 11),
BPF_F_ADJ_ROOM_DECAP_IPXIP6 = (1ULL << 12),
};
enum {
@@ -6336,9 +6410,10 @@ enum {
/* Flags for bpf_redirect and bpf_redirect_map helpers */
enum {
BPF_F_INGRESS = (1ULL << 0), /* used for skb path */
BPF_F_EGRESS = (1ULL << 1), /* used for skb path */
BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */
BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */
#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS)
#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_EGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS)
};
#define __bpf_md_ptr(type, name) \
@@ -6839,6 +6914,15 @@ struct bpf_link_info {
__u32 flags;
__u32 pid;
} uprobe_multi;
struct {
__u32 attach_type;
__u32 count; /* in/out: tracing_multi target count */
__u32 btf_obj_id;
__u32 :32;
__aligned_u64 ids;
__aligned_u64 addrs;
__aligned_u64 cookies;
} tracing_multi;
struct {
__u32 type; /* enum bpf_perf_event_type */
__u32 :32;
@@ -7327,6 +7411,8 @@ enum {
BPF_FIB_LOOKUP_TBID = (1U << 3),
BPF_FIB_LOOKUP_SRC = (1U << 4),
BPF_FIB_LOOKUP_MARK = (1U << 5),
BPF_FIB_LOOKUP_VLAN = (1U << 6),
BPF_FIB_LOOKUP_VLAN_INPUT = (1U << 7),
};
enum {
@@ -7340,6 +7426,7 @@ enum {
BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */
BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */
BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */
BPF_FIB_LKUP_RET_VLAN_FAILURE, /* VLAN egress, parent unresolvable */
};
struct bpf_fib_lookup {
@@ -7393,7 +7480,13 @@ struct bpf_fib_lookup {
union {
struct {
/* output */
/*
* output with BPF_FIB_LOOKUP_VLAN: set from the
* resolved egress VLAN device (see the flag); zeroed
* on other successful lookups. input with
* BPF_FIB_LOOKUP_VLAN_INPUT: the VLAN tag to scope
* the lookup by.
*/
__be16 h_vlan_proto;
__be16 h_vlan_TCI;
};

4
tools/lib/bpf/bpf_gen_internal.h
View File

@@ -51,7 +51,6 @@ struct bpf_gen {
__u32 nr_ksyms;
int fd_array;
int nr_fd_array;
int hash_insn_offset[SHA256_DWORD_SIZE];
};
void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps);
@@ -66,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
enum bpf_prog_type prog_type, const char *prog_name,
const char *license, struct bpf_insn *insns, size_t insn_cnt,
struct bpf_prog_load_opts *load_attr, int prog_idx);
void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
__u64 flags);
void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,

20
tools/lib/bpf/btf.c
View File

@@ -589,7 +589,7 @@ static int btf_parse_type_sec(struct btf *btf)
if (type_size < 0)
return type_size;
if (next_type + type_size > end_type) {
pr_warn("BTF type [%d] is malformed\n", btf->start_id + btf->nr_types);
pr_warn("BTF type [%u] is malformed\n", btf->start_id + btf->nr_types);
return -EINVAL;
}
@@ -1424,7 +1424,7 @@ static int btf_find_elf_sections(Elf *elf, const char *path, struct btf_elf_secs
continue;
if (sh.sh_type != SHT_PROGBITS) {
pr_warn("unexpected section type (%d) of section(%d, %s) from %s\n",
pr_warn("unexpected section type (%u) of section(%d, %s) from %s\n",
sh.sh_type, idx, name, path);
goto err;
}
@@ -1506,9 +1506,6 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
dist_base_btf = NULL;
}
if (dist_base_btf)
btf->owns_base = true;
switch (gelf_getclass(elf)) {
case ELFCLASS32:
btf__set_pointer_size(btf, 4);
@@ -1523,13 +1520,16 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
if (btf_ext && secs.btf_ext_data) {
*btf_ext = btf_ext__new(secs.btf_ext_data->d_buf, secs.btf_ext_data->d_size);
if (IS_ERR(*btf_ext)) {
err = PTR_ERR(*btf_ext);
if (!*btf_ext) {
err = -errno;
goto done;
}
} else if (btf_ext) {
*btf_ext = NULL;
}
if (dist_base_btf)
btf->owns_base = true;
done:
if (elf)
elf_end(elf);
@@ -4854,7 +4854,7 @@ static bool btf_dedup_identical_types(struct btf_dedup *d, __u32 id1, __u32 id2,
continue;
if (!btf_dedup_identical_types(d, m1->type, m2->type, depth - 1)) {
if (t1->name_off) {
pr_debug("%s '%s' size=%d vlen=%d id1[%u] id2[%u] shallow-equal but not identical for field#%d '%s'\n",
pr_debug("%s '%s' size=%u vlen=%u id1[%u] id2[%u] shallow-equal but not identical for field#%d '%s'\n",
k1 == BTF_KIND_STRUCT ? "STRUCT" : "UNION",
btf__name_by_offset(d->btf, t1->name_off),
t1->size, btf_vlen(t1), id1, id2, i,
@@ -5104,7 +5104,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id,
eq = btf_dedup_is_equiv(d, cand_m->type, canon_m->type);
if (eq <= 0) {
if (cand_type->name_off) {
pr_debug("%s '%s' size=%d vlen=%d cand_id[%u] canon_id[%u] shallow-equal but not equiv for field#%d '%s': %d\n",
pr_debug("%s '%s' size=%u vlen=%u cand_id[%u] canon_id[%u] shallow-equal but not equiv for field#%d '%s': %d\n",
cand_kind == BTF_KIND_STRUCT ? "STRUCT" : "UNION",
btf__name_by_offset(d->btf, cand_type->name_off),
cand_type->size, vlen, cand_id, canon_id, i,
@@ -6069,7 +6069,7 @@ static int btf_add_distilled_types(struct btf_distill *dist)
err = btf_add_type(&dist->pipe, t);
break;
default:
pr_warn("unexpected kind when adding base type '%s'[%u] of kind [%u] to distilled base BTF.\n",
pr_warn("unexpected kind when adding base type '%s'[%d] of kind [%d] to distilled base BTF.\n",
name, i, kind);
return -EINVAL;

2
tools/lib/bpf/btf.h
View File

@@ -172,6 +172,8 @@ LIBBPF_API __s32 btf__find_by_name(const struct btf *btf,
const char *type_name);
LIBBPF_API __s32 btf__find_by_name_kind(const struct btf *btf,
const char *type_name, __u32 kind);
LIBBPF_API __s32 btf__find_by_name_kind_own(const struct btf *btf,
const char *type_name, __u32 kind);
LIBBPF_API __u32 btf__type_cnt(const struct btf *btf);
LIBBPF_API const struct btf *btf__base_btf(const struct btf *btf);
LIBBPF_API const struct btf_type *btf__type_by_id(const struct btf *btf,

4
tools/lib/bpf/btf_dump.c
View File

@@ -1776,7 +1776,7 @@ static int btf_dump_get_bitfield_value(struct btf_dump *d,
/* Maximum supported bitfield size is 64 bits */
if (t->size > 8) {
pr_warn("unexpected bitfield size %d\n", t->size);
pr_warn("unexpected bitfield size %u\n", t->size);
return -EINVAL;
}
@@ -2251,7 +2251,7 @@ static int btf_dump_get_enum_value(struct btf_dump *d,
*value = is_signed ? *(__s8 *)data : *(__u8 *)data;
return 0;
default:
pr_warn("unexpected size %d for enum, id:[%u]\n", t->size, id);
pr_warn("unexpected size %u for enum, id:[%u]\n", t->size, id);
return -EINVAL;
}
}

8
tools/lib/bpf/btf_relocate.c
View File

@@ -280,7 +280,7 @@ static int btf_relocate_map_distilled_base(struct btf_relocate *r)
cmp_btf_name_size(&base_info, dist_info) == 0;
dist_info++) {
if (!dist_info->id || dist_info->id >= r->nr_dist_base_types) {
pr_warn("base BTF id [%d] maps to invalid distilled base BTF id [%d]\n",
pr_warn("base BTF id [%u] maps to invalid distilled base BTF id [%u]\n",
id, dist_info->id);
err = -EINVAL;
goto done;
@@ -368,7 +368,7 @@ static int btf_relocate_map_distilled_base(struct btf_relocate *r)
continue;
dist_t = btf_type_by_id(r->dist_base_btf, id);
name = btf__name_by_offset(r->dist_base_btf, dist_t->name_off);
pr_warn("distilled base BTF type '%s' [%d] is not mapped to base BTF id\n",
pr_warn("distilled base BTF type '%s' [%u] is not mapped to base BTF id\n",
name, id);
err = -EINVAL;
break;
@@ -397,11 +397,11 @@ static int btf_relocate_validate_distilled_base(struct btf_relocate *r)
case BTF_KIND_FWD:
if (t->name_off)
break;
pr_warn("type [%d], kind [%d] is invalid for distilled base BTF; it is anonymous\n",
pr_warn("type [%u], kind [%d] is invalid for distilled base BTF; it is anonymous\n",
i, kind);
return -EINVAL;
default:
pr_warn("type [%d] in distilled based BTF has unexpected kind [%d]\n",
pr_warn("type [%u] in distilled based BTF has unexpected kind [%d]\n",
i, kind);
return -EINVAL;
}

2
tools/lib/bpf/elf.c
View File

@@ -354,7 +354,7 @@ long elf_find_func_offset(Elf *elf, const char *binary_path, const char *name)
if (ret > 0) {
pr_debug("elf: symbol address match for '%s' in '%s': 0x%lx\n", name, binary_path,
ret);
(unsigned long)ret);
} else {
if (ret == 0) {
pr_warn("elf: '%s' is 0 in symtab for '%s': %s\n", name, binary_path,

35
tools/lib/bpf/features.c
View File

@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
return probe_sys_bpf_ext();
}
static int probe_kern_percpu_data(int token_fd)
{
struct bpf_insn insns[] = {
BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
BPF_EXIT_INSN(),
};
LIBBPF_OPTS(bpf_map_create_opts, map_opts,
.token_fd = token_fd,
.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
);
LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
.token_fd = token_fd,
.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
);
int ret, map, insn_cnt = ARRAY_SIZE(insns);
map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
&map_opts);
if (map < 0) {
pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
__func__, errstr(map));
return map;
}
insns[0].imm = map;
ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
close(map);
return probe_fd(ret);
}
typedef int (*feature_probe_fn)(int /* token_fd */);
static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
},
[FEAT_PERCPU_DATA] = {
"kernel supports percpu data", probe_kern_percpu_data,
},
};
bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)

95
tools/lib/bpf/gen_loader.c
View File

@@ -111,7 +111,6 @@ static void emit2(struct bpf_gen *gen, struct bpf_insn insn1, struct bpf_insn in
static int add_data(struct bpf_gen *gen, const void *data, __u32 size);
static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off);
static void emit_signature_match(struct bpf_gen *gen);
void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps)
{
@@ -154,8 +153,6 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps
/* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */
emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_7));
emit(gen, BPF_EXIT_INSN());
if (OPTS_GET(gen->opts, gen_hash, false))
emit_signature_match(gen);
}
static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
@@ -377,14 +374,12 @@ static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off)
__emit_sys_close(gen);
}
static void compute_sha_update_offsets(struct bpf_gen *gen);
int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
{
int i;
if (nr_progs < gen->nr_progs || nr_maps != gen->nr_maps) {
pr_warn("nr_progs %d/%d nr_maps %d/%d mismatch\n",
pr_warn("nr_progs %d/%u nr_maps %d/%u mismatch\n",
nr_progs, gen->nr_progs, nr_maps, gen->nr_maps);
gen->error = -EFAULT;
return gen->error;
@@ -408,9 +403,6 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
if (!gen->error) {
struct gen_loader_opts *opts = gen->opts;
if (OPTS_GET(opts, gen_hash, false))
compute_sha_update_offsets(gen);
opts->insns = gen->insn_start;
opts->insns_sz = gen->insn_cur - gen->insn_start;
opts->data = gen->data_start;
@@ -460,22 +452,6 @@ void bpf_gen__free(struct bpf_gen *gen)
_val; \
})
static void compute_sha_update_offsets(struct bpf_gen *gen)
{
__u64 sha[SHA256_DWORD_SIZE];
__u64 sha_dw;
int i;
libbpf_sha256(gen->data_start, gen->data_cur - gen->data_start, (__u8 *)sha);
for (i = 0; i < SHA256_DWORD_SIZE; i++) {
struct bpf_insn *insn =
(struct bpf_insn *)(gen->insn_start + gen->hash_insn_offset[i]);
sha_dw = tgt_endian(sha[i]);
insn[0].imm = (__u32)sha_dw;
insn[1].imm = sha_dw >> 32;
}
}
void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data,
__u32 btf_raw_size)
{
@@ -488,7 +464,7 @@ void bpf_gen__load_btf(struct bpf_gen *gen, const void *btf_raw_data,
attr.btf_size = tgt_endian(btf_raw_size);
btf_load_attr = add_data(gen, &attr, attr_size);
pr_debug("gen: load_btf: off %d size %d, attr: off %d size %d\n",
pr_debug("gen: load_btf: off %d size %u, attr: off %d size %d\n",
btf_data, btf_raw_size, btf_load_attr, attr_size);
/* populate union bpf_attr with user provided log details */
@@ -534,7 +510,7 @@ void bpf_gen__map_create(struct bpf_gen *gen,
attr.btf_value_type_id = tgt_endian(map_attr->btf_value_type_id);
map_create_attr = add_data(gen, &attr, attr_size);
pr_debug("gen: map_create: %s idx %d type %d value_type_id %d, attr: off %d size %d\n",
pr_debug("gen: map_create: %s idx %d type %u value_type_id %u, attr: off %d size %d\n",
map_name, map_idx, map_type, map_attr->btf_value_type_id,
map_create_attr, attr_size);
@@ -557,8 +533,9 @@ void bpf_gen__map_create(struct bpf_gen *gen,
* Conditionally update max_entries from the host-supplied loader
* ctx. This sizes the map at runtime, but for a signed loader
* (gen_hash) it would let an untrusted host re-dimension the
* program's maps after emit_signature_match(), outside what the
* signature attests to. Keep the signer-provided max_entries
* program's maps, outside what the signature attests to: the
* metadata blob is covered by the program signature and verified
* by the kernel at load time. Keep the signer-provided max_entries
* baked into the blob in that case.
*/
if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false))
@@ -596,45 +573,6 @@ void bpf_gen__map_create(struct bpf_gen *gen,
emit_sys_close_stack(gen, stack_off(inner_map_fd));
}
static void emit_signature_match(struct bpf_gen *gen)
{
__s64 off;
int i;
/*
* Reject if the metadata map is not exclusive. Without exclusivity
* the cached map->sha[] verified above can be stale: another BPF
* program with map access could have mutated the contents between
* BPF_OBJ_GET_INFO_BY_FD and loader execution.
*/
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
0, 0, 0, 0));
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH));
off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2;
if (is_simm16(off)) {
emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL));
emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off));
} else {
gen->error = -ERANGE;
}
for (i = 0; i < SHA256_DWORD_SIZE; i++) {
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
0, 0, 0, 0));
emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, i * sizeof(__u64)));
gen->hash_insn_offset[i] = gen->insn_cur - gen->insn_start;
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_3, 0, 0, 0, 0, 0));
off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2;
if (is_simm16(off)) {
emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL));
emit(gen, BPF_JMP_REG(BPF_JNE, BPF_REG_2, BPF_REG_3, off));
} else {
gen->error = -ERANGE;
}
}
}
void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *attach_name,
enum bpf_attach_type type)
{
@@ -1082,7 +1020,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
license_off = add_data(gen, license, strlen(license) + 1);
/* add insns to blob of bytes */
insns_off = add_data(gen, insns, insn_cnt * sizeof(struct bpf_insn));
pr_debug("gen: prog_load: prog_idx %d type %d insn off %d insns_cnt %zd license off %d\n",
pr_debug("gen: prog_load: prog_idx %d type %u insn off %d insns_cnt %zu license off %d\n",
prog_idx, prog_type, insns_off, insn_cnt, license_off);
/* convert blob insns to target endianness */
@@ -1105,21 +1043,21 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
attr.func_info_rec_size = tgt_endian(load_attr->func_info_rec_size);
attr.func_info_cnt = tgt_endian(load_attr->func_info_cnt);
func_info = add_data(gen, load_attr->func_info, func_info_tot_sz);
pr_debug("gen: prog_load: func_info: off %d cnt %d rec size %d\n",
pr_debug("gen: prog_load: func_info: off %d cnt %u rec size %u\n",
func_info, load_attr->func_info_cnt,
load_attr->func_info_rec_size);
attr.line_info_rec_size = tgt_endian(load_attr->line_info_rec_size);
attr.line_info_cnt = tgt_endian(load_attr->line_info_cnt);
line_info = add_data(gen, load_attr->line_info, line_info_tot_sz);
pr_debug("gen: prog_load: line_info: off %d cnt %d rec size %d\n",
pr_debug("gen: prog_load: line_info: off %d cnt %u rec size %u\n",
line_info, load_attr->line_info_cnt,
load_attr->line_info_rec_size);
attr.core_relo_rec_size = tgt_endian((__u32)sizeof(struct bpf_core_relo));
attr.core_relo_cnt = tgt_endian(gen->core_relo_cnt);
core_relos = add_data(gen, gen->core_relos, core_relo_tot_sz);
pr_debug("gen: prog_load: core_relos: off %d cnt %d rec size %zd\n",
pr_debug("gen: prog_load: core_relos: off %d cnt %d rec size %zu\n",
core_relos, gen->core_relo_cnt,
sizeof(struct bpf_core_relo));
@@ -1190,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
}
void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
__u32 value_size)
__u32 value_size, __u64 flags)
{
int attr_size = offsetofend(union bpf_attr, flags);
int map_update_attr, value, key;
@@ -1198,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
int zero = 0;
memset(&attr, 0, attr_size);
attr.flags = tgt_endian(flags);
value = add_data(gen, pvalue, value_size);
key = add_data(gen, &zero, sizeof(zero));
@@ -1211,10 +1150,10 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
* }
*
* The runtime initial_value comes from the host-supplied loader
* ctx and would overwrite the blob value after emit_signature_match()
* has already validated map->sha[]. For a signed loader (gen_hash)
* the attested blob value must be authoritative, so skip the override
* and leave the hashed value in place.
* ctx and would overwrite the blob value that the program signature
* covers and the kernel verifies at load time. For a signed loader
* (gen_hash) the attested blob value must be authoritative, so skip
* the override and leave the signed value in place.
*/
if (!OPTS_GET(gen->opts, gen_hash, false)) {
emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
@@ -1234,7 +1173,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
}
map_update_attr = add_data(gen, &attr, attr_size);
pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n",
pr_debug("gen: map_update_elem: idx %d, value: off %d size %u, attr: off %d size %d\n",
map_idx, value, value_size, map_update_attr, attr_size);
move_blob2blob(gen, attr_field(map_update_attr, map_fd), 4,
blob_fd_array_off(gen, map_idx));

183
tools/lib/bpf/libbpf.c
View File

@@ -541,6 +541,7 @@ struct bpf_struct_ops {
};
#define DATA_SEC ".data"
#define PERCPU_SEC ".percpu"
#define BSS_SEC ".bss"
#define RODATA_SEC ".rodata"
#define KCONFIG_SEC ".kconfig"
@@ -555,6 +556,7 @@ enum libbpf_map_type {
LIBBPF_MAP_BSS,
LIBBPF_MAP_RODATA,
LIBBPF_MAP_KCONFIG,
LIBBPF_MAP_PERCPU,
};
struct bpf_map_def {
@@ -666,6 +668,7 @@ enum sec_type {
SEC_DATA,
SEC_RODATA,
SEC_ST_OPS,
SEC_PERCPU,
};
struct elf_sec_desc {
@@ -1486,7 +1489,7 @@ static int init_struct_ops_maps(struct bpf_object *obj, const char *sec_name,
type->size);
st_ops->type_id = type_id;
pr_debug("struct_ops init: struct %s(type_id=%u) %s found at offset %u\n",
pr_debug("struct_ops init: struct %s(type_id=%d) %s found at offset %u\n",
tname, type_id, var_name, vsi->offset);
}
@@ -1838,6 +1841,7 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
switch (map->def.type) {
case BPF_MAP_TYPE_ARRAY:
case BPF_MAP_TYPE_PERCPU_ARRAY:
return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
case BPF_MAP_TYPE_ARENA:
return page_sz * map->def.max_entries;
@@ -1866,7 +1870,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
return 0;
}
static char *internal_map_name(struct bpf_object *obj, const char *real_name)
static char *internal_map_name(struct bpf_object *obj, const char *real_name,
enum libbpf_map_type type)
{
char map_name[BPF_OBJ_NAME_LEN], *p;
int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1912,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
if (sfx_len >= BPF_OBJ_NAME_LEN)
sfx_len = BPF_OBJ_NAME_LEN - 1;
/* if there are two or more dots in map name, it's a custom dot map */
if (strchr(real_name + 1, '.') != NULL)
/*
* Don't prefix the bpf_object name if this is a custom dot map
* (containing two or more dots) or a percpu data map.
*/
if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)
pfx_len = 0;
else
pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1941,6 +1949,13 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
if (!map->btf_value_type_id)
return false;
/*
* The internal PERCPU maps are not mmapable because the underlying
* percpu_array maps do not have mmap support.
*/
if (map->libbpf_type == LIBBPF_MAP_PERCPU)
return false;
t = btf__type_by_id(obj->btf, map->btf_value_type_id);
if (!btf_is_datasec(t))
return false;
@@ -1962,6 +1977,7 @@ static int
bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
const char *real_name, int sec_idx, void *data, size_t data_sz)
{
bool is_percpu = type == LIBBPF_MAP_PERCPU;
struct bpf_map_def *def;
struct bpf_map *map;
size_t mmap_sz;
@@ -1975,7 +1991,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
map->sec_idx = sec_idx;
map->sec_offset = 0;
map->real_name = strdup(real_name);
map->name = internal_map_name(obj, real_name);
map->name = internal_map_name(obj, real_name, type);
if (!map->real_name || !map->name) {
zfree(&map->real_name);
zfree(&map->name);
@@ -1983,7 +1999,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
}
def = &map->def;
def->type = BPF_MAP_TYPE_ARRAY;
def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;
def->key_size = sizeof(int);
def->value_size = data_sz;
def->max_entries = 1;
@@ -1996,8 +2012,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
if (map_is_mmapable(obj, map))
def->map_flags |= BPF_F_MMAPABLE;
pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
map->name, map->sec_idx, map->sec_offset, def->map_flags);
pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
map->name, is_percpu ? "percpu " : "", map->sec_idx,
map->sec_offset, def->map_flags);
mmap_sz = bpf_map_mmap_sz(map);
map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
@@ -2057,6 +2074,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
NULL,
sec_desc->data->d_size);
break;
case SEC_PERCPU:
sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,
sec_name, sec_idx,
sec_desc->data->d_buf,
sec_desc->data->d_size);
break;
default:
/* skip */
break;
@@ -2626,7 +2650,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf,
t = btf__type_by_id(btf, m->type);
if (!t) {
pr_warn("map '%s': key type [%d] not found.\n",
pr_warn("map '%s': key type [%u] not found.\n",
map_name, m->type);
return -EINVAL;
}
@@ -2666,7 +2690,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf,
t = btf__type_by_id(btf, m->type);
if (!t) {
pr_warn("map '%s': value type [%d] not found.\n",
pr_warn("map '%s': value type [%u] not found.\n",
map_name, m->type);
return -EINVAL;
}
@@ -2720,7 +2744,7 @@ int parse_btf_map_def(const char *map_name, struct btf *btf,
map_def->value_size = 4;
t = btf__type_by_id(btf, m->type);
if (!t) {
pr_warn("map '%s': %s type [%d] not found.\n",
pr_warn("map '%s': %s type [%u] not found.\n",
map_name, desc, m->type);
return -EINVAL;
}
@@ -2806,7 +2830,7 @@ static size_t adjust_ringbuf_sz(size_t sz)
return 0;
/* Kernel expects BPF_MAP_TYPE_RINGBUF's max_entries to be
* a power-of-2 multiple of kernel's page size. If user diligently
* satisified these conditions, pass the size through.
* satisfied these conditions, pass the size through.
*/
if ((sz % page_sz) == 0 && is_pow_of_2(sz / page_sz))
return sz;
@@ -3476,7 +3500,7 @@ static int btf_fixup_datasec(struct bpf_object *obj, struct btf *btf,
var_name = btf__name_by_offset(btf, t_var->name_off);
if (!var_name) {
pr_debug("sec '%s': failed to find name of DATASEC's member #%d\n",
pr_debug("sec '%s': failed to find name of DATASEC's member #%u\n",
sec_name, i);
return -ENOENT;
}
@@ -3971,7 +3995,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
if (!data)
return -LIBBPF_ERRNO__FORMAT;
pr_debug("elf: section(%d) %s, size %ld, link %d, flags %lx, type=%d\n",
pr_debug("elf: section(%d) %s, size %lu, link %d, flags %lx, type=%d\n",
idx, name, (unsigned long)data->d_size,
(int)sh->sh_link, (unsigned long)sh->sh_flags,
(int)sh->sh_type);
@@ -4016,6 +4040,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
sec_desc->sec_type = SEC_RODATA;
sec_desc->shdr = sh;
sec_desc->data = data;
} else if (strcmp(name, PERCPU_SEC) == 0 ||
str_has_pfx(name, PERCPU_SEC ".")) {
sec_desc->sec_type = SEC_PERCPU;
sec_desc->shdr = sh;
sec_desc->data = data;
} else if (strcmp(name, STRUCT_OPS_SEC) == 0 ||
strcmp(name, STRUCT_OPS_LINK_SEC) == 0 ||
strcmp(name, "?" STRUCT_OPS_SEC) == 0 ||
@@ -4494,7 +4523,7 @@ static int bpf_object__collect_externs(struct bpf_object *obj)
ext->kcfg.data_off = roundup(off, ext->kcfg.align);
off = ext->kcfg.data_off + ext->kcfg.sz;
pr_debug("extern (kcfg) #%d: symbol %d, off %u, name %s\n",
pr_debug("extern (kcfg) #%d: symbol %d, off %d, name %s\n",
i, ext->sym_idx, ext->kcfg.data_off, ext->name);
}
sec->size = off;
@@ -4544,6 +4573,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
case SEC_BSS:
case SEC_DATA:
case SEC_RODATA:
case SEC_PERCPU:
return true;
default:
return false;
@@ -4569,6 +4599,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)
return LIBBPF_MAP_DATA;
case SEC_RODATA:
return LIBBPF_MAP_RODATA;
case SEC_PERCPU:
return LIBBPF_MAP_PERCPU;
default:
return LIBBPF_MAP_UNSPEC;
}
@@ -4626,7 +4658,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
struct bpf_map *map;
if (!is_call_insn(insn) && !is_ldimm64_insn(insn)) {
pr_warn("prog '%s': invalid relo against '%s' for insns[%d].code 0x%x\n",
pr_warn("prog '%s': invalid relo against '%s' for insns[%u].code 0x%x\n",
prog->name, sym_name, insn_idx, insn->code);
return -LIBBPF_ERRNO__RELOC;
}
@@ -4749,7 +4781,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
map->sec_idx != sym->st_shndx ||
map->sec_offset != sym->st_value)
continue;
pr_debug("prog '%s': found map %zd (%s, sec %d, off %zu) for insn #%u\n",
pr_debug("prog '%s': found map %zu (%s, sec %d, off %zu) for insn #%u\n",
prog->name, map_idx, map->name, map->sec_idx,
map->sec_offset, insn_idx);
break;
@@ -4776,7 +4808,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
map = &obj->maps[map_idx];
if (map->libbpf_type != type || map->sec_idx != sym->st_shndx)
continue;
pr_debug("prog '%s': found data map %zd (%s, sec %d, off %zu) for insn %u\n",
pr_debug("prog '%s': found data map %zu (%s, sec %d, off %zu) for insn %u\n",
prog->name, map_idx, map->name, map->sec_idx,
map->sec_offset, insn_idx);
break;
@@ -4944,7 +4976,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
/*
* LLVM annotates global data differently in BTF, that is,
* only as '.data', '.bss' or '.rodata'.
* only as '.data', '.bss', '.percpu' or '.rodata'.
*/
if (!bpf_map__is_internal(map))
return -ENOENT;
@@ -4985,7 +5017,7 @@ static int bpf_get_map_info_from_fdinfo(int fd, struct bpf_map_info *info)
info->value_size = val;
else if (sscanf(buff, "max_entries:\t%u", &val) == 1)
info->max_entries = val;
else if (sscanf(buff, "map_flags:\t%i", &val) == 1)
else if (sscanf(buff, "map_flags:\t%x", &val) == 1)
info->map_flags = val;
}
@@ -5172,12 +5204,8 @@ bpf_object__probe_loading(struct bpf_object *obj)
BPF_EXIT_INSN(),
};
int ret, insn_cnt = ARRAY_SIZE(insns);
LIBBPF_OPTS(bpf_prog_load_opts, opts,
.token_fd = obj->token_fd,
.prog_flags = obj->token_fd ? BPF_F_TOKEN_FD : 0,
);
if (obj->gen_loader)
if (obj->gen_loader || obj->token_fd)
return 0;
ret = bump_rlimit_memlock();
@@ -5186,9 +5214,9 @@ bpf_object__probe_loading(struct bpf_object *obj)
errstr(ret));
/* make sure basic loading works */
ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &opts);
ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, NULL);
if (ret < 0)
ret = bpf_prog_load(BPF_PROG_TYPE_TRACEPOINT, NULL, "GPL", insns, insn_cnt, &opts);
ret = bpf_prog_load(BPF_PROG_TYPE_TRACEPOINT, NULL, "GPL", insns, insn_cnt, NULL);
if (ret < 0) {
ret = errno;
pr_warn("Error in %s(): %s. Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.\n",
@@ -5297,18 +5325,20 @@ static int
bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
{
enum libbpf_map_type map_type = map->libbpf_type;
bool is_percpu = map_type == LIBBPF_MAP_PERCPU;
const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;
int err, zero = 0;
size_t mmap_sz;
if (obj->gen_loader) {
bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
map->mmaped, map->def.value_size);
map->mmaped, map->def.value_size, update_flags);
if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
return 0;
}
err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
if (err) {
err = -errno;
pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5353,6 +5383,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
return err;
}
map->mmaped = mmaped;
} else if (is_percpu) {
if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
err = -errno;
pr_warn("map '%s': failed to mprotect() contents: %s\n",
bpf_map__name(map), errstr(err));
return err;
}
} else if (map->mmaped) {
munmap(map->mmaped, mmap_sz);
map->mmaped = NULL;
@@ -5521,11 +5558,11 @@ static int init_map_in_map_slots(struct bpf_object *obj, struct bpf_map *map)
}
if (err) {
err = -errno;
pr_warn("map '%s': failed to initialize slot [%d] to map '%s' fd=%d: %s\n",
pr_warn("map '%s': failed to initialize slot [%u] to map '%s' fd=%d: %s\n",
map->name, i, targ_map->name, fd, errstr(err));
return err;
}
pr_debug("map '%s': slot [%d] set to map '%s' fd=%d\n",
pr_debug("map '%s': slot [%u] set to map '%s' fd=%d\n",
map->name, i, targ_map->name, fd);
}
@@ -5554,11 +5591,11 @@ static int init_prog_array_slots(struct bpf_object *obj, struct bpf_map *map)
err = bpf_map_update_elem(map->fd, &i, &fd, 0);
if (err) {
err = -errno;
pr_warn("map '%s': failed to initialize slot [%d] to prog '%s' fd=%d: %s\n",
pr_warn("map '%s': failed to initialize slot [%u] to prog '%s' fd=%d: %s\n",
map->name, i, targ_prog->name, fd, errstr(err));
return err;
}
pr_debug("map '%s': slot [%d] set to prog '%s' fd=%d\n",
pr_debug("map '%s': slot [%u] set to prog '%s' fd=%d\n",
map->name, i, targ_prog->name, fd);
}
@@ -5628,9 +5665,16 @@ bpf_object__create_maps(struct bpf_object *obj)
* runtime due to bpf_program__set_autoload(prog, false),
* bpf_object loading will succeed just fine even on old
* kernels.
* Same skipping applies to percpu data.
*/
if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA))
map->autocreate = false;
if (bpf_map__is_internal(map)) {
bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU;
enum kern_feature_id feat_id;
feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA;
if (!kernel_supports(obj, feat_id))
map->autocreate = false;
}
if (!map->autocreate) {
pr_debug("map '%s': skipped auto-creating...\n", map->name);
@@ -5788,7 +5832,7 @@ int bpf_core_add_cands(struct bpf_core_cand *local_cand,
if (strncmp(local_name, targ_name, local_essent_len) != 0)
continue;
pr_debug("CO-RE relocating [%d] %s %s: found target candidate [%d] %s %s in [%s]\n",
pr_debug("CO-RE relocating [%u] %s %s: found target candidate [%d] %s %s in [%s]\n",
local_cand->id, btf_kind_str(local_t),
local_name, i, btf_kind_str(t), targ_name,
targ_btf_name);
@@ -5848,7 +5892,7 @@ static int load_module_btfs(struct bpf_object *obj)
if (errno == ENOENT)
continue; /* expected race: BTF was unloaded */
err = -errno;
pr_warn("failed to get BTF object #%d FD: %s\n", id, errstr(err));
pr_warn("failed to get BTF object #%u FD: %s\n", id, errstr(err));
return err;
}
@@ -5861,7 +5905,7 @@ static int load_module_btfs(struct bpf_object *obj)
err = bpf_btf_get_info_by_fd(fd, &info, &len);
if (err) {
err = -errno;
pr_warn("failed to get BTF object #%d info: %s\n", id, errstr(err));
pr_warn("failed to get BTF object #%u info: %s\n", id, errstr(err));
break;
}
@@ -5874,7 +5918,7 @@ static int load_module_btfs(struct bpf_object *obj)
btf = btf_get_from_fd(fd, obj->btf_vmlinux);
err = libbpf_get_error(btf);
if (err) {
pr_warn("failed to load module [%s]'s BTF object #%d: %s\n",
pr_warn("failed to load module [%s]'s BTF object #%u: %s\n",
name, id, errstr(err));
break;
}
@@ -6067,7 +6111,7 @@ static int bpf_core_resolve_relo(struct bpf_program *prog,
!hashmap__find(cand_cache, local_id, &cands)) {
cands = bpf_core_find_cands(prog->obj, local_btf, local_id);
if (IS_ERR(cands)) {
pr_warn("prog '%s': relo #%d: target candidate search failed for [%d] %s %s: %ld\n",
pr_warn("prog '%s': relo #%d: target candidate search failed for [%u] %s %s: %ld\n",
prog_name, relo_idx, local_id, btf_kind_str(local_type),
local_name, PTR_ERR(cands));
return PTR_ERR(cands);
@@ -6127,7 +6171,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path)
goto out;
}
pr_debug("sec '%s': found %d CO-RE relocations\n", sec_name, sec->num_info);
pr_debug("sec '%s': found %u CO-RE relocations\n", sec_name, sec->num_info);
for_each_btf_ext_rec(seg, sec, i, rec) {
if (rec->insn_off % BPF_INSN_SZ)
@@ -6181,7 +6225,7 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path)
err = bpf_core_patch_insn(prog->name, insn, insn_idx, rec, i, &targ_res);
if (err) {
pr_warn("prog '%s': relo #%d: failed to patch insn #%u: %s\n",
pr_warn("prog '%s': relo #%d: failed to patch insn #%d: %s\n",
prog->name, i, insn_idx, errstr(err));
goto out;
}
@@ -6346,7 +6390,7 @@ static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struc
goto err_close;
}
if (sym_off + jt_size > obj->jumptables_data_sz) {
pr_warn("map '.jumptables': jumptables_data size is %zd, trying to access %d\n",
pr_warn("map '.jumptables': jumptables_data size is %zu, trying to access %u\n",
obj->jumptables_data_sz, sym_off + jt_size);
err = -EINVAL;
goto err_close;
@@ -6381,7 +6425,7 @@ static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struc
*/
if (insn_off > UINT32_MAX) {
pr_warn("map '.jumptables': invalid jump table value 0x%llx at offset %u\n",
(long long)jt[i], sym_off + i * jt_entry_size);
(unsigned long long)jt[i], sym_off + i * jt_entry_size);
err = -EINVAL;
goto err_close;
}
@@ -6517,7 +6561,7 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
}
break;
default:
pr_warn("prog '%s': relo #%d: bad relo type %d\n",
pr_warn("prog '%s': relo #%d: bad relo type %u\n",
prog->name, i, relo->type);
return -EINVAL;
}
@@ -6797,7 +6841,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
*/
continue;
if (relo && relo->type != RELO_CALL && relo->type != RELO_SUBPROG_ADDR) {
pr_warn("prog '%s': unexpected relo for insn #%zu, type %d\n",
pr_warn("prog '%s': unexpected relo for insn #%zu, type %u\n",
prog->name, insn_idx, relo->type);
return -LIBBPF_ERRNO__RELOC;
}
@@ -6944,7 +6988,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog,
* +-----------+------+------+
*
* At this point, we relocate subA calls, then go one level up and finish with
* relocatin mainA calls. mainA is done.
* relocation mainA calls. mainA is done.
*
* For mainB process is similar but results in different order. We start with
* mainB and skip subA and subB, as mainB never calls them (at least
@@ -7587,7 +7631,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj,
}
name = elf_sym_str(obj, sym->st_name) ?: "<?>";
pr_debug(".maps relo #%d: for %zd value %zd rel->r_offset %zu name %d ('%s')\n",
pr_debug(".maps relo #%d: for %zd value %zu rel->r_offset %zu name %u ('%s')\n",
i, (ssize_t)(rel->r_info >> 32), (size_t)sym->st_value,
(size_t)rel->r_offset, sym->st_name, name);
@@ -7678,7 +7722,7 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj,
}
map->init_slots[moff] = is_map_in_map ? (void *)targ_map : (void *)targ_prog;
pr_debug(".maps relo #%d: map '%s' slot [%d] points to %s '%s'\n",
pr_debug(".maps relo #%d: map '%s' slot [%u] points to %s '%s'\n",
i, map->name, moff, type, name);
}
@@ -7891,7 +7935,7 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
prog->attach_btf_id = btf_type_id;
/* but by now libbpf common logic is not utilizing
* prog->atach_btf_obj_fd/prog->attach_btf_id anymore because
* prog->attach_btf_obj_fd/prog->attach_btf_id anymore because
* this callback is called after opts were populated by
* libbpf, so this callback has to update opts explicitly here
*/
@@ -8738,7 +8782,7 @@ static int bpf_object__resolve_ksym_var_btf_id(struct bpf_object *obj,
local_name = btf__name_by_offset(obj->btf, local_type->name_off);
targ_name = btf__name_by_offset(btf, targ_type->name_off);
pr_warn("extern (var ksym) '%s': incompatible types, expected [%d] %s %s, but kernel has [%d] %s %s\n",
pr_warn("extern (var ksym) '%s': incompatible types, expected [%u] %s %s, but kernel has [%u] %s %s\n",
ext->name, local_type_id,
btf_kind_str(local_type), local_name, targ_type_id,
btf_kind_str(targ_type), targ_name);
@@ -8915,7 +8959,7 @@ static int bpf_object__resolve_externs(struct bpf_object *obj,
if (err)
return err;
pr_debug("extern (kcfg) '%s': set to 0x%llx\n",
ext->name, (long long)value);
ext->name, (unsigned long long)value);
} else {
pr_warn("extern '%s': unrecognized extern kind\n", ext->name);
return -EINVAL;
@@ -10494,7 +10538,7 @@ static int bpf_object__collect_st_ops_relos(struct bpf_object *obj,
moff = rel->r_offset - map->sec_offset;
shdr_idx = sym->st_shndx;
st_ops = map->st_ops;
pr_debug("struct_ops reloc %s: for %lld value %lld shdr_idx %u rel->r_offset %zu map->sec_offset %zu name %d (\'%s\')\n",
pr_debug("struct_ops reloc %s: for %lld value %lld shdr_idx %u rel->r_offset %zu map->sec_offset %zu name %u (\'%s\')\n",
map->name,
(long long)(rel->r_info >> 32),
(long long)sym->st_value,
@@ -10643,7 +10687,7 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd, int t
memset(&info, 0, info_len);
err = bpf_prog_get_info_by_fd(attach_prog_fd, &info, &info_len);
if (err) {
pr_warn("failed bpf_prog_get_info_by_fd for FD %d: %s\n",
pr_warn("failed bpf_prog_get_info_by_fd for FD %u: %s\n",
attach_prog_fd, errstr(err));
return err;
}
@@ -10656,7 +10700,7 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd, int t
btf = btf_load_from_kernel(info.btf_id, NULL, token_fd);
err = libbpf_get_error(btf);
if (err) {
pr_warn("Failed to get BTF %d of the program: %s\n", info.btf_id, errstr(err));
pr_warn("Failed to get BTF %u of the program: %s\n", info.btf_id, errstr(err));
goto out;
}
err = btf__find_by_name_kind(btf, name, BTF_KIND_FUNC);
@@ -10738,7 +10782,7 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attac
}
err = libbpf_find_prog_btf_id(attach_name, attach_prog_fd, prog->obj->token_fd);
if (err < 0) {
pr_warn("prog '%s': failed to find BPF program (FD %d) BTF ID for '%s': %s\n",
pr_warn("prog '%s': failed to find BPF program (FD %u) BTF ID for '%s': %s\n",
prog->name, attach_prog_fd, attach_name, errstr(err));
return err;
}
@@ -10811,11 +10855,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
* such map's corresponding ELF section name as a map name.
* This check distinguishes .data/.rodata from .data.* and .rodata.*
* maps to know which name has to be returned to the user.
* Map name of the custom .percpu.* maps might be truncated to
* BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
* maps must use real name for their user-visible name.
*/
if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
return true;
if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
return true;
if (map->libbpf_type == LIBBPF_MAP_PERCPU)
return true;
return false;
}
@@ -10980,7 +11029,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size)
size_t mmap_old_sz, mmap_new_sz;
int err;
if (map->def.type != BPF_MAP_TYPE_ARRAY)
if (map->def.type != BPF_MAP_TYPE_ARRAY &&
map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY)
return libbpf_err(-EOPNOTSUPP);
mmap_old_sz = bpf_map_mmap_sz(map);
@@ -11233,7 +11283,7 @@ static int validate_map_op(const struct bpf_map *map, size_t key_sz,
}
if (value_sz != num_cpu * elem_sz) {
pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zd\n",
pr_warn("map '%s': unexpected value size %zu provided for per-CPU map, expected %d * %zu = %zu\n",
map->name, value_sz, num_cpu, elem_sz, num_cpu * elem_sz);
return -EINVAL;
}
@@ -11774,7 +11824,7 @@ static void gen_probe_legacy_event_name(char *buf, size_t buf_sz,
static int index = 0;
int i;
snprintf(buf, buf_sz, "libbpf_%u_%d_%s_0x%zx", getpid(),
snprintf(buf, buf_sz, "libbpf_%d_%d_%s_0x%zx", getpid(),
__sync_fetch_and_add(&index, 1), name, offset);
/* sanitize name in the probe name */
@@ -12924,8 +12974,8 @@ static long elf_find_func_offset_from_archive(const char *archive_path, const ch
ret = elf_find_func_offset(elf, file_name, func_name);
if (ret > 0) {
pr_debug("elf: symbol address match for %s of %s in %s: 0x%x + 0x%lx = 0x%lx\n",
func_name, file_name, archive_path, entry.data_offset, ret,
ret + entry.data_offset);
func_name, file_name, archive_path, entry.data_offset, (unsigned long)ret,
(unsigned long)(ret + entry.data_offset));
ret += entry.data_offset;
}
elf_end(elf);
@@ -12973,13 +13023,14 @@ static const char *arch_specific_lib_paths(void)
/* Get full path to program/shared library. */
static int resolve_full_path(const char *file, char *result, size_t result_sz)
{
const char *search_paths[3] = {};
const char *search_paths[4] = {};
int i, perm;
if (str_has_sfx(file, ".so") || strstr(file, ".so.")) {
search_paths[0] = getenv("LD_LIBRARY_PATH");
search_paths[1] = "/usr/lib64:/usr/lib";
search_paths[2] = arch_specific_lib_paths();
search_paths[3] = "/lib64:/lib";
perm = R_OK;
} else {
search_paths[0] = getenv("PATH");
@@ -14167,7 +14218,7 @@ perf_event_read_simple(void *mmap_mem, size_t mmap_size, size_t page_size,
if (((void *)ehdr) + ehdr_size > base + mmap_size) {
void *copy_start = ehdr;
size_t len_first = base + mmap_size - copy_start;
size_t len_secnd = ehdr_size - len_first;
size_t len_second = ehdr_size - len_first;
if (*copy_size < ehdr_size) {
free(*copy_mem);
@@ -14181,7 +14232,7 @@ perf_event_read_simple(void *mmap_mem, size_t mmap_size, size_t page_size,
}
memcpy(*copy_mem, copy_start, len_first);
memcpy(*copy_mem + len_first, base, len_secnd);
memcpy(*copy_mem + len_first, base, len_second);
ehdr = *copy_mem;
}
@@ -14199,7 +14250,7 @@ struct perf_buffer;
struct perf_buffer_params {
struct perf_event_attr *attr;
/* if event_cb is specified, it takes precendence */
/* if event_cb is specified, it takes precedence */
perf_buffer_event_fn event_cb;
/* sample_cb and lost_cb are higher-level common-case callbacks */
perf_buffer_sample_fn sample_cb;
@@ -14570,7 +14621,7 @@ perf_buffer__process_record(struct perf_event_header *e, void *ctx)
break;
}
default:
pr_warn("unknown perf sample type %d\n", e->type);
pr_warn("unknown perf sample type %u\n", e->type);
return LIBBPF_PERF_EVENT_ERROR;
}
return LIBBPF_PERF_EVENT_CONT;

1
tools/lib/bpf/libbpf.map
View File

@@ -460,5 +460,6 @@ LIBBPF_1.8.0 {
global:
bpf_program__attach_tracing_multi;
bpf_program__clone;
btf__find_by_name_kind_own;
btf__new_empty_opts;
} LIBBPF_1.7.0;

5
tools/lib/bpf/libbpf_internal.h
View File

@@ -401,6 +401,8 @@ enum kern_feature_id {
FEAT_BTF_LAYOUT,
/* Kernel supports BPF syscall common attributes */
FEAT_BPF_SYSCALL_COMMON_ATTRS,
/* Kernel supports percpu data */
FEAT_PERCPU_DATA,
__FEAT_CNT,
};
@@ -596,8 +598,6 @@ typedef int (*type_id_visit_fn)(__u32 *type_id, void *ctx);
typedef int (*str_off_visit_fn)(__u32 *str_off, void *ctx);
int btf_ext_visit_type_ids(struct btf_ext *btf_ext, type_id_visit_fn visit, void *ctx);
int btf_ext_visit_str_offs(struct btf_ext *btf_ext, str_off_visit_fn visit, void *ctx);
__s32 btf__find_by_name_kind_own(const struct btf *btf, const char *type_name,
__u32 kind);
/* handle direct returned errors */
static inline int libbpf_err(int ret)
@@ -768,7 +768,6 @@ int elf_resolve_pattern_offsets(const char *binary_path, const char *pattern,
int probe_fd(int fd);
#define SHA256_DIGEST_LENGTH 32
#define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64)
void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]);
int probe_sys_bpf_ext(void);

2
tools/lib/bpf/nlattr.c
View File

@@ -123,7 +123,7 @@ int libbpf_nla_parse(struct nlattr *tb[], int maxtype, struct nlattr *head,
if (tb[type]) {
pr_warn("Attribute of type %#x found multiple times in message, "
"previous attribute is being ignored.\n", type);
"previous attribute is being ignored.\n", (unsigned)type);
}
tb[type] = nla;

22
tools/lib/bpf/relo_core.c
View File

@@ -216,7 +216,7 @@ int __bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id,
goto recur;
}
default:
pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n",
pr_warn("unexpected kind %s relocated, local [%u], target [%u]\n",
btf_kind_str(local_type), local_id, targ_id);
return 0;
}
@@ -384,7 +384,7 @@ int bpf_core_parse_spec(const char *prog_name, const struct btf *btf,
return sz;
spec->bit_offset += access_idx * sz * 8;
} else {
pr_warn("prog '%s': relo for [%u] %s (at idx %d) captures type [%d] of unexpected kind %s\n",
pr_warn("prog '%s': relo for [%u] %s (at idx %d) captures type [%u] of unexpected kind %s\n",
prog_name, relo->type_id, spec_str, i, id, btf_kind_str(t));
return -EINVAL;
}
@@ -725,7 +725,7 @@ static int bpf_core_calc_field_relo(const char *prog_name,
return -EINVAL;
*val = sz;
} else {
pr_warn("prog '%s': relo %d at insn #%d can't be applied to array access\n",
pr_warn("prog '%s': relo %u at insn #%u can't be applied to array access\n",
prog_name, relo->kind, relo->insn_off / 8);
return -EINVAL;
}
@@ -747,7 +747,7 @@ static int bpf_core_calc_field_relo(const char *prog_name,
while (bit_off + bit_sz - byte_off * 8 > byte_sz * 8) {
if (byte_sz >= 8) {
/* bitfield can't be read with 64-bit read */
pr_warn("prog '%s': relo %d at insn #%d can't be satisfied for bitfield\n",
pr_warn("prog '%s': relo %u at insn #%u can't be satisfied for bitfield\n",
prog_name, relo->kind, relo->insn_off / 8);
return -E2BIG;
}
@@ -971,7 +971,7 @@ static int bpf_core_calc_relo(const char *prog_name,
err = 0;
} else if (err == -EOPNOTSUPP) {
/* EOPNOTSUPP means unknown/unsupported relocation */
pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%d) at insn #%d\n",
pr_warn("prog '%s': relo #%d: unrecognized CO-RE relocation %s (%u) at insn #%u\n",
prog_name, relo_idx, core_relo_kind_str(relo->kind),
relo->kind, relo->insn_off / 8);
}
@@ -1067,7 +1067,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
if (BPF_SRC(insn->code) != BPF_K)
return -EINVAL;
if (res->validate && insn->imm != orig_val) {
pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %llu -> %llu\n",
pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %d, exp %llu -> %llu\n",
prog_name, relo_idx,
insn_idx, insn->imm, (unsigned long long)orig_val,
(unsigned long long)new_val);
@@ -1083,7 +1083,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
case BPF_ST:
case BPF_STX:
if (res->validate && insn->off != orig_val) {
pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %llu -> %llu\n",
pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %d, exp %llu -> %llu\n",
prog_name, relo_idx, insn_idx, insn->off, (unsigned long long)orig_val,
(unsigned long long)new_val);
return -EINVAL;
@@ -1159,7 +1159,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
default:
pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n",
prog_name, relo_idx, insn_idx, insn->code,
insn->src_reg, insn->dst_reg, insn->off, insn->imm);
(unsigned)insn->src_reg, (unsigned)insn->dst_reg, (unsigned)insn->off, (unsigned)insn->imm);
return -EINVAL;
}
@@ -1323,7 +1323,7 @@ int bpf_core_calc_relo_insn(const char *prog_name,
const char *spec_str;
spec_str = btf__name_by_offset(local_btf, relo->access_str_off);
pr_warn("prog '%s': relo #%d: parsing [%d] %s %s + %s failed: %d\n",
pr_warn("prog '%s': relo #%d: parsing [%u] %s %s + %s failed: %d\n",
prog_name, relo_idx, local_id, btf_kind_str(local_type),
str_is_empty(local_name) ? "<anon>" : local_name,
spec_str ?: "<?>", err);
@@ -1346,7 +1346,7 @@ int bpf_core_calc_relo_insn(const char *prog_name,
/* libbpf doesn't support candidate search for anonymous types */
if (str_is_empty(local_name)) {
pr_warn("prog '%s': relo #%d: <%s> (%d) relocation doesn't support anonymous types\n",
pr_warn("prog '%s': relo #%d: <%s> (%u) relocation doesn't support anonymous types\n",
prog_name, relo_idx, core_relo_kind_str(relo->kind), relo->kind);
return -EOPNOTSUPP;
}
@@ -1697,7 +1697,7 @@ int __bpf_core_types_match(const struct btf *local_btf, __u32 local_id, const st
goto recur;
}
default:
pr_warn("unexpected kind %s relocated, local [%d], target [%d]\n",
pr_warn("unexpected kind %s relocated, local [%u], target [%u]\n",
btf_kind_str(local_t), local_id, targ_id);
return 0;
}

2
tools/lib/bpf/ringbuf.c
View File

@@ -244,7 +244,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
do {
got_new_data = false;
prod_pos = smp_load_acquire(r->producer_pos);
while (cons_pos < prod_pos) {
while (prod_pos - cons_pos > 0) {
len_ptr = r->data + (cons_pos & r->mask);
len = smp_load_acquire(len_ptr);

Some files were not shown because too many files have changed in this diff Show More