mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-07-16 17:57:38 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Browse Source 
Pull bpf updates from Alexei Starovoitov:
 "Major changes:

   - Recover from BPF arena page faults using a scratch page and add
     ptep_try_set() for lockless empty-slot installs on x86 and arm64.

     This allows BPF kfuncs to access arena pointers directly.

     The 'arena_direct_access' stable branch was created for this work
     and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar
     Kartikeya Dwivedi)

   - Lift old restriction and support 6+ arguments in BPF programs and
     kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan)

  Other features and fixes:

   - Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease
     addition of new BTF kinds (Alan Maguire)

   - Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei
     Starovoitov)

   - Refactor object relationship tracking in the verifier and fix a
     dynptr use-after-free bug (Amery Hung)

   - Harden the signed program loader and reject exclusive maps as inner
     maps (Daniel Borkmann)

   - Replace the verifier min/max bounds fields with a circular number
     (cnum) representation and improve 32->64 bit range refinements
     (Eduard Zingerman)

   - Introduce the arena library and runtime (libarena) with a buddy
     allocator, rbtree and SPMC queue data structures, ASAN support and
     a parallel test harness. Allow subprograms to return arena pointers
     and switch to a BTF type-tag based __arena annotation (Emil
     Tsalapatis)

   - Cache build IDs in the sleepable stackmap path and avoid faultable
     build ID reads under mm locks (Ihor Solodrai)

   - Introduce the tracing_multi link to attach a single BPF program to
     many kernel functions at once. Allow specifying the uprobe_multi
     target via FD (Jiri Olsa)

   - Extend the bpf_list family of kfuncs with bpf_list_add/del(), and
     bpf_list_is_first/is_last/empty() (Kaitao Cheng)

   - Extend the BPF syscall with common attributes support for
     prog_load, btf_load and map_create (Leon Hwang)

   - Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu)

   - Add sleepable support for tracepoint programs and fix deadlocks in
     LRU map due to NMI reentry (Mykyta Yatsenko)

   - Fix OOB access in bpf_flow_keys, fix nullness analysis of inner
     arrays, enforce write checks for global subprograms (Nuoqi Gui)

   - Report the maximum combined stack depth and print a breakdown of
     instructions processed per subprogram (Paul Chaignon)

   - Add an XDP load-balancer benchmark and arm64 JIT support for stack
     arguments (Puranjay Mohan)

   - Add kfuncs to traverse over wakeup_sources (Samuel Wu)

   - Allow sleepable BPF programs to use LPM trie maps directly (Vlad
     Poenaru)

   - Many more fixes and cleanups across the verifier, BTF, sockmap,
     devmap, bpffs, security hooks, s390/riscv/loongarch JITs,
     rqspinlock, libbpf, bpftool, selftests"

* tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits)
  selftests/bpf: Work around llvm stack overflow in crypto progs
  selftests/bpf: add test for bpf_msg_pop_data() overflow
  bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check
  sockmap: Fix use-after-free in udp_bpf_recvmsg()
  bpf, sockmap: keep sk_msg copy state in sync
  bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data()
  bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data()
  selftsets/bpf: Retry map update on helper_fill_hashmap()
  selftests/bpf: Add test for sleepable lsm_cgroup rejection
  selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper
  bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket
  selftests/bpf: Avoid static LLVM linking for cross builds
  selftests/bpf: Use common CFLAGS for urandom_read
  selftests/bpf: Initialize operation name before use
  tools/bpf: build: Append extra cflags
  libbpf: Initialize CFLAGS before including Makefile.include
  bpftool: Append extra host flags
  bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS
  bpftool: Pass host flags to bootstrap libbpf
  selftests/bpf: correct CONFIG_PPC64 macro name in comment
  ...
This commit is contained in:
Linus Torvalds
2026-06-17 09:18:14 +01:00
parent b85966adbf e4287bf34f
commit 9c87e61e3c
349 changed files with 27853 additions and 4974 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Documentation/bpf/btf.rst
View File

@@ -97,10 +97,8 @@ Each type contains the following common data::
struct btf_type {
__u32 name_off;
/* "info" bits arrangement
* bits 0-15: vlen (e.g. # of struct's members)
* bits 16-23: unused
* bits 24-28: kind (e.g. int, ptr, array...etc)
* bits 29-30: unused
* bits 0-23: vlen (e.g. # of struct's members)
* bits 24-30: kind (e.g. int, ptr, array...etc)
* bit 31: kind_flag, currently used by
* struct, union, enum, fwd, enum64,
* decl_tag and type_tag

36
Documentation/bpf/kfuncs.rst
View File

@@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least
buffer__szk bytes in size. The kfunc is responsible for checking if the buffer
is NULL before using it.
2.3.5 __str Annotation
----------------------------
2.3.5 __nonown_allowed Annotation
---------------------------------
This annotation is used to indicate that the parameter may be a non-owning reference.
An example is given below::
__bpf_kfunc int bpf_list_add(..., struct bpf_list_node
*prev__nonown_allowed, ...)
{
...
}
For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``),
suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also
permits a non-owning reference with no ref_obj_id (e.g. the return value of
bpf_list_front() / bpf_list_back()).
2.3.6 __str Annotation
----------------------
This annotation is used to indicate that the argument is a constant string.
An example is given below::
@@ -462,6 +480,20 @@ In order to accommodate such requirements, the verifier will enforce strict
PTR_TO_BTF_ID type matching if two types have the exact same name, with one
being suffixed with ``___init``.
2.8 Accessing arena memory through kfunc arguments
--------------------------------------------------
A read or write at any address inside an arena does not oops the kernel.
Unallocated arena pages are lazily backed by a scratch page and the
access is reported through the program's BPF stream as an error. Only
the BPF program's correctness is affected; the kernel itself remains
intact.
The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that
is also covered by this recovery. A kfunc handed an arena pointer may
therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
against the arena. Larger accesses must verify the range explicitly.
.. _BPF_kfunc_lifecycle_expectations:
3. kfunc lifecycle expectations

44
Documentation/bpf/map_lru_hash_update.dot
View File

@@ -21,10 +21,18 @@ digraph {
// names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c.
// Number suffixes and errno suffixes handle subsections of the corresponding
// logic in the function as of the writing of this dot.
//
// All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock
// or contention timeout); on failure the corresponding helper returns NULL
// and the caller propagates -ENOMEM. The "rqspinlock acquire failed"
// terminal below is reached via the dashed arrows from each acquire site.
rqspinlock_failed [shape=rectangle,
label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"]
// cf. __local_list_pop_free() / bpf_percpu_lru_pop_free()
local_freelist_check [shape=diamond,fillcolor=1,
label="Local freelist\nnode available?"];
label="Local freelist\nnode available?\n(lockless free_llist)"];
use_local_node [shape=rectangle,
label="Use node owned\nby this CPU"]
@@ -82,6 +90,15 @@ digraph {
// fn__local_list_pop_pending()
}
// Post-steal: re-acquire local loc_l->lock to insert the stolen node into
// the local pending list. If the acquire fails, the stolen node is published
// to the lockless local free_llist so the next pop on this CPU picks it up
// instead of orphaning it.
post_steal_lock [shape=diamond,fillcolor=1,
label="Acquire local\nloc_l->lock\nto add pending"]
post_steal_to_free_llist [shape=rectangle,
label="Publish stolen node to\nlocal free_llist (lockless)"]
fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle,
label="Use node that was\nnot recently referenced"]
local_freelist_check4 [shape=rectangle,
@@ -97,10 +114,19 @@ digraph {
fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"]
begin -> local_freelist_check
// The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is
// acquired before the local freelist check; rqspinlock failure here exits
// directly to -ENOMEM (no recovery needed: nothing was removed yet).
local_freelist_check -> rqspinlock_failed [style=dashed,
xlabel="acquire fails"]
local_freelist_check -> use_local_node [xlabel="Y"]
local_freelist_check -> common_lru_check [xlabel="N"]
common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"]
common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"]
// Global lru_list lock acquire failure in pop_free_to_local: skip refill,
// fall through to the steal path. Not ENOMEM by itself.
fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed,
xlabel="global lru_lock\nacquire fails"]
fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free
fn___bpf_lru_node_move_to_free ->
fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"]
@@ -120,13 +146,27 @@ digraph {
local_freelist_check6 -> local_freelist_check7
local_freelist_check7 -> fn_htab_lru_map_update_elem
fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"]
// Steal-loop victim lock failure is silent: treat as "no node found here"
// and continue to next CPU; same edge as the existing "N" path.
local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed,
xlabel="victim's lock\nfails: skip"]
// After a successful steal, re-acquire the local loc_l->lock. On failure
// the stolen node is published to free_llist (recovered, not orphaned)
// and the update returns -ENOMEM.
fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"]
post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"]
post_steal_lock -> post_steal_to_free_llist [style=dashed,
xlabel="loc_l->lock\nacquire fails"]
post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM
fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"]
fn_htab_lru_map_update_elem2 ->
fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"]
fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"]
fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4
// Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit.
rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM
use_local_node -> fn_htab_lru_map_update_elem4
fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4
local_freelist_check4 -> fn_htab_lru_map_update_elem4

34
MAINTAINERS
View File

@@ -4797,13 +4797,6 @@ S: Supported
F: arch/x86/net/
X: arch/x86/net/bpf_jit_comp32.c
BPF [BTF]
M: Martin KaFai Lau <martin.lau@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: include/linux/btf*
F: kernel/bpf/btf.c
BPF [CORE]
M: Alexei Starovoitov <ast@kernel.org>
M: Daniel Borkmann <daniel@iogearbox.net>
@@ -4840,12 +4833,13 @@ BPF [GENERAL] (Safe Dynamic Programs and Tools)
M: Alexei Starovoitov <ast@kernel.org>
M: Daniel Borkmann <daniel@iogearbox.net>
M: Andrii Nakryiko <andrii@kernel.org>
R: Martin KaFai Lau <martin.lau@linux.dev>
M: Eduard Zingerman <eddyz87@gmail.com>
M: Kumar Kartikeya Dwivedi <memxor@gmail.com>
R: Martin KaFai Lau <martin.lau@linux.dev>
R: Song Liu <song@kernel.org>
R: Yonghong Song <yonghong.song@linux.dev>
R: Jiri Olsa <jolsa@kernel.org>
R: Emil Tsalapatis <emil@etsalapatis.com>
L: bpf@vger.kernel.org
S: Supported
W: https://bpf.io/
@@ -4855,7 +4849,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git
F: Documentation/bpf/
F: Documentation/networking/filter.rst
F: Documentation/userspace-api/ebpf/
F: arch/*/include/asm/rqspinlock.h
F: arch/*/net/*
F: include/asm-generic/rqspinlock.h
F: include/linux/bpf*
F: include/linux/btf*
F: include/linux/buildid.h
@@ -4867,17 +4863,15 @@ F: include/uapi/linux/filter.h
F: kernel/bpf/
F: kernel/trace/bpf_trace.c
F: lib/buildid.c
F: arch/*/include/asm/rqspinlock.h
F: include/asm-generic/rqspinlock.h
F: lib/test_bpf.c
F: net/bpf/
F: net/core/filter.c
F: net/sched/act_bpf.c
F: net/sched/cls_bpf.c
F: samples/bpf/
F: scripts/Makefile.btf
F: scripts/bpf_doc.py
F: scripts/gen-btf.sh
F: scripts/Makefile.btf
F: scripts/pahole-version.sh
F: tools/bpf/
F: tools/lib/bpf/
@@ -4892,6 +4886,7 @@ F: kernel/bpf/*iter.c
BPF [L7 FRAMEWORK] (sockmap)
M: John Fastabend <john.fastabend@gmail.com>
M: Jakub Sitnicki <jakub@cloudflare.com>
M: Jiayuan Chen <jiayuan.chen@linux.dev>
L: netdev@vger.kernel.org
L: bpf@vger.kernel.org
S: Maintained
@@ -4935,18 +4930,11 @@ F: tools/testing/selftests/bpf/prog_tests/tc_netkit.c
F: tools/testing/selftests/drivers/net/hw/nk_qlease.py
F: tools/testing/selftests/net/nk_qlease.py
BPF [NETWORKING] (struct_ops, reuseport)
M: Martin KaFai Lau <martin.lau@linux.dev>
L: bpf@vger.kernel.org
L: netdev@vger.kernel.org
S: Maintained
F: kernel/bpf/bpf_struct*
BPF [NETWORKING] (tcx & tc BPF, sock_addr)
M: Martin KaFai Lau <martin.lau@linux.dev>
M: Daniel Borkmann <daniel@iogearbox.net>
R: John Fastabend <john.fastabend@gmail.com>
R: Stanislav Fomichev <sdf@fomichev.me>
R: Martin KaFai Lau <martin.lau@linux.dev>
L: bpf@vger.kernel.org
L: netdev@vger.kernel.org
S: Maintained
@@ -4981,14 +4969,6 @@ L: bpf@vger.kernel.org
S: Maintained
F: tools/testing/selftests/bpf/
BPF [STORAGE & CGROUPS]
M: Martin KaFai Lau <martin.lau@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: kernel/bpf/*storage.c
F: kernel/bpf/bpf_lru*
F: kernel/bpf/cgroup.c
BPF [TOOLING] (bpftool)
M: Quentin Monnet <qmo@kernel.org>
L: bpf@vger.kernel.org

28
arch/arm64/include/asm/pgtable.h
View File

@@ -1830,6 +1830,34 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return __ptep_get_and_clear(mm, addr, ptep);
}
static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
{
pteval_t old = 0;
if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte)))
return false;
/*
* The store must be complete by the time this returns, but the caller
* may be in lazy MMU mode, where __set_pte_complete() would defer the
* barriers. Issue them directly.
*/
emit_pte_barriers();
return true;
}
#define ptep_try_set ptep_try_set
/*
* arm64 mandates break-before-make: a cleared kernel PTE must have its TLB
* invalidated before a different page is installed in its place. The broadcast
* TLBI is an instruction, not an IPI, so this is safe with interrupts disabled.
*/
static inline void flush_tlb_before_set(unsigned long addr)
{
flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
}
#define flush_tlb_before_set flush_tlb_before_set
#define test_and_clear_young_ptes test_and_clear_young_ptes
static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep, unsigned int nr)

10
arch/arm64/mm/fault.c
View File

@@ -9,6 +9,7 @@
#include <linux/acpi.h>
#include <linux/bitfield.h>
#include <linux/bpf_defs.h>
#include <linux/extable.h>
#include <linux/kfence.h>
#include <linux/signal.h>
@@ -436,9 +437,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr,
} else if (is_pkvm_stage2_abort(esr)) {
msg = "access to hypervisor-protected memory";
} else {
if (esr_fsc_is_translation_fault(esr) &&
kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
return;
if (esr_fsc_is_translation_fault(esr)) {
if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs))
return;
if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc))
return;
}
msg = "paging request";
}

152
arch/arm64/net/bpf_jit_comp.c
View File

@@ -47,7 +47,7 @@
/* Map BPF registers to A64 registers */
static const int bpf2a64[] = {
/* return value from in-kernel function, and exit value from eBPF */
[BPF_REG_0] = A64_R(7),
[BPF_REG_0] = A64_R(8),
/* arguments from eBPF program to in-kernel function */
[BPF_REG_1] = A64_R(0),
[BPF_REG_2] = A64_R(1),
@@ -86,6 +86,7 @@ struct jit_ctx {
__le32 *image;
__le32 *ro_image;
u32 stack_size;
u16 stack_arg_size;
u64 user_vm_start;
u64 arena_vm_start;
bool fp_used;
@@ -533,13 +534,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
* | |
* +-----+ <= (BPF_FP - prog->aux->stack_depth)
* |RSVD | padding
* current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size)
* +-----+ <= (BPF_FP - ctx->stack_size)
* | |
* | ... | outgoing stack args (9+, if any)
* | |
* current A64_SP => +-----+
* | |
* | ... | Function call stack
* | |
* +-----+
* low
*
* Stack args 6-8 are passed in x5-x7, args 9+ at [SP].
* Incoming args 9+ are at [A64_FP + 16], [A64_FP + 24], ...
* (above the saved FP/LR pair pushed in the callee prologue).
*/
emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
@@ -613,6 +621,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
if (ctx->stack_arg_size)
emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
if (ctx->arena_vm_start)
emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx);
@@ -673,6 +684,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)
/* Update tail_call_cnt if the slot is populated. */
emit(A64_STR64I(tcc, ptr, 0), ctx);
if (ctx->stack_arg_size)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
/* restore SP */
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1034,6 +1048,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
const u8 r0 = bpf2a64[BPF_REG_0];
const u8 ptr = bpf2a64[TCCNT_PTR];
if (ctx->stack_arg_size)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx);
/* We're done with BPF stack */
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
@@ -1048,7 +1065,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
/* Restore FP/LR registers */
emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx);
/* Move the return value from bpf:r0 (aka x7) to x0 */
/* Move the return value from bpf:r0 (aka x8) to x0 */
emit(A64_MOV(1, A64_R(0), r0), ctx);
/* Authenticate lr */
@@ -1191,6 +1208,42 @@ static int add_exception_handler(const struct bpf_insn *insn,
return 0;
}
static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx)
{
int idx = bpf_off / sizeof(u64) - 1;
if (idx < NR_STACK_ARG_REGS)
emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx);
else
emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx);
}
static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx)
{
int idx = -bpf_off / sizeof(u64) - 1;
if (idx < NR_STACK_ARG_REGS)
emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx);
else
emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
}
static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx)
{
int idx = -bpf_off / sizeof(u64) - 1;
if (idx < NR_STACK_ARG_REGS) {
emit_a64_mov_i(1, stack_arg_reg[idx], imm, ctx);
} else {
emit_a64_mov_i(1, tmp, imm, ctx);
emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx);
}
}
/* JITs an eBPF instruction.
* Returns:
* 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1646,6 +1699,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
case BPF_LDX | BPF_MEM | BPF_H:
case BPF_LDX | BPF_MEM | BPF_B:
case BPF_LDX | BPF_MEM | BPF_DW:
if (insn->src_reg == BPF_REG_PARAMS) {
emit_stack_arg_load(dst, off, ctx);
break;
}
fallthrough;
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
case BPF_LDX | BPF_PROBE_MEM | BPF_W:
case BPF_LDX | BPF_PROBE_MEM | BPF_H:
@@ -1672,6 +1730,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
if (src == fp) {
src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
} else {
src_adj = src;
off_adj = off;
@@ -1752,6 +1812,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
case BPF_ST | BPF_MEM | BPF_H:
case BPF_ST | BPF_MEM | BPF_B:
case BPF_ST | BPF_MEM | BPF_DW:
if (insn->dst_reg == BPF_REG_PARAMS) {
emit_stack_arg_store_imm(imm, off, tmp, ctx);
break;
}
fallthrough;
case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
@@ -1763,6 +1828,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
if (dst == fp) {
dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
} else {
dst_adj = dst;
off_adj = off;
@@ -1814,6 +1881,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_B:
case BPF_STX | BPF_MEM | BPF_DW:
if (insn->dst_reg == BPF_REG_PARAMS) {
emit_stack_arg_store(src, off, ctx);
break;
}
fallthrough;
case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
@@ -1825,6 +1897,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
if (dst == fp) {
dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
} else {
dst_adj = dst;
off_adj = off;
@@ -2018,6 +2092,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
u8 *ro_image_ptr;
int body_idx;
int exentry_idx;
int out_cnt;
if (!prog->jit_requested)
return prog;
@@ -2065,6 +2140,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
out_cnt = bpf_out_stack_arg_cnt(env, prog);
if (out_cnt) {
int nr_on_stack = out_cnt - NR_STACK_ARG_REGS;
if (nr_on_stack > 0)
ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16);
}
if (priv_stack_ptr)
ctx.priv_sp_used = true;
@@ -2229,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (!aarch64_insn_copy(dst, src, len))
@@ -2247,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
int bargs_off, int retval_off, int run_ctx_off,
bool save_ret)
{
__le32 *branch;
u64 enter_prog;
u64 exit_prog;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = node->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
enter_prog = (u64)bpf_trampoline_enter(p);
exit_prog = (u64)bpf_trampoline_exit(p);
if (l->cookie == 0) {
if (node->cookie == 0) {
/* if cookie is zero, one instruction is enough to store it */
emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx);
} else {
emit_a64_mov_i64(A64_R(10), l->cookie, ctx);
emit_a64_mov_i64(A64_R(10), node->cookie, ctx);
emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off),
ctx);
}
@@ -2314,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
emit_call(exit_prog, ctx);
}
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int bargs_off, int retval_off, int run_ctx_off,
__le32 **branches)
{
@@ -2324,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
* Set this to 0 to avoid confusing the program.
*/
emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx);
for (i = 0; i < tl->nr_links; i++) {
invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off,
for (i = 0; i < tn->nr_nodes; i++) {
invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off,
run_ctx_off, true);
/* if (*(u64 *)(sp + retval_off) != 0)
* goto do_fexit;
@@ -2456,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs)
}
}
static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links)
static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes)
{
return fentry_links->nr_links == 1 &&
fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS;
return fentry_nodes->nr_nodes == 1 &&
fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS;
}
static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)
@@ -2480,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of
*
*/
static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_links *tlinks, void *func_addr,
struct bpf_tramp_nodes *tnodes, void *func_addr,
const struct btf_func_model *m,
const struct arg_aux *a,
u32 flags)
@@ -2496,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
int run_ctx_off;
int oargs_off;
int nfuncargs;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
bool save_ret;
__le32 **branches = NULL;
bool is_struct_ops = is_struct_ops_tramp(fentry);
int cookie_off, cookie_cnt, cookie_bargs_off;
int fsession_cnt = bpf_fsession_cnt(tlinks);
int fsession_cnt = bpf_fsession_cnt(tnodes);
u64 func_meta;
/* trampoline stack layout:
@@ -2549,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
cookie_off = stack_size;
/* room for session cookies */
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
stack_size += cookie_cnt * 8;
ip_off = stack_size;
@@ -2646,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
cookie_bargs_off = (bargs_off - cookie_off) / 8;
for (i = 0; i < fentry->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentry->links[i])) {
for (i = 0; i < fentry->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fentry->links[i], bargs_off,
invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off,
retval_off, run_ctx_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET);
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *),
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *),
GFP_KERNEL);
if (!branches)
return -ENOMEM;
@@ -2683,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
/* update the branches saved in invoke_bpf_mod_ret with cbnz */
for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) {
for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) {
int offset = &ctx->image[ctx->idx] - branches[i];
*branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset));
}
@@ -2694,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
store_func_meta(ctx, func_meta, func_meta_off);
cookie_bargs_off = (bargs_off - cookie_off) / 8;
for (i = 0; i < fexit->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fexit->links[i])) {
for (i = 0; i < fexit->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(ctx, meta, func_meta_off);
cookie_bargs_off--;
}
invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off,
invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off,
run_ctx_off, false);
}
@@ -2759,7 +2847,7 @@ bool bpf_jit_supports_fsession(void)
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct jit_ctx ctx = {
.image = NULL,
@@ -2773,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
if (ret < 0)
return ret;
ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags);
ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags);
if (ret < 0)
return ret;
@@ -2797,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
u32 size = ro_image_end - ro_image;
@@ -2824,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ret = calc_arg_aux(m, &aaux);
if (ret)
goto out;
ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags);
ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags);
if (ret > 0 && validate_code(&ctx) < 0) {
ret = -EINVAL;

8
arch/arm64/net/bpf_timed_may_goto.S
View File

@@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
stp x29, x30, [sp, #-64]!
mov x29, sp
/* Save BPF registers R0 - R5 (x7, x0-x4)*/
stp x7, x0, [sp, #16]
/* Save BPF registers R0 - R5 (x8, x0-x4)*/
stp x8, x0, [sp, #16]
stp x1, x2, [sp, #32]
stp x3, x4, [sp, #48]
@@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
/* BPF_REG_AX(x9) will be stored into count, so move return value to it. */
mov x9, x0
/* Restore BPF registers R0 - R5 (x7, x0-x4) */
ldp x7, x0, [sp, #16]
/* Restore BPF registers R0 - R5 (x8, x0-x4) */
ldp x8, x0, [sp, #16]
ldp x1, x2, [sp, #32]
ldp x3, x4, [sp, #48]

52
arch/loongarch/net/bpf_jit.c
View File

@@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off,
}
}
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n,
int args_off, int retval_off, int run_ctx_off, bool save_ret)
{
int ret;
u32 *branch;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = n->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
if (l->cookie)
if (n->cookie)
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1,
-run_ctx_off + cookie_off, l->cookie);
-run_ctx_off + cookie_off, n->cookie);
else
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
@@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l,
return ret;
}
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl,
static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn,
int args_off, int retval_off, int run_ctx_off,
int func_meta_off, bool save_ret, u64 func_meta, int cookie_off)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
for (i = 0; i < tl->nr_links; i++) {
for (i = 0; i < tn->nr_nodes; i++) {
int err;
if (bpf_prog_calls_session_cookie(tl->links[i])) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta);
cur_cookie--;
}
err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret);
err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret);
if (err)
return err;
}
@@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign)
}
static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
const struct btf_func_model *m, struct bpf_tramp_links *tlinks,
const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes,
void *func_addr, u32 flags)
{
int i, ret, save_ret;
@@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
unsigned long long func_meta;
bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
void *orig_call = func_addr;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
u32 **branches = NULL;
/*
@@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
ip_off = stack_size;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* Room for session cookies */
stack_size += cookie_cnt * 8;
@@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
store_args(ctx, nr_arg_slots, args_off);
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all session cookies' value */
for (i = 0; i < cookie_cnt; i++)
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i);
@@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
return ret;
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off);
if (ret)
return ret;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL);
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL);
if (!branches)
return -ENOMEM;
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off);
for (i = 0; i < fmod_ret->nr_links; i++) {
ret = invoke_bpf_prog(ctx, fmod_ret->links[i],
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i],
args_off, retval_off, run_ctx_off, true);
if (ret)
goto out;
@@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
emit_insn(ctx, nop);
}
for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) {
for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) {
int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i];
*branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset);
}
/* Set "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off,
func_meta_off, false, func_meta, cookie_off);
if (ret)
@@ -2115,7 +2115,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks, void *func_addr)
u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr)
{
int ret, size;
void *image, *tmp;
@@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ctx.idx = 0;
jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));
ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags);
ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags);
if (ret < 0)
goto out;
@@ -2152,7 +2152,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
int ret;
struct jit_ctx ctx;
@@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
ctx.image = NULL;
ctx.idx = 0;
ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags);
return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
}

54
arch/powerpc/net/bpf_jit_comp.c
View File

@@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
}
static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_link *l, int regs_off, int retval_off,
struct bpf_tramp_node *n, int regs_off, int retval_off,
int run_ctx_off, bool save_ret)
{
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = n->link->prog;
ppc_inst_t branch_insn;
u32 jmp_idx;
int ret = 0;
/* Save cookie */
if (IS_ENABLED(CONFIG_PPC64)) {
PPC_LI64(_R3, l->cookie);
PPC_LI64(_R3, n->cookie);
EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx,
bpf_cookie)));
} else {
PPC_LI32(_R3, l->cookie >> 32);
PPC_LI32(_R4, l->cookie);
PPC_LI32(_R3, n->cookie >> 32);
PPC_LI32(_R4, n->cookie);
EMIT(PPC_RAW_STW(_R3, _R1,
run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie)));
EMIT(PPC_RAW_STW(_R4, _R1,
@@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct
}
static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,
struct bpf_tramp_links *tl, int regs_off, int retval_off,
struct bpf_tramp_nodes *tn, int regs_off, int retval_off,
int run_ctx_off, u32 *branches)
{
int i;
@@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context
*/
EMIT(PPC_RAW_LI(_R3, 0));
EMIT(PPC_RAW_STL(_R3, _R1, retval_off));
for (i = 0; i < tl->nr_links; i++) {
if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off,
for (i = 0; i < tn->nr_nodes; i++) {
if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off,
run_ctx_off, true))
return -EINVAL;
@@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
void *rw_image_end, void *ro_image,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off;
int nvr_off, alt_lr_off, r4_off = 0;
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0;
struct codegen_context codegen_ctx, *ctx;
int cookie_off, cookie_cnt, cookie_ctx_off;
int fsession_cnt = bpf_fsession_cnt(tlinks);
int fsession_cnt = bpf_fsession_cnt(tnodes);
u64 func_meta;
u32 *image = (u32 *)rw_image;
ppc_inst_t branch_insn;
@@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* room for session cookies */
cookie_off = bpf_frame_size;
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
bpf_frame_size += cookie_cnt * 8;
/* Room for IP address argument */
@@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off = (regs_off - cookie_off) / 8;
for (i = 0; i < fentry->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentry->links[i])) {
for (i = 0; i < fentry->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentry->nodes[i])) {
u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(image, ctx, meta, func_meta_off);
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off,
if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off,
run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET))
return -EINVAL;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL);
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL);
if (!branches)
return -ENOMEM;
@@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
/* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */
for (i = 0; i < fmod_ret->nr_links && image; i++) {
for (i = 0; i < fmod_ret->nr_nodes && image; i++) {
if (create_cond_branch(&branch_insn, &image[branches[i]],
(unsigned long)&image[ctx->idx], COND_NE << 16)) {
ret = -EINVAL;
@@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
cookie_ctx_off = (regs_off - cookie_off) / 8;
for (i = 0; i < fexit->nr_links; i++) {
if (bpf_prog_calls_session_cookie(fexit->links[i])) {
for (i = 0; i < fexit->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fexit->nodes[i])) {
u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT);
store_func_meta(image, ctx, meta, func_meta_off);
cookie_ctx_off--;
}
if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off,
if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off,
run_ctx_off, false)) {
ret = -EINVAL;
goto cleanup;
@@ -1185,18 +1185,18 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
int ret;
ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr);
ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr);
return ret;
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
u32 size = image_end - image;
@@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
flags, tlinks, func_addr);
flags, tnodes, func_addr);
if (ret < 0)
goto out;

61
arch/riscv/net/bpf_jit_comp64.c
View File

@@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64,
emit_sd(RV_REG_FP, stack_off, reg, ctx);
}
static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off,
static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off,
int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
{
int ret, branch_off;
struct bpf_prog *p = l->link.prog;
struct bpf_prog *p = node->link->prog;
int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
if (l->cookie)
emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx);
if (node->cookie)
emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx);
else
emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx);
@@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of
return ret;
}
static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,
static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,
int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta,
int cookie_off, struct rv_jit_context *ctx)
{
int i, cur_cookie = (cookie_off - args_off) / 8;
for (i = 0; i < tl->nr_links; i++) {
for (i = 0; i < tn->nr_nodes; i++) {
int err;
if (bpf_prog_calls_session_cookie(tl->links[i])) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx);
cur_cookie--;
}
err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off,
err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off,
save_ret, ctx);
if (err)
return err;
@@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off,
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
const struct btf_func_model *m,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr, u32 flags,
struct rv_jit_context *ctx)
{
@@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
int stack_size = 0, nr_arg_slots = 0;
int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
int cookie_off, cookie_cnt;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
void *orig_call = func_addr;
bool save_ret;
@@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
ip_off = stack_size;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* room for session cookies */
stack_size += cookie_cnt * 8;
cookie_off = stack_size;
@@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
store_args(nr_arg_slots, args_off, ctx);
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all session cookies' value */
for (i = 0; i < cookie_cnt; i++)
emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx);
@@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
return ret;
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx);
if (ret)
return ret;
}
if (fmod_ret->nr_links) {
branches_off = kzalloc_objs(int, fmod_ret->nr_links);
if (fmod_ret->nr_nodes) {
branches_off = kzalloc_objs(int, fmod_ret->nr_nodes);
if (!branches_off)
return -ENOMEM;
/* cleanup to avoid garbage return value confusion */
emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx);
for (i = 0; i < fmod_ret->nr_links; i++) {
ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off,
for (i = 0; i < fmod_ret->nr_nodes; i++) {
ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off,
run_ctx_off, true, ctx);
if (ret)
goto out;
@@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
/* update branches saved in invoke_bpf_mod_ret with bnez */
for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) {
for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) {
offset = ninsns_rvoff(ctx->ninsns - branches_off[i]);
insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1);
*(u32 *)(ctx->insns + branches_off[i]) = insn;
@@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
/* set "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off,
false, func_meta, cookie_off, ctx);
if (ret)
@@ -1305,7 +1305,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
struct rv_jit_context ctx;
@@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
ctx.ninsns = 0;
ctx.insns = NULL;
ctx.ro_insns = NULL;
ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx);
ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx);
return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);
}
@@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int ret;
@@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
ctx.ninsns = 0;
ctx.insns = image;
ctx.ro_insns = ro_image;
ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx);
ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx);
if (ret < 0)
goto out;
@@ -1808,6 +1808,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
break;
}
/* Implement helper call to bpf_get_current_task/_btf() inline */
if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
insn->imm == BPF_FUNC_get_current_task_btf)) {
emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_TP, ctx);
break;
}
mark_call(ctx);
ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
&addr, &fixed_addr);
@@ -2138,6 +2145,8 @@ bool bpf_jit_inlines_helper_call(s32 imm)
{
switch (imm) {
case BPF_FUNC_get_smp_processor_id:
case BPF_FUNC_get_current_task:
case BPF_FUNC_get_current_task_btf:
return true;
default:
return false;

81
arch/s390/net/bpf_jit_comp.c
View File

@@ -27,6 +27,7 @@
#include <asm/extable.h>
#include <asm/dis.h>
#include <asm/facility.h>
#include <asm/lowcore.h>
#include <asm/nospec-branch.h>
#include <asm/set_memory.h>
#include <asm/text-patching.h>
@@ -1777,6 +1778,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
int j, ret;
u64 func;
/* Implement helper call to bpf_get_smp_processor_id() inline */
if (insn->src_reg == 0 &&
insn->imm == BPF_FUNC_get_smp_processor_id) {
const u32 *cpu_nr = &get_lowcore()->cpu_nr;
/* ly %b0, cpu_nr */
EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0,
(unsigned long)cpu_nr);
break;
}
/* Implement helper call to bpf_get_current_task/_btf() inline */
if (insn->src_reg == 0 &&
(insn->imm == BPF_FUNC_get_current_task ||
insn->imm == BPF_FUNC_get_current_task_btf)) {
const u64 *current_task =
&get_lowcore()->current_task;
/* lg %b0, current_task */
EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_0, REG_0, REG_0,
(unsigned long)current_task);
break;
}
ret = bpf_jit_get_func_addr(fp, insn, extra_pass,
&func, &func_addr_fixed);
if (ret < 0)
@@ -2512,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o
static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
struct bpf_tramp_link *tlink, bool save_ret)
struct bpf_tramp_node *node, bool save_ret)
{
struct bpf_jit *jit = &tjit->common;
int cookie_off = tjit->run_ctx_off +
offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = tlink->link.prog;
struct bpf_prog *p = node->link->prog;
int patch;
/*
* run_ctx.cookie = tlink->cookie;
* run_ctx.cookie = node->cookie;
*/
emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie);
emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie);
/*
* if ((start = __bpf_prog_enter(p, &run_ctx)) == 0)
@@ -2584,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit,
static int invoke_bpf(struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
struct bpf_tramp_links *tl, bool save_ret,
struct bpf_tramp_nodes *tn, bool save_ret,
u64 func_meta, int cookie_off)
{
int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64);
struct bpf_jit *jit = &tjit->common;
for (i = 0; i < tl->nr_links; i++) {
if (bpf_prog_calls_session_cookie(tl->links[i])) {
for (i = 0; i < tn->nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(tn->nodes[i])) {
u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT);
emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta);
cur_cookie--;
}
if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret))
if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret))
return -EINVAL;
}
@@ -2626,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
struct bpf_tramp_jit *tjit,
const struct btf_func_model *m,
u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
int nr_bpf_args, nr_reg_args, nr_stack_args;
int cookie_cnt, cookie_off, fsession_cnt;
struct bpf_jit *jit = &tjit->common;
@@ -2668,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
return -ENOTSUPP;
}
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
fsession_cnt = bpf_fsession_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
fsession_cnt = bpf_fsession_cnt(tnodes);
/*
* Calculate the stack layout.
@@ -2804,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
func_meta, cookie_off))
return -EINVAL;
if (fmod_ret->nr_links) {
if (fmod_ret->nr_nodes) {
/*
* retval = 0;
*/
@@ -2813,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
_EMIT6(0xd707f000 | tjit->retval_off,
0xf000 | tjit->retval_off);
for (i = 0; i < fmod_ret->nr_links; i++) {
if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true))
for (i = 0; i < fmod_ret->nr_nodes; i++) {
if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true))
return -EINVAL;
/*
@@ -2939,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *orig_call)
struct bpf_tramp_nodes *tnodes, void *orig_call)
{
struct bpf_tramp_image im;
struct bpf_tramp_jit tjit;
@@ -2948,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
memset(&tjit, 0, sizeof(tjit));
ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags,
tlinks, orig_call);
tnodes, orig_call);
return ret < 0 ? ret : tjit.common.prg;
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
void *image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
u32 flags, struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
struct bpf_tramp_jit tjit;
@@ -2964,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
/* Compute offsets, check whether the code fits. */
memset(&tjit, 0, sizeof(tjit));
ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
tlinks, func_addr);
tnodes, func_addr);
if (ret < 0)
return ret;
@@ -2978,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
tjit.common.prg = 0;
tjit.common.prg_buf = image;
ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags,
tlinks, func_addr);
tnodes, func_addr);
return ret < 0 ? ret : tjit.common.prg;
}
@@ -3057,3 +3082,15 @@ bool bpf_jit_supports_timed_may_goto(void)
{
return true;
}
bool bpf_jit_inlines_helper_call(s32 imm)
{
switch (imm) {
case BPF_FUNC_get_smp_processor_id:
case BPF_FUNC_get_current_task:
case BPF_FUNC_get_current_task_btf:
return true;
default:
return false;
}
}

12
arch/x86/include/asm/pgtable.h
View File

@@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
} while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte));
}
/*
* Note: strictly-zero compare is narrower than pte_none(), but the gap is
* harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs.
*/
static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
{
pte_t old_pte = __pte(0);
return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte);
}
#define ptep_try_set ptep_try_set
#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0)
#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS

12
arch/x86/mm/fault.c
View File

@@ -8,6 +8,7 @@
#include <linux/sched/task_stack.h> /* task_stack_*(), ... */
#include <linux/kdebug.h> /* oops_begin/end, ... */
#include <linux/memblock.h> /* max_low_pfn */
#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */
#include <linux/kfence.h> /* kfence_handle_page_fault */
#include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */
#include <linux/mmiotrace.h> /* kmmio_handler, ... */
@@ -693,10 +694,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code,
if (IS_ENABLED(CONFIG_EFI))
efi_crash_gracefully_on_page_fault(address, regs);
/* Only not-present faults should be handled by KFENCE. */
if (!(error_code & X86_PF_PROT) &&
kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
return;
/* Only not-present faults should be handled by KFENCE or BPF arena. */
if (!(error_code & X86_PF_PROT)) {
if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs))
return;
if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip))
return;
}
oops:
/*

205
arch/x86/net/bpf_jit_comp.c
View File

@@ -10,6 +10,7 @@
#include <linux/if_vlan.h>
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/memory.h>
#include <linux/sort.h>
#include <asm/extable.h>
@@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
*pprog = prog;
}
/* add rsp, depth */
static void emit_add_rsp(u8 **pprog, u16 depth)
{
u8 *prog = *pprog;
if (!depth)
return;
if (is_imm8(depth))
EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
else
EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
*pprog = prog;
}
/* sub rsp, depth */
static void emit_sub_rsp(u8 **pprog, u16 depth)
{
u8 *prog = *pprog;
if (!depth)
return;
if (is_imm8(depth))
EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
else
EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
*pprog = prog;
}
static void emit_nops(u8 **pprog, int len)
{
u8 *prog = *pprog;
@@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
bool seen_exit = false;
u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
void __percpu *priv_frame_ptr = NULL;
u16 out_stack_arg_cnt, outgoing_rsp;
u64 arena_vm_start, user_vm_start;
void __percpu *priv_stack_ptr;
int i, excnt = 0;
int ilen, proglen = 0;
u8 *ip, *prog = temp;
u32 stack_depth;
int callee_saved_size;
s32 outgoing_arg_base;
int err;
stack_depth = bpf_prog->aux->stack_depth;
out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
if (priv_stack_ptr) {
priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);
stack_depth = 0;
}
/*
* Follow x86-64 calling convention for both BPF-to-BPF and
* kfunc calls:
* - Arg 6 is passed in R9 register
* - Args 7+ are passed on the stack at [rsp]
*
* Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9).
* Incoming args 7+ are read from [rbp + 16], [rbp + 24], ...
* (BPF r11+16, r11+24, ... map directly with no offset change).
*
* tail_call_reachable is rejected by the verifier and priv_stack
* is disabled by the JIT when stack args exist, so R9 is always
* available.
*
* Stack layout (high to low):
* [rbp + 16 + ...] incoming stack args 7+ (from caller)
* [rbp + 8] return address
* [rbp] saved rbp
* [rbp - prog_stack] program stack
* [below] callee-saved regs
* [below] outgoing args 7+ (= rsp)
*/
arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
@@ -1700,6 +1755,44 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
push_r12(&prog);
push_callee_regs(&prog, callee_regs_used);
}
/* Compute callee-saved register area size. */
callee_saved_size = 0;
if (bpf_prog->aux->exception_boundary || arena_vm_start)
callee_saved_size += 8; /* r12 */
if (bpf_prog->aux->exception_boundary) {
callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
} else {
int j;
for (j = 0; j < 4; j++)
if (callee_regs_used[j])
callee_saved_size += 8;
}
/*
* Base offset from rbp for translating BPF outgoing args 7+
* to native offsets. BPF uses negative offsets from r11
* (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses
* positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for
* arg8, ...). Arg 6 goes to R9 directly.
*
* The translation reverses direction:
* native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16
*
* Note that tail_call_reachable is guaranteed to be false when
* stack args exist, so tcc pushes need not be accounted for.
*/
outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
/*
* Allocate outgoing stack arg area for args 7+ only.
* Arg 6 goes into r9 register, not on stack.
*/
outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
if (bpf_prog->aux->exception_boundary)
bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
emit_sub_rsp(&prog, outgoing_rsp);
if (arena_vm_start)
emit_mov_imm64(&prog, X86_REG_R12,
arena_vm_start >> 32, (u32) arena_vm_start);
@@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
u8 b2 = 0, b3 = 0;
u8 *start_of_ldx;
s64 jmp_offset;
s16 insn_off;
s32 insn_off;
u8 jmp_cond;
u8 *func;
int nops;
@@ -2134,12 +2227,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
EMIT1(0xC7);
goto st;
case BPF_ST | BPF_MEM | BPF_DW:
if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
/* Arg 6: store immediate in r9 register */
emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
break;
}
EMIT2(add_1mod(0x48, dst_reg), 0xC7);
st: if (is_imm8(insn->off))
EMIT2(add_1reg(0x40, dst_reg), insn->off);
st: insn_off = insn->off;
if (dst_reg == BPF_REG_PARAMS) {
/*
* Args 7+: reverse BPF negative offsets to
* x86 positive rsp offsets.
* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
*/
insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
dst_reg = BPF_REG_FP;
}
if (is_imm8(insn_off))
EMIT2(add_1reg(0x40, dst_reg), insn_off);
else
EMIT1_off32(add_1reg(0x80, dst_reg), insn->off);
EMIT1_off32(add_1reg(0x80, dst_reg), insn_off);
EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code)));
break;
@@ -2149,7 +2257,17 @@ st: if (is_imm8(insn->off))
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_DW:
emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
/* Arg 6: store register value in r9 */
EMIT_mov(X86_REG_R9, src_reg);
break;
}
insn_off = insn->off;
if (dst_reg == BPF_REG_PARAMS) {
insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
dst_reg = BPF_REG_FP;
}
emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
break;
case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
@@ -2248,6 +2366,19 @@ st: if (is_imm8(insn->off))
case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
insn_off = insn->off;
if (src_reg == BPF_REG_PARAMS) {
if (insn_off == 8) {
/* Incoming arg 6: read from r9 */
EMIT_mov(dst_reg, X86_REG_R9);
break;
}
src_reg = BPF_REG_FP;
/*
* Incoming args 7+: native_off == bpf_off
* (r11+16 → [rbp+16], r11+24 → [rbp+24], ...)
* No offset adjustment needed.
*/
}
if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
@@ -2736,6 +2867,8 @@ st: if (is_imm8(insn->off))
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
return -EINVAL;
}
/* Deallocate outgoing args 7+ area. */
emit_add_rsp(&prog, outgoing_rsp);
if (bpf_prog->aux->exception_boundary) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
@@ -2971,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
}
static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_link *l, int stack_size,
struct bpf_tramp_node *node, int stack_size,
int run_ctx_off, bool save_ret,
void *image, void *rw_image)
{
u8 *prog = *pprog;
u8 *jmp_insn;
int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
struct bpf_prog *p = l->link.prog;
u64 cookie = l->cookie;
struct bpf_prog *p = node->link->prog;
u64 cookie = node->cookie;
/* mov rdi, cookie */
emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
@@ -3086,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
}
static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_links *tl, int stack_size,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, int func_meta_off, bool save_ret,
void *image, void *rw_image, u64 func_meta,
int cookie_off)
@@ -3094,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
int i, cur_cookie = (cookie_off - stack_size) / 8;
u8 *prog = *pprog;
for (i = 0; i < tl->nr_links; i++) {
if (tl->links[i]->link.prog->call_session_cookie) {
for (i = 0; i < tl->nr_nodes; i++) {
if (tl->nodes[i]->link->prog->call_session_cookie) {
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
cur_cookie--;
}
if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size,
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
run_ctx_off, save_ret, image, rw_image))
return -EINVAL;
}
@@ -3109,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
}
static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
struct bpf_tramp_links *tl, int stack_size,
struct bpf_tramp_nodes *tl, int stack_size,
int run_ctx_off, u8 **branches,
void *image, void *rw_image)
{
@@ -3121,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
*/
emit_mov_imm32(&prog, false, BPF_REG_0, 0);
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
for (i = 0; i < tl->nr_links; i++) {
if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true,
for (i = 0; i < tl->nr_nodes; i++) {
if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
image, rw_image))
return -EINVAL;
@@ -3213,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,
void *rw_image_end, void *image,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
int i, ret, nr_regs = m->nr_args, stack_size = 0;
int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
void *orig_call = func_addr;
int cookie_off, cookie_cnt;
u8 **branches = NULL;
@@ -3292,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
ip_off = stack_size;
cookie_cnt = bpf_fsession_cookie_cnt(tlinks);
cookie_cnt = bpf_fsession_cookie_cnt(tnodes);
/* room for session cookies */
stack_size += cookie_cnt * 8;
cookie_off = stack_size;
@@ -3385,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
}
if (bpf_fsession_cnt(tlinks)) {
if (bpf_fsession_cnt(tnodes)) {
/* clear all the session cookies' value */
for (int i = 0; i < cookie_cnt; i++)
emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
@@ -3393,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
}
if (fentry->nr_links) {
if (fentry->nr_nodes) {
if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
func_meta, cookie_off))
return -EINVAL;
}
if (fmod_ret->nr_links) {
branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *),
if (fmod_ret->nr_nodes) {
branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *),
GFP_KERNEL);
if (!branches)
return -ENOMEM;
@@ -3440,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_nops(&prog, X86_PATCH_SIZE);
}
if (fmod_ret->nr_links) {
if (fmod_ret->nr_nodes) {
/* From Intel 64 and IA-32 Architectures Optimization
* Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler
* Coding Rule 11: All branch targets should be 16-byte
@@ -3450,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* Update the branches saved in invoke_bpf_mod_ret with the
* aligned address of do_fexit.
*/
for (i = 0; i < fmod_ret->nr_links; i++) {
for (i = 0; i < fmod_ret->nr_nodes; i++) {
emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
image + (branches[i] - (u8 *)rw_image), X86_JNE);
}
@@ -3458,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* set the "is_return" flag for fsession */
func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
if (bpf_fsession_cnt(tlinks))
if (bpf_fsession_cnt(tnodes))
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
if (fexit->nr_links) {
if (fexit->nr_nodes) {
if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
false, image, rw_image, func_meta, cookie_off)) {
ret = -EINVAL;
@@ -3535,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size)
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
void *rw_image, *tmp;
@@ -3550,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,
flags, tlinks, func_addr);
flags, tnodes, func_addr);
if (ret < 0)
goto out;
@@ -3563,7 +3696,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
}
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
struct bpf_tramp_image im;
void *image;
@@ -3581,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
return -ENOMEM;
ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
m, flags, tlinks, func_addr);
m, flags, tnodes, func_addr);
bpf_jit_free_exec(image);
return ret;
}
@@ -3793,7 +3926,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
for (pass = 0; pass < MAX_PASSES || image; pass++) {
if (!padding && pass >= PADDING_PASSES)
padding = true;
proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding);
proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen,
&ctx, padding);
if (proglen <= 0) {
out_image:
image = NULL;
@@ -3910,6 +4044,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (text_poke_copy(dst, src, len) == NULL)

7
drivers/base/power/power.h
View File

@@ -168,3 +168,10 @@ static inline void device_pm_init(struct device *dev)
device_pm_sleep_init(dev);
pm_runtime_init(dev);
}
#ifdef CONFIG_BPF_SYSCALL
struct bpf_ws_lock { };
struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void);
void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock);
void *bpf_wakeup_sources_get_head(void);
#endif

71
drivers/base/power/wakeup.c
View File

@@ -1168,11 +1168,78 @@ static const struct file_operations wakeup_sources_stats_fops = {
.release = seq_release_private,
};
static int __init wakeup_sources_debugfs_init(void)
#ifdef CONFIG_BPF_SYSCALL
#include <linux/btf.h>
__bpf_kfunc_start_defs();
/**
* bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources
*
* The underlying SRCU lock returns an integer index. However, the BPF verifier
* requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired
* resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque
* structure pointer (struct bpf_ws_lock *) to satisfy the verifier while
* safely encoding the integer index within the pointer address itself.
*
* Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL).
*/
__bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void)
{
return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1);
}
/**
* bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources
* @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock()
*
* The BPF verifier guarantees that @lock is a valid, unreleased pointer from
* the acquire function. We decode the pointer back into the integer SRCU index
* by subtracting 1 and release the lock.
*/
__bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock)
{
wakeup_sources_read_unlock((int)(long)lock - 1);
}
/**
* bpf_wakeup_sources_get_head - Get the head of the wakeup sources list
*
* Return: The head of the wakeup sources list.
*/
__bpf_kfunc void *bpf_wakeup_sources_get_head(void)
{
return &wakeup_sources;
}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(wakeup_source_kfunc_ids)
BTF_ID_FLAGS(func, bpf_wakeup_sources_read_lock, KF_ACQUIRE)
BTF_ID_FLAGS(func, bpf_wakeup_sources_read_unlock, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_wakeup_sources_get_head)
BTF_KFUNCS_END(wakeup_source_kfunc_ids)
static const struct btf_kfunc_id_set wakeup_source_kfunc_set = {
.set = &wakeup_source_kfunc_ids,
};
static void __init wakeup_sources_bpf_init(void)
{
if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set))
pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n");
}
#else
static inline void wakeup_sources_bpf_init(void) {}
#endif /* CONFIG_BPF_SYSCALL */
static int __init wakeup_sources_init(void)
{
debugfs_create_file("wakeup_sources", 0444, NULL, NULL,
&wakeup_sources_stats_fops);
wakeup_sources_bpf_init();
return 0;
}
postcore_initcall(wakeup_sources_debugfs_init);
postcore_initcall(wakeup_sources_init);

8
drivers/net/ethernet/netronome/nfp/bpf/verifier.c
View File

@@ -561,10 +561,10 @@ nfp_bpf_check_alu(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta,
const struct bpf_reg_state *dreg =
cur_regs(env) + meta->insn.dst_reg;
meta->umin_src = min(meta->umin_src, sreg->umin_value);
meta->umax_src = max(meta->umax_src, sreg->umax_value);
meta->umin_dst = min(meta->umin_dst, dreg->umin_value);
meta->umax_dst = max(meta->umax_dst, dreg->umax_value);
meta->umin_src = min(meta->umin_src, reg_umin(sreg));
meta->umax_src = max(meta->umax_src, reg_umax(sreg));
meta->umin_dst = min(meta->umin_dst, reg_umin(dreg));
meta->umax_dst = max(meta->umax_dst, reg_umax(dreg));
/* NFP supports u16 and u32 multiplication.
*

2
fs/bpf_fs_kfuncs.c
View File

@@ -200,7 +200,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str,
const struct bpf_dynptr *value_p, int flags)
{
struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
struct inode *inode = d_inode(dentry);
const void *value;
u32 value_len;

4
fs/verity/measure.c
View File

@@ -118,9 +118,9 @@ __bpf_kfunc_start_defs();
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p)
__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, const struct bpf_dynptr *digest_p)
{
struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
const struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
const struct inode *inode = file_inode(file);
u32 dynptr_sz = __bpf_dynptr_size(digest_ptr);
struct fsverity_digest *arg;

14
include/asm-generic/rqspinlock.h
View File

@@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock)
({ \
int __ret; \
local_irq_save(flags); \
__ret = raw_res_spin_lock(lock); \
if (__ret) \
preempt_disable(); \
__ret = res_spin_lock(lock); \
if (__ret) { \
local_irq_restore(flags); \
preempt_enable(); \
} \
__ret; \
})
#define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); })
#define raw_res_spin_unlock_irqrestore(lock, flags) \
({ \
res_spin_unlock(lock); \
local_irq_restore(flags); \
preempt_enable(); \
})
#endif /* __ASM_GENERIC_RQSPINLOCK_H */

5
include/linux/bpf-cgroup.h
View File

@@ -421,7 +421,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr,
enum bpf_prog_type ptype);
int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
union bpf_attr __user *uattr, u32 uattr_size);
const struct bpf_func_proto *
cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
@@ -452,7 +452,8 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
}
static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
union bpf_attr __user *uattr,
u32 uattr_size)
{
return -EINVAL;
}

282
include/linux/bpf.h
View File

@@ -6,6 +6,7 @@
#include <uapi/linux/bpf.h>
#include <uapi/linux/filter.h>
#include <linux/bpf_defs.h>
#include <crypto/sha2.h>
#include <linux/workqueue.h>
@@ -32,6 +33,8 @@
#include <linux/memcontrol.h>
#include <linux/cfi.h>
#include <linux/xattr.h>
#include <linux/key.h>
#include <linux/ftrace.h>
#include <asm/rqspinlock.h>
struct bpf_verifier_env;
@@ -111,7 +114,7 @@ struct bpf_map_ops {
long (*map_pop_elem)(struct bpf_map *map, void *value);
long (*map_peek_elem)(struct bpf_map *map, void *value);
void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);
int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf);
int (*map_get_hash)(struct bpf_map *map);
/* funcs called by prog_array and perf_event_array map */
void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
@@ -296,6 +299,7 @@ struct bpf_map_owner {
struct bpf_map {
u8 sha[SHA256_DIGEST_SIZE];
u32 excl;
const struct bpf_map_ops *ops;
struct bpf_map *inner_map_meta;
#ifdef CONFIG_SECURITY
@@ -489,6 +493,35 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f
return rec->field_mask & type;
}
static inline bool btf_field_is_nmi_safe(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
case BPF_RES_SPIN_LOCK:
case BPF_TIMER:
case BPF_WORKQUEUE:
case BPF_TASK_WORK:
case BPF_KPTR_UNREF:
case BPF_REFCOUNT:
return true;
default:
return false;
}
}
static inline bool btf_record_has_nmi_unsafe_fields(const struct btf_record *rec)
{
int i;
if (IS_ERR_OR_NULL(rec))
return false;
for (i = 0; i < rec->cnt; i++) {
if (!btf_field_is_nmi_safe(rec->fields[i].type))
return true;
}
return false;
}
static inline void bpf_obj_init(const struct btf_record *rec, void *obj)
{
int i;
@@ -618,6 +651,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock);
u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
struct bpf_offload_dev;
@@ -679,6 +714,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
u64 flags);
void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
u64 flags);
#else
static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
int node_id, u64 flags)
@@ -689,6 +726,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr
static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
{
}
static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
int node_id, u64 flags)
{
return NULL;
}
#endif
extern const struct bpf_map_ops bpf_map_offload_ops;
@@ -1052,7 +1095,7 @@ struct bpf_insn_access_aux {
struct {
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
u32 ref_id;
};
};
struct bpf_verifier_log *log; /* for verbose logs */
@@ -1152,6 +1195,11 @@ struct bpf_prog_offload {
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
* Also reuse this macro for maximum number of arguments a BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots. The JIT may map some stack arg slots to registers based
* on the native calling convention (e.g., arg 6 to R9 on x86-64).
*/
#define MAX_BPF_FUNC_ARGS 12
@@ -1234,9 +1282,9 @@ enum {
#define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
#define BPF_TRAMP_IS_RETURN_SHIFT 63
struct bpf_tramp_links {
struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];
int nr_links;
struct bpf_tramp_nodes {
struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS];
int nr_nodes;
};
struct bpf_tramp_run_ctx;
@@ -1264,13 +1312,13 @@ struct bpf_tramp_run_ctx;
struct bpf_tramp_image;
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr);
void *arch_alloc_bpf_trampoline(unsigned int size);
void arch_free_bpf_trampoline(void *image, unsigned int size);
int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr);
struct bpf_tramp_nodes *tnodes, void *func_addr);
u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
@@ -1336,8 +1384,6 @@ struct bpf_trampoline {
/* hlist for trampoline_ip_table */
struct hlist_node hlist_ip;
struct ftrace_ops *fops;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
u32 flags;
u64 key;
@@ -1358,6 +1404,11 @@ struct bpf_trampoline {
int progs_cnt[BPF_TRAMP_MAX];
/* Executable image of trampoline */
struct bpf_tramp_image *cur_image;
/* Used as temporary old image storage for multi_attach */
struct {
struct bpf_tramp_image *old_image;
u32 old_flags;
} multi_attach;
};
struct bpf_attach_target_info {
@@ -1455,11 +1506,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6
return 0;
}
struct bpf_tracing_multi_link;
#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
@@ -1467,6 +1520,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link);
int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link);
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
@@ -1544,14 +1602,15 @@ void bpf_jit_uncharge_modmem(u32 size);
bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
int insn_idx);
u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
@@ -1578,6 +1637,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog,
struct bpf_tracing_multi_link *link)
{
return -ENOTSUPP;
}
#endif
struct bpf_func_info_aux {
@@ -1615,7 +1684,7 @@ struct bpf_ctx_arg_aux {
enum bpf_reg_type reg_type;
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
u32 ref_id;
bool refcounted;
};
@@ -1657,6 +1726,19 @@ struct bpf_stream_stage {
int len;
};
enum bpf_sig_verdict {
BPF_SIG_UNSIGNED = 0,
BPF_SIG_VERIFIED,
};
enum bpf_sig_keyring {
BPF_SIG_KEYRING_NONE = 0,
BPF_SIG_KEYRING_BUILTIN,
BPF_SIG_KEYRING_SECONDARY,
BPF_SIG_KEYRING_PLATFORM,
BPF_SIG_KEYRING_USER,
};
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
@@ -1699,6 +1781,11 @@ struct bpf_prog_aux {
bool changes_pkt_data;
bool might_sleep;
bool kprobe_write_ctx;
struct {
s32 keyring_serial;
u8 keyring_type;
u8 verdict;
} sig;
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
struct bpf_arena *arena;
@@ -1731,6 +1818,7 @@ struct bpf_prog_aux {
struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
#ifdef CONFIG_SECURITY
void *security;
#endif
@@ -1874,12 +1962,17 @@ struct bpf_link_ops {
__poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
};
struct bpf_tramp_link {
struct bpf_link link;
struct bpf_tramp_node {
struct bpf_link *link;
struct hlist_node tramp_hlist;
u64 cookie;
};
struct bpf_tramp_link {
struct bpf_link link;
struct bpf_tramp_node node;
};
struct bpf_shim_tramp_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
@@ -1887,13 +1980,31 @@ struct bpf_shim_tramp_link {
struct bpf_tracing_link {
struct bpf_tramp_link link;
struct bpf_tramp_node fexit;
struct bpf_trampoline *trampoline;
struct bpf_prog *tgt_prog;
};
struct bpf_fsession_link {
struct bpf_tracing_link link;
struct bpf_tramp_link fexit;
struct bpf_tracing_multi_node {
struct bpf_tramp_node node;
struct bpf_trampoline *trampoline;
struct ftrace_func_entry entry;
};
struct bpf_tracing_multi_data {
struct ftrace_hash *unreg;
struct ftrace_hash *modify;
struct ftrace_hash *reg;
struct ftrace_func_entry *entry;
};
struct bpf_tracing_multi_link {
struct bpf_link link;
struct bpf_tracing_multi_data data;
u64 *cookies;
struct bpf_tramp_node *fexits;
int nodes_cnt;
struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt);
};
struct bpf_raw_tp_link {
@@ -2079,6 +2190,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog)
#endif
}
static inline bool is_tracing_multi(enum bpf_attach_type type)
{
return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
type == BPF_TRACE_FSESSION_MULTI;
}
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
/* This macro helps developer to register a struct_ops type and generate
* type information correctly. Developers should use this macro to register
@@ -2099,8 +2216,8 @@ void bpf_struct_ops_put(const void *kdata);
int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
struct bpf_tramp_node *node,
const struct btf_func_model *model,
void *stub_func,
void **image, u32 *image_off,
@@ -2125,6 +2242,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);
void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
u32 bpf_struct_ops_id(const void *kdata);
int bpf_struct_ops_for_each_prog(const void *kdata,
int (*cb)(struct bpf_prog *prog, void *data),
void *data);
#ifdef CONFIG_NET
/* Define it here to avoid the use of forward declaration */
@@ -2192,31 +2312,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op
#endif
static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)
static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)
for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION)
cnt++;
if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI)
cnt++;
}
return cnt;
}
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node)
{
return link->link.prog->call_session_cookie;
return node->link->prog->call_session_cookie;
}
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentries.links[i]))
for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) {
if (bpf_prog_calls_session_cookie(fentries.nodes[i]))
cnt++;
}
@@ -2598,6 +2720,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r
void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
void bpf_obj_cancel_fields(struct bpf_map *map, void *obj);
void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
@@ -2764,6 +2887,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable);
void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie);
int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
int bpf_link_settle(struct bpf_link_primer *primer);
void bpf_link_cleanup(struct bpf_link_primer *primer);
@@ -2917,7 +3043,9 @@ int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
size_t actual_size);
/* verify correctness of eBPF program */
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size);
struct bpf_log_attr;
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr,
struct bpf_log_attr *attr_log);
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
@@ -3088,6 +3216,56 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
static __always_inline u32
bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
const void *ctx, bpf_prog_run_fn run_prog)
{
const struct bpf_prog_array_item *item;
struct bpf_prog *prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
u32 ret = 1;
if (unlikely(!array))
return ret;
migrate_disable();
run_ctx.is_uprobe = false;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
/* Skip dummy_bpf_prog placeholder (len == 0) */
if (unlikely(!prog->len)) {
item++;
continue;
}
if (unlikely(!bpf_prog_get_recursion_context(prog))) {
bpf_prog_inc_misses_counter(prog);
bpf_prog_put_recursion_context(prog);
item++;
continue;
}
run_ctx.bpf_cookie = item->bpf_cookie;
if (!prog->sleepable) {
guard(rcu)();
ret &= run_prog(prog, ctx);
} else {
ret &= run_prog(prog, ctx);
}
bpf_prog_put_recursion_context(prog);
item++;
}
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
return ret;
}
#else /* !CONFIG_BPF_SYSCALL */
static inline struct bpf_prog *bpf_prog_get(u32 ufd)
{
@@ -3135,6 +3313,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_
{
}
static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie)
{
}
static inline int bpf_link_prime(struct bpf_link *link,
struct bpf_link_primer *primer)
{
@@ -3626,15 +3810,25 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map,
#endif /* CONFIG_BPF_SYSCALL */
#endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
#ifdef CONFIG_KEYS
struct bpf_key {
struct key *key;
bool has_ref;
};
#endif /* CONFIG_KEYS */
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
struct bpf_key *bpf_lookup_system_key(u64 id);
void bpf_key_put(struct bpf_key *bkey);
int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring);
static inline s32 bpf_key_serial(const struct bpf_key *key)
{
return key->has_ref ? key->key->serial : 0;
}
#else
static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
{
@@ -3650,12 +3844,17 @@ static inline void bpf_key_put(struct bpf_key *bkey)
{
}
static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
static inline int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
return -EOPNOTSUPP;
}
static inline s32 bpf_key_serial(const struct bpf_key *key)
{
return 0;
}
#endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */
/* verifier prototypes for helper functions called from eBPF programs */
@@ -3931,15 +4130,6 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {}
static inline void bpf_cgroup_atype_put(int cgroup_atype) {}
#endif /* CONFIG_BPF_LSM */
struct key;
#ifdef CONFIG_KEYS
struct bpf_key {
struct key *key;
bool has_ref;
};
#endif /* CONFIG_KEYS */
static inline bool type_is_alloc(u32 type)
{
return type & MEM_ALLOC;

19
include/linux/bpf_defs.h Normal file
View File

@@ -0,0 +1,19 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Subset of bpf.h declarations, split out so files that need only these
* declarations can avoid bpf.h's full include cost.
*/
#ifndef _LINUX_BPF_DEFS_H
#define _LINUX_BPF_DEFS_H
#ifdef CONFIG_BPF_SYSCALL
bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip);
#else
static inline bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
unsigned long fault_ip)
{
return false;
}
#endif
#endif /* _LINUX_BPF_DEFS_H */

6
include/linux/bpf_lsm.h
View File

@@ -52,6 +52,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str,
const struct bpf_dynptr *value_p, int flags);
int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str);
bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog);
bool bpf_lsm_hook_returns_errno(u32 btf_id);
#else /* !CONFIG_BPF_LSM */
@@ -104,6 +105,11 @@ static inline bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog)
{
return false;
}
static inline bool bpf_lsm_hook_returns_errno(u32 btf_id)
{
return true;
}
#endif /* CONFIG_BPF_LSM */
#endif /* _LINUX_BPF_LSM_H */

2
include/linux/bpf_types.h
View File

@@ -134,6 +134,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_RHASH, rhtab_map_ops)
BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
@@ -155,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf)
BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi)
BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops)
BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi)
BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi)

301
include/linux/bpf_verifier.h
View File

@@ -8,6 +8,7 @@
#include <linux/btf.h> /* for struct btf and btf_id() */
#include <linux/filter.h> /* for MAX_BPF_STACK */
#include <linux/tnum.h>
#include <linux/cnum.h>
/* Maximum variable offset umax_value permitted when resolving memory accesses.
* In practice this is far bigger than any realistic pointer offset; this limit
@@ -65,7 +66,6 @@ struct bpf_reg_state {
struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */
u32 mem_size;
u32 dynptr_id; /* for dynptr slices */
};
/* For dynptr stack slots */
@@ -120,14 +120,8 @@ struct bpf_reg_state {
* These refer to the same value as var_off, not necessarily the actual
* contents of the register.
*/
s64 smin_value; /* minimum possible (s64)value */
s64 smax_value; /* maximum possible (s64)value */
u64 umin_value; /* minimum possible (u64)value */
u64 umax_value; /* maximum possible (u64)value */
s32 s32_min_value; /* minimum possible (s32)value */
s32 s32_max_value; /* maximum possible (s32)value */
u32 u32_min_value; /* minimum possible (u32)value */
u32 u32_max_value; /* maximum possible (u32)value */
struct cnum64 r64; /* 64-bit range as circular number */
struct cnum32 r32; /* 32-bit range as circular number */
/* For PTR_TO_PACKET, used to find other pointers with the same variable
* offset, so they can share range knowledge.
* For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we
@@ -153,46 +147,14 @@ struct bpf_reg_state {
#define BPF_ADD_CONST32 (1U << 30)
#define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32)
u32 id;
/* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned
* from a pointer-cast helper, bpf_sk_fullsock() and
* bpf_tcp_sock().
*
* Consider the following where "sk" is a reference counted
* pointer returned from "sk = bpf_sk_lookup_tcp();":
*
* 1: sk = bpf_sk_lookup_tcp();
* 2: if (!sk) { return 0; }
* 3: fullsock = bpf_sk_fullsock(sk);
* 4: if (!fullsock) { bpf_sk_release(sk); return 0; }
* 5: tp = bpf_tcp_sock(fullsock);
* 6: if (!tp) { bpf_sk_release(sk); return 0; }
* 7: bpf_sk_release(sk);
* 8: snd_cwnd = tp->snd_cwnd; // verifier will complain
*
* After bpf_sk_release(sk) at line 7, both "fullsock" ptr and
* "tp" ptr should be invalidated also. In order to do that,
* the reg holding "fullsock" and "sk" need to remember
* the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id
* such that the verifier can reset all regs which have
* ref_obj_id matching the sk_reg->id.
*
* sk_reg->ref_obj_id is set to sk_reg->id at line 1.
* sk_reg->id will stay as NULL-marking purpose only.
* After NULL-marking is done, sk_reg->id can be reset to 0.
*
* After "fullsock = bpf_sk_fullsock(sk);" at line 3,
* fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id.
*
* After "tp = bpf_tcp_sock(fullsock);" at line 5,
* tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id
* which is the same as sk_reg->ref_obj_id.
*
* From the verifier perspective, if sk, fullsock and tp
* are not NULL, they are the same ptr with different
* reg->type. In particular, bpf_sk_release(tp) is also
* allowed and has the same effect as bpf_sk_release(sk).
/*
* Tracks the parent object this register was derived from.
* Used for cascading invalidation: when the parent object is
* released or invalidated, all registers with matching parent_id
* are also invalidated. For example, a slice from bpf_dynptr_data()
* gets parent_id set to the dynptr's id.
*/
u32 ref_obj_id;
u32 parent_id;
/* Inside the callee two registers can be both PTR_TO_STACK like
* R1=fp-8 and R2=fp-8, but one of them points to this function stack
* while another to the caller's stack. To differentiate them 'frameno'
@@ -209,6 +171,66 @@ struct bpf_reg_state {
bool precise;
};
static inline s64 reg_smin(const struct bpf_reg_state *reg)
{
return cnum64_smin(reg->r64);
}
static inline s64 reg_smax(const struct bpf_reg_state *reg)
{
return cnum64_smax(reg->r64);
}
static inline u64 reg_umin(const struct bpf_reg_state *reg)
{
return cnum64_umin(reg->r64);
}
static inline u64 reg_umax(const struct bpf_reg_state *reg)
{
return cnum64_umax(reg->r64);
}
static inline s32 reg_s32_min(const struct bpf_reg_state *reg)
{
return cnum32_smin(reg->r32);
}
static inline s32 reg_s32_max(const struct bpf_reg_state *reg)
{
return cnum32_smax(reg->r32);
}
static inline u32 reg_u32_min(const struct bpf_reg_state *reg)
{
return cnum32_umin(reg->r32);
}
static inline u32 reg_u32_max(const struct bpf_reg_state *reg)
{
return cnum32_umax(reg->r32);
}
static inline void reg_set_srange32(struct bpf_reg_state *reg, s32 smin, s32 smax)
{
reg->r32 = cnum32_from_srange(smin, smax);
}
static inline void reg_set_urange32(struct bpf_reg_state *reg, u32 umin, u32 umax)
{
reg->r32 = cnum32_from_urange(umin, umax);
}
static inline void reg_set_srange64(struct bpf_reg_state *reg, s64 smin, s64 smax)
{
reg->r64 = cnum64_from_srange(smin, smax);
}
static inline void reg_set_urange64(struct bpf_reg_state *reg, u64 umin, u64 umax)
{
reg->r64 = cnum64_from_urange(umin, umax);
}
enum bpf_stack_slot_type {
STACK_INVALID, /* nothing was stored in this stack slot */
STACK_SPILL, /* register spilled into stack */
@@ -309,10 +331,14 @@ struct bpf_reference_state {
* is used purely to inform the user of a reference leak.
*/
int insn_idx;
/* Use to keep track of the source object of a lock, to ensure
* it matches on unlock.
*/
void *ptr;
union {
/* For REF_TYPE_PTR */
int parent_id;
/* Use to keep track of the source object of a lock, to ensure
* it matches on unlock.
*/
void *ptr;
};
};
struct bpf_retval_range {
@@ -347,6 +373,7 @@ struct bpf_func_state {
bool in_callback_fn;
bool in_async_callback_fn;
bool in_exception_callback_fn;
bool no_stack_arg_load;
/* For callback calling functions that limit number of possible
* callback executions (e.g. bpf_loop) keeps track of current
* simulated iteration number.
@@ -372,46 +399,49 @@ struct bpf_func_state {
* `stack`. allocated_stack is always a multiple of BPF_REG_SIZE.
*/
int allocated_stack;
u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */
struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */
};
#define MAX_CALL_FRAMES 8
#define MAX_CALL_FRAMES 16
/* instruction history flags, used in bpf_jmp_history_entry.flags field */
/* instruction history flags, used in bpf_jmp_history_entry.flags field.
* Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
*/
enum {
/* instruction references stack slot through PTR_TO_STACK register;
* we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8)
* and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512,
* 8 bytes per slot, so slot index (spi) is [0, 63])
*/
INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */
INSN_F_STACK_ACCESS = BIT(0),
INSN_F_SPI_MASK = 0x3f, /* 6 bits */
INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */
INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
INSN_F_STACK_ACCESS = BIT(9),
INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */
INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */
/* total 12 bits are used now. */
INSN_F_STACK_ARG_ACCESS = BIT(3),
};
static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES);
static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8);
struct bpf_jmp_history_entry {
u32 idx;
/* insn idx can't be bigger than 1 million */
u32 idx : 20;
u32 frame : 4; /* stack access frame number */
u32 spi : 6; /* stack slot index (0..63) */
u32 : 2;
u32 prev_idx : 20;
/* special INSN_F_xxx flags */
u32 flags : 12;
/* additional registers that need precision tracking when this
* jump is backtracked, vector of six 10-bit records
u32 flags : 4;
u32 : 8;
/*
* additional registers that need precision tracking when this
* jump is backtracked, vector of five 11-bit records
*/
u64 linked_regs;
};
/* Maximum number of register states that can exist at once */
#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES)
static_assert(MAX_CALL_FRAMES <= (1 << 4));
static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
/* Maximum number of bpf_reg_state objects that can exist at once */
#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
struct bpf_verifier_state {
/* call stack tracking */
struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -497,10 +527,23 @@ struct bpf_verifier_state {
u32 may_goto_depth;
};
#define bpf_get_spilled_reg(slot, frame, mask) \
(((slot < frame->allocated_stack / BPF_REG_SIZE) && \
((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \
? &frame->stack[slot].spilled_ptr : NULL)
static inline struct bpf_reg_state *
bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
{
if (slot < frame->allocated_stack / BPF_REG_SIZE &&
(1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
return &frame->stack[slot].spilled_ptr;
return NULL;
}
static inline struct bpf_reg_state *
bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
{
if (slot < frame->out_stack_arg_cnt &&
frame->stack_arg_regs[slot].type != NOT_INIT)
return &frame->stack_arg_regs[slot];
return NULL;
}
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
#define bpf_for_each_spilled_reg(iter, frame, reg, mask) \
@@ -508,7 +551,13 @@ struct bpf_verifier_state {
iter < frame->allocated_stack / BPF_REG_SIZE; \
iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
#define bpf_for_each_spilled_stack_arg(iter, frame, reg) \
for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame); \
iter < frame->out_stack_arg_cnt; \
iter++, reg = bpf_get_spilled_stack_arg(iter, frame))
#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __stack, __mask, __expr) \
({ \
struct bpf_verifier_state *___vstate = __vst; \
int ___i, ___j; \
@@ -516,6 +565,7 @@ struct bpf_verifier_state {
struct bpf_reg_state *___regs; \
__state = ___vstate->frame[___i]; \
___regs = __state->regs; \
__stack = NULL; \
for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \
__reg = &___regs[___j]; \
(void)(__expr); \
@@ -523,14 +573,27 @@ struct bpf_verifier_state {
bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \
if (!__reg) \
continue; \
__stack = &__state->stack[___j]; \
(void)(__expr); \
} \
__stack = NULL; \
bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \
if (!__reg) \
continue; \
(void)(__expr); \
} \
} \
(void)__stack; \
})
/* Invoke __expr over regsiters in __vst, setting __state and __reg */
#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \
bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, 1 << STACK_SPILL, __expr)
#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \
({ \
struct bpf_stack_state * ___stack; \
(void)___stack; \
bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, ___stack,\
1 << STACK_SPILL, __expr); \
})
/* linked list of verifier states used to prune search */
struct bpf_verifier_state_list {
@@ -700,6 +763,22 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log)
return log && log->level;
}
struct bpf_log_attr {
char __user *ubuf;
u32 size;
u32 level;
u32 offsetof_true_size;
bpfptr_t uattr;
};
int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
bpfptr_t uattr_common, u32 size_common);
struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
struct bpf_common_attr *common, bpfptr_t uattr,
u32 size);
int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
#define BPF_MAX_SUBPROGS 256
struct bpf_subprog_arg_info {
@@ -724,6 +803,7 @@ struct bpf_subprog_info {
u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
u32 insn_processed;
/* offsets in range [stack_depth .. fastcall_stack_off)
* are used for bpf_fastcall spills and fills.
*/
@@ -740,12 +820,21 @@ struct bpf_subprog_info {
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
bool might_sleep: 1;
u8 arg_cnt:3;
u8 arg_cnt:4;
enum priv_stack_mode priv_stack_mode;
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS];
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
u16 stack_arg_cnt; /* incoming + max outgoing */
u16 max_out_stack_arg_cnt;
};
static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
{
if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS)
return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS;
return 0;
}
struct bpf_verifier_env;
struct backtrack_state {
@@ -753,6 +842,7 @@ struct backtrack_state {
u32 frame;
u32 reg_masks[MAX_CALL_FRAMES];
u64 stack_masks[MAX_CALL_FRAMES];
u8 stack_arg_masks[MAX_CALL_FRAMES];
};
struct bpf_id_pair {
@@ -881,6 +971,8 @@ struct bpf_verifier_env {
u32 prev_insn_processed, insn_processed;
/* number of jmps, calls, exits analyzed so far */
u32 prev_jmps_processed, jmps_processed;
/* maximum combined stack depth */
u32 max_stack_depth;
/* total verification time */
u64 verification_time;
/* maximum number of verifier states kept in 'branching' instructions */
@@ -914,6 +1006,7 @@ struct bpf_verifier_env {
* e.g., in reg_type_str() to generate reg_type string
*/
char tmp_str_buf[TMP_STR_BUF_LEN];
char tmp_arg_name[32];
struct bpf_insn insn_buf[INSN_BUF_SIZE];
struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
struct bpf_scc_callchain callchain_buf;
@@ -1087,7 +1180,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
void bpf_free_backedges(struct bpf_scc_visit *visit);
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
int insn_flags, u64 linked_regs);
int insn_flags, int spi, int frame, u64 linked_regs);
void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg);
@@ -1150,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame,
bt->stack_masks[frame] |= 1ull << slot;
}
static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
bt->stack_arg_masks[frame] |= 1 << slot;
}
static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
{
return bt->reg_masks[frame] & (1 << reg);
@@ -1321,6 +1419,25 @@ struct bpf_map_desc {
int uid;
};
/* The last initialized dynptr; Populated by process_dynptr_func() */
struct bpf_dynptr_desc {
enum bpf_dynptr_type type;
u32 id;
u32 parent_id;
};
/*
* The last seen rereferenced object; Updated by update_ref_obj() when a register refers to a
* referenced object. Used when the helper or kfunc is casting a referenced object, returning
* allocated memory derived from referenced object or creating a dynptr with a referenced
* object as parent.
*/
struct ref_obj_desc {
u32 id;
u32 parent_id;
u8 cnt;
};
struct bpf_kfunc_call_arg_meta {
/* In parameters */
struct btf *btf;
@@ -1329,7 +1446,6 @@ struct bpf_kfunc_call_arg_meta {
const struct btf_type *func_proto;
const char *func_name;
/* Out parameters */
u32 ref_obj_id;
u8 release_regno;
bool r0_rdonly;
u32 ret_btf_id;
@@ -1361,16 +1477,13 @@ struct bpf_kfunc_call_arg_meta {
struct {
struct btf_field *field;
} arg_rbtree_root;
struct {
enum bpf_dynptr_type type;
u32 id;
u32 ref_obj_id;
} initialized_dynptr;
struct {
u8 spi;
u8 frameno;
} iter;
struct bpf_map_desc map;
struct bpf_dynptr_desc dynptr;
struct ref_obj_desc ref_obj;
u64 mem_size;
};
@@ -1479,6 +1592,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset);
int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_insn *insn_buf, int insn_idx, int *cnt);
/* Functions exported from verifier.c, used by trampoline.c */
int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,
struct bpf_attach_target_info *tgt_info);
/* Functions in fixups.c, called from bpf_check() */
int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env);
int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);

7
include/linux/btf.h
View File

@@ -145,7 +145,8 @@ const char *btf_get_name(const struct btf *btf);
void btf_get(struct btf *btf);
void btf_put(struct btf *btf);
const struct btf_header *btf_header(const struct btf *btf);
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
struct bpf_log_attr;
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log);
struct btf *btf_get_by_fd(int fd);
int btf_get_info_by_fd(const struct btf *btf,
const union bpf_attr *attr,
@@ -415,12 +416,12 @@ static inline bool btf_type_is_array(const struct btf_type *t)
return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
}
static inline u16 btf_type_vlen(const struct btf_type *t)
static inline u32 btf_type_vlen(const struct btf_type *t)
{
return BTF_INFO_VLEN(t->info);
}
static inline u16 btf_vlen(const struct btf_type *t)
static inline u32 btf_vlen(const struct btf_type *t)
{
return btf_type_vlen(t);
}

1
include/linux/btf_ids.h
View File

@@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[];
extern u32 bpf_local_storage_map_btf_id[];
extern u32 btf_bpf_map_id[];
extern u32 bpf_kmem_cache_btf_id[];
extern u32 bpf_multi_func_btf_id[];
#endif

82
include/linux/cnum.h Normal file
View File

@@ -0,0 +1,82 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef _LINUX_CNUM_H
#define _LINUX_CNUM_H
#include <linux/types.h>
/*
* cnum32: a circular number.
* A unified representation for signed and unsigned ranges.
*
* Assume that a 32-bit range is a circle, with 0 being in the 12 o'clock
* position, numbers placed sequentially in clockwise order and U32_MAX
* in the 11 o'clock position. Signed values map onto the same circle:
* S32_MAX sits at 5 o'clock, S32_MIN sits at 6 o'clock (opposite 0),
* negative values occupy the left half and positive values the right half.
*
* @cnum32 represents an arc on this circle drawn clockwise.
* @base corresponds to the first value of the range.
* @size corresponds to the number of integers in the range excluding @base.
* (The @base is excluded to avoid integer overflow when representing the full
* 0..U32_MAX range, which corresponds to 2^32, which can't be stored in u32).
*
* For example: {U32_MAX, 1} corresponds to signed range [-1, 0],
* {S32_MAX, 1} corresponds to unsigned range [S32_MAX, S32_MIN].
*/
struct cnum32 {
u32 base;
u32 size;
};
#define CNUM32_UNBOUNDED ((struct cnum32){ .base = 0, .size = U32_MAX })
#define CNUM32_EMPTY ((struct cnum32){ .base = U32_MAX, .size = U32_MAX })
struct cnum32 cnum32_from_urange(u32 min, u32 max);
struct cnum32 cnum32_from_srange(s32 min, s32 max);
u32 cnum32_umin(struct cnum32 cnum);
u32 cnum32_umax(struct cnum32 cnum);
s32 cnum32_smin(struct cnum32 cnum);
s32 cnum32_smax(struct cnum32 cnum);
struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b);
void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src);
void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max);
void cnum32_intersect_with_srange(struct cnum32 *dst, s32 min, s32 max);
bool cnum32_contains(struct cnum32 cnum, u32 v);
bool cnum32_is_const(struct cnum32 cnum);
bool cnum32_is_empty(struct cnum32 cnum);
struct cnum32 cnum32_add(struct cnum32 a, struct cnum32 b);
struct cnum32 cnum32_negate(struct cnum32 a);
bool cnum32_is_subset(struct cnum32 outer, struct cnum32 inner);
/* Same as cnum32 but for 64-bit ranges */
struct cnum64 {
u64 base;
u64 size;
};
#define CNUM64_UNBOUNDED ((struct cnum64){ .base = 0, .size = U64_MAX })
#define CNUM64_EMPTY ((struct cnum64){ .base = U64_MAX, .size = U64_MAX })
struct cnum64 cnum64_from_urange(u64 min, u64 max);
struct cnum64 cnum64_from_srange(s64 min, s64 max);
u64 cnum64_umin(struct cnum64 cnum);
u64 cnum64_umax(struct cnum64 cnum);
s64 cnum64_smin(struct cnum64 cnum);
s64 cnum64_smax(struct cnum64 cnum);
struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b);
void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src);
void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max);
void cnum64_intersect_with_srange(struct cnum64 *dst, s64 min, s64 max);
bool cnum64_contains(struct cnum64 cnum, u64 v);
bool cnum64_is_const(struct cnum64 cnum);
bool cnum64_is_empty(struct cnum64 cnum);
struct cnum64 cnum64_add(struct cnum64 a, struct cnum64 b);
struct cnum64 cnum64_negate(struct cnum64 a);
bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner);
struct cnum32 cnum32_from_cnum64(struct cnum64 cnum);
struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b);
#endif /* _LINUX_CNUM_H */

27
include/linux/filter.h
View File

@@ -58,8 +58,9 @@ struct ctl_table_header;
#define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */
/* Kernel hidden auxiliary/helper register. */
#define BPF_REG_AX MAX_BPF_REG
#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1)
#define BPF_REG_PARAMS MAX_BPF_REG
#define BPF_REG_AX (MAX_BPF_REG + 1)
#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
/* unused opcode to mark special call to bpf_tail_call() helper */
@@ -748,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog,
return ret;
}
static inline bool is_stack_arg_ldx(const struct bpf_insn *insn)
{
return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) &&
insn->src_reg == BPF_REG_PARAMS &&
insn->off > 0 && insn->off % 8 == 0;
}
static inline bool is_stack_arg_st(const struct bpf_insn *insn)
{
return insn->code == (BPF_ST | BPF_MEM | BPF_DW) &&
insn->dst_reg == BPF_REG_PARAMS &&
insn->off < 0 && insn->off % 8 == 0;
}
static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
{
return insn->code == (BPF_STX | BPF_MEM | BPF_DW) &&
insn->dst_reg == BPF_REG_PARAMS &&
insn->off < 0 && insn->off % 8 == 0;
}
#define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
struct bpf_skb_data_end {
@@ -1159,6 +1181,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);

9
include/linux/ftrace.h
View File

@@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits);
void free_ftrace_hash(struct ftrace_hash *hash);
struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,
unsigned long ip, unsigned long direct);
void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry);
void ftrace_hash_remove(struct ftrace_hash *hash);
/* The hash used to know what functions callbacks trace */
struct ftrace_ops_hash {
@@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b
void ftrace_stub_direct_tramp(void);
unsigned long ftrace_hash_count(struct ftrace_hash *hash);
#else
struct ftrace_ops;
static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
@@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace
return -ENODEV;
}
static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash)
{
return 0;
}
/*
* This must be implemented by the architecture.
* It is the way the ftrace direct_ops helper, when called

43
include/linux/pgtable.h
View File

@@ -1070,6 +1070,49 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres
}
#endif
#ifndef ptep_try_set
/**
* ptep_try_set - atomically set an empty kernel PTE
* @ptep: page table entry
* @new_pte: value to install
*
* Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on
* success, false if the slot was already populated or the arch has no
* implementation.
*
* For special kernel page tables only - never user page tables. The caller must
* prevent concurrent teardown of @ptep and must accept that other writers may
* race. Concurrent clearers must use ptep_get_and_clear() so racing accesses
* agree on the outcome.
*
* Architectures opt in by providing a cmpxchg-based override and defining
* ptep_try_set as an identity macro. The generic stub returns false, which is
* correct for callers that fall through to oops on failure.
*/
static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
{
return false;
}
#endif
#ifndef flush_tlb_before_set
/**
* flush_tlb_before_set - invalidate a kernel PTE's TLB before re-setting it
* @addr: kernel virtual address whose PTE was just cleared
*
* Some architectures (e.g. arm64) do not allow a live page-table entry to be
* repointed at a different page in one step. The old entry must first be made
* invalid and its translation flushed from every TLB, and only then may the new
* entry be written.
*
* This is only for the lockless atomic kernel-PTE installers (ptep_try_set()).
* It must be callable with interrupts disabled.
*/
static inline void flush_tlb_before_set(unsigned long addr)
{
}
#endif
#ifndef wrprotect_ptes
/**
* wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same

4
include/linux/rhashtable.h
View File

@@ -263,6 +263,8 @@ struct rhash_lock_head __rcu **__rht_bucket_nested(
struct rhash_lock_head __rcu **rht_bucket_nested_insert(
struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash);
void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key);
#define rht_dereference(p, ht) \
rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht))
@@ -1117,7 +1119,7 @@ static __always_inline int __rhashtable_remove_fast_one(
atomic_dec(&ht->nelems);
if (unlikely(ht->p.automatic_shrinking &&
rht_shrink_below_30(ht, tbl)))
schedule_work(&ht->run_work);
irq_work_queue(&ht->run_irq_work);
err = 0;
}

3
include/linux/syscalls.h
View File

@@ -940,7 +940,8 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags,
asmlinkage long sys_getrandom(char __user *buf, size_t count,
unsigned int flags);
asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags);
asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size);
asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size,
struct bpf_common_attr __user *attr_common, unsigned int size_common);
asmlinkage long sys_execveat(int dfd, const char __user *filename,
const char __user *const __user *argv,
const char __user *const __user *envp, int flags);

12
include/linux/trace_events.h
View File

@@ -770,6 +770,7 @@ trace_trigger_soft_disabled(struct trace_event_file *file)
#ifdef CONFIG_BPF_EVENTS
unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx);
unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx);
int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie);
void perf_event_detach_bpf_prog(struct perf_event *event);
int perf_event_query_prog_array(struct perf_event *event, void __user *info);
@@ -786,12 +787,18 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
unsigned long *missed);
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr);
#else
static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
{
return 1;
}
static inline unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
{
return 1;
}
static inline int
perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie)
{
@@ -838,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static inline int
bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
#endif
enum {

2
include/trace/bpf_probe.h
View File

@@ -58,9 +58,7 @@ static notrace void \
__bpf_trace_##call(void *__data, proto) \
{ \
might_fault(); \
preempt_disable_notrace(); \
CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \
preempt_enable_notrace(); \
}
#undef DECLARE_EVENT_SYSCALL_CLASS

32
include/uapi/linux/bpf.h
View File

@@ -994,6 +994,7 @@ enum bpf_cmd {
BPF_PROG_STREAM_READ_BY_FD,
BPF_PROG_ASSOC_STRUCT_OPS,
__MAX_BPF_CMD,
BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
};
enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_CGRP_STORAGE,
BPF_MAP_TYPE_ARENA,
BPF_MAP_TYPE_INSN_ARRAY,
BPF_MAP_TYPE_RHASH,
__MAX_BPF_MAP_TYPE
};
@@ -1154,6 +1156,9 @@ enum bpf_attach_type {
BPF_TRACE_KPROBE_SESSION,
BPF_TRACE_UPROBE_SESSION,
BPF_TRACE_FSESSION,
BPF_TRACE_FENTRY_MULTI,
BPF_TRACE_FEXIT_MULTI,
BPF_TRACE_FSESSION_MULTI,
__MAX_BPF_ATTACH_TYPE
};
@@ -1178,6 +1183,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_UPROBE_MULTI = 12,
BPF_LINK_TYPE_NETKIT = 13,
BPF_LINK_TYPE_SOCKMAP = 14,
BPF_LINK_TYPE_TRACING_MULTI = 15,
__MAX_BPF_LINK_TYPE,
};
@@ -1321,7 +1327,11 @@ enum {
* BPF_TRACE_UPROBE_MULTI attach type to create return probe.
*/
enum {
BPF_F_UPROBE_MULTI_RETURN = (1U << 0)
/* Get return uprobe. */
BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
/* Get path from provided path_fd. */
BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
};
/* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id {
};
};
struct bpf_common_attr {
__aligned_u64 log_buf;
__u32 log_size;
__u32 log_level;
__u32 log_true_size;
};
#define BPF_OBJ_NAME_LEN 16U
enum {
@@ -1537,6 +1554,11 @@ union bpf_attr {
*
* BPF_MAP_TYPE_ARENA - contains the address where user space
* is going to mmap() the arena. It has to be page aligned.
*
* BPF_MAP_TYPE_RHASH - initial table size hint
* (nelem_hint). 0 = use rhashtable default. Must be
* <= min(max_entries, U16_MAX). Upper 32 bits reserved,
* must be zero.
*/
__u64 map_extra;
@@ -1846,6 +1868,7 @@ union bpf_attr {
__u32 cnt;
__u32 flags;
__u32 pid;
__u32 path_fd;
} uprobe_multi;
struct {
union {
@@ -1861,6 +1884,11 @@ union bpf_attr {
};
__u64 expected_revision;
} cgroup;
struct {
__aligned_u64 ids;
__aligned_u64 cookies;
__u32 cnt;
} tracing_multi;
};
} link_create;
@@ -6698,6 +6726,7 @@ struct bpf_prog_info {
__u32 verified_insns;
__u32 attach_btf_obj_id;
__u32 attach_btf_id;
__u32 :32;
} __attribute__((aligned(8)));
struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info {
__u64 map_extra;
__aligned_u64 hash;
__u32 hash_size;
__u32 :32;
} __attribute__((aligned(8)));
struct bpf_btf_info {

26
include/uapi/linux/btf.h
View File

@@ -33,20 +33,22 @@ struct btf_header {
__u32 layout_len; /* length of layout section */
};
/* Max # of type identifier */
#define BTF_MAX_TYPE 0x000fffff
/* Max offset into the string section */
#define BTF_MAX_NAME_OFFSET 0x00ffffff
/* Max # of struct/union/enum members or func args */
#define BTF_MAX_VLEN 0xffff
enum btf_max {
/* Max possible kind */
BTF_MAX_KIND = 0x0000007f,
/* Max # of type identifier */
BTF_MAX_TYPE = 0x000fffff,
/* Max offset into the string section */
BTF_MAX_NAME_OFFSET = 0x00ffffff,
/* Max # of struct/union/enum members or func args */
BTF_MAX_VLEN = 0x00ffffff,
};
struct btf_type {
__u32 name_off;
/* "info" bits arrangement
* bits 0-15: vlen (e.g. # of struct's members)
* bits 16-23: unused
* bits 24-28: kind (e.g. int, ptr, array...etc)
* bits 29-30: unused
* bits 0-23: vlen (e.g. # of struct's members)
* bits 24-30: kind (e.g. int, ptr, array...etc)
* bit 31: kind_flag, currently used by
* struct, union, enum, fwd, enum64,
* decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type {
};
};
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)
#define BTF_INFO_VLEN(info) ((info) & 0xffff)
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
#define BTF_INFO_KFLAG(info) ((info) >> 31)
enum {

2
kernel/bpf/Makefile
View File

@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
endif
CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o const_fold.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o

315
kernel/bpf/arena.c
View File

@@ -53,12 +53,15 @@ struct bpf_arena {
u64 user_vm_start;
u64 user_vm_end;
struct vm_struct *kern_vm;
struct page *scratch_page;
struct range_tree rt;
/* protects rt */
rqspinlock_t spinlock;
struct list_head vma_list;
/* protects vma_list */
struct mutex lock;
u64 zap_gen;
struct mutex zap_mutex;
struct irq_work free_irq;
struct work_struct free_work;
struct llist_head free_spans;
@@ -83,6 +86,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena)
return arena ? arena->user_vm_start : 0;
}
/**
* bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map *
* @map: a BPF_MAP_TYPE_ARENA map
*
* Return @map's kern_vm_start.
*/
u64 bpf_arena_map_kern_vm_start(struct bpf_map *map)
{
return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map));
}
/**
* bpf_prog_arena - return the bpf_map of the arena referenced by @prog
* @prog: a loaded BPF program
*
* The verifier enforces at most one arena per program and stores it in
* prog->aux->arena. Return that arena's underlying bpf_map, or NULL if
* @prog does not reference an arena.
*/
struct bpf_map *bpf_prog_arena(struct bpf_prog *prog)
{
struct bpf_arena *arena = prog->aux->arena;
return arena ? &arena->map : NULL;
}
static long arena_map_peek_elem(struct bpf_map *map, void *value)
{
return -EOPNOTSUPP;
@@ -115,26 +144,57 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)
struct apply_range_data {
struct page **pages;
struct page *scratch_page;
int i;
};
struct clear_range_data {
struct llist_head *free_pages;
struct page *scratch_page;
};
static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
{
struct apply_range_data *d = data;
struct page *page;
pte_t pteval;
if (!data)
return 0;
/* sanity check */
if (unlikely(!pte_none(ptep_get(pte))))
return -EBUSY;
page = d->pages[d->i];
/* paranoia, similar to vmap_pages_pte_range() */
if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
return -EINVAL;
set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL));
pteval = mk_pte(page, PAGE_KERNEL);
#ifdef ptep_try_set
/*
* Kernel-fault recovery may have installed the scratch page here, and
* some architectures (arm64) prohibit valid->valid PTE transitions.
* Install atomically into a none slot. If scratch is present, clear it
* and flush_tlb_before_set() (break-before-make) before retrying.
*/
while (!ptep_try_set(pte, pteval)) {
pte_t old = ptep_get(pte);
if (pte_none(old))
continue;
if (WARN_ON_ONCE(pte_page(old) != d->scratch_page))
return -EBUSY;
ptep_get_and_clear(&init_mm, addr, pte);
flush_tlb_before_set(addr);
}
#else
/*
* Without ptep_try_set() there is no atomic installer, but such arches
* also do not wire up bpf_arena_handle_page_fault(), so no scratch page
* is ever installed and the slot is always none here.
*/
if (unlikely(!pte_none(ptep_get(pte))))
return -EBUSY;
set_pte_at(&init_mm, addr, pte, pteval);
#endif
d->i++;
return 0;
}
@@ -144,33 +204,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size)
flush_cache_vmap(start, start + size);
}
static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages)
static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)
{
struct clear_range_data *d = data;
pte_t old_pte;
struct page *page;
/* sanity check */
old_pte = ptep_get(pte);
/*
* Pairs with ptep_try_set() in the kernel-fault scratch installer.
* Both sides must be atomic.
*/
old_pte = ptep_get_and_clear(&init_mm, addr, pte);
if (pte_none(old_pte) || !pte_present(old_pte))
return 0; /* nothing to do */
return 0;
page = pte_page(old_pte);
if (WARN_ON_ONCE(!page))
return -EINVAL;
pte_clear(&init_mm, addr, pte);
/*
* Skip the per-arena scratch page. A kernel fault on an unallocated uaddr
* scratches its PTE. A later bpf_arena_free_pages() over that range walks
* here. Without the skip, scratch_page would be freed.
*/
if (page == d->scratch_page)
return 0;
/* Add page to the list so it is freed later */
if (free_pages)
__llist_add(&page->pcp_llist, free_pages);
__llist_add(&page->pcp_llist, d->free_pages);
return 0;
}
static int apply_range_set_scratch_cb(pte_t *pte, unsigned long addr, void *data)
{
struct page *scratch_page = data;
if (!pte_none(ptep_get(pte)))
return 0;
/*
* Best-effort install. ptep_try_set() returns false only if another
* installer (real allocation or concurrent fault) won the cmpxchg.
* Their PTE is already valid, so the access retry succeeds.
*
* No flush_tlb_kernel_range() needed. Stale "not mapped" entries just
* cause one extra re-fault through this same path.
*/
ptep_try_set(pte, mk_pte(scratch_page, PAGE_KERNEL));
return 0;
}
static int populate_pgtable_except_pte(struct bpf_arena *arena)
{
/* Populate intermediates for the recovery range (4 GiB + upper half-guard). */
return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL);
SZ_4G + GUARD_SZ / 2, apply_range_set_cb, NULL);
}
static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
@@ -221,22 +307,30 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
init_irq_work(&arena->free_irq, arena_free_irq);
INIT_WORK(&arena->free_work, arena_free_worker);
bpf_map_init_from_attr(&arena->map, attr);
err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page);
if (err)
goto err_free_arena;
range_tree_init(&arena->rt);
err = range_tree_set(&arena->rt, 0, attr->max_entries);
if (err) {
bpf_map_area_free(arena);
goto err;
}
if (err)
goto err_free_scratch;
mutex_init(&arena->lock);
mutex_init(&arena->zap_mutex);
raw_res_spin_lock_init(&arena->spinlock);
err = populate_pgtable_except_pte(arena);
if (err) {
range_tree_destroy(&arena->rt);
bpf_map_area_free(arena);
goto err;
}
if (err)
goto err_destroy_rt;
return &arena->map;
err_destroy_rt:
range_tree_destroy(&arena->rt);
err_free_scratch:
__free_page(arena->scratch_page);
err_free_arena:
bpf_map_area_free(arena);
err:
free_vm_area(kern_vm);
return ERR_PTR(err);
@@ -244,6 +338,7 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data)
{
struct bpf_arena *arena = data;
struct page *page;
pte_t pte;
@@ -251,6 +346,12 @@ static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data)
if (!pte_present(pte)) /* sanity check */
return 0;
page = pte_page(pte);
/*
* Skip the scratch page. The walk is page-table-driven, not range-tree-driven,
* so it can visit scratch PTEs at uaddrs the BPF program never allocated.
*/
if (page == arena->scratch_page)
return 0;
/*
* We do not update pte here:
* 1. Nobody should be accessing bpf_arena's range outside of a kernel bug
@@ -286,9 +387,10 @@ static void arena_map_free(struct bpf_map *map)
* free those pages.
*/
apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena),
KERN_VM_SZ - GUARD_SZ, existing_page_cb, NULL);
SZ_4G + GUARD_SZ / 2, existing_page_cb, arena);
free_vm_area(arena->kern_vm);
range_tree_destroy(&arena->rt);
__free_page(arena->scratch_page);
bpf_map_area_free(arena);
}
@@ -318,6 +420,7 @@ struct vma_list {
struct vm_area_struct *vma;
struct list_head head;
refcount_t mmap_count;
u64 zap_gen;
};
static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma)
@@ -330,6 +433,7 @@ static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma)
refcount_set(&vml->mmap_count, 1);
vma->vm_private_data = vml;
vml->vma = vma;
vml->zap_gen = 0;
list_add(&vml->head, &arena->vma_list);
return 0;
}
@@ -384,33 +488,38 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
return VM_FAULT_RETRY;
page = vmalloc_to_page((void *)kaddr);
if (page)
if (page) {
if (page == arena->scratch_page)
/* BPF triggered scratch here; don't lazy-alloc over it */
goto out_sigsegv;
/* already have a page vmap-ed */
goto out;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)
/* User space requested to segfault when page is not allocated by bpf prog */
goto out_unlock_sigsegv;
goto out_sigsegv_memcg;
ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
if (ret)
goto out_unlock_sigsegv;
goto out_sigsegv_memcg;
struct apply_range_data data = { .pages = &page, .i = 0 };
struct apply_range_data data = { .pages = &page, .i = 0,
.scratch_page = arena->scratch_page };
/* Account into memcg of the process that created bpf_arena */
ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
if (ret) {
range_tree_set(&arena->rt, vmf->pgoff, 1);
goto out_unlock_sigsegv;
goto out_sigsegv_memcg;
}
ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
if (ret) {
range_tree_set(&arena->rt, vmf->pgoff, 1);
free_pages_nolock(page, 0);
goto out_unlock_sigsegv;
goto out_sigsegv_memcg;
}
flush_vmap_cache(kaddr, PAGE_SIZE);
bpf_map_memcg_exit(old_memcg, new_memcg);
@@ -419,8 +528,9 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
vmf->page = page;
return 0;
out_unlock_sigsegv:
out_sigsegv_memcg:
bpf_map_memcg_exit(old_memcg, new_memcg);
out_sigsegv:
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return VM_FAULT_SIGSEGV;
}
@@ -587,6 +697,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
return 0;
}
data.pages = pages;
data.scratch_page = arena->scratch_page;
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
goto out_free_pages;
@@ -668,12 +779,60 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
*/
static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
{
unsigned long size = (unsigned long)page_cnt << PAGE_SHIFT;
struct vm_area_struct *vma;
struct mm_struct *mm;
struct vma_list *vml;
unsigned long vm_start;
u64 my_gen;
guard(mutex)(&arena->lock);
/* iterate link list under lock */
list_for_each_entry(vml, &arena->vma_list, head)
zap_vma_range(vml->vma, uaddr, PAGE_SIZE * page_cnt);
/*
* Taking mmap_read_lock() under arena->lock would deadlock against
* arena_vm_close(), which runs with mmap_write_lock held and then
* acquires arena->lock. Drop arena->lock for mmap_read_lock().
*
* Use per-call my_gen, recorded in vml->zap_gen, to remember which
* vmls this invocation has already processed across the lock drop.
* Hold zap_mutex around the whole walk so concurrent zap_pages()
* callers cannot overwrite each other's marks on shared vmls --
* otherwise call B's mark would make call A skip a vml that A has
* not yet zapped for A's uaddr range.
*/
mutex_lock(&arena->zap_mutex);
mutex_lock(&arena->lock);
my_gen = ++arena->zap_gen;
for (;;) {
mm = NULL;
list_for_each_entry(vml, &arena->vma_list, head) {
if (vml->zap_gen >= my_gen)
continue;
vml->zap_gen = my_gen;
if (!mmget_not_zero(vml->vma->vm_mm))
continue;
mm = vml->vma->vm_mm;
vm_start = vml->vma->vm_start;
break;
}
if (!mm)
break;
mutex_unlock(&arena->lock);
mmap_read_lock(mm);
/*
* Re-resolve: while we waited the VMA could have been unmapped
* and a different mapping installed at the same address.
*/
vma = find_vma(mm, vm_start);
if (vma && vma->vm_start == vm_start &&
vma->vm_file && vma->vm_file->private_data == &arena->map)
zap_vma_range(vma, uaddr, size);
mmap_read_unlock(mm);
mmput(mm);
mutex_lock(&arena->lock);
}
mutex_unlock(&arena->lock);
mutex_unlock(&arena->zap_mutex);
}
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
@@ -685,6 +844,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
struct llist_head free_pages;
struct llist_node *pos, *t;
struct arena_free_span *s;
struct clear_range_data cdata;
unsigned long flags;
int ret = 0;
@@ -713,9 +873,11 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
range_tree_set(&arena->rt, pgoff, page_cnt);
init_llist_head(&free_pages);
cdata.free_pages = &free_pages;
cdata.scratch_page = arena->scratch_page;
/* clear ptes and collect struct pages */
apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
apply_range_clear_cb, &free_pages);
apply_range_clear_cb, &cdata);
/* drop the lock to do the tlb flush and zap pages */
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
@@ -805,6 +967,7 @@ static void arena_free_worker(struct work_struct *work)
struct arena_free_span *s;
u64 arena_vm_start, user_vm_start;
struct llist_head free_pages;
struct clear_range_data cdata;
struct page *page;
unsigned long full_uaddr;
long kaddr, page_cnt, pgoff;
@@ -818,6 +981,8 @@ static void arena_free_worker(struct work_struct *work)
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
init_llist_head(&free_pages);
cdata.free_pages = &free_pages;
cdata.scratch_page = arena->scratch_page;
arena_vm_start = bpf_arena_get_kern_vm_start(arena);
user_vm_start = bpf_arena_get_user_vm_start(arena);
@@ -830,7 +995,7 @@ static void arena_free_worker(struct work_struct *work)
/* clear ptes and collect pages in free_pages llist */
apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
apply_range_clear_cb, &free_pages);
apply_range_clear_cb, &cdata);
range_tree_set(&arena->rt, pgoff, page_cnt);
}
@@ -893,6 +1058,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag
return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false);
}
void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
int node_id, u64 flags)
{
struct bpf_map *map = p__map;
struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt)
return NULL;
return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true);
}
__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt)
{
struct bpf_map *map = p__map;
@@ -945,23 +1123,12 @@ static int __init kfunc_init(void)
}
late_initcall(kfunc_init);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)
static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write,
unsigned long addr, unsigned long fault_ip)
{
struct bpf_stream_stage ss;
struct bpf_prog *prog;
u64 user_vm_start;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it will not
* disappear while we are handling the fault.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(fault_ip);
rcu_read_unlock();
if (!prog)
return;
/* Use main prog for stream access */
prog = prog->aux->main_prog_aux->prog;
@@ -974,3 +1141,53 @@ void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned lo
bpf_stream_dump_stack(ss);
}));
}
bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip)
{
struct bpf_arena *arena;
struct bpf_prog *prog;
unsigned long kbase;
unsigned long page_addr = addr & PAGE_MASK;
prog = bpf_prog_find_from_stack();
if (!prog)
return false;
arena = prog->aux->arena;
/* a prog not using arena may be on stack, so arena can be NULL */
if (!arena)
return false;
kbase = bpf_arena_get_kern_vm_start(arena);
/*
* Recovery covers the 4 GiB mappable band plus the upper half-guard.
* Lower guard is unreachable from kfuncs; an address there indicates
* a different bug class - leave it to the regular kernel oops path.
*/
if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2)
return false;
apply_to_page_range(&init_mm, page_addr, PAGE_SIZE,
apply_range_set_scratch_cb, arena->scratch_page);
flush_vmap_cache(page_addr, PAGE_SIZE);
__bpf_prog_report_arena_violation(prog, is_write, page_addr - kbase, fault_ip);
return true;
}
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip)
{
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it will not
* disappear while we are handling the fault.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(fault_ip);
rcu_read_unlock();
if (!prog)
return;
__bpf_prog_report_arena_violation(prog, write, addr, fault_ip);
}

14
kernel/bpf/arraymap.c
View File

@@ -175,14 +175,12 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key)
return array->value + (u64)array->elem_size * (index & array->index_mask);
}
static int array_map_get_hash(struct bpf_map *map, u32 hash_buf_size,
void *hash_buf)
static int array_map_get_hash(struct bpf_map *map)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
sha256(array->value, (u64)array->elem_size * array->map.max_entries,
hash_buf);
memcpy(array->map.sha, hash_buf, sizeof(array->map.sha));
array->map.sha);
return 0;
}
@@ -386,7 +384,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value,
if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
val = this_cpu_ptr(array->pptrs[index & array->index_mask]);
copy_map_value(map, val, value);
bpf_obj_free_fields(array->map.record, val);
bpf_obj_cancel_fields(map, val);
} else {
val = array->value +
(u64)array->elem_size * (index & array->index_mask);
@@ -394,7 +392,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value,
copy_map_value_locked(map, val, value, false);
else
copy_map_value(map, val, value);
bpf_obj_free_fields(array->map.record, val);
bpf_obj_cancel_fields(map, val);
}
return 0;
}
@@ -434,14 +432,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
cpu = map_flags >> 32;
ptr = per_cpu_ptr(pptr, cpu);
copy_map_value(map, ptr, value);
bpf_obj_free_fields(array->map.record, ptr);
bpf_obj_cancel_fields(map, ptr);
goto unlock;
}
for_each_possible_cpu(cpu) {
ptr = per_cpu_ptr(pptr, cpu);
val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
copy_map_value(map, ptr, val);
bpf_obj_free_fields(array->map.record, ptr);
bpf_obj_cancel_fields(map, ptr);
}
unlock:
rcu_read_unlock();

82
kernel/bpf/backtrack.c
View File

@@ -9,7 +9,7 @@
/* for any branch, call, exit record the history of jmps in the given state */
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
int insn_flags, u64 linked_regs)
int insn_flags, int spi, int frame, u64 linked_regs)
{
u32 cnt = cur->jmp_history_cnt;
struct bpf_jmp_history_entry *p;
@@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
env, "insn history: insn_idx %d cur flags %x new flags %x",
env->insn_idx, env->cur_hist_ent->flags, insn_flags);
env->cur_hist_ent->flags |= insn_flags;
env->cur_hist_ent->spi = spi;
env->cur_hist_ent->frame = frame;
verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
"insn history: insn_idx %d linked_regs: %#llx",
env->insn_idx, env->cur_hist_ent->linked_regs);
@@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->idx = env->insn_idx;
p->prev_idx = env->prev_insn_idx;
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
p->linked_regs = linked_regs;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
(insn->imm & BPF_FETCH);
}
static int insn_stack_access_spi(int insn_flags)
{
return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK;
}
static int insn_stack_access_frameno(int insn_flags)
{
return insn_flags & INSN_F_FRAMENO_MASK;
}
/* Backtrack one insn at a time. If idx is not at the top of recorded
* history then previous instruction came from straight line execution.
* Return -ENOENT if we exhausted all instructions within given state.
@@ -135,11 +129,21 @@ static inline u32 bt_empty(struct backtrack_state *bt)
int i;
for (i = 0; i <= bt->frame; i++)
mask |= bt->reg_masks[i] | bt->stack_masks[i];
mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
return mask == 0;
}
static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
bt->stack_arg_masks[frame] &= ~(1 << slot);
}
static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
{
return bt->stack_arg_masks[frame] & (1 << slot);
}
static inline int bt_subprog_enter(struct backtrack_state *bt)
{
if (bt->frame == MAX_CALL_FRAMES - 1) {
@@ -200,6 +204,11 @@ static inline u64 bt_stack_mask(struct backtrack_state *bt)
return bt->stack_masks[bt->frame];
}
static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
{
return bt->stack_arg_masks[bt->frame];
}
static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
{
return bt->reg_masks[bt->frame] & (1 << reg);
@@ -341,6 +350,19 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return 0;
bt_clear_reg(bt, load_reg);
if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
spi = hist->spi;
/*
* Stack arg read: callee reads from r11+off, but
* the data lives in the caller's stack_arg_regs.
* Set the mask in the caller frame so precision
* is marked in the caller's slot at the callee
* entry checkpoint.
*/
bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
return 0;
}
/* scalars can only be spilled into stack w/o losing precision.
* Load from any other memory can be zero extended.
* The desire to keep that precision is already indicated
@@ -353,8 +375,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* that [fp - off] slot contains scalar that needs to be
* tracked with precision
*/
spi = insn_stack_access_spi(hist->flags);
fr = insn_stack_access_frameno(hist->flags);
spi = hist->spi;
fr = hist->frame;
bpf_bt_set_frame_slot(bt, fr, spi);
} else if (class == BPF_STX || class == BPF_ST) {
if (bt_is_reg_set(bt, dreg))
@@ -363,11 +385,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* encountered a case of pointer subtraction.
*/
return -ENOTSUPP;
if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
spi = hist->spi;
if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
return 0;
bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
if (class == BPF_STX)
bt_set_reg(bt, sreg);
return 0;
}
/* scalars can only be spilled into stack */
if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
return 0;
spi = insn_stack_access_spi(hist->flags);
fr = insn_stack_access_frameno(hist->flags);
spi = hist->spi;
fr = hist->frame;
if (!bt_is_frame_slot_set(bt, fr, spi))
return 0;
bt_clear_frame_slot(bt, fr, spi);
@@ -431,6 +464,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
}
}
if (bt_stack_arg_mask(bt)) {
verifier_bug(env,
"static subprog leftover stack arg slots %x",
bt_stack_arg_mask(bt));
return -EFAULT;
}
if (bt_subprog_exit(bt))
return -EFAULT;
return 0;
@@ -901,6 +940,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
*changed = true;
}
}
for (i = 0; i < func->out_stack_arg_cnt; i++) {
if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
continue;
reg = &func->stack_arg_regs[i];
if (reg->type != SCALAR_VALUE || reg->precise) {
bt_clear_frame_stack_arg_slot(bt, fr, i);
} else {
reg->precise = true;
*changed = true;
}
}
if (env->log.level & BPF_LOG_LEVEL2) {
fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
bt_frame_reg_mask(bt, fr));

165
kernel/bpf/bpf_lru_list.c
View File

@@ -13,23 +13,8 @@
#define PERCPU_FREE_TARGET (4)
#define PERCPU_NR_SCANS PERCPU_FREE_TARGET
/* Helpers to get the local list index */
#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET)
#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE)
#define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING)
#define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET)
/* Local list helpers */
static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l)
{
return &loc_l->lists[LOCAL_FREE_LIST_IDX];
}
static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l)
{
return &loc_l->lists[LOCAL_PENDING_LIST_IDX];
}
/* bpf_lru_node helpers */
static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node)
{
@@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l,
bpf_lru_list_count_dec(l, node->type);
node->type = tgt_free_type;
WRITE_ONCE(node->pending_free, 0);
list_move(&node->list, free_list);
}
@@ -87,6 +73,9 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l,
bpf_lru_list_count_inc(l, tgt_type);
node->type = tgt_type;
bpf_lru_node_clear_ref(node);
/* Reset pending_free only when moving to the free list */
if (tgt_type == BPF_LRU_LIST_T_FREE)
WRITE_ONCE(node->pending_free, 0);
list_move(&node->list, &l->lists[tgt_type]);
}
@@ -212,9 +201,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru,
unsigned int i = 0;
list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) {
if (bpf_lru_node_is_ref(node)) {
if (bpf_lru_node_is_ref(node) &&
!READ_ONCE(node->pending_free)) {
__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE);
} else if (lru->del_from_htab(lru->del_arg, node)) {
} else if (READ_ONCE(node->pending_free) ||
lru->del_from_htab(lru->del_arg, node)) {
__bpf_lru_node_move_to_free(l, node, free_list,
tgt_free_type);
if (++nshrinked == tgt_nshrink)
@@ -273,7 +264,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru,
list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list,
list) {
if (lru->del_from_htab(lru->del_arg, node)) {
if (READ_ONCE(node->pending_free) ||
lru->del_from_htab(lru->del_arg, node)) {
__bpf_lru_node_move_to_free(l, node, free_list,
tgt_free_type);
return 1;
@@ -290,8 +282,10 @@ static void __local_list_flush(struct bpf_lru_list *l,
struct bpf_lru_node *node, *tmp_node;
list_for_each_entry_safe_reverse(node, tmp_node,
local_pending_list(loc_l), list) {
if (bpf_lru_node_is_ref(node))
&loc_l->pending_list, list) {
if (READ_ONCE(node->pending_free))
__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE);
else if (bpf_lru_node_is_ref(node))
__bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE);
else
__bpf_lru_node_move_in(l, node,
@@ -307,9 +301,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l,
if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
return;
raw_spin_lock_irqsave(&l->lock, flags);
if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
WRITE_ONCE(node->pending_free, 1);
return;
}
__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
raw_spin_unlock_irqrestore(&l->lock, flags);
raw_res_spin_unlock_irqrestore(&l->lock, flags);
}
static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
@@ -318,8 +315,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
struct bpf_lru_list *l = &lru->common_lru.lru_list;
struct bpf_lru_node *node, *tmp_node;
unsigned int nfree = 0;
LIST_HEAD(tmp_free);
raw_spin_lock(&l->lock);
if (raw_res_spin_lock(&l->lock))
return;
__local_list_flush(l, loc_l);
@@ -327,7 +326,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE],
list) {
__bpf_lru_node_move_to_free(l, node, local_free_list(loc_l),
__bpf_lru_node_move_to_free(l, node, &tmp_free,
BPF_LRU_LOCAL_LIST_T_FREE);
if (++nfree == lru->target_free)
break;
@@ -335,10 +334,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru,
if (nfree < lru->target_free)
__bpf_lru_list_shrink(lru, l, lru->target_free - nfree,
local_free_list(loc_l),
&tmp_free,
BPF_LRU_LOCAL_LIST_T_FREE);
raw_spin_unlock(&l->lock);
raw_res_spin_unlock(&l->lock);
/*
* Transfer the harvested nodes from the temporary list_head into
* the lockless per-CPU free llist.
*/
list_for_each_entry_safe(node, tmp_node, &tmp_free, list) {
list_del(&node->list);
llist_add(&node->llist, &loc_l->free_llist);
}
}
static void __local_list_add_pending(struct bpf_lru *lru,
@@ -350,22 +358,21 @@ static void __local_list_add_pending(struct bpf_lru *lru,
*(u32 *)((void *)node + lru->hash_offset) = hash;
node->cpu = cpu;
node->type = BPF_LRU_LOCAL_LIST_T_PENDING;
WRITE_ONCE(node->pending_free, 0);
bpf_lru_node_clear_ref(node);
list_add(&node->list, local_pending_list(loc_l));
list_add(&node->list, &loc_l->pending_list);
}
static struct bpf_lru_node *
__local_list_pop_free(struct bpf_lru_locallist *loc_l)
{
struct bpf_lru_node *node;
struct llist_node *llnode;
node = list_first_entry_or_null(local_free_list(loc_l),
struct bpf_lru_node,
list);
if (node)
list_del(&node->list);
llnode = llist_del_first(&loc_l->free_llist);
if (!llnode)
return NULL;
return node;
return container_of(llnode, struct bpf_lru_node, llist);
}
static struct bpf_lru_node *
@@ -376,10 +383,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l)
ignore_ref:
/* Get from the tail (i.e. older element) of the pending list. */
list_for_each_entry_reverse(node, local_pending_list(loc_l),
list) {
list_for_each_entry_reverse(node, &loc_l->pending_list, list) {
if ((!bpf_lru_node_is_ref(node) || force) &&
lru->del_from_htab(lru->del_arg, node)) {
(READ_ONCE(node->pending_free) ||
lru->del_from_htab(lru->del_arg, node))) {
list_del(&node->list);
return node;
}
@@ -404,7 +411,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru,
l = per_cpu_ptr(lru->percpu_lru, cpu);
raw_spin_lock_irqsave(&l->lock, flags);
if (raw_res_spin_lock_irqsave(&l->lock, flags))
return NULL;
__bpf_lru_list_rotate(lru, l);
@@ -420,7 +428,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru,
__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE);
}
raw_spin_unlock_irqrestore(&l->lock, flags);
raw_res_spin_unlock_irqrestore(&l->lock, flags);
return node;
}
@@ -437,7 +445,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
loc_l = per_cpu_ptr(clru->local_list, cpu);
raw_spin_lock_irqsave(&loc_l->lock, flags);
if (raw_res_spin_lock_irqsave(&loc_l->lock, flags))
return NULL;
node = __local_list_pop_free(loc_l);
if (!node) {
@@ -448,17 +457,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
if (node)
__local_list_add_pending(lru, loc_l, cpu, node, hash);
raw_spin_unlock_irqrestore(&loc_l->lock, flags);
raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
if (node)
return node;
/* No free nodes found from the local free list and
/*
* No free nodes found from the local free list and
* the global LRU list.
*
* Steal from the local free/pending list of the
* current CPU and remote CPU in RR. It starts
* with the loc_l->next_steal CPU.
*
* Acquire the victim's lock before touching either list. On
* acquisition failure (rqspinlock AA or timeout) skip the victim
* and try the next CPU.
*/
first_steal = loc_l->next_steal;
@@ -466,24 +480,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru,
do {
steal_loc_l = per_cpu_ptr(clru->local_list, steal);
raw_spin_lock_irqsave(&steal_loc_l->lock, flags);
node = __local_list_pop_free(steal_loc_l);
if (!node)
node = __local_list_pop_pending(lru, steal_loc_l);
raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) {
node = __local_list_pop_free(steal_loc_l);
if (!node)
node = __local_list_pop_pending(lru, steal_loc_l);
raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags);
}
steal = cpumask_next_wrap(steal, cpu_possible_mask);
} while (!node && steal != first_steal);
loc_l->next_steal = steal;
if (node) {
raw_spin_lock_irqsave(&loc_l->lock, flags);
__local_list_add_pending(lru, loc_l, cpu, node, hash);
raw_spin_unlock_irqrestore(&loc_l->lock, flags);
if (!node)
return NULL;
if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
/*
* The local pending lock can't be acquired (rqspinlock AA
* or timeout). Return the stolen node to the per-CPU
* free_llist instead of orphaning it; the next pop_free on
* this CPU will pick it up.
*/
node->type = BPF_LRU_LOCAL_LIST_T_FREE;
bpf_lru_node_clear_ref(node);
WRITE_ONCE(node->pending_free, 0);
llist_add(&node->llist, &loc_l->free_llist);
return NULL;
}
__local_list_add_pending(lru, loc_l, cpu, node, hash);
raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
return node;
}
@@ -511,18 +537,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru,
loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu);
raw_spin_lock_irqsave(&loc_l->lock, flags);
if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) {
WRITE_ONCE(node->pending_free, 1);
return;
}
if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
raw_spin_unlock_irqrestore(&loc_l->lock, flags);
raw_res_spin_unlock_irqrestore(&loc_l->lock,
flags);
goto check_lru_list;
}
node->type = BPF_LRU_LOCAL_LIST_T_FREE;
bpf_lru_node_clear_ref(node);
list_move(&node->list, local_free_list(loc_l));
list_del(&node->list);
raw_spin_unlock_irqrestore(&loc_l->lock, flags);
raw_res_spin_unlock_irqrestore(&loc_l->lock, flags);
llist_add(&node->llist, &loc_l->free_llist);
return;
}
@@ -538,11 +570,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru,
l = per_cpu_ptr(lru->percpu_lru, node->cpu);
raw_spin_lock_irqsave(&l->lock, flags);
if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
WRITE_ONCE(node->pending_free, 1);
return;
}
__bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE);
raw_spin_unlock_irqrestore(&l->lock, flags);
raw_res_spin_unlock_irqrestore(&l->lock, flags);
}
void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node)
@@ -565,6 +600,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf,
node = (struct bpf_lru_node *)(buf + node_offset);
node->type = BPF_LRU_LIST_T_FREE;
node->pending_free = 0;
bpf_lru_node_clear_ref(node);
list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
buf += elem_size;
@@ -594,6 +630,7 @@ static void bpf_percpu_lru_populate(struct bpf_lru *lru, void *buf,
node = (struct bpf_lru_node *)(buf + node_offset);
node->cpu = cpu;
node->type = BPF_LRU_LIST_T_FREE;
node->pending_free = 0;
bpf_lru_node_clear_ref(node);
list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]);
i++;
@@ -618,14 +655,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset,
static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu)
{
int i;
for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++)
INIT_LIST_HEAD(&loc_l->lists[i]);
INIT_LIST_HEAD(&loc_l->pending_list);
init_llist_head(&loc_l->free_llist);
loc_l->next_steal = cpu;
raw_spin_lock_init(&loc_l->lock);
raw_res_spin_lock_init(&loc_l->lock);
}
static void bpf_lru_list_init(struct bpf_lru_list *l)
@@ -640,7 +675,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l)
l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE];
raw_spin_lock_init(&l->lock);
raw_res_spin_lock_init(&l->lock);
}
int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset,

25
kernel/bpf/bpf_lru_list.h
View File

@@ -6,11 +6,11 @@
#include <linux/cache.h>
#include <linux/list.h>
#include <linux/spinlock_types.h>
#include <linux/llist.h>
#include <asm/rqspinlock.h>
#define NR_BPF_LRU_LIST_T (3)
#define NR_BPF_LRU_LIST_COUNT (2)
#define NR_BPF_LRU_LOCAL_LIST_T (2)
#define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T
enum bpf_lru_list_type {
@@ -22,10 +22,22 @@ enum bpf_lru_list_type {
};
struct bpf_lru_node {
struct list_head list;
/*
* A node is in at most one list at a time. The free path on the
* per-CPU locallist uses an llist, so share storage via a union.
*/
union {
struct list_head list;
struct llist_node llist;
};
u16 cpu;
u8 type;
u8 ref;
/*
* Marks nodes whose *_push_free() lock acquire failed; reclaimed
* by flush/shrink which honor the flag instead of del_from_htab().
*/
u8 pending_free;
};
struct bpf_lru_list {
@@ -34,13 +46,14 @@ struct bpf_lru_list {
/* The next inactive list rotation starts from here */
struct list_head *next_inactive_rotation;
raw_spinlock_t lock ____cacheline_aligned_in_smp;
rqspinlock_t lock ____cacheline_aligned_in_smp;
};
struct bpf_lru_locallist {
struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T];
struct list_head pending_list;
struct llist_head free_llist;
u16 next_steal;
raw_spinlock_t lock;
rqspinlock_t lock;
};
struct bpf_common_lru {

20
kernel/bpf/bpf_lsm.c
View File

@@ -427,6 +427,26 @@ BTF_ID(func, bpf_lsm_audit_rule_known)
BTF_ID(func, bpf_lsm_inode_xattr_skipcap)
BTF_SET_END(bool_lsm_hooks)
/* hooks returning void */
#define LSM_HOOK_void(DEFAULT, NAME, ...) BTF_ID(func, bpf_lsm_##NAME)
#define LSM_HOOK_int(DEFAULT, NAME, ...) /* nothing */
#define LSM_HOOK(RET, DEFAULT, NAME, ...) LSM_HOOK_##RET(DEFAULT, NAME, __VA_ARGS__)
BTF_SET_START(void_lsm_hooks)
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
#undef LSM_HOOK_void
#undef LSM_HOOK_int
BTF_SET_END(void_lsm_hooks)
bool bpf_lsm_hook_returns_errno(u32 btf_id)
{
if (btf_id_set_contains(&bool_lsm_hooks, btf_id))
return false;
if (btf_id_set_contains(&void_lsm_hooks, btf_id))
return false;
return true;
}
int bpf_lsm_get_retval_range(const struct bpf_prog *prog,
struct bpf_retval_range *retval_range)
{

63
kernel/bpf/bpf_struct_ops.c
View File

@@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = {
.dealloc = bpf_struct_ops_link_dealloc,
};
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,
struct bpf_tramp_node *node,
const struct btf_func_model *model,
void *stub_func,
void **_image, u32 *_image_off,
@@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
void *image = *_image;
int size;
tlinks[BPF_TRAMP_FENTRY].links[0] = link;
tlinks[BPF_TRAMP_FENTRY].nr_links = 1;
tnodes[BPF_TRAMP_FENTRY].nodes[0] = node;
tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1;
if (model->ret_size > 0)
flags |= BPF_TRAMP_F_RET_FENTRY_RET;
size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func);
size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func);
if (size <= 0)
return size ? : -EFAULT;
@@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
size = arch_prepare_bpf_trampoline(NULL, image + image_off,
image + image_off + size,
model, flags, tlinks, stub_func);
model, flags, tnodes, stub_func);
if (size <= 0) {
if (image != *_image)
bpf_struct_ops_image_free(image);
@@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
const struct btf_type *module_type;
const struct btf_member *member;
const struct btf_type *t = st_ops_desc->type;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_nodes *tnodes;
void *udata, *kdata;
int prog_fd, err;
u32 i, trampoline_start, image_off = 0;
@@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
if (uvalue->common.state || refcount_read(&uvalue->common.refcnt))
return -EINVAL;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks)
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes)
return -ENOMEM;
uvalue = (struct bpf_struct_ops_value *)st_map->uvalue;
@@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
err = -ENOMEM;
goto reset_unlock;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS,
&bpf_struct_ops_link_lops, prog, prog->expected_attach_type);
bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS,
&bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0);
*plink++ = &link->link;
/* Poison pointer on error instead of return for backward compatibility */
@@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
*pksym++ = ksym;
trampoline_start = image_off;
err = bpf_struct_ops_prepare_trampoline(tlinks, link,
err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
&st_ops->func_models[i],
*(void **)(st_ops->cfi_stubs + moff),
&image, &image_off,
@@ -911,7 +912,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
memset(uvalue, 0, map->value_size);
memset(kvalue, 0, map->value_size);
unlock:
kfree(tlinks);
kfree(tnodes);
mutex_unlock(&st_map->lock);
if (!err)
bpf_struct_ops_map_add_ksyms(st_map);
@@ -1204,6 +1205,42 @@ u32 bpf_struct_ops_id(const void *kdata)
}
EXPORT_SYMBOL_GPL(bpf_struct_ops_id);
/**
* bpf_struct_ops_for_each_prog - Invoke @cb for each member prog
* @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg)
* @cb: callback invoked once per member prog; non-zero return stops iteration
* @data: opaque argument passed to @cb
*
* Walks the struct_ops member progs registered on the map containing @kdata.
* Intended for use from struct_ops ->reg() callbacks (and similar) that need to
* inspect the loaded BPF programs (for example to discover maps they reference
* via @prog->aux->used_maps).
*
* Return 0 if iteration completed, otherwise the first non-zero @cb return.
*/
int bpf_struct_ops_for_each_prog(const void *kdata,
int (*cb)(struct bpf_prog *prog, void *data),
void *data)
{
struct bpf_struct_ops_value *kvalue;
struct bpf_struct_ops_map *st_map;
u32 i;
int ret;
kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
for (i = 0; i < st_map->funcs_cnt; i++) {
if (!st_map->links[i])
continue;
ret = cb(st_map->links[i]->prog, data);
if (ret)
return ret;
}
return 0;
}
EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog);
static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;

318
kernel/bpf/btf.c
View File

@@ -182,7 +182,6 @@
#define BITS_ROUNDUP_BYTES(bits) \
(BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
#define BTF_INFO_MASK 0x9f00ffff
#define BTF_INT_MASK 0x0fffffff
#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
@@ -289,7 +288,7 @@ enum verifier_phase {
struct resolve_vertex {
const struct btf_type *t;
u32 type_id;
u16 next_member;
u32 next_member;
};
enum visit_state {
@@ -2031,7 +2030,7 @@ static int env_stack_push(struct btf_verifier_env *env,
}
static void env_stack_set_next_member(struct btf_verifier_env *env,
u16 next_member)
u32 next_member)
{
env->stack[env->top_stack - 1].next_member = next_member;
}
@@ -3293,7 +3292,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env,
struct btf *btf = env->btf;
u32 struct_size = t->size;
u32 offset;
u16 i;
u32 i;
meta_needed = btf_type_vlen(t) * sizeof(*member);
if (meta_left < meta_needed) {
@@ -3369,7 +3368,7 @@ static int btf_struct_resolve(struct btf_verifier_env *env,
{
const struct btf_member *member;
int err;
u16 i;
u32 i;
/* Before continue resolving the next_member,
* ensure the last member is indeed resolved to a
@@ -3668,7 +3667,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_
static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
u32 field_cnt, u32 repeat_cnt, u32 elem_size)
{
u32 i, j;
u32 i, j, total_cnt, total_repeats;
u32 cur;
/* Ensure not repeating fields that should not be repeated. */
@@ -3686,10 +3685,9 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
}
}
/* The type of struct size or variable size is u32,
* so the multiplication will not overflow.
*/
if (field_cnt * (repeat_cnt + 1) > info_cnt)
if (check_add_overflow(repeat_cnt, 1, &total_repeats) ||
check_mul_overflow(field_cnt, total_repeats, &total_cnt) ||
total_cnt > (u32)info_cnt)
return -E2BIG;
cur = field_cnt;
@@ -4447,7 +4445,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env,
const struct btf_enum *enums = btf_type_enum(t);
struct btf *btf = env->btf;
const char *fmt_str;
u16 i, nr_enums;
u32 i, nr_enums;
u32 meta_needed;
nr_enums = btf_type_vlen(t);
@@ -4555,7 +4553,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
const struct btf_enum64 *enums = btf_type_enum64(t);
struct btf *btf = env->btf;
const char *fmt_str;
u16 i, nr_enums;
u32 i, nr_enums;
u32 meta_needed;
nr_enums = btf_type_vlen(t);
@@ -4683,7 +4681,7 @@ static void btf_func_proto_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
const struct btf_param *args = (const struct btf_param *)(t + 1);
u16 nr_args = btf_type_vlen(t), i;
u32 nr_args = btf_type_vlen(t), i;
btf_verifier_log(env, "return=%u args=(", t->type);
if (!nr_args) {
@@ -4929,7 +4927,7 @@ static int btf_datasec_resolve(struct btf_verifier_env *env,
{
const struct btf_var_secinfo *vsi;
struct btf *btf = env->btf;
u16 i;
u32 i;
env->resolve_mode = RESOLVE_TBD;
for_each_vsi_from(i, v->next_member, v->t, vsi) {
@@ -5183,7 +5181,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env,
const struct btf_type *ret_type;
const struct btf_param *args;
const struct btf *btf;
u16 nr_args, i;
u32 nr_args, i;
int err;
btf = env->btf;
@@ -5278,7 +5276,7 @@ static int btf_func_check(struct btf_verifier_env *env,
const struct btf_type *proto_type;
const struct btf_param *args;
const struct btf *btf;
u16 nr_args, i;
u32 nr_args, i;
btf = env->btf;
proto_type = btf_type_by_id(btf, t->type);
@@ -5336,12 +5334,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env,
}
meta_left -= sizeof(*t);
if (t->info & ~BTF_INFO_MASK) {
btf_verifier_log(env, "[%u] Invalid btf_info:%x",
env->log_type_id, t->info);
return -EINVAL;
}
if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
btf_verifier_log(env, "[%u] Invalid kind:%u",
@@ -5914,25 +5906,10 @@ static int btf_check_type_tags(struct btf_verifier_env *env,
return 0;
}
static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)
{
u32 log_true_size;
int err;
err = bpf_vlog_finalize(log, &log_true_size);
if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) &&
copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size),
&log_true_size, sizeof(log_true_size)))
err = -EFAULT;
return err;
}
static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,
struct bpf_log_attr *attr_log)
{
bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel);
char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf);
struct btf_struct_metas *struct_meta_tab;
struct btf_verifier_env *env = NULL;
struct btf *btf = NULL;
@@ -5949,8 +5926,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat
/* user could have requested verbose verifier output
* and supplied buffer to store the verification trace
*/
err = bpf_vlog_init(&env->log, attr->btf_log_level,
log_ubuf, attr->btf_log_size);
err = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size);
if (err)
goto errout_free;
@@ -6015,7 +5991,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat
}
}
err = finalize_log(&env->log, uattr, uattr_size);
err = bpf_log_attr_finalize(attr_log, &env->log);
if (err)
goto errout_free;
@@ -6027,7 +6003,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat
btf_free_struct_meta_tab(btf);
errout:
/* overwrite err with -ENOSPC or -EFAULT */
ret = finalize_log(&env->log, uattr, uattr_size);
ret = bpf_log_attr_finalize(attr_log, &env->log);
if (ret)
err = ret;
errout_free:
@@ -6980,7 +6956,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
info->reg_type = ctx_arg_info->reg_type;
info->btf = ctx_arg_info->btf ? : btf_vmlinux;
info->btf_id = ctx_arg_info->btf_id;
info->ref_obj_id = ctx_arg_info->ref_obj_id;
info->ref_id = ctx_arg_info->ref_id;
return true;
}
}
@@ -7825,6 +7801,134 @@ enum btf_arg_tag {
ARG_TAG_ARENA = BIT_ULL(5),
};
static int btf_scan_decl_tags(struct bpf_verifier_env *env,
const struct btf *btf,
const struct btf_type *fn_t,
u32 arg_idx, bool is_global, u32 *tags)
{
int id = btf_named_start_id(btf, false) - 1;
const char tag_key[] = "arg:";
static const struct {
const char *tag_value;
enum btf_arg_tag arg_tag;
} tag_values[] = {
{ "ctx", ARG_TAG_CTX },
{ "trusted", ARG_TAG_TRUSTED },
{ "untrusted", ARG_TAG_UNTRUSTED },
{ "nonnull", ARG_TAG_NONNULL },
{ "nullable", ARG_TAG_NULLABLE },
{ "arena", ARG_TAG_ARENA },
};
/*
* The 'arg:<tag>' decl_tag takes precedence over the derivation
* of the register type from the BTF type itself.
*/
while ((id = btf_find_next_decl_tag(btf, fn_t, arg_idx, tag_key, id)) > 0) {
const struct btf_type *tag_t;
const char *tag;
int i;
bool found;
/* disallow arg tags in static subprogs */
if (!is_global) {
bpf_log(&env->log,
"arg#%d type tag is not supported in static functions\n",
arg_idx);
return -EOPNOTSUPP;
}
tag_t = btf_type_by_id(btf, id);
tag = __btf_name_by_offset(btf, tag_t->name_off) + (sizeof(tag_key) - 1);
found = false;
for (i = 0; i < ARRAY_SIZE(tag_values); ++i) {
if (!strcmp(tag, tag_values[i].tag_value)) {
*tags |= tag_values[i].arg_tag;
found = true;
break;
}
}
if (!found) {
bpf_log(&env->log, "arg#%d has unsupported set of tags\n", arg_idx);
return -EOPNOTSUPP;
}
}
if (id != -ENOENT) {
bpf_log(&env->log, "arg#%d type tag fetching failure: %d\n", arg_idx, id);
return id;
}
return 0;
}
static int btf_scan_type_tags(struct bpf_verifier_env *env,
const struct btf *btf, u32 type_id,
u32 *tags)
{
const struct btf_type *t;
/* Find the first pointer type in the chain. */
t = btf_type_skip_modifiers(btf, type_id, NULL);
/*
* We currently reject type tags on non-pointer types,
* which neither LLVM nor GCC support anyway.
*/
if (!t || !btf_type_is_ptr(t))
return 0;
/* We got a pointer, get all associated type tags. */
for (t = btf_type_by_id(btf, t->type); t && btf_type_is_modifier(t);
t = btf_type_by_id(btf, t->type)) {
/* Skip non-type tag modifiers. */
if (!btf_type_is_type_tag(t))
continue;
const char *tag = __btf_name_by_offset(btf, t->name_off);
if (strcmp(tag, "arena") == 0) {
*tags |= ARG_TAG_ARENA;
} else {
bpf_log(&env->log, "function signature member has unsupported type tag '%s'\n",
tag);
return -EOPNOTSUPP;
}
}
return 0;
}
/* Check whether the type is a valid return type. */
static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
const struct btf_type *t, int subprog)
{
u32 tags = 0;
int err;
err = btf_scan_type_tags(env, btf, t->type, &tags);
if (err)
return err;
t = btf_type_skip_modifiers(btf, t->type, NULL);
/*
* We allow all subprogs except for the main one to return any kind of arena pointer.
* General arena variables are not allowed, since it makes no sense to return by value
* a variable that's on the heap in the first place.
*/
if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t))
return 0;
/* We always accept void or scalars. */
if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
return 0;
return -EOPNOTSUPP;
}
/* Process BTF of a function to produce high-level expectation of function
* arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information
* is cached in subprog info for reuse.
@@ -7843,6 +7947,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
struct btf *btf = prog->aux->btf;
const struct btf_param *args;
const struct btf_type *t, *ref_t, *fn_t;
int err;
u32 i, nargs, btf_id;
const char *tname;
@@ -7887,25 +7992,36 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
}
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
sub->arg_cnt = nargs;
if (nargs > MAX_BPF_FUNC_ARGS) {
bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
MAX_BPF_FUNC_ARGS, tname, nargs);
return -EFAULT;
}
if (nargs > MAX_BPF_FUNC_REG_ARGS) {
if (!is_global)
return -EINVAL;
bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
if (!bpf_jit_supports_stack_args()) {
bpf_log(log, "JIT does not support function %s() with %d args\n",
tname, nargs);
return -EFAULT;
}
sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
}
if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) {
bpf_log(log, "global function %s has %d > %d args, stack args not supported\n",
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
/* check that function is void or returns int, exception cb also requires this */
t = btf_type_by_id(btf, t->type);
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) {
if (!is_global)
return -EINVAL;
bpf_log(log,
"Global function %s() return value not void or scalar. "
"Only those are supported.\n",
tname);
return -EINVAL;
err = btf_validate_return_type(env, btf, t, subprog);
if (err) {
if (is_global) {
bpf_log(log,
"Global function %s() return value not void or scalar. "
"Only those are supported.\n",
tname);
}
return err;
}
/* Convert BTF function arguments into verifier types.
@@ -7913,42 +8029,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
*/
for (i = 0; i < nargs; i++) {
u32 tags = 0;
int id = btf_named_start_id(btf, false) - 1;
err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags);
if (err)
return err;
/* 'arg:<tag>' decl_tag takes precedence over derivation of
* register type from BTF type itself
*/
while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) {
const struct btf_type *tag_t = btf_type_by_id(btf, id);
const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4;
/* disallow arg tags in static subprogs */
if (!is_global) {
bpf_log(log, "arg#%d type tag is not supported in static functions\n", i);
return -EOPNOTSUPP;
}
if (strcmp(tag, "ctx") == 0) {
tags |= ARG_TAG_CTX;
} else if (strcmp(tag, "trusted") == 0) {
tags |= ARG_TAG_TRUSTED;
} else if (strcmp(tag, "untrusted") == 0) {
tags |= ARG_TAG_UNTRUSTED;
} else if (strcmp(tag, "nonnull") == 0) {
tags |= ARG_TAG_NONNULL;
} else if (strcmp(tag, "nullable") == 0) {
tags |= ARG_TAG_NULLABLE;
} else if (strcmp(tag, "arena") == 0) {
tags |= ARG_TAG_ARENA;
} else {
bpf_log(log, "arg#%d has unsupported set of tags\n", i);
return -EOPNOTSUPP;
}
}
if (id != -ENOENT) {
bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id);
return id;
}
err = btf_scan_type_tags(env, btf, args[i].type, &tags);
if (err)
return err;
t = btf_type_by_id(btf, args[i].type);
while (btf_type_is_modifier(t))
@@ -7973,7 +8060,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY;
sub->args[i].arg_type = ARG_PTR_TO_DYNPTR;
continue;
}
if (tags & ARG_TAG_TRUSTED) {
@@ -8074,7 +8161,6 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return -EINVAL;
}
sub->arg_cnt = nargs;
sub->args_cached = true;
return 0;
@@ -8196,12 +8282,12 @@ static int __btf_new_fd(struct btf *btf)
return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
}
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
{
struct btf *btf;
int ret;
btf = btf_parse(attr, uattr, uattr_size);
btf = btf_parse(attr, uattr, attr_log);
if (IS_ERR(btf))
return PTR_ERR(btf);
@@ -8684,6 +8770,39 @@ static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name,
return 0;
}
static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind)
{
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
struct btf_module *btf_mod, *tmp;
#endif
s32 id;
if (!btf_is_module(btf))
return 0;
id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind);
if (id >= 0) {
pr_err("kfunc %s (id: %d) is already present in vmlinux.\n",
func_name, id);
return -EINVAL;
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
guard(mutex)(&btf_module_mutex);
list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
if (btf_mod->btf == btf)
continue;
id = btf_find_by_name_kind(btf_mod->btf, func_name, kind);
if (id >= 0) {
pr_err("kfunc %s (id: %d) is already present in module %s.\n",
func_name, id, btf_mod->module->name);
return -EINVAL;
}
}
#endif
return 0;
}
static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)
{
const struct btf_type *func;
@@ -8697,7 +8816,8 @@ static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)
/* sanity check kfunc name */
func_name = btf_name_by_offset(btf, func->name_off);
if (!func_name || !func_name[0])
if (!func_name || !func_name[0] ||
btf_check_kfunc_name(btf, func_name, BTF_INFO_KIND(func->info)))
return -EINVAL;
func = btf_type_by_id(btf, func->type);

65
kernel/bpf/cgroup.c
View File

@@ -55,6 +55,28 @@ void __init cgroup_bpf_lifetime_notifier_init(void)
&cgroup_bpf_lifetime_nb));
}
#ifdef CONFIG_BPF_LSM
struct cgroup_lsm_atype {
u32 attach_btf_id;
int refcnt;
bool returns_errno;
};
static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
{
if (atype >= CGROUP_LSM_START && atype <= CGROUP_LSM_END)
return READ_ONCE(cgroup_lsm_atype[atype - CGROUP_LSM_START].returns_errno);
return true;
}
#else
static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype)
{
return true;
}
#endif
/* __always_inline is necessary to prevent indirect call through run_prog
* function pointer.
*/
@@ -83,7 +105,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp,
*(ret_flags) |= (func_ret >> 1);
func_ret &= 1;
}
if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval))
if (!func_ret && cgroup_bpf_hook_returns_errno(atype) &&
!IS_ERR_VALUE((long)run_ctx.retval))
run_ctx.retval = -EPERM;
item++;
}
@@ -156,13 +179,6 @@ unsigned int __cgroup_bpf_run_lsm_current(const void *ctx,
}
#ifdef CONFIG_BPF_LSM
struct cgroup_lsm_atype {
u32 attach_btf_id;
int refcnt;
};
static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM];
static enum cgroup_bpf_attach_type
bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id)
{
@@ -191,10 +207,13 @@ void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype)
lockdep_assert_held(&cgroup_mutex);
WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id &&
cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);
cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;
if (!cgroup_lsm_atype[i].attach_btf_id) {
cgroup_lsm_atype[i].attach_btf_id = attach_btf_id;
WRITE_ONCE(cgroup_lsm_atype[i].returns_errno,
bpf_lsm_hook_returns_errno(attach_btf_id));
} else {
WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id != attach_btf_id);
}
cgroup_lsm_atype[i].refcnt++;
}
@@ -203,8 +222,10 @@ void bpf_cgroup_atype_put(int cgroup_atype)
int i = cgroup_atype - CGROUP_LSM_START;
cgroup_lock();
if (--cgroup_lsm_atype[i].refcnt <= 0)
if (--cgroup_lsm_atype[i].refcnt <= 0) {
WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, true);
cgroup_lsm_atype[i].attach_btf_id = 0;
}
WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0);
cgroup_unlock();
}
@@ -1208,7 +1229,7 @@ static int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
/* Must be called with cgroup_mutex held to avoid races. */
static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
union bpf_attr __user *uattr)
union bpf_attr __user *uattr, u32 uattr_size)
{
__u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags);
bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE;
@@ -1259,7 +1280,8 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
return -EFAULT;
if (!effective_query && from_atype == to_atype)
revision = cgrp->bpf.revisions[from_atype];
if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
if (uattr_size >= offsetofend(union bpf_attr, query.revision) &&
copy_to_user(&uattr->query.revision, &revision, sizeof(revision)))
return -EFAULT;
if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt)
/* return early if user requested only program count + flags */
@@ -1312,12 +1334,12 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
}
static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
union bpf_attr __user *uattr)
union bpf_attr __user *uattr, u32 uattr_size)
{
int ret;
cgroup_lock();
ret = __cgroup_bpf_query(cgrp, attr, uattr);
ret = __cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
cgroup_unlock();
return ret;
}
@@ -1520,7 +1542,7 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
}
int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
union bpf_attr __user *uattr, u32 uattr_size)
{
struct cgroup *cgrp;
int ret;
@@ -1529,7 +1551,7 @@ int cgroup_bpf_prog_query(const union bpf_attr *attr,
if (IS_ERR(cgrp))
return PTR_ERR(cgrp);
ret = cgroup_bpf_query(cgrp, attr, uattr);
ret = cgroup_bpf_query(cgrp, attr, uattr, uattr_size);
cgroup_put(cgrp);
return ret;
@@ -1935,8 +1957,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head,
kfree(ctx.cur_val);
if (ret == 1 && ctx.new_updated) {
kfree(*buf);
if (!ret && ctx.new_updated) {
kvfree(*buf);
*buf = ctx.new_val;
*pcount = ctx.new_len;
} else {
@@ -2342,6 +2364,7 @@ BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx,
return -E2BIG;
memcpy(ctx->new_val, buf, buf_len);
((char *)ctx->new_val)[buf_len] = '\0';
ctx->new_len = buf_len;
ctx->new_updated = 1;

120
kernel/bpf/cnum.c Normal file
View File

@@ -0,0 +1,120 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bits.h>
#define T 32
#include "cnum_defs.h"
#undef T
#define T 64
#include "cnum_defs.h"
#undef T
struct cnum32 cnum32_from_cnum64(struct cnum64 cnum)
{
if (cnum64_is_empty(cnum))
return CNUM32_EMPTY;
if (cnum.size >= U32_MAX)
return (struct cnum32){ .base = 0, .size = U32_MAX };
else
return (struct cnum32){ .base = (u32)cnum.base, .size = cnum.size };
}
/*
* Suppose 'a' and 'b' are laid out as follows:
*
* 64-bit number axis --->
*
* N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32
* ||------|---|=====|-------||----------|=====|-------||----------|=====|----|--||
* | |< b >| |< b >| |< b >| |
* | | | |
* |<--+--------------------------- a ---------------------------+--->|
* | |
* |<-------------------------- t -------------------------->|
*
* In such a case it is possible to infer a more tight representation t
* such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t.
*/
struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b)
{
/*
* To simplify reasoning, rotate the circles so that [virtual] a1 starts
* at u32 boundary, b1 represents b in this new frame of reference.
*/
struct cnum32 b1 = { b.base - (u32)a.base, b.size };
struct cnum64 t = a;
u64 d, b1_max;
if (cnum64_is_empty(a) || cnum32_is_empty(b))
return CNUM64_EMPTY;
if (cnum32_urange_overflow(b1)) {
b1_max = (u32)b1.base + (u32)b1.size; /* overflow here is fine and necessary */
if ((u32)a.size > b1_max && (u32)a.size < b1.base) {
/*
* N*2^32 (N+1)*2^32
* ||=====|------------|=====||=====|---------|---|=====||
* |b1 ->| |<- b1||b1 ->| | |<- b1|
* |<----------------- a1 ------------------>|
* |<-------------- t ------------>|<-- d -->| (after adjustment)
* ^
* b1_max
*/
d = (u32)a.size - b1_max;
t.size -= d;
} else {
/*
* No adjustments possible in the following cases:
*
* ||=====|------------|=====||===|=|-------------|=|===||
* |b1 ->| |<- b1||b1 +>| |<+ b1|
* |<----------------- a1 ------>| |
* |<----------------- (or) a1 ------------------->|
*/
}
} else {
if (t.size < b1.base)
/*
* N*2^32 (N+1)*2^32
* ||----------|--|=======|--||------>
* |<-- a1 -->| |<- b ->|
*/
return CNUM64_EMPTY;
/*
* N*2^32 (N+1)*2^32
* ||-------------|========|-||-----| -------|========|-||
* | |<- b1 ->| | |<- b1 ->|
* |<------------+ a1 ------------>|
* |<------ t ------>| (after adjustment)
*/
t.base += b1.base;
t.size -= b1.base;
b1_max = b1.base + b1.size;
d = 0;
if ((u32)a.size < b1.base)
/*
* N*2^32 (N+1)*2^32
* ||-------------|========|-||------|-------|========|-||
* | |<- b1 ->| | |<- b1 ->|
* |<------------+-- a1 --+-------->|
* |<- t ->|<-- d -->| (after adjustment)
*/
d = (u32)a.size + (BIT_ULL(32) - b1_max);
else if ((u32)a.size >= b1_max)
/*
* N*2^32 (N+1)*2^32
* ||--|========|------------||--|========|-------|-----||
* | |<- b1 ->| |<- b1 ->| |
* |<-+------------------ a1 ------------+------>|
* |<-------------- t --------------->|<- d ->| (after adjustment)
*/
d = (u32)a.size - b1_max;
if (t.size < d)
return CNUM64_EMPTY;
t.size -= d;
}
return t;
}

247
kernel/bpf/cnum_defs.h Normal file
View File

@@ -0,0 +1,247 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#ifndef T
#error "Define T (bit width: 32, 64) before including cnum_defs.h"
#endif
#include <linux/cnum.h>
#include <linux/kernel.h>
#include <linux/limits.h>
#include <linux/minmax.h>
#include <linux/compiler_types.h>
#define cnum_t __PASTE(cnum, T)
#define ut __PASTE(u, T)
#define st __PASTE(s, T)
#define UT_MAX __PASTE(__PASTE(U, T), _MAX)
#define ST_MAX __PASTE(__PASTE(S, T), _MAX)
#define ST_MIN __PASTE(__PASTE(S, T), _MIN)
#define EMPTY __PASTE(__PASTE(CNUM, T), _EMPTY)
#define FN(name) __PASTE(__PASTE(cnum, T), __PASTE(_, name))
struct cnum_t FN(from_urange)(ut min, ut max)
{
return (struct cnum_t){ .base = min, .size = (ut)max - min };
}
struct cnum_t FN(from_srange)(st min, st max)
{
ut size = (ut)max - (ut)min;
ut base = size == UT_MAX ? 0 : (ut)min;
return (struct cnum_t){ .base = base, .size = size };
}
/* True if this cnum represents two unsigned ranges. */
static inline bool FN(urange_overflow)(struct cnum_t cnum)
{
/* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */
return cnum.size > UT_MAX - (ut)cnum.base;
}
/*
* cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum.
* If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range
* [0..UT_MAX] is returned.
*/
ut FN(umin)(struct cnum_t cnum)
{
return FN(urange_overflow)(cnum) ? 0 : cnum.base;
}
EXPORT_SYMBOL_GPL(FN(umin));
ut FN(umax)(struct cnum_t cnum)
{
return FN(urange_overflow)(cnum) ? UT_MAX : cnum.base + cnum.size;
}
EXPORT_SYMBOL_GPL(FN(umax));
/* True if this cnum represents two signed ranges. */
static inline bool FN(srange_overflow)(struct cnum_t cnum)
{
return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN);
}
/*
* cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum.
* If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range
* [ST_MIN..ST_MAX] is returned.
*/
st FN(smin)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MIN
: min((st)cnum.base, (st)(cnum.base + cnum.size));
}
st FN(smax)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MAX
: max((st)cnum.base, (st)(cnum.base + cnum.size));
}
/*
* Returns a possibly empty intersection of cnums 'a' and 'b'.
* If 'a' and 'b' intersect in two sub-arcs, the function over-approximates
* and returns either 'a' or 'b', whichever is smaller.
*/
struct cnum_t FN(intersect)(struct cnum_t a, struct cnum_t b)
{
struct cnum_t b1;
ut dbase;
if (FN(is_empty)(a) || FN(is_empty)(b))
return EMPTY;
if (a.base > b.base)
swap(a, b);
/*
* Rotate frame of reference such that a.base is 0.
* 'b1' is 'b' in this frame of reference.
*/
dbase = b.base - a.base;
b1 = (struct cnum_t){ dbase, b.size };
if (FN(urange_overflow)(b1)) {
if (b1.base <= a.size) {
/*
* Rotated frame (a.base at origin):
*
* 0 UT_MAX
* |--------------------------------------------|
* [=== a ==========================] |
* [= b1 tail =] [========= b1 main ==========>]
* ^-- b1.base <= a.size
*
* 'a' and 'b' intersect in two disjoint arcs,
* can't represent as single cnum, over-approximate
* the result.
*/
return a.size <= b.size ? a : b;
} else {
/*
* Rotated frame (a.base at origin):
*
* 0 UT_MAX
* |--------------------------------------------|
* [=== a =============] | |
* [= b1 tail =] [======= b1 main ====>]
* ^-- b1.base > a.size
*
* Only 'b' tail intersects 'a'.
*/
return (struct cnum_t) {
.base = a.base,
.size = min(a.size, (ut)(b1.base + b1.size)),
};
}
} else if (a.size >= b1.base) {
/*
* Rotated frame (a.base at origin):
*
* 0 UT_MAX
* |--------------------------------------------------|
* [=== a ==================================] |
* [== b1 =====================]
*
* 0 UT_MAX
* |--------------------------------------------------|
* [=== a ==================================] |
* [== b1 ====]
* ^-- b1.base <= a.size
* |<-- a.size - dbase -->|
*
* 'a' and 'b' intersect as one cnum.
*/
return (struct cnum_t) {
.base = b.base,
.size = min((ut)(a.size - dbase), b.size),
};
} else {
return EMPTY;
}
}
void FN(intersect_with)(struct cnum_t *dst, struct cnum_t src)
{
*dst = FN(intersect)(*dst, src);
}
void FN(intersect_with_urange)(struct cnum_t *dst, ut min, ut max)
{
FN(intersect_with)(dst, FN(from_urange)(min, max));
}
void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max)
{
FN(intersect_with)(dst, FN(from_srange)(min, max));
}
static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
{
if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX)
cnum.base = 0;
return cnum;
}
struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b)
{
if (FN(is_empty)(a) || FN(is_empty)(b))
return EMPTY;
if (a.size > UT_MAX - b.size)
return (struct cnum_t){ 0, (ut)UT_MAX };
else
return FN(normalize)((struct cnum_t){ a.base + b.base, a.size + b.size });
}
struct cnum_t FN(negate)(struct cnum_t a)
{
if (FN(is_empty)(a))
return EMPTY;
return FN(normalize)((struct cnum_t){ -((ut)a.base + a.size), a.size });
}
bool FN(is_empty)(struct cnum_t cnum)
{
return cnum.base == EMPTY.base && cnum.size == EMPTY.size;
}
bool FN(contains)(struct cnum_t cnum, ut v)
{
if (FN(is_empty)(cnum))
return false;
if (FN(urange_overflow)(cnum))
return v >= cnum.base || v <= (ut)cnum.base + cnum.size;
else
return v >= cnum.base && v <= (ut)cnum.base + cnum.size;
}
bool FN(is_const)(struct cnum_t cnum)
{
return cnum.size == 0;
}
bool FN(is_subset)(struct cnum_t bigger, struct cnum_t smaller)
{
if (FN(is_empty(smaller)))
return true;
if (FN(is_empty(bigger)))
return false;
/* rotate both arcs such that 'bigger' starts at origin, hence does not overflow */
smaller.base -= bigger.base;
bigger.base = 0;
if (FN(urange_overflow)(smaller) && bigger.size < UT_MAX)
return false;
return smaller.base + smaller.size <= bigger.size;
}
#undef EMPTY
#undef cnum_t
#undef ut
#undef st
#undef UT_MAX
#undef ST_MAX
#undef ST_MIN
#undef FN

8
kernel/bpf/const_fold.c
View File

@@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code);
int r;
/* Stack arg stores (r11-based) are outside the tracked register set. */
if (is_stack_arg_st(insn) || is_stack_arg_stx(insn))
return;
if (is_stack_arg_ldx(insn)) {
ci_out[insn->dst_reg] = unknown;
return;
}
switch (class) {
case BPF_ALU:
case BPF_ALU64:

27
kernel/bpf/core.c
View File

@@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
u32 imm_rnd = get_random_u32();
s16 off;
BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
/* Constraints on AX register:
*
@@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc
insn_idx += prog->aux->subprog_start;
return env->insn_aux_data[insn_idx].indirect_target;
}
u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog)
{
const struct bpf_subprog_info *sub;
if (!env)
return 0;
sub = &env->subprog_info[prog->aux->func_idx];
return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub);
}
#endif /* CONFIG_BPF_JIT */
/* Base function for offset calculation. Needs to go into .text section,
@@ -2471,7 +2481,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map,
cookie = aux->cgroup_storage[i] ?
aux->cgroup_storage[i]->cookie : 0;
ret = map->owner->storage_cookie[i] == cookie ||
!cookie;
(!cookie && !aux->tail_call_reachable);
}
if (ret &&
map->owner->attach_func_proto != aux->attach_func_proto) {
@@ -3228,6 +3238,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
return false;
}
bool __weak bpf_jit_supports_stack_args(void)
{
return false;
}
bool __weak bpf_jit_supports_far_kfunc_call(void)
{
return false;
@@ -3363,6 +3378,12 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)
}
#ifdef CONFIG_BPF_SYSCALL
__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
unsigned long fault_ip)
{
return false;
}
static int __init bpf_global_ma_init(void)
{
int ret;

19
kernel/bpf/devmap.c
View File

@@ -581,6 +581,10 @@ static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj,
{
struct xdp_frame *nxdpf;
/* Frags live outside the linear frame and cannot be cloned safely. */
if (unlikely(xdp_frame_has_frags(xdpf)))
return -EOPNOTSUPP;
nxdpf = xdpf_clone(xdpf);
if (!nxdpf)
return -ENOMEM;
@@ -706,6 +710,18 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,
if (unlikely(err))
return err;
if (dst->xdp_prog && skb_cloned(skb)) {
struct sk_buff *nskb;
nskb = skb_copy(skb, GFP_ATOMIC);
if (!nskb)
return -ENOMEM;
nskb->mac_len = skb->mac_len;
consume_skb(skb);
skb = nskb;
}
/* Redirect has already succeeded semantically at this point, so we just
* return 0 even if packet is dropped. Helper below takes care of
* freeing skb.
@@ -726,6 +742,9 @@ static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst,
struct sk_buff *nskb;
int err;
if (unlikely(skb_is_nonlinear(skb)))
return -EOPNOTSUPP;
nskb = skb_clone(skb, GFP_ATOMIC);
if (!nskb)
return -ENOMEM;

27
kernel/bpf/fixups.c
View File

@@ -870,7 +870,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env)
case PTR_TO_BTF_ID:
case PTR_TO_BTF_ID | PTR_UNTRUSTED:
/* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike
* PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot
* PTR_TO_BTF_ID, and an active referenced id, but the same cannot
* be said once it is marked PTR_UNTRUSTED, hence we must handle
* any faults for loads into such types. BPF_WRITE is disallowed
* for this case.
@@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
prog->aux->real_func_cnt = env->subprog_cnt;
prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
prog->aux->exception_boundary = func[0]->aux->exception_boundary;
prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
bpf_prog_jit_attempt_done(prog);
return 0;
out_free:
@@ -1378,9 +1379,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
struct bpf_prog *prog = env->prog;
struct bpf_insn *insn = prog->insnsi;
bool has_kfunc_call = bpf_prog_has_kfunc_call(prog);
int i, depth;
int depth;
#endif
int err = 0;
int i, err = 0;
for (i = 0; i < env->subprog_cnt; i++) {
struct bpf_subprog_info *subprog = &env->subprog_info[i];
u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog);
if (subprog->max_out_stack_arg_cnt > outgoing) {
verbose(env,
"func#%d writes %u stack arg slots, but calls only require %u\n",
i, subprog->max_out_stack_arg_cnt, outgoing);
return -EINVAL;
}
}
if (env->prog->jit_requested &&
!bpf_prog_is_offloaded(env->prog->aux)) {
@@ -1395,6 +1408,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
verbose(env, "calling kernel functions are not allowed in non-JITed programs\n");
return -EINVAL;
}
for (i = 0; i < env->subprog_cnt; i++) {
if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) {
verbose(env, "stack args are not supported in non-JITed programs\n");
return -EINVAL;
}
}
if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
/* When JIT fails the progs with bpf2bpf calls and tail_calls
* have to be rejected, since interpreter doesn't support them yet.
@@ -2167,6 +2186,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
insn->imm == BPF_FUNC_get_func_ret) {
if (eatype == BPF_TRACE_FEXIT ||
eatype == BPF_TRACE_FSESSION ||
eatype == BPF_TRACE_FEXIT_MULTI ||
eatype == BPF_TRACE_FSESSION_MULTI ||
eatype == BPF_MODIFY_RETURN) {
/* Load nr_args from ctx - 8 */
insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8);

840
kernel/bpf/hashtab.c
View File

@@ -9,6 +9,7 @@
#include <linux/rculist_nulls.h>
#include <linux/rcupdate_wait.h>
#include <linux/random.h>
#include <linux/rhashtable.h>
#include <uapi/linux/btf.h>
#include <linux/rcupdate_trace.h>
#include <linux/btf_ids.h>
@@ -242,6 +243,10 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab)
if (IS_ERR_OR_NULL(htab->map.record))
return;
/*
* Preallocated maps do not have a bpf_mem_alloc destructor, so fully
* destroy every element, including the extra elements.
*/
if (htab_has_extra_elems(htab))
num_entries += num_possible_cpus();
for (i = 0; i < num_entries; i++) {
@@ -496,28 +501,26 @@ static void htab_dtor_ctx_free(void *ctx)
kfree(ctx);
}
static int htab_set_dtor(struct bpf_htab *htab, void (*dtor)(void *, void *))
static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma,
void (*dtor)(void *, void *))
{
u32 key_size = htab->map.key_size;
struct bpf_mem_alloc *ma;
struct htab_btf_record *hrec;
int err;
/* No need for dtors. */
if (IS_ERR_OR_NULL(htab->map.record))
if (IS_ERR_OR_NULL(map->record))
return 0;
hrec = kzalloc(sizeof(*hrec), GFP_KERNEL);
if (!hrec)
return -ENOMEM;
hrec->key_size = key_size;
hrec->record = btf_record_dup(htab->map.record);
hrec->key_size = map->key_size;
hrec->record = btf_record_dup(map->record);
if (IS_ERR(hrec->record)) {
err = PTR_ERR(hrec->record);
kfree(hrec);
return err;
}
ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma;
bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec);
return 0;
}
@@ -534,9 +537,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf,
* populated in htab_map_alloc(), so it will always appear as NULL.
*/
if (htab_is_percpu(htab))
return htab_set_dtor(htab, htab_pcpu_mem_dtor);
return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor);
else
return htab_set_dtor(htab, htab_mem_dtor);
return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor);
}
static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
@@ -834,8 +837,8 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map,
return insn - insn_buf;
}
static void check_and_free_fields(struct bpf_htab *htab,
struct htab_elem *elem)
static void check_and_cancel_fields(struct bpf_htab *htab,
struct htab_elem *elem)
{
if (IS_ERR_OR_NULL(htab->map.record))
return;
@@ -845,11 +848,11 @@ static void check_and_free_fields(struct bpf_htab *htab,
int cpu;
for_each_possible_cpu(cpu)
bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu));
bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu));
} else {
void *map_value = htab_elem_value(elem, htab->map.key_size);
bpf_obj_free_fields(htab->map.record, map_value);
bpf_obj_cancel_fields(&htab->map, map_value);
}
}
@@ -884,7 +887,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node)
htab_unlock_bucket(b, flags);
if (l == tgt_l)
check_and_free_fields(htab, l);
check_and_cancel_fields(htab, l);
return l == tgt_l;
}
@@ -949,7 +952,7 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
{
check_and_free_fields(htab, l);
check_and_cancel_fields(htab, l);
if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);
@@ -1002,7 +1005,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
if (htab_is_prealloc(htab)) {
bpf_map_dec_elem_count(&htab->map);
check_and_free_fields(htab, l);
check_and_cancel_fields(htab, l);
pcpu_freelist_push(&htab->freelist, &l->fnode);
} else {
dec_elem_count(htab);
@@ -1019,7 +1022,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
/* copy true value_size bytes */
ptr = this_cpu_ptr(pptr);
copy_map_value(&htab->map, ptr, value);
bpf_obj_free_fields(htab->map.record, ptr);
bpf_obj_cancel_fields(&htab->map, ptr);
} else {
u32 size = round_up(htab->map.value_size, 8);
void *val;
@@ -1029,7 +1032,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
cpu = map_flags >> 32;
ptr = per_cpu_ptr(pptr, cpu);
copy_map_value(&htab->map, ptr, value);
bpf_obj_free_fields(htab->map.record, ptr);
bpf_obj_cancel_fields(&htab->map, ptr);
return;
}
@@ -1037,7 +1040,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr,
ptr = per_cpu_ptr(pptr, cpu);
val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu;
copy_map_value(&htab->map, ptr, val);
bpf_obj_free_fields(htab->map.record, ptr);
bpf_obj_cancel_fields(&htab->map, ptr);
}
}
}
@@ -1253,11 +1256,11 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value,
if (l_old) {
hlist_nulls_del_rcu(&l_old->hash_node);
/* l_old has already been stashed in htab->extra_elems, free
* its special fields before it is available for reuse.
/* l_old has already been stashed in htab->extra_elems, cancel
* its reusable special fields before it is available for reuse.
*/
if (htab_is_prealloc(htab))
check_and_free_fields(htab, l_old);
check_and_cancel_fields(htab, l_old);
}
htab_unlock_bucket(b, flags);
if (l_old && !htab_is_prealloc(htab))
@@ -1270,7 +1273,7 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value,
static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem)
{
check_and_free_fields(htab, elem);
check_and_cancel_fields(htab, elem);
bpf_map_dec_elem_count(&htab->map);
bpf_lru_push_free(&htab->lru, &elem->lru_node);
}
@@ -2739,3 +2742,794 @@ const struct bpf_map_ops htab_of_maps_map_ops = {
BATCH_OPS(htab),
.map_btf_id = &htab_map_btf_ids[0],
};
struct rhtab_elem {
struct rhash_head node;
/* key bytes, then value bytes follow */
u8 data[] __aligned(8);
};
struct bpf_rhtab {
struct bpf_map map;
struct rhashtable ht;
struct bpf_mem_alloc ma;
u32 elem_size;
bool freeing_internal;
};
static const struct rhashtable_params rhtab_params = {
.head_offset = offsetof(struct rhtab_elem, node),
.key_offset = offsetof(struct rhtab_elem, data),
};
static inline void *rhtab_elem_value(struct rhtab_elem *l, u32 key_size)
{
return l->data + round_up(key_size, 8);
}
/* Specialize hash function and objcmp for long sized key */
static __always_inline int rhtab_key_cmp_long(struct rhashtable_compare_arg *arg,
const void *ptr)
{
const unsigned long key1 = *(const unsigned long *)arg->key;
const struct rhtab_elem *key2 = ptr;
return key1 != *(const unsigned long *)key2->data;
}
static __always_inline u32 rhtab_hashfn_long(const void *data, u32 len, u32 seed)
{
u64 k = *(const unsigned long *)data;
return (u32)(k ^ (k >> 32)) ^ seed;
}
static const struct rhashtable_params rhtab_params_long = {
.head_offset = offsetof(struct rhtab_elem, node),
.key_offset = offsetof(struct rhtab_elem, data),
.key_len = sizeof(long),
.hashfn = rhtab_hashfn_long,
.obj_cmpfn = rhtab_key_cmp_long,
};
static struct bpf_map *rhtab_map_alloc(union bpf_attr *attr)
{
struct rhashtable_params params;
struct bpf_rhtab *rhtab;
int err = 0;
rhtab = bpf_map_area_alloc(sizeof(*rhtab), NUMA_NO_NODE);
if (!rhtab)
return ERR_PTR(-ENOMEM);
bpf_map_init_from_attr(&rhtab->map, attr);
if (rhtab->map.max_entries > 1UL << 31) {
err = -E2BIG;
goto free_rhtab;
}
rhtab->elem_size = sizeof(struct rhtab_elem) + round_up(rhtab->map.key_size, 8) +
round_up(rhtab->map.value_size, 8);
params = rhtab_params;
params.key_len = rhtab->map.key_size;
params.nelem_hint = (u32)attr->map_extra;
params.automatic_shrinking = true;
if (rhtab->map.key_size == sizeof(long)) {
params.hashfn = rhtab_hashfn_long;
params.obj_cmpfn = rhtab_key_cmp_long;
}
err = rhashtable_init(&rhtab->ht, &params);
if (err)
goto free_rhtab;
/* Set max_elems after rhashtable_init() since init zeroes the struct */
rhtab->ht.max_elems = rhtab->map.max_entries;
err = bpf_mem_alloc_init(&rhtab->ma, rhtab->elem_size, false);
if (err)
goto destroy_rhtab;
return &rhtab->map;
destroy_rhtab:
rhashtable_destroy(&rhtab->ht);
free_rhtab:
bpf_map_area_free(rhtab);
return ERR_PTR(err);
}
static int rhtab_map_alloc_check(union bpf_attr *attr)
{
if (!(attr->map_flags & BPF_F_NO_PREALLOC))
return -EINVAL;
if (attr->map_flags & BPF_F_ZERO_SEED)
return -EINVAL;
if (attr->key_size > U16_MAX)
return -E2BIG;
if (attr->map_extra >> 32)
return -EINVAL;
if ((u32)attr->map_extra > U16_MAX)
return -E2BIG;
if ((u32)attr->map_extra > attr->max_entries)
return -EINVAL;
return htab_map_alloc_check(attr);
}
static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab,
struct rhtab_elem *elem)
{
if (IS_ERR_OR_NULL(rhtab->map.record))
return;
bpf_obj_free_fields(rhtab->map.record,
rhtab_elem_value(elem, rhtab->map.key_size));
}
static void rhtab_mem_dtor(void *obj, void *ctx)
{
struct htab_btf_record *hrec = ctx;
struct rhtab_elem *elem = obj;
if (IS_ERR_OR_NULL(hrec->record))
return;
bpf_obj_free_fields(hrec->record,
rhtab_elem_value(elem, hrec->key_size));
}
static void rhtab_free_elem(void *ptr, void *arg)
{
struct bpf_rhtab *rhtab = arg;
struct rhtab_elem *elem = ptr;
bpf_map_free_internal_structs(&rhtab->map, rhtab_elem_value(elem, rhtab->map.key_size));
bpf_mem_cache_free_rcu(&rhtab->ma, elem);
}
static void rhtab_map_free(struct bpf_map *map)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
rhashtable_free_and_destroy(&rhtab->ht, rhtab_free_elem, rhtab);
bpf_mem_alloc_destroy(&rhtab->ma);
bpf_map_area_free(rhtab);
}
static void *rhtab_lookup_elem(struct bpf_map *map, void *key)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
/* Hold RCU lock in case sleepable program calls via gen_lookup */
guard(rcu)();
if (map->key_size == sizeof(long))
return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params_long);
return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params);
}
static void *rhtab_map_lookup_elem(struct bpf_map *map, void *key) __must_hold(RCU)
{
struct rhtab_elem *l;
l = rhtab_lookup_elem(map, key);
return l ? rhtab_elem_value(l, map->key_size) : NULL;
}
static void rhtab_read_elem_value(struct bpf_map *map, void *dst, struct rhtab_elem *elem,
u64 flags)
{
void *src = rhtab_elem_value(elem, map->key_size);
if (flags & BPF_F_LOCK)
copy_map_value_locked(map, dst, src, true);
else
copy_map_value(map, dst, src);
}
static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, void *copy,
u64 flags)
{
int err;
/*
* disable_instrumentation() mitigates the deadlock for programs running in NMI context.
* rhashtable locks bucket with local_irq_save(). Only NMI programs may reenter
* rhashtable code, bpf_disable_instrumentation() disables programs running in NMI, except
* raw tracepoints, which we don't have in rhashtable.
*/
bpf_disable_instrumentation();
if (rhtab->map.key_size == sizeof(long))
err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params_long);
else
err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params);
bpf_enable_instrumentation();
if (err)
return err;
if (copy) {
rhtab_read_elem_value(&rhtab->map, copy, elem, flags);
check_and_init_map_value(&rhtab->map, copy);
}
/* Release internal structs: kptr, bpf_timer, task_work, wq */
rhtab_check_and_free_fields(rhtab, elem);
bpf_mem_cache_free_rcu(&rhtab->ma, elem);
return 0;
}
static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
struct rhtab_elem *elem;
guard(rcu)();
elem = rhtab_lookup_elem(map, key);
if (!elem)
return -ENOENT;
return rhtab_delete_elem(rhtab, elem, NULL, 0);
}
static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void *value, u64 flags)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
struct rhtab_elem *elem;
int err;
err = bpf_map_check_op_flags(map, flags, BPF_F_LOCK);
if (err)
return err;
guard(rcu)();
elem = rhtab_lookup_elem(map, key);
if (!elem)
return -ENOENT;
return rhtab_delete_elem(rhtab, elem, value, flags);
}
static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value,
u64 map_flags)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
void *old_val = rhtab_elem_value(elem, map->key_size);
if (map_flags & BPF_NOEXIST)
return -EEXIST;
if (map_flags & BPF_F_LOCK)
copy_map_value_locked(map, old_val, value, false);
else
copy_map_value(map, old_val, value);
/*
* Torn reads: a concurrent reader without BPF_F_LOCK may observe
* the value mid-copy. Callers requiring consistent reads must use
* BPF_F_LOCK, matching arraymap semantics.
*
* copy_map_value() skips special-field offsets, so old timers/
* kptrs/etc. still sit in the slot. Cancel them after the copy
* to match arraymap's update semantics.
*/
rhtab_check_and_free_fields(rhtab, elem);
return 0;
}
static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
struct rhtab_elem *elem, *tmp;
if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST))
return -EINVAL;
if ((map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
return -EINVAL;
guard(rcu)();
elem = rhtab_lookup_elem(map, key);
if (elem)
return rhtab_map_update_existing(map, elem, value, map_flags);
if (map_flags & BPF_EXIST)
return -ENOENT;
/*
* Reject new insertions while map_release_uref cleanup walks the
* table. Without this, new elements could keep triggering rehash
* and prevent the walk from terminating.
*/
if (READ_ONCE(rhtab->freeing_internal))
return -EBUSY;
/* Check max_entries limit before inserting new element */
if (atomic_read(&rhtab->ht.nelems) >= map->max_entries)
return -E2BIG;
elem = bpf_mem_cache_alloc(&rhtab->ma);
if (!elem)
return -ENOMEM;
memcpy(elem->data, key, map->key_size);
copy_map_value(map, rhtab_elem_value(elem, map->key_size), value);
check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size));
/* Prevent deadlock for NMI programs attempting to take bucket lock */
bpf_disable_instrumentation();
if (map->key_size == sizeof(long))
tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params_long);
else
tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params);
bpf_enable_instrumentation();
if (tmp) {
bpf_mem_cache_free(&rhtab->ma, elem);
if (IS_ERR(tmp))
return PTR_ERR(tmp);
return rhtab_map_update_existing(map, tmp, value, map_flags);
}
return 0;
}
static int rhtab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
{
struct bpf_insn *insn = insn_buf;
const int ret = BPF_REG_0;
BUILD_BUG_ON(!__same_type(&rhtab_lookup_elem,
(void *(*)(struct bpf_map *map, void *key)) NULL));
*insn++ = BPF_EMIT_CALL(rhtab_lookup_elem);
*insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1);
*insn++ = BPF_ALU64_IMM(BPF_ADD, ret,
offsetof(struct rhtab_elem, data) + round_up(map->key_size, 8));
return insn - insn_buf;
}
static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf,
const struct btf_type *key_type,
const struct btf_type *value_type)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor);
}
static void rhtab_map_free_internal_structs(struct bpf_map *map)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
struct rhashtable_iter iter;
struct rhtab_elem *elem;
if (!bpf_map_has_internal_structs(map))
return;
/*
* Block new insertions. Once observed, no new growth is triggered,
* so any in-flight rehash will drain and the walker is guaranteed
* to stop returning -EAGAIN. Treat -EAGAIN as "rehash in progress,
* retry"; do not wait for the worker.
*/
WRITE_ONCE(rhtab->freeing_internal, true);
rhashtable_walk_enter(&rhtab->ht, &iter);
rhashtable_walk_start(&iter);
while ((elem = rhashtable_walk_next(&iter))) {
if (IS_ERR(elem)) {
if (PTR_ERR(elem) == -EAGAIN)
continue;
break;
}
bpf_map_free_internal_structs(map, rhtab_elem_value(elem, map->key_size));
if (need_resched()) { /* Avoid stalls on large maps */
rhashtable_walk_stop(&iter);
cond_resched();
rhashtable_walk_start(&iter);
}
}
rhashtable_walk_stop(&iter);
rhashtable_walk_exit(&iter);
WRITE_ONCE(rhtab->freeing_internal, false);
}
static int rhtab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
__must_hold_shared(RCU)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
struct rhtab_elem *elem;
elem = rhashtable_next_key(&rhtab->ht, key);
/* if not found, return the first key */
if (PTR_ERR(elem) == -ENOENT)
elem = rhashtable_next_key(&rhtab->ht, NULL);
if (IS_ERR(elem))
return PTR_ERR(elem);
if (!elem)
return -ENOENT;
memcpy(next_key, elem->data, map->key_size);
return 0;
}
static void rhtab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m)
{
void *value;
/* Guarantee that hashtab value is not freed */
guard(rcu)();
value = rhtab_map_lookup_elem(map, key);
if (!value)
return;
btf_type_seq_show(map->btf, map->btf_key_type_id, key, m);
seq_puts(m, ": ");
btf_type_seq_show(map->btf, map->btf_value_type_id, value, m);
seq_putc(m, '\n');
}
static long bpf_each_rhash_elem(struct bpf_map *map, bpf_callback_t callback_fn,
void *callback_ctx, u64 flags)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
void *prev_key = NULL;
struct rhtab_elem *elem;
int num_elems = 0;
u64 ret = 0;
cant_migrate();
if (flags != 0)
return -EINVAL;
rcu_read_lock();
/*
* Best-effort iteration: if rhashtable is concurrently resized or
* elements are deleted/inserted, there may be missed or duplicate
* elements visited.
*/
while ((elem = rhashtable_next_key(&rhtab->ht, prev_key))) {
if (IS_ERR(elem))
break;
num_elems++;
ret = callback_fn((u64)(long)map,
(u64)(long)elem->data,
(u64)(long)rhtab_elem_value(elem, map->key_size),
(u64)(long)callback_ctx, 0);
if (ret)
break;
prev_key = elem->data; /* valid while RCU held */
}
rcu_read_unlock();
return num_elems;
}
static u64 rhtab_map_mem_usage(const struct bpf_map *map)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
u64 num_entries;
/* Excludes rhashtable bucket overhead (~ nelems * sizeof(void *) at 75% load). */
num_entries = atomic_read(&rhtab->ht.nelems);
return sizeof(struct bpf_rhtab) + rhtab->elem_size * num_entries;
}
static int __rhtab_map_lookup_and_delete_batch(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr,
bool do_delete)
{
struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
void __user *uvalues = u64_to_user_ptr(attr->batch.values);
void __user *ukeys = u64_to_user_ptr(attr->batch.keys);
void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch);
void *cursor = NULL, *keys = NULL, *values = NULL, *dst_key, *dst_val;
struct rhtab_elem **del_elems = NULL;
u32 max_count, total, key_size, value_size, i;
bool has_next_cursor = false;
struct rhtab_elem *elem;
u64 elem_map_flags, map_flags;
int ret = 0;
elem_map_flags = attr->batch.elem_flags;
ret = bpf_map_check_op_flags(map, elem_map_flags, BPF_F_LOCK);
if (ret)
return ret;
map_flags = attr->batch.flags;
if (map_flags)
return -EINVAL;
max_count = attr->batch.count;
if (!max_count)
return 0;
if (put_user(0, &uattr->batch.count))
return -EFAULT;
key_size = map->key_size;
value_size = map->value_size;
keys = kvmalloc_array(max_count, key_size, GFP_USER | __GFP_NOWARN);
values = kvmalloc_array(max_count, value_size, GFP_USER | __GFP_NOWARN);
if (do_delete)
del_elems = kvmalloc_array(max_count, sizeof(void *),
GFP_USER | __GFP_NOWARN);
cursor = kmalloc(key_size, GFP_USER | __GFP_NOWARN);
if (!keys || !values || !cursor || (do_delete && !del_elems)) {
ret = -ENOMEM;
goto free;
}
if (ubatch && copy_from_user(cursor, ubatch, key_size)) {
ret = -EFAULT;
goto free;
}
dst_key = keys;
dst_val = values;
total = 0;
rcu_read_lock();
/*
* Cursor stores the key of the next-to-process element (stashed by
* the previous batch). Look it up directly so the element is included
* here rather than skipped by next_key(). If the cursor was deleted
* concurrently (or by the previous do_delete batch), return -EAGAIN
* so userspace can distinguish a lost cursor from end-of-iteration
* (-ENOENT) and restart from a NULL cursor.
*/
if (ubatch) {
elem = rhtab_lookup_elem(map, cursor);
if (!elem) {
rcu_read_unlock();
ret = -EAGAIN;
goto free;
}
} else {
elem = rhashtable_next_key(&rhtab->ht, NULL);
}
while (elem && !IS_ERR(elem) && total < max_count) {
memcpy(dst_key, elem->data, key_size);
rhtab_read_elem_value(map, dst_val, elem, elem_map_flags);
check_and_init_map_value(map, dst_val);
if (do_delete)
del_elems[total] = elem;
elem = rhashtable_next_key(&rhtab->ht, dst_key);
dst_key += key_size;
dst_val += value_size;
total++;
/* Bail to userspace to avoid stalls. */
if (need_resched())
break;
}
if (elem && !IS_ERR(elem)) {
/* Stash next-to-process key as cursor for the next batch. */
memcpy(cursor, elem->data, key_size);
has_next_cursor = true;
}
if (do_delete) {
for (i = 0; i < total; i++)
rhtab_delete_elem(rhtab, del_elems[i], NULL, 0);
}
rcu_read_unlock();
if (total == 0) {
ret = -ENOENT;
goto free;
}
/* No more elements after this batch. */
if (!has_next_cursor)
ret = -ENOENT;
if (copy_to_user(ukeys, keys, (size_t)total * key_size) ||
copy_to_user(uvalues, values, (size_t)total * value_size) ||
put_user(total, &uattr->batch.count) ||
(has_next_cursor &&
copy_to_user(u64_to_user_ptr(attr->batch.out_batch),
cursor, key_size))) {
ret = -EFAULT;
goto free;
}
free:
kfree(cursor);
kvfree(keys);
kvfree(values);
kvfree(del_elems);
return ret;
}
static int rhtab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, false);
}
static int rhtab_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, true);
}
struct bpf_iter_seq_rhash_map_info {
struct bpf_map *map;
struct bpf_rhtab *rhtab;
struct rhashtable_iter iter;
};
static void *bpf_rhash_map_seq_start(struct seq_file *seq, loff_t *pos)
__acquires(RCU)
{
struct bpf_iter_seq_rhash_map_info *info = seq->private;
struct rhtab_elem *elem;
rhashtable_walk_start(&info->iter);
/*
* Re-deliver the element returned by walk_next() at the end of the
* previous read() — bpf_seq_read may have stopped before show()
* consumed it. Rehash rewinds the walker; retry on -EAGAIN.
*/
do {
elem = rhashtable_walk_peek(&info->iter);
} while (PTR_ERR(elem) == -EAGAIN);
if (IS_ERR(elem))
return NULL;
if (elem && *pos == 0)
++*pos;
return elem;
}
static void *bpf_rhash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{
struct bpf_iter_seq_rhash_map_info *info = seq->private;
struct rhtab_elem *elem;
++*pos;
/* Rehash rewinds the walker; retry until it stops returning -EAGAIN. */
do {
elem = rhashtable_walk_next(&info->iter);
} while (PTR_ERR(elem) == -EAGAIN);
if (IS_ERR(elem))
return NULL;
return elem;
}
static int __bpf_rhash_map_seq_show(struct seq_file *seq,
struct rhtab_elem *elem)
{
struct bpf_iter_seq_rhash_map_info *info = seq->private;
struct bpf_iter__bpf_map_elem ctx = {};
struct bpf_iter_meta meta;
struct bpf_prog *prog;
int ret = 0;
meta.seq = seq;
prog = bpf_iter_get_info(&meta, elem == NULL);
if (prog) {
ctx.meta = &meta;
ctx.map = info->map;
if (elem) {
ctx.key = elem->data;
ctx.value = rhtab_elem_value(elem, info->map->key_size);
}
ret = bpf_iter_run_prog(prog, &ctx);
}
return ret;
}
static int bpf_rhash_map_seq_show(struct seq_file *seq, void *v)
{
return __bpf_rhash_map_seq_show(seq, v);
}
static void bpf_rhash_map_seq_stop(struct seq_file *seq, void *v)
__releases(RCU)
{
struct bpf_iter_seq_rhash_map_info *info = seq->private;
if (!v)
(void)__bpf_rhash_map_seq_show(seq, NULL);
rhashtable_walk_stop(&info->iter);
}
static int bpf_iter_init_rhash_map(void *priv_data, struct bpf_iter_aux_info *aux)
{
struct bpf_iter_seq_rhash_map_info *info = priv_data;
struct bpf_map *map = aux->map;
bpf_map_inc_with_uref(map);
info->map = map;
info->rhtab = container_of(map, struct bpf_rhtab, map);
rhashtable_walk_enter(&info->rhtab->ht, &info->iter);
return 0;
}
static void bpf_iter_fini_rhash_map(void *priv_data)
{
struct bpf_iter_seq_rhash_map_info *info = priv_data;
rhashtable_walk_exit(&info->iter);
bpf_map_put_with_uref(info->map);
}
static const struct seq_operations bpf_rhash_map_seq_ops = {
.start = bpf_rhash_map_seq_start,
.next = bpf_rhash_map_seq_next,
.stop = bpf_rhash_map_seq_stop,
.show = bpf_rhash_map_seq_show,
};
static const struct bpf_iter_seq_info rhash_iter_seq_info = {
.seq_ops = &bpf_rhash_map_seq_ops,
.init_seq_private = bpf_iter_init_rhash_map,
.fini_seq_private = bpf_iter_fini_rhash_map,
.seq_priv_size = sizeof(struct bpf_iter_seq_rhash_map_info),
};
BTF_ID_LIST_SINGLE(rhtab_map_btf_ids, struct, bpf_rhtab)
const struct bpf_map_ops rhtab_map_ops = {
.map_meta_equal = bpf_map_meta_equal,
.map_alloc_check = rhtab_map_alloc_check,
.map_alloc = rhtab_map_alloc,
.map_free = rhtab_map_free,
.map_get_next_key = rhtab_map_get_next_key,
.map_release_uref = rhtab_map_free_internal_structs,
.map_check_btf = rhtab_map_check_btf,
.map_lookup_elem = rhtab_map_lookup_elem,
.map_lookup_and_delete_elem = rhtab_map_lookup_and_delete_elem,
.map_update_elem = rhtab_map_update_elem,
.map_delete_elem = rhtab_map_delete_elem,
.map_gen_lookup = rhtab_map_gen_lookup,
.map_seq_show_elem = rhtab_map_seq_show_elem,
.map_set_for_each_callback_args = map_set_for_each_callback_args,
.map_for_each_callback = bpf_each_rhash_elem,
.map_mem_usage = rhtab_map_mem_usage,
BATCH_OPS(rhtab),
.map_btf_id = &rhtab_map_btf_ids[0],
.iter_seq_info = &rhash_iter_seq_info,
};

204
kernel/bpf/helpers.c
View File

@@ -1944,7 +1944,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg3_type = ARG_PTR_TO_DYNPTR,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@@ -2001,7 +2001,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_DYNPTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
@@ -2044,7 +2044,7 @@ static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_DYNPTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
@@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
struct list_head *head = list_head, drain, *pos, *n;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
INIT_LIST_HEAD(&drain);
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
@@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
list_for_each_safe(pos, n, head) {
struct bpf_list_node_kern *node;
node = container_of(pos, struct bpf_list_node_kern, list_head);
WRITE_ONCE(node->owner, BPF_PTR_POISON);
list_move_tail(pos, &drain);
}
unlock:
INIT_LIST_HEAD(orig_head);
INIT_LIST_HEAD(head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
while (!list_empty(&drain)) {
struct bpf_list_node_kern *node;
obj -= field->graph_root.node_offset;
head = head->next;
pos = drain.next;
node = container_of(pos, struct bpf_list_node_kern, list_head);
list_del_init(pos);
/* Ensure __bpf_list_add() sees the node as unlinked. */
smp_store_release(&node->owner, NULL);
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
__bpf_obj_drop_impl((char *)pos - field->graph_root.node_offset,
field->graph_root.value_rec, false);
}
}
@@ -2295,6 +2306,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct bpf_rb_node_kern *node;
struct rb_node *pos, *n;
void *obj;
@@ -2303,14 +2315,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
WRITE_ONCE(node->owner, BPF_PTR_POISON);
}
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
RB_CLEAR_NODE(pos);
/* Ensure __bpf_rbtree_add() sees the node as unlinked. */
smp_store_release(&node->owner, NULL);
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
@@ -2467,9 +2485,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
struct list_head **prev_ptr,
struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
struct list_head *prev;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
@@ -2477,19 +2497,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node,
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
prev = *prev_ptr;
/* When prev is not the list head, it must be a node in this list. */
if (prev != h) {
struct bpf_list_node_kern *prev_kn =
container_of(prev, struct bpf_list_node_kern, list_head);
if (unlikely(READ_ONCE(prev_kn->owner) != head))
goto fail;
}
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON))
goto fail;
tail ? list_add_tail(n, h) : list_add(n, h);
list_add(n, prev);
WRITE_ONCE(node->owner, head);
return 0;
fail:
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
/**
@@ -2510,8 +2542,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
struct list_head *h = (void *)head;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
return __bpf_list_add(n, head, &h, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
@@ -2539,8 +2572,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
struct list_head *h = (void *)head;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
return __bpf_list_add(n, head, &h->prev, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
@@ -2550,37 +2584,63 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new,
struct bpf_list_node *prev__nonown_allowed,
struct btf_struct_meta *meta, u64 off)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed;
struct list_head *prev_ptr = &p->list_head;
return __bpf_list_add(n, head, &prev_ptr, meta ? meta->record : NULL, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head,
struct list_head *n)
{
struct list_head *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
if (unlikely(!h->next)) {
INIT_LIST_HEAD(h);
return NULL;
}
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
if (unlikely(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
/* Ensure __bpf_list_add() sees the node as unlinked. */
smp_store_release(&node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
struct list_head *h = (void *)head;
return __bpf_list_del(head, h->next);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
struct list_head *h = (void *)head;
return __bpf_list_del(head, h->prev);
}
__bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head,
struct bpf_list_node *node__nonown_allowed)
{
struct bpf_list_node_kern *kn = (void *)node__nonown_allowed;
/* verifier guarantees node is a list node rather than list head */
return __bpf_list_del(head, &kn->list_head);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
@@ -2603,6 +2663,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head,
struct bpf_list_node *node__nonown_allowed)
{
struct list_head *h = (struct list_head *)head;
struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
if (READ_ONCE(kn->owner) != head)
return false;
return list_is_first(&kn->list_head, h);
}
__bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head,
struct bpf_list_node *node__nonown_allowed)
{
struct list_head *h = (struct list_head *)head;
struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed;
if (READ_ONCE(kn->owner) != head)
return false;
return list_is_last(&kn->list_head, h);
}
__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
return list_empty(h);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
@@ -2912,11 +3009,13 @@ __bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
guard(rcu)();
if (!task_active_pid_ns(current))
return NULL;
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
@@ -3072,7 +3171,7 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
__bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
@@ -3093,14 +3192,14 @@ __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
@@ -3110,7 +3209,7 @@ __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
@@ -3122,7 +3221,7 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
@@ -3145,11 +3244,11 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
__bpf_kfunc int bpf_dynptr_copy(const struct bpf_dynptr *dst_ptr, u64 dst_off,
const struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
const struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
const struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
@@ -3200,9 +3299,9 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
__bpf_kfunc int bpf_dynptr_memset(const struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
@@ -3301,7 +3400,7 @@ __bpf_kfunc void bpf_throw(u64 cookie)
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
@@ -4214,13 +4313,13 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
__bpf_kfunc int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
const struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
@@ -4718,10 +4817,15 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_is_first)
BTF_ID_FLAGS(func, bpf_list_is_last)
BTF_ID_FLAGS(func, bpf_list_empty)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
@@ -4862,7 +4966,7 @@ BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)

4
kernel/bpf/inode.c
View File

@@ -845,6 +845,10 @@ static void bpf_destroy_inode(struct inode *inode)
simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL);
}
/*
* Called after RCU grace period - safe to free inode and anything
* that might be accessed by RCU pathwalk (inode fields, i_link).
*/
static void bpf_free_inode(struct inode *inode)
{
if (S_ISLNK(inode->i_mode))

181
kernel/bpf/liveness.c
View File

@@ -610,6 +610,21 @@ enum arg_track_state {
/* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */
#define MAX_ARG_SPILL_SLOTS 64
/*
* Combined register + stack arg tracking: R0-R10 at indices 0-10,
* outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6.
*/
#define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS)
static int stack_arg_off_to_slot(s16 off)
{
int aoff = off < 0 ? -off : off;
if (aoff / 8 > MAX_STACK_ARG_SLOTS)
return -1;
return aoff / 8 - 1;
}
static bool arg_is_visited(const struct arg_track *at)
{
return at->frame != ARG_UNVISITED;
@@ -791,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in
return true;
verbose(env, "arg JOIN insn %d -> %d ", idx, target);
if (r >= 0)
if (r >= MAX_BPF_REG)
verbose(env, "sa%d: ", r - MAX_BPF_REG);
else if (r >= 0)
verbose(env, "r%d: ", r);
else
verbose(env, "fp%+d: ", r * 8);
@@ -1032,6 +1049,21 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);
verbose(env, " -> "); verbose_arg_track(env, &at_out[i]);
}
/* Log outgoing stack arg slot transitions at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1 */
for (i = 0; i < MAX_STACK_ARG_SLOTS; i++) {
int ai = MAX_BPF_REG + i;
if (arg_track_eq(&at_out[ai], &at_in[ai]))
continue;
if (!printed) {
verbose(env, "%3d: ", idx);
bpf_verbose_insn(env, insn);
bpf_vlog_reset(&env->log, env->log.end_pos - 1);
printed = true;
}
verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]);
verbose(env, " -> "); verbose_arg_track(env, &at_out[ai]);
}
for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) {
if (arg_track_eq(&at_stack_out[i], &at_stack_in[i]))
continue;
@@ -1062,6 +1094,7 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at)
static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
int insn_idx,
struct arg_track *at_out, struct arg_track *at_stack_out,
const struct arg_track *at_stack_arg_entry,
struct func_instance *instance,
u32 *callsites)
{
@@ -1071,9 +1104,21 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct arg_track *dst = &at_out[insn->dst_reg];
struct arg_track *src = &at_out[insn->src_reg];
struct arg_track none = { .frame = ARG_NONE };
int r;
int r, slot;
if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
/* Handle stack arg stores and loads. */
if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {
slot = stack_arg_off_to_slot(insn->off);
if (slot >= 0) {
if (is_stack_arg_stx(insn))
at_out[MAX_BPF_REG + slot] = at_out[insn->src_reg];
else
at_out[MAX_BPF_REG + slot] = none;
}
} else if (is_stack_arg_ldx(insn)) {
slot = stack_arg_off_to_slot(insn->off);
at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none;
} else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
if (code == BPF_MOV) {
*dst = none;
} else if (dst->frame >= 0) {
@@ -1297,6 +1342,16 @@ static int record_load_store_access(struct bpf_verifier_env *env,
struct arg_track resolved, *ptr;
int oi;
/*
* Stack arg insns use dst_reg/src_reg=BPF_REG_PARAMS(11). Since at[]
* is extended to MAX_AT_TRACK_REGS, at[11] holds the arg_track for
* outgoing stack arg slot 0 — not the pointer used for the memory
* access. Skip so the slot's tracked value isn't confused with the
* base register that record_stack_access() expects.
*/
if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn))
return 0;
switch (class) {
case BPF_LDX:
ptr = &at[insn->src_reg];
@@ -1343,6 +1398,42 @@ static int record_load_store_access(struct bpf_verifier_env *env,
return 0;
}
static int record_arg_access(struct bpf_verifier_env *env,
struct func_instance *instance,
struct bpf_insn *insn,
struct arg_track *at, int arg_idx,
int insn_idx)
{
int depth = instance->depth;
int frame = at->frame;
int err = 0;
s64 bytes;
if (!arg_is_fp(at))
return 0;
if (bpf_helper_call(insn)) {
bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx);
} else if (bpf_pseudo_kfunc_call(insn)) {
bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
} else {
for (int f = 0; f <= depth; f++) {
err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
if (err)
return err;
}
return 0;
}
if (bytes == 0)
return 0;
if (frame >= 0 && frame <= depth)
err = record_stack_access(instance, at, bytes, frame, insn_idx);
else if (frame == ARG_IMPRECISE)
err = record_imprecise(instance, at->mask, insn_idx);
return err;
}
/* Record stack access for a given 'at' state of helper/kfunc 'insn' */
static int record_call_access(struct bpf_verifier_env *env,
struct func_instance *instance,
@@ -1350,9 +1441,8 @@ static int record_call_access(struct bpf_verifier_env *env,
int insn_idx)
{
struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
int depth = instance->depth;
struct bpf_call_summary cs;
int r, err = 0, num_params = 5;
int r, err, num_params = 5;
if (bpf_pseudo_call(insn))
return 0;
@@ -1360,32 +1450,15 @@ static int record_call_access(struct bpf_verifier_env *env,
if (bpf_get_call_summary(env, insn, &cs))
num_params = cs.num_params;
for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) {
int frame = at[r].frame;
s64 bytes;
for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) {
err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx);
if (err)
return err;
}
if (!arg_is_fp(&at[r]))
continue;
if (bpf_helper_call(insn)) {
bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx);
} else if (bpf_pseudo_kfunc_call(insn)) {
bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx);
} else {
for (int f = 0; f <= depth; f++) {
err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
if (err)
return err;
}
return 0;
}
if (bytes == 0)
continue;
if (frame >= 0 && frame <= depth)
err = record_stack_access(instance, &at[r], bytes, frame, insn_idx);
else if (frame == ARG_IMPRECISE)
err = record_imprecise(instance, at[r].mask, insn_idx);
for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) {
err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r],
r + MAX_BPF_FUNC_REG_ARGS, insn_idx);
if (err)
return err;
}
@@ -1445,7 +1518,7 @@ static int find_callback_subprog(struct bpf_verifier_env *env,
/* Per-subprog intermediate state kept alive across analysis phases */
struct subprog_at_info {
struct arg_track (*at_in)[MAX_BPF_REG];
struct arg_track (*at_in)[MAX_AT_TRACK_REGS];
int len;
};
@@ -1479,6 +1552,9 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
for (r = 0; r < MAX_BPF_REG - 1; r++)
if (arg_is_fp(&info->at_in[i][r]))
has_extra = true;
for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
if (arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
has_extra = true;
}
if (is_ldx_stx_call) {
for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
@@ -1503,6 +1579,12 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
verbose(env, " r%d=", r);
verbose_arg_track(env, &info->at_in[i][r]);
}
for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) {
if (!arg_is_fp(&info->at_in[i][MAX_BPF_REG + r]))
continue;
verbose(env, " sa%d=", r);
verbose_arg_track(env, &info->at_in[i][MAX_BPF_REG + r]);
}
}
if (is_ldx_stx_call) {
@@ -1525,7 +1607,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
* Runs forward fixed-point with arg_track_xfer(), then records
* memory accesses in a single linear pass over converged state.
*
* @callee_entry: pre-populated entry state for R1-R5
* @callee_entry: pre-populated entry state for R1-R5 and stack args
* NULL for main (subprog 0).
* @info: stores at_in, len for debug printing.
*/
@@ -1543,10 +1625,11 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
int end = env->subprog_info[subprog + 1].start;
int po_end = env->subprog_info[subprog + 1].postorder_start;
int len = end - start;
struct arg_track (*at_in)[MAX_BPF_REG] = NULL;
struct arg_track at_out[MAX_BPF_REG];
struct arg_track (*at_in)[MAX_AT_TRACK_REGS] = NULL;
struct arg_track at_out[MAX_AT_TRACK_REGS];
struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL;
struct arg_track *at_stack_out = NULL;
struct arg_track at_stack_arg_entry[MAX_STACK_ARG_SLOTS];
struct arg_track unvisited = { .frame = ARG_UNVISITED };
struct arg_track none = { .frame = ARG_NONE };
bool changed;
@@ -1565,13 +1648,13 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
goto err_free;
for (i = 0; i < len; i++) {
for (r = 0; r < MAX_BPF_REG; r++)
for (r = 0; r < MAX_AT_TRACK_REGS; r++)
at_in[i][r] = unvisited;
for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
at_stack_in[i][r] = unvisited;
}
for (r = 0; r < MAX_BPF_REG; r++)
for (r = 0; r < MAX_AT_TRACK_REGS; r++)
at_in[0][r] = none;
/* Entry: R10 is always precisely the current frame's FP */
@@ -1587,6 +1670,10 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++)
at_stack_in[0][r] = none;
/* Entry: incoming stack args from caller, or ARG_NONE for main */
for (r = 0; r < MAX_STACK_ARG_SLOTS; r++)
at_stack_arg_entry[r] = callee_entry ? callee_entry[MAX_BPF_REG + r] : none;
if (env->log.level & BPF_LOG_LEVEL2)
verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth);
@@ -1605,7 +1692,8 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
memcpy(at_out, at_in[i], sizeof(at_out));
memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out));
arg_track_xfer(env, insn, idx, at_out, at_stack_out, instance, callsites);
arg_track_xfer(env, insn, idx, at_out, at_stack_out,
at_stack_arg_entry, instance, callsites);
arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out);
/* Propagate to successors within this subprogram */
@@ -1619,7 +1707,7 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
continue;
ti = target - start;
for (r = 0; r < MAX_BPF_REG; r++)
for (r = 0; r < MAX_AT_TRACK_REGS; r++)
changed |= arg_track_join(env, idx, target, r,
&at_in[ti][r], at_out[r]);
@@ -1674,11 +1762,14 @@ static int compute_subprog_args(struct bpf_verifier_env *env,
return err;
}
/* Return true if any of R1-R5 is derived from a frame pointer. */
/* Return true if any of R1-R5 or stack args is derived from a frame pointer. */
static bool has_fp_args(struct arg_track *args)
{
for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
if (args[r].frame != ARG_NONE)
if (arg_is_fp(&args[r]))
return true;
for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
if (arg_is_fp(&args[MAX_BPF_REG + r]))
return true;
return false;
}
@@ -1803,7 +1894,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
/* For each reachable call site in the subprog, recurse into callees */
for (int p = po_start; p < po_end; p++) {
int idx = env->cfg.insn_postorder[p];
struct arg_track callee_args[BPF_REG_5 + 1];
struct arg_track callee_args[MAX_AT_TRACK_REGS] = {};
struct arg_track none = { .frame = ARG_NONE };
struct bpf_insn *insn = &insns[idx];
struct func_instance *callee_instance;
@@ -1818,9 +1909,11 @@ static int analyze_subprog(struct bpf_verifier_env *env,
if (callee < 0)
continue;
/* Build entry args: R1-R5 from at_in at call site */
/* Build entry args: R1-R5 and stack args from at_in at call site */
for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
callee_args[r] = info[subprog].at_in[j][r];
for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r];
} else if (bpf_calls_callback(env, idx)) {
callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg);
if (callee == -2) {
@@ -1842,6 +1935,8 @@ static int analyze_subprog(struct bpf_verifier_env *env,
for (int r = BPF_REG_1; r <= BPF_REG_5; r++)
callee_args[r] = none;
for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++)
callee_args[MAX_BPF_REG + r] = none;
callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg];
} else {
continue;
@@ -2085,7 +2180,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
def = ALL_CALLER_SAVED_REGS;
use = def & ~BIT(BPF_REG_0);
if (bpf_get_call_summary(env, insn, &cs))
use = GENMASK(cs.num_params, 1);
use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
break;
default:
def = 0;

132
kernel/bpf/log.c
View File

@@ -13,17 +13,17 @@
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
static bool bpf_verifier_log_attr_valid(const struct bpf_verifier_log *log)
static bool bpf_verifier_log_attr_valid(u32 log_level, char __user *log_buf, u32 log_size)
{
/* ubuf and len_total should both be specified (or not) together */
if (!!log->ubuf != !!log->len_total)
if (!!log_buf != !!log_size)
return false;
/* log buf without log_level is meaningless */
if (log->ubuf && log->level == 0)
if (log_buf && log_level == 0)
return false;
if (log->level & ~BPF_LOG_MASK)
if (log_level & ~BPF_LOG_MASK)
return false;
if (log->len_total > UINT_MAX >> 2)
if (log_size > UINT_MAX >> 2)
return false;
return true;
}
@@ -36,7 +36,7 @@ int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level,
log->len_total = log_size;
/* log attributes have to be sane */
if (!bpf_verifier_log_attr_valid(log))
if (!bpf_verifier_log_attr_valid(log_level, log_buf, log_size))
return -EINVAL;
return 0;
@@ -571,20 +571,20 @@ static void print_scalar_ranges(struct bpf_verifier_env *env,
u64 val;
bool omit;
} minmaxs[] = {
{"smin", reg->smin_value, reg->smin_value == S64_MIN},
{"smax", reg->smax_value, reg->smax_value == S64_MAX},
{"umin", reg->umin_value, reg->umin_value == 0},
{"umax", reg->umax_value, reg->umax_value == U64_MAX},
{"smin", reg_smin(reg), reg_smin(reg) == S64_MIN},
{"smax", reg_smax(reg), reg_smax(reg) == S64_MAX},
{"umin", reg_umin(reg), reg_umin(reg) == 0},
{"umax", reg_umax(reg), reg_umax(reg) == U64_MAX},
{"smin32",
is_snum_decimal((s64)reg->s32_min_value)
? (s64)reg->s32_min_value
: (u32)reg->s32_min_value, reg->s32_min_value == S32_MIN},
is_snum_decimal((s64)reg_s32_min(reg))
? (s64)reg_s32_min(reg)
: (u32)reg_s32_min(reg), reg_s32_min(reg) == S32_MIN},
{"smax32",
is_snum_decimal((s64)reg->s32_max_value)
? (s64)reg->s32_max_value
: (u32)reg->s32_max_value, reg->s32_max_value == S32_MAX},
{"umin32", reg->u32_min_value, reg->u32_min_value == 0},
{"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX},
is_snum_decimal((s64)reg_s32_max(reg))
? (s64)reg_s32_max(reg)
: (u32)reg_s32_max(reg), reg_s32_max(reg) == S32_MAX},
{"umin32", reg_u32_min(reg), reg_u32_min(reg) == 0},
{"umax32", reg_u32_max(reg), reg_u32_max(reg) == U32_MAX},
}, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)];
bool neg1, neg2;
@@ -665,8 +665,8 @@ static void print_reg_state(struct bpf_verifier_env *env,
verbose_a("id=%d", reg->id & ~BPF_ADD_CONST);
if (reg->id & BPF_ADD_CONST)
verbose(env, "%+d", reg->delta);
if (reg->ref_obj_id)
verbose_a("ref_obj_id=%d", reg->ref_obj_id);
if (reg->parent_id)
verbose_a("parent_id=%d", reg->parent_id);
if (type_is_non_owning_ref(reg->type))
verbose_a("%s", "non_own_ref");
if (type_is_map_ptr(t)) {
@@ -768,21 +768,19 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
if (reg->id)
verbose_a("id=%d", reg->id);
if (reg->ref_obj_id)
verbose_a("ref_id=%d", reg->ref_obj_id);
if (reg->dynptr_id)
verbose_a("dynptr_id=%d", reg->dynptr_id);
if (reg->parent_id)
verbose_a("parent_id=%d", reg->parent_id);
verbose(env, ")");
break;
case STACK_ITER:
/* only main slot has ref_obj_id set; skip others */
if (!reg->ref_obj_id)
/* only main slot has id set; skip others */
if (!reg->id)
continue;
verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)",
verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
(-i - 1) * BPF_REG_SIZE,
iter_type_str(reg->iter.btf, reg->iter.btf_id),
reg->ref_obj_id, iter_state_str(reg->iter.state),
reg->id, iter_state_str(reg->iter.state),
reg->iter.depth);
break;
case STACK_MISC:
@@ -825,3 +823,81 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st
}
print_verifier_state(env, vstate, frameno, false);
}
int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level,
u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common,
bpfptr_t uattr_common, u32 size_common)
{
char __user *ubuf_common = u64_to_user_ptr(common->log_buf);
char __user *ubuf = u64_to_user_ptr(log_buf);
if (!bpf_verifier_log_attr_valid(common->log_level, ubuf_common, common->log_size) ||
!bpf_verifier_log_attr_valid(log_level, ubuf, log_size))
return -EINVAL;
if (ubuf && ubuf_common && (ubuf != ubuf_common || log_size != common->log_size ||
log_level != common->log_level))
return -EINVAL;
memset(log, 0, sizeof(*log));
log->ubuf = ubuf;
log->size = log_size;
log->level = log_level;
log->offsetof_true_size = offsetof_log_true_size;
log->uattr = uattr;
if (!ubuf && ubuf_common) {
log->ubuf = ubuf_common;
log->size = common->log_size;
log->level = common->log_level;
log->uattr = uattr_common;
log->offsetof_true_size = 0;
if (size_common >= offsetofend(struct bpf_common_attr, log_true_size))
log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
}
return 0;
}
struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
struct bpf_common_attr *common, bpfptr_t uattr,
u32 size)
{
struct bpf_verifier_log *log;
int err;
memset(attr_log, 0, sizeof(*attr_log));
attr_log->uattr = uattr;
if (size >= offsetofend(struct bpf_common_attr, log_true_size))
attr_log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size);
if (!size)
return NULL;
log = kzalloc_obj(*log, GFP_KERNEL);
if (!log)
return ERR_PTR(-ENOMEM);
err = bpf_vlog_init(log, common->log_level, u64_to_user_ptr(common->log_buf),
common->log_size);
if (err) {
kfree(log);
return ERR_PTR(err);
}
return log;
}
int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log)
{
u32 log_true_size;
int err;
err = bpf_vlog_finalize(log, &log_true_size);
if (attr->offsetof_true_size &&
copy_to_bpfptr_offset(attr->uattr, attr->offsetof_true_size, &log_true_size,
sizeof(log_true_size)))
return -EFAULT;
return err;
}

8
kernel/bpf/lpm_trie.c
View File

@@ -246,7 +246,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key)
/* Start walking the trie from the root node ... */
for (node = rcu_dereference_check(trie->root, rcu_read_lock_bh_held());
for (node = rcu_dereference_check(trie->root, bpf_rcu_lock_held());
node;) {
unsigned int next_bit;
size_t matchlen;
@@ -280,7 +280,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key)
*/
next_bit = extract_bit(key->data, node->prefixlen);
node = rcu_dereference_check(node->child[next_bit],
rcu_read_lock_bh_held());
bpf_rcu_lock_held());
}
if (!found)
@@ -359,7 +359,7 @@ static long trie_update_elem(struct bpf_map *map,
*/
slot = &trie->root;
while ((node = rcu_dereference(*slot))) {
while ((node = rcu_dereference_protected(*slot, 1))) {
matchlen = longest_prefix_match(trie, node, key);
if (node->prefixlen != matchlen ||
@@ -482,7 +482,7 @@ static long trie_delete_elem(struct bpf_map *map, void *_key)
trim = &trie->root;
trim2 = trim;
parent = NULL;
while ((node = rcu_dereference(*trim))) {
while ((node = rcu_dereference_protected(*trim, 1))) {
matchlen = longest_prefix_match(trie, node, key);
if (node->prefixlen != matchlen ||

5
kernel/bpf/map_in_map.c
View File

@@ -20,7 +20,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
/* Does not support >1 level map-in-map */
if (inner_map->inner_map_meta)
return ERR_PTR(-EINVAL);
if (inner_map->excl_prog_sha)
return ERR_PTR(-ENOTSUPP);
if (!inner_map->ops->map_meta_equal)
return ERR_PTR(-ENOTSUPP);
@@ -101,6 +102,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map,
inner_map = __bpf_map_get(f);
if (IS_ERR(inner_map))
return inner_map;
if (inner_map->excl_prog_sha)
return ERR_PTR(-ENOTSUPP);
inner_map_meta = map->inner_map_meta;
if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))

7
kernel/bpf/map_iter.c
View File

@@ -112,6 +112,10 @@ static int bpf_iter_attach_map(struct bpf_prog *prog,
map = bpf_map_get_with_uref(linfo->map.map_fd);
if (IS_ERR(map))
return PTR_ERR(map);
if (map->excl_prog_sha) {
err = -EPERM;
goto put_map;
}
if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
@@ -119,7 +123,8 @@ static int bpf_iter_attach_map(struct bpf_prog *prog,
is_percpu = true;
else if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_LRU_HASH &&
map->map_type != BPF_MAP_TYPE_ARRAY)
map->map_type != BPF_MAP_TYPE_ARRAY &&
map->map_type != BPF_MAP_TYPE_RHASH)
goto put_map;
key_acc_size = prog->aux->max_rdonly_access;

221
kernel/bpf/stackmap.c
View File

@@ -9,6 +9,7 @@
#include <linux/perf_event.h>
#include <linux/btf_ids.h>
#include <linux/buildid.h>
#include <linux/mmap_lock.h>
#include "percpu_freelist.h"
#include "mmap_unlock_work.h"
@@ -152,6 +153,180 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b
: build_id_parse_nofault(vma, build_id, NULL);
}
static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id)
{
id->status = BPF_STACK_BUILD_ID_IP;
memset(id->build_id, 0, BUILD_ID_SIZE_MAX);
}
static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff,
unsigned long vm_start, u64 ip)
{
return (vm_pgoff << PAGE_SHIFT) + ip - vm_start;
}
static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id,
u64 offset,
const unsigned char *build_id)
{
id->status = BPF_STACK_BUILD_ID_VALID;
id->offset = offset;
if (id->build_id != build_id)
memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX);
}
struct stack_map_vma_lock {
struct vm_area_struct *vma;
struct mm_struct *mm;
};
/*
* Acquire a stable read-side reference on the VMA covering @ip.
*
* With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read
* lock held and mmap_lock dropped, so the caller may sleep.
*
* With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still
* held; the caller must snapshot any fields it needs and pin vm_file
* with get_file() before stack_map_unlock_vma() drops mmap_lock, as
* the VMA may be split, merged, or freed after that.
*
* Returns NULL on failure, in which case no lock is held.
*/
static struct vm_area_struct *
stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip)
{
struct mm_struct *mm = lock->mm;
struct vm_area_struct *vma;
/* noop under !CONFIG_PER_VMA_LOCK */
vma = lock_vma_under_rcu(mm, ip);
if (vma) {
lock->vma = vma;
return vma;
}
/*
* Taking mmap_read_lock() is unsafe here, because the caller BPF
* program might already hold it, causing a deadlock.
*/
if (!mmap_read_trylock(mm))
return NULL;
vma = vma_lookup(mm, ip);
if (!vma) {
mmap_read_unlock(mm);
return NULL;
}
#ifdef CONFIG_PER_VMA_LOCK
if (!vma_start_read_locked(vma)) {
mmap_read_unlock(mm);
return NULL;
}
mmap_read_unlock(mm);
#endif
lock->vma = vma;
return vma;
}
static void stack_map_unlock_vma(struct stack_map_vma_lock *lock)
{
#ifdef CONFIG_PER_VMA_LOCK
vma_end_read(lock->vma);
#else
mmap_read_unlock(lock->mm);
#endif
lock->vma = NULL;
}
static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs,
u32 trace_nr)
{
struct mm_struct *mm = current->mm;
struct stack_map_vma_lock lock = { .mm = mm };
struct {
struct file *file;
const unsigned char *build_id;
unsigned long vm_start;
unsigned long vm_end;
unsigned long vm_pgoff;
} cache = {};
unsigned long vm_pgoff, vm_start, vm_end;
struct vm_area_struct *vma;
struct file *file;
u64 offset;
u64 ip;
for (u32 i = 0; i < trace_nr; i++) {
ip = READ_ONCE(id_offs[i].ip);
/*
* Range cache fast path: if ip falls within the previously
* resolved VMA range, reuse the cache build_id without
* re-acquiring the VMA lock.
*/
if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) {
offset = stack_map_build_id_offset(cache.vm_pgoff, cache.vm_start, ip);
stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
continue;
}
vma = stack_map_lock_vma(&lock, ip);
if (!vma) {
stack_map_build_id_set_ip(&id_offs[i]);
continue;
}
if (vma_is_anonymous(vma) || !vma->vm_file) {
stack_map_build_id_set_ip(&id_offs[i]);
stack_map_unlock_vma(&lock);
continue;
}
file = vma->vm_file;
vm_pgoff = vma->vm_pgoff;
vm_start = vma->vm_start;
vm_end = vma->vm_end;
offset = stack_map_build_id_offset(vm_pgoff, vm_start, ip);
/*
* Same backing file as previous (e.g. different VMAs
* of the same ELF binary). Reuse the cache build_id.
*/
if (file == cache.file) {
stack_map_unlock_vma(&lock);
stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id);
cache.vm_start = vm_start;
cache.vm_end = vm_end;
cache.vm_pgoff = vm_pgoff;
continue;
}
file = get_file(file);
stack_map_unlock_vma(&lock);
/* build_id_parse_file() may block on filesystem reads */
if (build_id_parse_file(file, id_offs[i].build_id, NULL)) {
stack_map_build_id_set_ip(&id_offs[i]);
fput(file);
continue;
}
stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
if (cache.file)
fput(cache.file);
cache.file = file;
cache.build_id = id_offs[i].build_id;
cache.vm_start = vm_start;
cache.vm_end = vm_end;
cache.vm_pgoff = vm_pgoff;
}
if (cache.file)
fput(cache.file);
}
/*
* Expects all id_offs[i].ip values to be set to correct initial IPs.
* They will be subsequently:
@@ -165,44 +340,50 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b
static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
u32 trace_nr, bool user, bool may_fault)
{
int i;
struct mmap_unlock_irq_work *work = NULL;
bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
bool has_user_ctx = user && current && current->mm;
struct vm_area_struct *vma, *prev_vma = NULL;
const char *prev_build_id;
const unsigned char *prev_build_id = NULL;
int i;
if (may_fault && has_user_ctx) {
stack_map_get_build_id_offset_sleepable(id_offs, trace_nr);
return;
}
/* If the irq_work is in use, fall back to report ips. Same
* fallback is used for kernel stack (!user) on a stackmap with
* build_id.
*/
if (!user || !current || !current->mm || irq_work_busy ||
!mmap_read_trylock(current->mm)) {
if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) {
/* cannot access current->mm, fall back to ips */
for (i = 0; i < trace_nr; i++) {
id_offs[i].status = BPF_STACK_BUILD_ID_IP;
memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
}
for (i = 0; i < trace_nr; i++)
stack_map_build_id_set_ip(&id_offs[i]);
return;
}
for (i = 0; i < trace_nr; i++) {
u64 ip = READ_ONCE(id_offs[i].ip);
u64 offset;
if (range_in_vma(prev_vma, ip, ip)) {
if (prev_build_id && range_in_vma(prev_vma, ip, ip)) {
vma = prev_vma;
memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX);
goto build_id_valid;
}
vma = find_vma(current->mm, ip);
if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) {
/* per entry fall back to ips */
id_offs[i].status = BPF_STACK_BUILD_ID_IP;
memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id);
continue;
}
build_id_valid:
id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start;
id_offs[i].status = BPF_STACK_BUILD_ID_VALID;
vma = find_vma(current->mm, ip);
if (!vma || vma_is_anonymous(vma) ||
fetch_build_id(vma, id_offs[i].build_id, may_fault)) {
/* per entry fall back to ips */
stack_map_build_id_set_ip(&id_offs[i]);
prev_vma = vma;
prev_build_id = NULL;
continue;
}
offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip);
stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id);
prev_vma = vma;
prev_build_id = id_offs[i].build_id;
}

67
kernel/bpf/states.c
View File

@@ -2,6 +2,7 @@
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/cnum.h>
#include <linux/filter.h>
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -301,14 +302,8 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s
static bool range_within(const struct bpf_reg_state *old,
const struct bpf_reg_state *cur)
{
return old->umin_value <= cur->umin_value &&
old->umax_value >= cur->umax_value &&
old->smin_value <= cur->smin_value &&
old->smax_value >= cur->smax_value &&
old->u32_min_value <= cur->u32_min_value &&
old->u32_max_value >= cur->u32_max_value &&
old->s32_min_value <= cur->s32_min_value &&
old->s32_max_value >= cur->s32_max_value;
return cnum64_is_subset(old->r64, cur->r64) &&
cnum32_is_subset(old->r32, cur->r32);
}
/* If in the old state two registers had the same id, then they need to have
@@ -348,8 +343,12 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
return true;
}
/* We ran out of idmap slots, which should be impossible */
WARN_ON_ONCE(1);
/*
* idmap slots are bounded by the number of registers and stack slots.
* Since referenced dynptrs acquire intermediate references that do
* not live in either, so the map can be exhausted. Since it is unlikely,
* fail the verification by treating the states as not equivalent.
*/
return false;
}
@@ -494,7 +493,7 @@ static bool regs_exact(const struct bpf_reg_state *rold,
{
return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
check_ids(rold->id, rcur->id, idmap) &&
check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);
check_ids(rold->parent_id, rcur->parent_id, idmap);
}
enum exact_level {
@@ -619,7 +618,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
range_within(rold, rcur) &&
tnum_in(rold->var_off, rcur->var_off) &&
check_ids(rold->id, rcur->id, idmap) &&
check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap);
check_ids(rold->parent_id, rcur->parent_id, idmap);
case PTR_TO_PACKET_META:
case PTR_TO_PACKET:
/* We must have at least as much range as the old ptr
@@ -799,7 +798,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
cur_reg = &cur->stack[spi].spilled_ptr;
if (old_reg->dynptr.type != cur_reg->dynptr.type ||
old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||
!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))
!check_ids(old_reg->id, cur_reg->id, idmap) ||
!check_ids(old_reg->parent_id, cur_reg->parent_id, idmap))
return false;
break;
case STACK_ITER:
@@ -815,13 +815,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
old_reg->iter.btf_id != cur_reg->iter.btf_id ||
old_reg->iter.state != cur_reg->iter.state ||
/* ignore {old_reg,cur_reg}->iter.depth, see above */
!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap))
!check_ids(old_reg->id, cur_reg->id, idmap))
return false;
break;
case STACK_IRQ_FLAG:
old_reg = &old->stack[spi].spilled_ptr;
cur_reg = &cur->stack[spi].spilled_ptr;
if (!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap) ||
if (!check_ids(old_reg->id, cur_reg->id, idmap) ||
old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)
return false;
break;
@@ -838,6 +838,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return true;
}
/*
* Compare stack arg slots between old and current states.
* Outgoing stack args are path-local state and must agree for pruning.
*/
static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
struct bpf_func_state *cur, struct bpf_idmap *idmap,
enum exact_level exact)
{
int i, nslots;
nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt);
for (i = 0; i < nslots; i++) {
struct bpf_reg_state *old_arg, *cur_arg;
struct bpf_reg_state not_init = { .type = NOT_INIT };
old_arg = i < old->out_stack_arg_cnt ?
&old->stack_arg_regs[i] : &not_init;
cur_arg = i < cur->out_stack_arg_cnt ?
&cur->stack_arg_regs[i] : &not_init;
if (!regsafe(env, old_arg, cur_arg, idmap, exact))
return false;
}
return true;
}
static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur,
struct bpf_idmap *idmap)
{
@@ -868,6 +894,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c
return false;
switch (old->refs[i].type) {
case REF_TYPE_PTR:
if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap))
return false;
break;
case REF_TYPE_IRQ:
break;
case REF_TYPE_LOCK:
@@ -920,6 +949,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
if (old->callback_depth > cur->callback_depth)
return false;
if (!old->no_stack_arg_load && cur->no_stack_arg_load)
return false;
for (i = 0; i < MAX_BPF_REG; i++)
if (((1 << i) & live_regs) &&
!regsafe(env, &old->regs[i], &cur->regs[i],
@@ -929,6 +961,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
return false;
if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
return false;
return true;
}
@@ -1376,7 +1411,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
err = 0;
if (bpf_is_jmp_point(env, env->insn_idx))
err = bpf_push_jmp_history(env, cur, 0, 0);
err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
err = err ? : propagate_precision(env, &sl->state, cur, NULL);
if (err)
return err;

312
kernel/bpf/syscall.c
View File

@@ -41,6 +41,7 @@
#include <linux/overflow.h>
#include <linux/cookie.h>
#include <linux/verification.h>
#include <linux/btf_ids.h>
#include <net/netfilter/nf_bpf_link.h>
#include <net/netkit.h>
@@ -807,6 +808,11 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj)
bpf_task_work_cancel_and_free(obj + rec->task_work_off);
}
void bpf_obj_cancel_fields(struct bpf_map *map, void *obj)
{
bpf_map_free_internal_structs(map, obj);
}
void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
{
const struct btf_field *fields;
@@ -1280,6 +1286,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
case BPF_SPIN_LOCK:
case BPF_RES_SPIN_LOCK:
if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_RHASH &&
map->map_type != BPF_MAP_TYPE_ARRAY &&
map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
@@ -1294,6 +1301,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
case BPF_WORKQUEUE:
case BPF_TASK_WORK:
if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_RHASH &&
map->map_type != BPF_MAP_TYPE_LRU_HASH &&
map->map_type != BPF_MAP_TYPE_ARRAY) {
ret = -EOPNOTSUPP;
@@ -1305,6 +1313,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
case BPF_KPTR_PERCPU:
case BPF_REFCOUNT:
if (map->map_type != BPF_MAP_TYPE_HASH &&
map->map_type != BPF_MAP_TYPE_RHASH &&
map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
map->map_type != BPF_MAP_TYPE_LRU_HASH &&
map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH &&
@@ -1359,7 +1368,8 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size
/* called via syscall */
static int map_create(union bpf_attr *attr, bpfptr_t uattr)
static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,
struct bpf_map **mapp, struct bpf_token **tokenp)
{
const struct bpf_map_ops *ops;
struct bpf_token *token = NULL;
@@ -1367,12 +1377,13 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
u32 map_type = attr->map_type;
struct bpf_map *map;
bool token_flag;
int f_flags;
int err;
err = CHECK_ATTR(BPF_MAP_CREATE);
if (err)
if (err) {
bpf_log(log, "Invalid attr.\n");
return -EINVAL;
}
/* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it
* to avoid per-map type checks tripping on unknown flag
@@ -1381,31 +1392,40 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
attr->map_flags &= ~BPF_F_TOKEN_FD;
if (attr->btf_vmlinux_value_type_id) {
if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS ||
attr->btf_key_type_id || attr->btf_value_type_id)
if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS) {
bpf_log(log, "btf_vmlinux_value_type_id can only be used with struct_ops maps.\n");
return -EINVAL;
}
if (attr->btf_key_type_id || attr->btf_value_type_id) {
bpf_log(log, "btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\n");
return -EINVAL;
}
} else if (attr->btf_key_type_id && !attr->btf_value_type_id) {
bpf_log(log, "Invalid btf_value_type_id.\n");
return -EINVAL;
}
if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER &&
attr->map_type != BPF_MAP_TYPE_ARENA &&
attr->map_extra != 0)
attr->map_type != BPF_MAP_TYPE_RHASH &&
attr->map_extra != 0) {
bpf_log(log, "Invalid map_extra.\n");
return -EINVAL;
f_flags = bpf_get_file_flag(attr->map_flags);
if (f_flags < 0)
return f_flags;
}
if (numa_node != NUMA_NO_NODE &&
((unsigned int)numa_node >= nr_node_ids ||
!node_online(numa_node)))
!node_online(numa_node))) {
bpf_log(log, "Invalid numa_node.\n");
return -EINVAL;
}
/* find map type and init map: hashtable vs rbtree vs bloom vs ... */
map_type = attr->map_type;
if (map_type >= ARRAY_SIZE(bpf_map_types))
if (map_type >= ARRAY_SIZE(bpf_map_types)) {
bpf_log(log, "Invalid map_type.\n");
return -EINVAL;
}
map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types));
ops = bpf_map_types[map_type];
if (!ops)
@@ -1423,8 +1443,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
if (token_flag) {
token = bpf_token_get_from_fd(attr->map_token_fd);
if (IS_ERR(token))
if (IS_ERR(token)) {
bpf_log(log, "Invalid map_token_fd.\n");
return PTR_ERR(token);
}
/* if current token doesn't grant map creation permissions,
* then we can't use this token, so ignore it and rely on
@@ -1457,6 +1479,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
case BPF_MAP_TYPE_CGROUP_ARRAY:
case BPF_MAP_TYPE_ARRAY_OF_MAPS:
case BPF_MAP_TYPE_HASH:
case BPF_MAP_TYPE_RHASH:
case BPF_MAP_TYPE_PERCPU_HASH:
case BPF_MAP_TYPE_HASH_OF_MAPS:
case BPF_MAP_TYPE_RINGBUF:
@@ -1507,8 +1530,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
err = bpf_obj_name_cpy(map->name, attr->map_name,
sizeof(attr->map_name));
if (err < 0)
if (err < 0) {
bpf_log(log, "Invalid map_name.\n");
goto free_map;
}
preempt_disable();
map->cookie = gen_cookie_next(&bpf_map_cookie);
@@ -1531,6 +1556,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
btf = btf_get_by_fd(attr->btf_fd);
if (IS_ERR(btf)) {
bpf_log(log, "Invalid btf_fd.\n");
err = PTR_ERR(btf);
goto free_map;
}
@@ -1558,6 +1584,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel);
if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) {
bpf_log(log, "Invalid excl_prog_hash_size.\n");
err = -EINVAL;
goto free_map;
}
@@ -1572,11 +1599,62 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
err = -EFAULT;
goto free_map;
}
/* See libbpf: emit_signature_match() */
BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE);
BUILD_BUG_ON(!__same_type(map->excl, u32));
BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0);
BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE]));
map->excl = 1;
} else if (attr->excl_prog_hash_size) {
bpf_log(log, "Invalid excl_prog_hash_size.\n");
err = -EINVAL;
goto free_map;
}
*mapp = map;
*tokenp = token;
return 0;
free_map:
bpf_map_free(map);
put_token:
bpf_token_put(token);
return err;
}
static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_attr *attr_common,
bpfptr_t uattr_common, u32 size_common)
{
struct bpf_token *token = NULL;
struct bpf_verifier_log *log;
struct bpf_log_attr attr_log;
struct bpf_map *map = NULL;
int err, ret;
int f_flags;
log = bpf_log_attr_create_vlog(&attr_log, attr_common, uattr_common, size_common);
if (IS_ERR(log))
return PTR_ERR(log);
err = map_create_alloc(attr, uattr, log, &map, &token);
/* preserve original error even if log finalization is successful */
ret = bpf_log_attr_finalize(&attr_log, log);
if (ret)
err = ret;
kfree(log);
if (err)
goto free_map;
f_flags = bpf_get_file_flag(attr->map_flags);
if (f_flags < 0) {
err = f_flags;
goto free_map;
}
err = security_bpf_map_create(map, attr, token, uattr.is_kernel);
if (err)
goto free_map_sec;
@@ -1605,8 +1683,8 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
free_map_sec:
security_bpf_map_free(map);
free_map:
bpf_map_free(map);
put_token:
if (map)
bpf_map_free(map);
bpf_token_put(token);
return err;
}
@@ -2192,6 +2270,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr)
map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
map->map_type == BPF_MAP_TYPE_LRU_HASH ||
map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
map->map_type == BPF_MAP_TYPE_RHASH ||
map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
if (!bpf_map_is_offloaded(map)) {
bpf_disable_instrumentation();
@@ -2646,7 +2725,8 @@ static int
bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
enum bpf_attach_type expected_attach_type,
struct btf *attach_btf, u32 btf_id,
struct bpf_prog *dst_prog)
struct bpf_prog *dst_prog,
bool multi_func)
{
if (btf_id) {
if (btf_id > BTF_MAX_TYPE)
@@ -2666,6 +2746,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
}
}
if (multi_func) {
if (prog_type != BPF_PROG_TYPE_TRACING)
return -EINVAL;
if (!attach_btf || btf_id)
return -EINVAL;
return 0;
}
if (attach_btf && (!btf_id || dst_prog))
return -EINVAL;
@@ -2798,8 +2886,22 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type)
}
}
static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id)
{
switch (keyring_id) {
case 0:
return BPF_SIG_KEYRING_BUILTIN;
case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING:
return BPF_SIG_KEYRING_SECONDARY;
case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING:
return BPF_SIG_KEYRING_PLATFORM;
default:
return BPF_SIG_KEYRING_USER;
}
}
static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr,
bool is_kernel)
bool is_kernel, s32 *keyring_serial)
{
bpfptr_t usig = make_bpfptr(attr->signature, is_kernel);
struct bpf_dynptr_kern sig_ptr, insns_ptr;
@@ -2835,7 +2937,8 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr
err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr,
(struct bpf_dynptr *)&sig_ptr, key);
if (!err)
*keyring_serial = bpf_key_serial(key);
bpf_key_put(key);
kvfree(sig);
return err;
@@ -2858,10 +2961,15 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog)
return 0;
}
extern int bpf_multi_func(void);
int __init __used bpf_multi_func(void) { return 0; }
BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
/* last field in 'union bpf_attr' used by this command */
#define BPF_PROG_LOAD_LAST_FIELD keyring_id
static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
{
enum bpf_prog_type type = attr->prog_type;
struct bpf_prog *prog, *dst_prog = NULL;
@@ -2870,6 +2978,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
bool bpf_cap;
int err;
char license[128];
bool multi_func;
if (CHECK_ATTR(BPF_PROG_LOAD))
return -EINVAL;
@@ -2936,6 +3045,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
goto put_token;
multi_func = is_tracing_multi(attr->expected_attach_type);
/* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
* or btf, we need to check which one it is
*/
@@ -2957,7 +3068,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
goto put_token;
}
}
} else if (attr->attach_btf_id) {
} else if (attr->attach_btf_id || multi_func) {
/* fall back to vmlinux BTF, if BTF type ID is specified */
attach_btf = bpf_get_btf_vmlinux();
if (IS_ERR(attach_btf)) {
@@ -2973,7 +3084,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
attach_btf, attr->attach_btf_id,
dst_prog)) {
dst_prog, multi_func)) {
if (dst_prog)
bpf_prog_put(dst_prog);
if (attach_btf)
@@ -2996,7 +3107,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
prog->expected_attach_type = attr->expected_attach_type;
prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
prog->aux->attach_btf = attach_btf;
prog->aux->attach_btf_id = attr->attach_btf_id;
prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id;
prog->aux->dst_prog = dst_prog;
prog->aux->dev_bound = !!attr->prog_ifindex;
prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
@@ -3022,13 +3133,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
/* eBPF programs must be GPL compatible to use GPL-ed functions */
prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;
if (attr->signature) {
err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel);
err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel,
&prog->aux->sig.keyring_serial);
if (err)
goto free_prog;
prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id);
prog->aux->sig.verdict = BPF_SIG_VERIFIED;
} else {
prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE;
prog->aux->sig.verdict = BPF_SIG_UNSIGNED;
}
prog->orig_prog = NULL;
prog->jited = 0;
@@ -3076,10 +3191,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);
if (err)
goto free_prog_sec;
goto free_prog;
/* run eBPF verifier */
err = bpf_check(&prog, attr, uattr, uattr_size);
err = bpf_check(&prog, attr, uattr, attr_log);
if (err < 0)
goto free_used_maps;
@@ -3122,8 +3237,6 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
__bpf_prog_put_noref(prog, prog->aux->real_func_cnt);
return err;
free_prog_sec:
security_bpf_prog_free(prog);
free_prog:
free_uid(prog->aux->user);
if (prog->aux->attach_btf)
@@ -3198,6 +3311,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
bpf_link_init_sleepable(link, type, ops, prog, attach_type, false);
}
void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, u64 cookie)
{
bpf_link_init(&link->link, type, ops, prog, attach_type);
link->node.link = &link->link;
link->node.cookie = cookie;
}
static void bpf_link_free_id(int id)
{
if (!id)
@@ -3358,7 +3480,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ?
"kretprobe_multi" : "kprobe_multi");
else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI)
seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_UPROBE_MULTI_RETURN ?
seq_printf(m, "link_type:\t%s\n", link->flags & BPF_F_UPROBE_MULTI_RETURN ?
"uretprobe_multi" : "uprobe_multi");
else
seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]);
@@ -3505,7 +3627,7 @@ static void bpf_tracing_link_release(struct bpf_link *link)
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link,
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node,
tr_link->trampoline,
tr_link->tgt_prog));
@@ -3518,8 +3640,7 @@ static void bpf_tracing_link_release(struct bpf_link *link)
static void bpf_tracing_link_dealloc(struct bpf_link *link)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
kfree(tr_link);
}
@@ -3527,8 +3648,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link)
static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
u32 target_btf_id, target_obj_id;
bpf_trampoline_unpack_key(tr_link->trampoline->key,
@@ -3541,17 +3662,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link,
link->attach_type,
target_obj_id,
target_btf_id,
tr_link->link.cookie);
tr_link->link.node.cookie);
}
static int bpf_tracing_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
struct bpf_tracing_link *tr_link =
container_of(link, struct bpf_tracing_link, link.link);
struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link);
info->tracing.attach_type = link->attach_type;
info->tracing.cookie = tr_link->link.cookie;
info->tracing.cookie = tr_link->link.node.cookie;
bpf_trampoline_unpack_key(tr_link->trampoline->key,
&info->tracing.target_obj_id,
&info->tracing.target_btf_id);
@@ -3633,29 +3753,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id);
}
if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
struct bpf_fsession_link *fslink;
fslink = kzalloc_obj(*fslink, GFP_USER);
if (fslink) {
bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type);
fslink->fexit.cookie = bpf_cookie;
link = &fslink->link;
} else {
link = NULL;
}
} else {
link = kzalloc_obj(*link, GFP_USER);
}
link = kzalloc_obj(*link, GFP_USER);
if (!link) {
err = -ENOMEM;
goto out_put_prog;
}
bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type);
bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING,
&bpf_tracing_link_lops, prog, attach_type, bpf_cookie);
link->link.cookie = bpf_cookie;
if (prog->expected_attach_type == BPF_TRACE_FSESSION) {
link->fexit.link = &link->link.link;
link->fexit.cookie = bpf_cookie;
}
mutex_lock(&prog->aux->dst_mutex);
@@ -3758,7 +3867,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog);
err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog);
if (err) {
bpf_link_cleanup(&link_primer);
link = NULL;
@@ -4281,6 +4390,11 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog,
if (!btp)
return -ENOENT;
if (prog->sleepable && !tracepoint_is_faultable(btp->tp)) {
bpf_put_raw_tracepoint(btp);
return -EINVAL;
}
link = kzalloc_obj(*link, GFP_USER);
if (!link) {
err = -ENOMEM;
@@ -4389,6 +4503,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type)
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
case BPF_TRACE_FSESSION_MULTI:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_MODIFY_RETURN:
return BPF_PROG_TYPE_TRACING;
case BPF_LSM_MAC:
@@ -4654,7 +4771,7 @@ static int bpf_prog_detach(const union bpf_attr *attr)
#define BPF_PROG_QUERY_LAST_FIELD query.revision
static int bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
union bpf_attr __user *uattr, u32 uattr_size)
{
if (!bpf_net_capable())
return -EPERM;
@@ -4693,7 +4810,7 @@ static int bpf_prog_query(const union bpf_attr *attr,
case BPF_CGROUP_GETSOCKOPT:
case BPF_CGROUP_SETSOCKOPT:
case BPF_LSM_CGROUP:
return cgroup_bpf_prog_query(attr, uattr);
return cgroup_bpf_prog_query(attr, uattr, uattr_size);
case BPF_LIRC_MODE2:
return lirc_prog_query(attr, uattr);
case BPF_FLOW_DISSECTOR:
@@ -5045,10 +5162,11 @@ static int bpf_prog_get_info_by_fd(struct file *file,
u32 info_len = attr->info.info_len;
struct bpf_prog_kstats stats;
char __user *uinsns;
u32 ulen;
u32 ulen, len;
int err;
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
len = offsetofend(struct bpf_prog_info, attach_btf_id);
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
if (err)
return err;
info_len = min_t(u32, sizeof(info), info_len);
@@ -5330,10 +5448,11 @@ static int bpf_map_get_info_by_fd(struct file *file,
{
struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info);
struct bpf_map_info info;
u32 info_len = attr->info.info_len;
u32 info_len = attr->info.info_len, len;
int err;
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len);
len = offsetofend(struct bpf_map_info, hash_size);
err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len);
if (err)
return err;
info_len = min_t(u32, sizeof(info), info_len);
@@ -5371,18 +5490,16 @@ static int bpf_map_get_info_by_fd(struct file *file,
if (!map->ops->map_get_hash)
return -EINVAL;
if (info.hash_size != SHA256_DIGEST_SIZE)
if (info.hash_size != sizeof(map->sha))
return -EINVAL;
if (!READ_ONCE(map->frozen))
return -EPERM;
err = map->ops->map_get_hash(map, SHA256_DIGEST_SIZE, map->sha);
err = map->ops->map_get_hash(map);
if (err != 0)
return err;
if (copy_to_user(uhash, map->sha, SHA256_DIGEST_SIZE) != 0)
if (copy_to_user(uhash, map->sha, sizeof(map->sha)) != 0)
return -EFAULT;
} else if (info.hash_size) {
return -EINVAL;
@@ -5495,7 +5612,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
#define BPF_BTF_LOAD_LAST_FIELD btf_token_fd
static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_size)
static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
{
struct bpf_token *token = NULL;
@@ -5522,7 +5639,7 @@ static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_
bpf_token_put(token);
return btf_new_fd(attr, uattr, uattr_size);
return btf_new_fd(attr, uattr, attr_log);
}
#define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd
@@ -5723,7 +5840,7 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
return err;
}
#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid
#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd
static int link_create(union bpf_attr *attr, bpfptr_t uattr)
{
struct bpf_prog *prog;
@@ -5774,6 +5891,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
ret = bpf_iter_link_attach(attr, uattr, prog);
else if (prog->expected_attach_type == BPF_LSM_CGROUP)
ret = cgroup_bpf_link_attach(attr, prog);
else if (is_tracing_multi(prog->expected_attach_type))
ret = bpf_tracing_multi_attach(prog, attr);
else
ret = bpf_tracing_prog_attach(prog,
attr->link_create.target_fd,
@@ -6232,8 +6351,12 @@ static int prog_assoc_struct_ops(union bpf_attr *attr)
return ret;
}
static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size,
bpfptr_t uattr_common, unsigned int size_common)
{
struct bpf_common_attr attr_common;
u32 offsetof_log_true_size = 0;
struct bpf_log_attr attr_log;
union bpf_attr attr;
int err;
@@ -6247,13 +6370,29 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
if (copy_from_bpfptr(&attr, uattr, size) != 0)
return -EFAULT;
memset(&attr_common, 0, sizeof(attr_common));
if (cmd & BPF_COMMON_ATTRS) {
err = bpf_check_uarg_tail_zero(uattr_common,
offsetofend(struct bpf_common_attr, log_true_size),
size_common);
if (err)
return err;
cmd &= ~BPF_COMMON_ATTRS;
size_common = min_t(u32, size_common, sizeof(attr_common));
if (copy_from_bpfptr(&attr_common, uattr_common, size_common) != 0)
return -EFAULT;
} else {
size_common = 0;
}
err = security_bpf(cmd, &attr, size, uattr.is_kernel);
if (err < 0)
return err;
switch (cmd) {
case BPF_MAP_CREATE:
err = map_create(&attr, uattr);
err = map_create(&attr, uattr, &attr_common, uattr_common, size_common);
break;
case BPF_MAP_LOOKUP_ELEM:
err = map_lookup_elem(&attr);
@@ -6271,7 +6410,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
err = map_freeze(&attr);
break;
case BPF_PROG_LOAD:
err = bpf_prog_load(&attr, uattr, size);
if (size >= offsetofend(union bpf_attr, log_true_size))
offsetof_log_true_size = offsetof(union bpf_attr, log_true_size);
err = bpf_log_attr_init(&attr_log, attr.log_buf, attr.log_size, attr.log_level,
offsetof_log_true_size, uattr, &attr_common, uattr_common,
size_common);
err = err ?: bpf_prog_load(&attr, uattr, &attr_log);
break;
case BPF_OBJ_PIN:
err = bpf_obj_pin(&attr);
@@ -6286,7 +6430,7 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
err = bpf_prog_detach(&attr);
break;
case BPF_PROG_QUERY:
err = bpf_prog_query(&attr, uattr.user);
err = bpf_prog_query(&attr, uattr.user, size);
break;
case BPF_PROG_TEST_RUN:
err = bpf_prog_test_run(&attr, uattr.user);
@@ -6316,7 +6460,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
err = bpf_raw_tracepoint_open(&attr);
break;
case BPF_BTF_LOAD:
err = bpf_btf_load(&attr, uattr, size);
if (size >= offsetofend(union bpf_attr, btf_log_true_size))
offsetof_log_true_size = offsetof(union bpf_attr, btf_log_true_size);
err = bpf_log_attr_init(&attr_log, attr.btf_log_buf, attr.btf_log_size,
attr.btf_log_level, offsetof_log_true_size, uattr,
&attr_common, uattr_common, size_common);
err = err ?: bpf_btf_load(&attr, uattr, &attr_log);
break;
case BPF_BTF_GET_FD_BY_ID:
err = bpf_btf_get_fd_by_id(&attr);
@@ -6382,9 +6531,10 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
return err;
}
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size,
struct bpf_common_attr __user *, uattr_common, unsigned int, size_common)
{
return __sys_bpf(cmd, USER_BPFPTR(uattr), size);
return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common);
}
static bool syscall_prog_is_valid_access(int off, int size,
@@ -6414,7 +6564,7 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size)
default:
return -EINVAL;
}
return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size);
return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
}

673
kernel/bpf/trampoline.c
View File

@@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE];
/* serializes access to trampoline tables */
static DEFINE_MUTEX(trampoline_mutex);
/*
* Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all()
* stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep
* class because trampoline_lock_all() takes all pool mutexes at once;
* otherwise lockdep would report recursive locking on same-class mutexes.
*/
#define TRAMPOLINE_LOCKS_BITS 5
#define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS)
static struct {
struct mutex mutex;
struct lock_class_key key;
} trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE];
static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr)
{
return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex;
}
static void trampoline_lock(struct bpf_trampoline *tr)
{
mutex_lock(select_trampoline_lock(tr));
}
static void trampoline_unlock(struct bpf_trampoline *tr)
{
mutex_unlock(select_trampoline_lock(tr));
}
struct bpf_trampoline_ops {
int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data);
int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data);
int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *data);
};
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
const struct bpf_trampoline_ops *ops, void *data);
static const struct bpf_trampoline_ops trampoline_ops;
#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
@@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
/* This is called inside register_ftrace_direct_multi(), so
* tr->mutex is already locked.
* trampoline's mutex is already locked.
*/
lockdep_assert_held_once(&tr->mutex);
lockdep_assert_held_once(select_trampoline_lock(tr));
/* Instead of updating the trampoline here, we propagate
* -EAGAIN to register_ftrace_direct(). Then we can
@@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
}
/* The normal locking order is
* tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
* select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c)
*
* The following two commands are called from
*
@@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
* cleanup_direct_functions_after_ipmodify
*
* In both cases, direct_mutex is already locked. Use
* mutex_trylock(&tr->mutex) to avoid deadlock in race condition
* (something else is making changes to this same trampoline).
* mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition
* (something else holds the same pool lock).
*/
if (!mutex_trylock(&tr->mutex)) {
/* sleep 1 ms to make sure whatever holding tr->mutex makes
* some progress.
if (!mutex_trylock(select_trampoline_lock(tr))) {
/* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr)
* makes some progress.
*/
msleep(1);
return -EAGAIN;
@@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) &&
!(tr->flags & BPF_TRAMP_F_ORIG_STACK))
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
&trampoline_ops, NULL);
break;
case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER:
tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY;
if (tr->flags & BPF_TRAMP_F_ORIG_STACK)
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */);
ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */,
&trampoline_ops, NULL);
break;
default:
ret = -EINVAL;
break;
}
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return ret;
}
#endif
@@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
switch (ptype) {
case BPF_PROG_TYPE_TRACING:
if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT ||
eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION)
eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION ||
eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI ||
eatype == BPF_TRACE_FSESSION_MULTI)
return true;
return false;
case BPF_PROG_TYPE_LSM:
@@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)];
hlist_add_head(&tr->hlist_ip, head);
refcount_set(&tr->refcnt, 1);
mutex_init(&tr->mutex);
for (i = 0; i < BPF_TRAMP_MAX; i++)
INIT_HLIST_HEAD(&tr->progs_hlist[i]);
out:
@@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag
return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr);
}
static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,
void *old_addr)
static void bpf_tramp_image_put(struct bpf_tramp_image *im);
static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused)
{
void *old_addr = tr->cur_image->image;
int ret;
if (tr->func.ftrace_managed)
@@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags,
else
ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL);
return ret;
if (ret)
return ret;
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = NULL;
return 0;
}
static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,
void *old_addr, void *new_addr,
bool lock_direct_mutex)
static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *data __maybe_unused)
{
void *old_addr = tr->cur_image->image;
void *new_addr = im->image;
int ret;
if (tr->func.ftrace_managed) {
@@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags,
ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr,
new_addr);
}
return ret;
if (ret)
return ret;
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = im;
return 0;
}
/* first time registering */
static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im,
void *data __maybe_unused)
{
void *new_addr = im->image;
void *ip = tr->func.addr;
unsigned long faddr;
int ret;
@@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr);
}
return ret;
if (ret)
return ret;
tr->cur_image = im;
return 0;
}
static struct bpf_tramp_links *
static const struct bpf_trampoline_ops trampoline_ops = {
.register_fentry = register_fentry,
.unregister_fentry = unregister_fentry,
.modify_fentry = modify_fentry,
};
static struct bpf_tramp_nodes *
bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg)
{
struct bpf_tramp_link *link;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_link **links;
struct bpf_tramp_node *node, **nodes;
struct bpf_tramp_nodes *tnodes;
int kind;
*total = 0;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks)
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes)
return ERR_PTR(-ENOMEM);
for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
tlinks[kind].nr_links = tr->progs_cnt[kind];
tnodes[kind].nr_nodes = tr->progs_cnt[kind];
*total += tr->progs_cnt[kind];
links = tlinks[kind].links;
nodes = tnodes[kind].nodes;
hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
*ip_arg |= link->link.prog->call_get_func_ip;
*links++ = link;
hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
*ip_arg |= node->link->prog->call_get_func_ip;
*nodes++ = node;
}
}
return tlinks;
return tnodes;
}
static void bpf_tramp_image_free(struct bpf_tramp_image *im)
@@ -604,30 +670,29 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)
return ERR_PTR(err);
}
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,
const struct bpf_trampoline_ops *ops, void *data)
{
struct bpf_tramp_image *im;
struct bpf_tramp_links *tlinks;
struct bpf_tramp_nodes *tnodes;
u32 orig_flags = tr->flags;
bool ip_arg = false;
int err, total, size;
tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg);
if (IS_ERR(tlinks))
return PTR_ERR(tlinks);
tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg);
if (IS_ERR(tnodes))
return PTR_ERR(tnodes);
if (total == 0) {
err = unregister_fentry(tr, orig_flags, tr->cur_image->image);
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = NULL;
err = ops->unregister_fentry(tr, orig_flags, data);
goto out;
}
/* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */
tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX);
if (tlinks[BPF_TRAMP_FEXIT].nr_links ||
tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) {
if (tnodes[BPF_TRAMP_FEXIT].nr_nodes ||
tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) {
/* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME
* should not be set together.
*/
@@ -658,7 +723,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
#endif
size = arch_bpf_trampoline_size(&tr->func.model, tr->flags,
tlinks, tr->func.addr);
tnodes, tr->func.addr);
if (size < 0) {
err = size;
goto out;
@@ -676,7 +741,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
}
err = arch_prepare_bpf_trampoline(im, im->image, im->image + size,
&tr->func.model, tr->flags, tlinks,
&tr->func.model, tr->flags, tnodes,
tr->func.addr);
if (err < 0)
goto out_free;
@@ -685,14 +750,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
if (err)
goto out_free;
WARN_ON(tr->cur_image && total == 0);
if (tr->cur_image)
/* progs already running at this address */
err = modify_fentry(tr, orig_flags, tr->cur_image->image,
im->image, lock_direct_mutex);
err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data);
else
/* first time registering */
err = register_fentry(tr, im->image);
err = ops->register_fentry(tr, im, data);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
if (err == -EAGAIN) {
@@ -704,34 +767,31 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
goto again;
}
#endif
if (err)
goto out_free;
if (tr->cur_image)
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = im;
out_free:
if (err)
bpf_tramp_image_free(im);
out:
/* If any error happens, restore previous flags */
if (err)
tr->flags = orig_flags;
kfree(tlinks);
kfree(tnodes);
return err;
out_free:
bpf_tramp_image_free(im);
goto out;
}
static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
{
switch (prog->expected_attach_type) {
case BPF_TRACE_FENTRY:
case BPF_TRACE_FENTRY_MULTI:
return BPF_TRAMP_FENTRY;
case BPF_MODIFY_RETURN:
return BPF_TRAMP_MODIFY_RETURN;
case BPF_TRACE_FEXIT:
case BPF_TRACE_FEXIT_MULTI:
return BPF_TRAMP_FEXIT;
case BPF_TRACE_FSESSION:
case BPF_TRACE_FSESSION_MULTI:
return BPF_TRAMP_FSESSION;
case BPF_LSM_MAC:
if (!prog->aux->attach_func_proto->type)
@@ -764,18 +824,95 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog)
return 0;
}
static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node)
{
if (node->link->type == BPF_LINK_TYPE_TRACING) {
struct bpf_tracing_link *link;
link = container_of(node->link, struct bpf_tracing_link, link.link);
return &link->fexit;
} else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) {
struct bpf_tracing_multi_link *link;
struct bpf_tracing_multi_node *mnode;
link = container_of(node->link, struct bpf_tracing_multi_link, link);
mnode = container_of(node, struct bpf_tracing_multi_node, node);
return &link->fexits[mnode - link->nodes];
}
return NULL;
}
static int bpf_trampoline_add_prog(struct bpf_trampoline *tr,
struct bpf_tramp_node *node,
int cnt)
{
struct bpf_fsession_link *fslink = NULL;
enum bpf_tramp_prog_type kind;
struct bpf_tramp_link *link_exiting;
struct bpf_tramp_node *node_existing, *fexit;
struct hlist_head *prog_list;
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_FSESSION) {
prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
cnt++;
} else {
prog_list = &tr->progs_hlist[kind];
}
if (cnt >= BPF_MAX_TRAMP_LINKS)
return -E2BIG;
if (!hlist_unhashed(&node->tramp_hlist))
/* prog already linked */
return -EBUSY;
hlist_for_each_entry(node_existing, prog_list, tramp_hlist) {
if (node_existing->link->prog != node->link->prog)
continue;
/* prog already linked */
return -EBUSY;
}
hlist_add_head(&node->tramp_hlist, prog_list);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]++;
fexit = fsession_exit(node);
if (WARN_ON_ONCE(!fexit))
return -EINVAL;
hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
tr->progs_cnt[BPF_TRAMP_FEXIT]++;
} else {
tr->progs_cnt[kind]++;
}
return 0;
}
static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr,
struct bpf_tramp_node *node)
{
enum bpf_tramp_prog_type kind;
struct bpf_tramp_node *fexit;
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_FSESSION) {
fexit = fsession_exit(node);
if (WARN_ON_ONCE(!fexit))
return;
hlist_del_init(&fexit->tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
kind = BPF_TRAMP_FENTRY;
}
hlist_del_init(&node->tramp_hlist);
tr->progs_cnt[kind]--;
}
static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog,
const struct bpf_trampoline_ops *ops,
void *data)
{
enum bpf_tramp_prog_type kind;
int err = 0;
int cnt = 0, i;
kind = bpf_attach_type_to_tramp(link->link.prog);
kind = bpf_attach_type_to_tramp(node->link->prog);
if (tr->extension_prog)
/* cannot attach fentry/fexit if extension prog is attached.
* cannot overwrite extension prog either.
@@ -792,72 +929,42 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
err = bpf_freplace_check_tgt_prog(tgt_prog);
if (err)
return err;
tr->extension_prog = link->link.prog;
tr->extension_prog = node->link->prog;
return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP,
BPF_MOD_JUMP, NULL,
link->link.prog->bpf_func);
}
if (kind == BPF_TRAMP_FSESSION) {
prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY];
cnt++;
} else {
prog_list = &tr->progs_hlist[kind];
}
if (cnt >= BPF_MAX_TRAMP_LINKS)
return -E2BIG;
if (!hlist_unhashed(&link->tramp_hlist))
/* prog already linked */
return -EBUSY;
hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) {
if (link_exiting->link.prog != link->link.prog)
continue;
/* prog already linked */
return -EBUSY;
}
hlist_add_head(&link->tramp_hlist, prog_list);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]++;
fslink = container_of(link, struct bpf_fsession_link, link.link);
hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]);
tr->progs_cnt[BPF_TRAMP_FEXIT]++;
} else {
tr->progs_cnt[kind]++;
}
err = bpf_trampoline_update(tr, true /* lock_direct_mutex */);
if (err) {
hlist_del_init(&link->tramp_hlist);
if (kind == BPF_TRAMP_FSESSION) {
tr->progs_cnt[BPF_TRAMP_FENTRY]--;
hlist_del_init(&fslink->fexit.tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
} else {
tr->progs_cnt[kind]--;
}
node->link->prog->bpf_func);
}
err = bpf_trampoline_add_prog(tr, node, cnt);
if (err)
return err;
err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
if (err)
bpf_trampoline_remove_prog(tr, node);
return err;
}
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
int bpf_trampoline_link_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
err = __bpf_trampoline_link_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
trampoline_lock(tr);
err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
trampoline_unlock(tr);
return err;
}
static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
struct bpf_prog *tgt_prog,
const struct bpf_trampoline_ops *ops,
void *data)
{
enum bpf_tramp_prog_type kind;
int err;
kind = bpf_attach_type_to_tramp(link->link.prog);
kind = bpf_attach_type_to_tramp(node->link->prog);
if (kind == BPF_TRAMP_REPLACE) {
WARN_ON_ONCE(!tr->extension_prog);
err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
@@ -867,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
guard(mutex)(&tgt_prog->aux->ext_mutex);
tgt_prog->aux->is_extended = false;
return err;
} else if (kind == BPF_TRAMP_FSESSION) {
struct bpf_fsession_link *fslink =
container_of(link, struct bpf_fsession_link, link.link);
hlist_del_init(&fslink->fexit.tramp_hlist);
tr->progs_cnt[BPF_TRAMP_FEXIT]--;
kind = BPF_TRAMP_FENTRY;
}
hlist_del_init(&link->tramp_hlist);
tr->progs_cnt[kind]--;
return bpf_trampoline_update(tr, true /* lock_direct_mutex */);
bpf_trampoline_remove_prog(tr, node);
return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data);
}
/* bpf_trampoline_unlink_prog() should never fail. */
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
trampoline_lock(tr);
err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL);
trampoline_unlock(tr);
return err;
}
@@ -903,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link)
if (!shim_link->trampoline)
return;
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL));
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL));
bpf_trampoline_put(shim_link->trampoline);
}
@@ -949,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog
p->type = BPF_PROG_TYPE_LSM;
p->expected_attach_type = BPF_LSM_MAC;
bpf_prog_inc(p);
bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC,
&bpf_shim_tramp_link_lops, p, attach_type);
bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC,
&bpf_shim_tramp_link_lops, p, attach_type, 0);
bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype);
return shim_link;
@@ -959,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog
static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr,
bpf_func_t bpf_func)
{
struct bpf_tramp_link *link;
struct bpf_tramp_node *node;
int kind;
for (kind = 0; kind < BPF_TRAMP_MAX; kind++) {
hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) {
struct bpf_prog *p = link->link.prog;
hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) {
struct bpf_prog *p = node->link->prog;
if (p->bpf_func == bpf_func)
return container_of(link, struct bpf_shim_tramp_link, link);
return container_of(node, struct bpf_shim_tramp_link, link.node);
}
}
@@ -999,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
if (!tr)
return -ENOMEM;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
shim_link = cgroup_shim_find(tr, bpf_func);
if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) {
/* Reusing existing shim attached by the other program. */
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
bpf_trampoline_put(tr); /* bpf_trampoline_get above */
return 0;
}
@@ -1017,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
goto err;
}
err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL);
err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL);
if (err)
goto err;
shim_link->trampoline = tr;
/* note, we're still holding tr refcnt from above */
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return 0;
err:
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
if (shim_link)
bpf_link_put(&shim_link->link.link);
/* have to release tr while _not_ holding its mutex */
/* have to release tr while _not_ holding pool mutex for trampoline */
bpf_trampoline_put(tr); /* bpf_trampoline_get above */
return err;
@@ -1054,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
if (WARN_ON_ONCE(!tr))
return;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
shim_link = cgroup_shim_find(tr, bpf_func);
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
if (shim_link)
bpf_link_put(&shim_link->link.link);
@@ -1074,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
if (!tr)
return NULL;
mutex_lock(&tr->mutex);
trampoline_lock(tr);
if (tr->func.addr)
goto out;
memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
tr->func.addr = (void *)tgt_info->tgt_addr;
out:
mutex_unlock(&tr->mutex);
trampoline_unlock(tr);
return tr;
}
@@ -1094,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
mutex_lock(&trampoline_mutex);
if (!refcount_dec_and_test(&tr->refcnt))
goto out;
WARN_ON_ONCE(mutex_is_locked(&tr->mutex));
for (i = 0; i < BPF_TRAMP_MAX; i++)
if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i])))
@@ -1333,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog)
int __weak
arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
struct bpf_tramp_nodes *tnodes,
void *func_addr)
{
return -ENOTSUPP;
@@ -1367,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size)
}
int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr)
struct bpf_tramp_nodes *tnodes, void *func_addr)
{
return -ENOTSUPP;
}
#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \
defined(CONFIG_BPF_SYSCALL)
static void trampoline_lock_all(void)
{
int i;
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
mutex_lock(&trampoline_locks[i].mutex);
}
static void trampoline_unlock_all(void)
{
int i;
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
mutex_unlock(&trampoline_locks[i].mutex);
}
static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
ftrace_hash_remove(data->reg);
ftrace_hash_remove(data->unreg);
ftrace_hash_remove(data->modify);
}
static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
remove_tracing_multi_data(data);
free_ftrace_hash(data->reg);
free_ftrace_hash(data->unreg);
free_ftrace_hash(data->modify);
}
static int init_tracing_multi_data(struct bpf_tracing_multi_data *data)
{
data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
if (!data->reg || !data->unreg || !data->modify) {
clear_tracing_multi_data(data);
return -ENOMEM;
}
return 0;
}
static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry,
unsigned long ip, unsigned long direct)
{
entry->ip = ip;
entry->direct = direct;
add_ftrace_hash_entry(hash, entry);
}
static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr)
{
unsigned long addr = (unsigned long) im->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->reg, data->entry, ip, addr);
tr->cur_image = im;
return 0;
}
static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr)
{
unsigned long addr = (unsigned long) tr->cur_image->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->unreg, data->entry, ip, addr);
tr->cur_image = NULL;
return 0;
}
static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im,
bool lock_direct_mutex, void *ptr)
{
unsigned long addr = (unsigned long) im->image;
unsigned long ip = ftrace_location(tr->ip);
struct bpf_tracing_multi_data *data = ptr;
if (bpf_trampoline_use_jmp(tr->flags))
addr = ftrace_jmp_set(addr);
ftrace_hash_add(data->modify, data->entry, ip, addr);
tr->cur_image = im;
return 0;
}
static const struct bpf_trampoline_ops trampoline_multi_ops = {
.register_fentry = register_fentry_multi,
.unregister_fentry = unregister_fentry_multi,
.modify_fentry = modify_fentry_multi,
};
static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr)
{
tr->multi_attach.old_image = tr->cur_image;
tr->multi_attach.old_flags = tr->flags;
}
static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr)
{
if (tr->multi_attach.old_image)
bpf_tramp_image_put(tr->multi_attach.old_image);
tr->multi_attach.old_image = NULL;
tr->multi_attach.old_flags = 0;
}
static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr)
{
if (tr->cur_image)
bpf_tramp_image_put(tr->cur_image);
tr->cur_image = tr->multi_attach.old_image;
tr->flags = tr->multi_attach.old_flags;
tr->multi_attach.old_image = NULL;
tr->multi_attach.old_flags = 0;
}
#define for_each_mnode_cnt(mnode, link, cnt) \
for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i])
#define for_each_mnode(mnode, link) \
for_each_mnode_cnt(mnode, link, link->nodes_cnt)
int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids,
struct bpf_tracing_multi_link *link)
{
struct bpf_tracing_multi_data *data = &link->data;
struct bpf_attach_target_info tgt_info = {};
struct btf *btf = prog->aux->attach_btf;
struct bpf_tracing_multi_node *mnode;
struct bpf_trampoline *tr;
int i, err, rollback_cnt;
u64 key;
for_each_mnode(mnode, link) {
rollback_cnt = i;
err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info);
if (err)
goto rollback_put;
key = bpf_trampoline_compute_key(NULL, btf, ids[i]);
tr = bpf_trampoline_get(key, &tgt_info);
if (!tr) {
err = -ENOMEM;
goto rollback_put;
}
mnode->trampoline = tr;
mnode->node.link = &link->link;
mnode->node.cookie = link->cookies ? link->cookies[i] : 0;
if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
link->fexits[i].link = &link->link;
link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0;
}
cond_resched();
}
err = init_tracing_multi_data(data);
if (err) {
rollback_cnt = link->nodes_cnt;
goto rollback_put;
}
trampoline_lock_all();
for_each_mnode(mnode, link) {
bpf_trampoline_multi_attach_init(mnode->trampoline);
data->entry = &mnode->entry;
err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL,
&trampoline_multi_ops, data);
if (err) {
rollback_cnt = i;
goto rollback_unlink;
}
}
rollback_cnt = link->nodes_cnt;
if (ftrace_hash_count(data->reg)) {
err = update_ftrace_direct_add(&direct_ops, data->reg);
if (err)
goto rollback_unlink;
}
if (ftrace_hash_count(data->modify)) {
err = update_ftrace_direct_mod(&direct_ops, data->modify, true);
if (err) {
if (ftrace_hash_count(data->reg))
WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg));
goto rollback_unlink;
}
}
for_each_mnode(mnode, link)
bpf_trampoline_multi_attach_free(mnode->trampoline);
trampoline_unlock_all();
remove_tracing_multi_data(data);
return 0;
rollback_unlink:
for_each_mnode_cnt(mnode, link, rollback_cnt) {
bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node);
bpf_trampoline_multi_attach_rollback(mnode->trampoline);
}
trampoline_unlock_all();
clear_tracing_multi_data(data);
rollback_cnt = link->nodes_cnt;
rollback_put:
for_each_mnode_cnt(mnode, link, rollback_cnt)
bpf_trampoline_put(mnode->trampoline);
return err;
}
int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link)
{
struct bpf_tracing_multi_data *data = &link->data;
struct bpf_tracing_multi_node *mnode;
int i;
trampoline_lock_all();
for_each_mnode(mnode, link) {
data->entry = &mnode->entry;
bpf_trampoline_multi_attach_init(mnode->trampoline);
WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline,
NULL, &trampoline_multi_ops, data));
}
if (ftrace_hash_count(data->unreg))
WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg));
if (ftrace_hash_count(data->modify))
WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true));
for_each_mnode(mnode, link)
bpf_trampoline_multi_attach_free(mnode->trampoline);
trampoline_unlock_all();
for_each_mnode(mnode, link)
bpf_trampoline_put(mnode->trampoline);
clear_tracing_multi_data(data);
return 0;
}
#undef for_each_mnode_cnt
#undef for_each_mnode
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS &&
CONFIG_BPF_SYSCALL */
static int __init init_trampolines(void)
{
int i;
@@ -1380,6 +1755,8 @@ static int __init init_trampolines(void)
INIT_HLIST_HEAD(&trampoline_key_table[i]);
for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
INIT_HLIST_HEAD(&trampoline_ip_table[i]);
for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++)
__mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key);
return 0;
}
late_initcall(init_trampolines);

4150
kernel/bpf/verifier.c
View File

File diff suppressed because it is too large Load Diff

9
kernel/events/core.c
View File

@@ -11712,6 +11712,15 @@ static int __perf_event_set_bpf_prog(struct perf_event *event,
/* only uprobe programs are allowed to be sleepable */
return -EINVAL;
if (prog->type == BPF_PROG_TYPE_TRACEPOINT && prog->sleepable) {
/*
* Sleepable tracepoint programs can only attach to faultable
* tracepoints. Currently only syscall tracepoints are faultable.
*/
if (!is_syscall_tp)
return -EINVAL;
}
/* Kprobe override only works for kprobes, not uprobes. */
if (prog->kprobe_override && !is_kprobe)
return -EINVAL;

334
kernel/trace/bpf_trace.c
View File

@@ -23,6 +23,7 @@
#include <linux/sort.h>
#include <linux/key.h>
#include <linux/namei.h>
#include <linux/file.h>
#include <net/bpf_sk_storage.h>
@@ -42,6 +43,7 @@
#define MAX_UPROBE_MULTI_CNT (1U << 20)
#define MAX_KPROBE_MULTI_CNT (1U << 20)
#define MAX_TRACING_MULTI_CNT (1U << 20)
#ifdef CONFIG_MODULES
struct bpf_trace_module {
@@ -152,6 +154,34 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
return ret;
}
/**
* trace_call_bpf_faultable - invoke BPF program in faultable context
* @call: tracepoint event
* @ctx: opaque context pointer
*
* Variant of trace_call_bpf() for faultable tracepoints (syscall
* tracepoints). Supports sleepable BPF programs by using rcu_tasks_trace
* for lifetime protection and bpf_prog_run_array_sleepable() for per-program
* RCU flavor selection, following the uprobe pattern.
*
* Per-program recursion protection is provided by
* bpf_prog_run_array_sleepable(). Global bpf_prog_active is not
* needed because syscall tracepoints cannot self-recurse.
*
* Must be called from a faultable/preemptible context.
*/
unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx)
{
struct bpf_prog_array *prog_array;
might_fault();
guard(rcu_tasks_trace)();
prog_array = rcu_dereference_check(call->prog_array,
rcu_read_lock_trace_held());
return bpf_prog_run_array_sleepable(prog_array, ctx, bpf_prog_run);
}
#ifdef CONFIG_BPF_KPROBE_OVERRIDE
BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)
{
@@ -1305,7 +1335,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog)
static inline bool is_trace_fsession(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_FSESSION;
(prog->expected_attach_type == BPF_TRACE_FSESSION ||
prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI);
}
static const struct bpf_func_proto *
@@ -2072,11 +2103,19 @@ void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp)
static __always_inline
void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
{
struct srcu_ctr __percpu *scp = NULL;
struct bpf_prog *prog = link->link.prog;
bool sleepable = prog->sleepable;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
rcu_read_lock_dont_migrate();
if (sleepable) {
scp = rcu_read_lock_tasks_trace();
migrate_disable();
} else {
rcu_read_lock_dont_migrate();
}
if (unlikely(!bpf_prog_get_recursion_context(prog))) {
bpf_prog_inc_misses_counter(prog);
goto out;
@@ -2085,12 +2124,18 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
run_ctx.bpf_cookie = link->cookie;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
(void) bpf_prog_run(prog, args);
(void)bpf_prog_run(prog, args);
bpf_reset_run_ctx(old_run_ctx);
out:
bpf_prog_put_recursion_context(prog);
rcu_read_unlock_migrate();
if (sleepable) {
migrate_enable();
rcu_read_unlock_tasks_trace(scp);
} else {
rcu_read_unlock_migrate();
}
}
#define UNPACK(...) __VA_ARGS__
@@ -3170,6 +3215,38 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
return run_ctx->uprobe->cookie;
}
static int bpf_uprobe_multi_get_path(const union bpf_attr *attr, struct path *path)
{
void __user *upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
u32 path_fd = attr->link_create.uprobe_multi.path_fd;
u32 flags = attr->link_create.uprobe_multi.flags;
if (flags & BPF_F_UPROBE_MULTI_PATH_FD) {
/*
* When BPF_F_UPROBE_MULTI_PATH_FD is set, the executable is
* identified by path_fd, upath must be NULL.
*/
if (upath)
return -EINVAL;
CLASS(fd, f)(path_fd);
if (fd_empty(f))
return -EBADF;
*path = fd_file(f)->f_path;
path_get(path);
return 0;
}
/*
* When BPF_F_UPROBE_MULTI_PATH_FD is not set, the path is resolved
* relative to the cwd (AT_FDCWD) or absolute using the upath string.
*/
if (!upath || path_fd)
return -EINVAL;
return user_path_at(AT_FDCWD, upath, LOOKUP_FOLLOW, path);
}
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_uprobe_multi_link *link = NULL;
@@ -3179,10 +3256,9 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
struct task_struct *task = NULL;
unsigned long __user *uoffsets;
u64 __user *ucookies;
void __user *upath;
unsigned long size;
u32 flags, cnt, i;
struct path path;
char *name;
pid_t pid;
int err;
@@ -3197,19 +3273,18 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
return -EINVAL;
flags = attr->link_create.uprobe_multi.flags;
if (flags & ~BPF_F_UPROBE_MULTI_RETURN)
if (flags & ~(BPF_F_UPROBE_MULTI_RETURN | BPF_F_UPROBE_MULTI_PATH_FD))
return -EINVAL;
/*
* path, offsets and cnt are mandatory,
* offsets and cnt are mandatory,
* ref_ctr_offsets and cookies are optional
*/
upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
cnt = attr->link_create.uprobe_multi.cnt;
pid = attr->link_create.uprobe_multi.pid;
if (!upath || !uoffsets || !cnt || pid < 0)
if (!uoffsets || !cnt || pid < 0)
return -EINVAL;
if (cnt > MAX_UPROBE_MULTI_CNT)
return -E2BIG;
@@ -3217,14 +3292,17 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
name = strndup_user(upath, PATH_MAX);
if (IS_ERR(name)) {
err = PTR_ERR(name);
return err;
}
/*
* All uoffsets/uref_ctr_offsets/ucookies arrays have the same value
* size, we need to check their address range is safe for __get_user
* calls.
*/
size = sizeof(*uoffsets) * cnt;
if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) ||
!access_ok(ucookies, size))
return -EFAULT;
err = kern_path(name, LOOKUP_FOLLOW, &path);
kfree(name);
err = bpf_uprobe_multi_get_path(attr, &path);
if (err)
return err;
@@ -3398,12 +3476,12 @@ typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struc
* direct calls into all the specific callback implementations
* (copy_user_data_sleepable, copy_user_data_nofault, and so on)
*/
static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size,
static __always_inline int __bpf_dynptr_copy_str(const struct bpf_dynptr *dptr, u64 doff, u64 size,
const void *unsafe_src,
copy_fn_t str_copy_fn,
struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
const struct bpf_dynptr_kern *dst;
u64 chunk_sz, off;
void *dst_slice;
int cnt, err;
@@ -3439,7 +3517,7 @@ static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64
u64 size, const void *unsafe_src,
copy_fn_t copy_fn, struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
const struct bpf_dynptr_kern *dst;
void *dst_slice;
char buf[256];
u64 off, chunk_sz;
@@ -3540,49 +3618,49 @@ __bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid
return bpf_send_signal_common(sig, type, task, value);
}
__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_probe_read_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_probe_read_kernel_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,
copy_kernel_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_probe_read_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,
copy_kernel_str_nofault, NULL);
}
__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_copy_from_user_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_copy_from_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_copy_from_user_task_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
@@ -3590,7 +3668,7 @@ __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,
copy_user_data_sleepable, tsk);
}
__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off,
__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
@@ -3599,3 +3677,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64
}
__bpf_kfunc_end_defs();
#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \
defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS)
static void bpf_tracing_multi_link_release(struct bpf_link *link)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link));
}
static void bpf_tracing_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
kvfree(tr_link->fexits);
kvfree(tr_link->cookies);
kvfree(tr_link);
}
#ifdef CONFIG_PROC_FS
static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_tracing_multi_link *tr_link =
container_of(link, struct bpf_tracing_multi_link, link);
bool has_cookies = !!tr_link->cookies;
seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type);
seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt);
seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func");
for (int i = 0; i < tr_link->nodes_cnt; i++) {
struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i];
u32 btf_id, obj_id;
bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id);
seq_printf(seq, "%u\t %u\t %llu\t %pS\n",
obj_id, btf_id,
has_cookies ? tr_link->cookies[i] : 0,
(void *) mnode->trampoline->ip);
cond_resched();
}
}
#endif
static const struct bpf_link_ops bpf_tracing_multi_link_lops = {
.release = bpf_tracing_multi_link_release,
.dealloc_deferred = bpf_tracing_multi_link_dealloc,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_tracing_multi_show_fdinfo,
#endif
};
static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused)
{
u32 a = *(u32 *) pa;
u32 b = *(u32 *) pb;
return (a > b) - (a < b);
}
static void ids_swap_r(void *a, void *b, int size __maybe_unused,
const void *priv __maybe_unused)
{
u64 *cookie_a, *cookie_b, *cookies;
u32 *id_a = a, *id_b = b, *ids;
void **data = (void **) priv;
ids = data[0];
cookies = data[1];
if (cookies) {
cookie_a = cookies + (id_a - ids);
cookie_b = cookies + (id_b - ids);
swap(*cookie_a, *cookie_b);
}
swap(*id_a, *id_b);
}
static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt)
{
void *data[2] = { ids, cookies };
int err = 0;
/*
* Sort ids array (together with cookies array if defined)
* and check it for duplicates. The ids and cookies arrays
* are left sorted.
*/
sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data);
for (int i = 1; i < cnt; i++) {
if (ids[i] == ids[i - 1]) {
err = -EINVAL;
break;
}
}
return err;
}
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
struct bpf_tracing_multi_link *link = NULL;
struct bpf_tramp_node *fexits = NULL;
struct bpf_link_primer link_primer;
u32 cnt, *ids = NULL;
u64 __user *ucookies;
u64 *cookies = NULL;
u32 __user *uids;
int err;
uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids);
cnt = attr->link_create.tracing_multi.cnt;
if (!cnt || !uids)
return -EINVAL;
if (cnt > MAX_TRACING_MULTI_CNT)
return -E2BIG;
if (attr->link_create.flags || attr->link_create.target_fd)
return -EINVAL;
ids = kvmalloc_objs(*ids, cnt);
if (!ids)
return -ENOMEM;
if (copy_from_user(ids, uids, cnt * sizeof(*ids))) {
err = -EFAULT;
goto error;
}
ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies);
if (ucookies) {
cookies = kvmalloc_objs(*cookies, cnt);
if (!cookies) {
err = -ENOMEM;
goto error;
}
if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) {
err = -EFAULT;
goto error;
}
}
err = check_dup_ids(ids, cookies, cnt);
if (err)
goto error;
if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) {
fexits = kvmalloc_objs(*fexits, cnt);
if (!fexits) {
err = -ENOMEM;
goto error;
}
}
link = kvzalloc_flex(*link, nodes, cnt);
if (!link) {
err = -ENOMEM;
goto error;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI,
&bpf_tracing_multi_link_lops, prog, prog->expected_attach_type);
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error;
link->nodes_cnt = cnt;
link->cookies = cookies;
link->fexits = fexits;
err = bpf_trampoline_multi_attach(prog, ids, link);
kvfree(ids);
if (err) {
bpf_link_cleanup(&link_primer);
return err;
}
return bpf_link_settle(&link_primer);
error:
kvfree(fexits);
kvfree(cookies);
kvfree(ids);
kvfree(link);
return err;
}
#else
int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */

35
kernel/trace/ftrace.c
View File

@@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip)
return __ftrace_lookup_ip(hash, ip);
}
static void __add_hash_entry(struct ftrace_hash *hash,
struct ftrace_func_entry *entry)
void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry)
{
struct hlist_head *hhd;
unsigned long key;
@@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne
entry->ip = ip;
entry->direct = direct;
__add_hash_entry(hash, entry);
add_ftrace_hash_entry(hash, entry);
return entry;
}
@@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash,
hash->count--;
}
void ftrace_hash_remove(struct ftrace_hash *hash)
{
struct ftrace_func_entry *entry;
struct hlist_head *hhd;
struct hlist_node *tn;
int size;
int i;
if (!hash || !hash->count)
return;
size = 1 << hash->size_bits;
for (i = 0; i < size; i++) {
hhd = &hash->buckets[i];
hlist_for_each_entry_safe(entry, tn, hhd, hlist)
remove_hash_entry(hash, entry);
}
FTRACE_WARN_ON(hash->count);
}
static void ftrace_hash_clear(struct ftrace_hash *hash)
{
struct hlist_head *hhd;
@@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size)
hhd = &src->buckets[i];
hlist_for_each_entry_safe(entry, tn, hhd, hlist) {
remove_hash_entry(src, entry);
__add_hash_entry(new_hash, entry);
add_ftrace_hash_entry(new_hash, entry);
}
}
return new_hash;
@@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper,
map->entry.ip = ip;
map->data = data;
__add_hash_entry(&mapper->hash, &map->entry);
add_ftrace_hash_entry(&mapper->hash, &map->entry);
return 0;
}
@@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
}
EXPORT_SYMBOL_GPL(modify_ftrace_direct);
static unsigned long hash_count(struct ftrace_hash *hash)
static inline unsigned long hash_count(struct ftrace_hash *hash)
{
return hash ? hash->count : 0;
}
unsigned long ftrace_hash_count(struct ftrace_hash *hash)
{
return hash_count(hash);
}
/**
* hash_add - adds two struct ftrace_hash and returns the result
* @a: struct ftrace_hash object

108
kernel/trace/trace_syscalls.c
View File

@@ -1371,33 +1371,33 @@ static DECLARE_BITMAP(enabled_perf_exit_syscalls, NR_syscalls);
static int sys_perf_refcount_enter;
static int sys_perf_refcount_exit;
static int perf_call_bpf_enter(struct trace_event_call *call, struct pt_regs *regs,
static int perf_call_bpf_enter(struct trace_event_call *call,
struct syscall_metadata *sys_data,
struct syscall_trace_enter *rec)
int syscall_nr, unsigned long *args)
{
struct syscall_tp_t {
struct trace_entry ent;
int syscall_nr;
unsigned long args[SYSCALL_DEFINE_MAXARGS];
} __aligned(8) param;
struct pt_regs regs = {};
int i;
BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *));
/* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */
perf_fetch_caller_regs(regs);
*(struct pt_regs **)&param = regs;
param.syscall_nr = rec->nr;
/* bpf prog requires 'regs' to be the first member in the ctx */
perf_fetch_caller_regs(&regs);
*(struct pt_regs **)&param = &regs;
param.syscall_nr = syscall_nr;
for (i = 0; i < sys_data->nb_args; i++)
param.args[i] = rec->args[i];
return trace_call_bpf(call, &param);
param.args[i] = args[i];
return trace_call_bpf_faultable(call, &param);
}
static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
{
struct syscall_metadata *sys_data;
struct syscall_trace_enter *rec;
struct pt_regs *fake_regs;
struct hlist_head *head;
unsigned long args[6];
bool valid_prog_array;
@@ -1410,12 +1410,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
int size = 0;
int uargs = 0;
/*
* Syscall probe called with preemption enabled, but the ring
* buffer and per-cpu data require preemption to be disabled.
*/
might_fault();
guard(preempt_notrace)();
syscall_nr = trace_get_syscall_nr(current, regs);
if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1429,6 +1424,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
syscall_get_arguments(current, regs, args);
/*
* Run BPF program in faultable context before per-cpu buffer
* allocation, allowing sleepable BPF programs to execute.
*/
valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
if (valid_prog_array &&
!perf_call_bpf_enter(sys_data->enter_event, sys_data,
syscall_nr, args))
return;
/*
* Per-cpu ring buffer and perf event list operations require
* preemption to be disabled.
*/
guard(preempt_notrace)();
head = this_cpu_ptr(sys_data->enter_event->perf_events);
if (hlist_empty(head))
return;
/* Check if this syscall event faults in user space memory */
mayfault = sys_data->user_mask != 0;
@@ -1438,17 +1453,12 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
return;
}
head = this_cpu_ptr(sys_data->enter_event->perf_events);
valid_prog_array = bpf_prog_array_valid(sys_data->enter_event);
if (!valid_prog_array && hlist_empty(head))
return;
/* get the size after alignment with the u32 buffer size field */
size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec);
size = ALIGN(size + sizeof(u32), sizeof(u64));
size -= sizeof(u32);
rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);
rec = perf_trace_buf_alloc(size, NULL, &rctx);
if (!rec)
return;
@@ -1458,13 +1468,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)
if (mayfault)
syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs);
if ((valid_prog_array &&
!perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) ||
hlist_empty(head)) {
perf_swevent_put_recursion_context(rctx);
return;
}
perf_trace_buf_submit(rec, size, rctx,
sys_data->enter_event->event.type, 1, regs,
head, NULL);
@@ -1514,40 +1517,35 @@ static void perf_sysenter_disable(struct trace_event_call *call)
syscall_fault_buffer_disable();
}
static int perf_call_bpf_exit(struct trace_event_call *call, struct pt_regs *regs,
struct syscall_trace_exit *rec)
static int perf_call_bpf_exit(struct trace_event_call *call,
int syscall_nr, long ret_val)
{
struct syscall_tp_t {
struct trace_entry ent;
int syscall_nr;
unsigned long ret;
} __aligned(8) param;
struct pt_regs regs = {};
/* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. &param) */
perf_fetch_caller_regs(regs);
*(struct pt_regs **)&param = regs;
param.syscall_nr = rec->nr;
param.ret = rec->ret;
return trace_call_bpf(call, &param);
/* bpf prog requires 'regs' to be the first member in the ctx */
perf_fetch_caller_regs(&regs);
*(struct pt_regs **)&param = &regs;
param.syscall_nr = syscall_nr;
param.ret = ret_val;
return trace_call_bpf_faultable(call, &param);
}
static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)
{
struct syscall_metadata *sys_data;
struct syscall_trace_exit *rec;
struct pt_regs *fake_regs;
struct hlist_head *head;
bool valid_prog_array;
int syscall_nr;
int rctx;
int size;
/*
* Syscall probe called with preemption enabled, but the ring
* buffer and per-cpu data require preemption to be disabled.
*/
might_fault();
guard(preempt_notrace)();
syscall_nr = trace_get_syscall_nr(current, regs);
if (syscall_nr < 0 || syscall_nr >= NR_syscalls)
@@ -1559,29 +1557,37 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)
if (!sys_data)
return;
head = this_cpu_ptr(sys_data->exit_event->perf_events);
/*
* Run BPF program in faultable context before per-cpu buffer
* allocation, allowing sleepable BPF programs to execute.
*/
valid_prog_array = bpf_prog_array_valid(sys_data->exit_event);
if (!valid_prog_array && hlist_empty(head))
if (valid_prog_array &&
!perf_call_bpf_exit(sys_data->exit_event, syscall_nr,
syscall_get_return_value(current, regs)))
return;
/*
* Per-cpu ring buffer and perf event list operations require
* preemption to be disabled.
*/
guard(preempt_notrace)();
head = this_cpu_ptr(sys_data->exit_event->perf_events);
if (hlist_empty(head))
return;
/* We can probably do that at build time */
size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64));
size -= sizeof(u32);
rec = perf_trace_buf_alloc(size, &fake_regs, &rctx);
rec = perf_trace_buf_alloc(size, NULL, &rctx);
if (!rec)
return;
rec->nr = syscall_nr;
rec->ret = syscall_get_return_value(current, regs);
if ((valid_prog_array &&
!perf_call_bpf_exit(sys_data->exit_event, fake_regs, rec)) ||
hlist_empty(head)) {
perf_swevent_put_recursion_context(rctx);
return;
}
perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type,
1, regs, head, NULL);
}

102
lib/rhashtable.c
View File

@@ -687,6 +687,108 @@ void *rhashtable_insert_slow(struct rhashtable *ht, const void *key,
}
EXPORT_SYMBOL_GPL(rhashtable_insert_slow);
/* Scan one element forward from prev_key's position in @tbl.
* Returns first rhash_head whose bucket > prev_key's bucket, or the
* element immediately after prev_key inside prev_key's bucket.
* Returns the first element if prev_key is NULL, NULL when @tbl is
* exhausted, or ERR_PTR(-ENOENT) if prev_key is not found in @tbl.
*/
static struct rhash_head *__rhashtable_next_in_table(
struct rhashtable *ht, struct bucket_table *tbl,
const void *prev_key)
{
struct rhashtable_compare_arg arg = { .ht = ht, .key = prev_key };
const struct rhashtable_params params = ht->p;
struct rhash_head *he;
unsigned int b = 0;
bool found = false;
if (prev_key) {
b = rht_key_hashfn(ht, tbl, prev_key, params);
rht_for_each_rcu(he, tbl, b) {
bool match = params.obj_cmpfn
? !params.obj_cmpfn(&arg, rht_obj(ht, he))
: !rhashtable_compare(&arg, rht_obj(ht, he));
if (found) {
if (match)
continue;
return he;
}
if (match)
found = true;
}
if (!found)
return ERR_PTR(-ENOENT);
b++;
}
for (; b < tbl->size; b++)
rht_for_each_rcu(he, tbl, b)
return he;
return NULL;
}
/**
* rhashtable_next_key - return next element after a given key
* @ht: hash table
* @prev_key: pointer to previous key, or NULL for the first element
*
* WARNING: this walk is highly unstable. Unlike rhashtable_walk_*(),
* it cannot detect a concurrent resize or rehash, so a full iteration
* is NOT guaranteed to terminate under adversarial or sustained
* rehashing. Callers MUST tolerate skipped and duplicated elements and
* SHOULD bound their loop externally.
*
* Returns the next element in best-effort iteration order, walking the
* @tbl chain (including any future_tbl in flight). Caller must hold RCU.
*
* Pass @prev_key == NULL to obtain the first element. To iterate, set
* @prev_key to the key of the previously returned element on each call,
* and stop when NULL is returned.
*
* Best-effort semantics:
* - Across the tbl->future_tbl chain, an element being migrated may
* transiently appear in both tables and be observed twice.
* - Concurrent inserts may or may not be observed.
* - Termination of a full iteration loop is NOT guaranteed under
* adversarial continuous rehash; callers MUST tolerate skips and
* repeats and SHOULD bound their loop externally.
* - Behavior on tables that contain duplicate keys is undefined:
* duplicates may be skipped, repeated, or trap the walk in a
* cycle. Callers requiring duplicate-key iteration must use
* rhashtable_walk_*() instead.
* - rhltable instances are not supported and return
* ERR_PTR(-EOPNOTSUPP).
* - If prev_key was concurrently deleted and is not present in any
* in-flight table, returns ERR_PTR(-ENOENT).
*
* Returns entry of the next element, or NULL when iteration is exhausted,
* or ERR_PTR(-ENOENT) if prev_key is not found, or
* ERR_PTR(-EOPNOTSUPP) if @ht is an rhltable.
*/
void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key)
{
struct bucket_table *tbl;
struct rhash_head *he;
if (unlikely(ht->rhlist))
return ERR_PTR(-EOPNOTSUPP);
tbl = rht_dereference_rcu(ht->tbl, ht);
do {
he = __rhashtable_next_in_table(ht, tbl, prev_key);
if (!IS_ERR_OR_NULL(he))
return rht_obj(ht, he);
if (!he)
prev_key = NULL;
/* See any new future_tbl attached during a rehash. */
smp_rmb();
tbl = rht_dereference_rcu(tbl->future_tbl, ht);
} while (tbl);
return he; /* NULL or -ENOENT */
}
EXPORT_SYMBOL_GPL(rhashtable_next_key);
/**
* rhashtable_walk_enter - Initialise an iterator
* @ht: Table to walk over

363
lib/test_bpf.c
View File

@@ -560,8 +560,23 @@ static int bpf_fill_max_jmp_never_taken(struct bpf_test *self)
}
/* ALU result computation used in tests */
static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op)
enum { F_ALU32 = 1, F_SIGNED = 2 };
static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op, u32 flags)
{
bool is_signed = flags & F_SIGNED;
/* Narrow operands for ALU32 */
if (flags & F_ALU32) {
if (is_signed) {
v1 = (u64)(s32)v1;
v2 = (u64)(s32)v2;
} else {
v1 = (u32)v1;
v2 = (u32)v2;
}
}
*res = 0;
switch (op) {
case BPF_MOV:
@@ -599,12 +614,28 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op)
case BPF_DIV:
if (v2 == 0)
return false;
*res = div64_u64(v1, v2);
if (!is_signed) {
*res = div64_u64(v1, v2);
} else {
if ((s64)v2 == -1) /* Handled by verifier */
return false;
*res = (u64)div64_s64(v1, v2);
}
break;
case BPF_MOD:
if (v2 == 0)
return false;
div64_u64_rem(v1, v2, res);
if (!is_signed) {
div64_u64_rem(v1, v2, res);
} else {
if ((s64)v2 == -1)
return false;
/*
* Avoid s64 % s64 which generates __moddi3 on
* 32-bit architectures. Use div64_s64 instead.
*/
*res = (u64)((s64)v1 - div64_s64(v1, v2) * (s64)v2);
}
break;
}
return true;
@@ -612,7 +643,7 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op)
/* Test an ALU shift operation for all valid shift values */
static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
u8 mode, bool alu32)
u8 mode, u32 flags)
{
static const s64 regs[] = {
0x0123456789abcdefLL, /* dword > 0, word < 0 */
@@ -620,7 +651,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
0xfedcba0198765432LL, /* dword < 0, word < 0 */
0x0123458967abcdefLL, /* dword > 0, word > 0 */
};
int bits = alu32 ? 32 : 64;
int bits = (flags & F_ALU32) ? 32 : 64;
int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3;
struct bpf_insn *insn;
int imm, k;
@@ -643,7 +674,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
/* Perform operation */
insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3);
insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm);
if (alu32) {
if (flags & F_ALU32) {
if (mode == BPF_K)
insn[i++] = BPF_ALU32_IMM(op, R1, imm);
else
@@ -653,14 +684,14 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
reg = (s32)reg;
else
reg = (u32)reg;
__bpf_alu_result(&val, reg, imm, op);
__bpf_alu_result(&val, reg, imm, op, 0);
val = (u32)val;
} else {
if (mode == BPF_K)
insn[i++] = BPF_ALU64_IMM(op, R1, imm);
else
insn[i++] = BPF_ALU64_REG(op, R1, R2);
__bpf_alu_result(&val, reg, imm, op);
__bpf_alu_result(&val, reg, imm, op, 0);
}
/*
@@ -688,62 +719,62 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op,
static int bpf_fill_alu64_lsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, false);
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, 0);
}
static int bpf_fill_alu64_rsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, false);
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, 0);
}
static int bpf_fill_alu64_arsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, false);
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, 0);
}
static int bpf_fill_alu64_lsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, false);
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, 0);
}
static int bpf_fill_alu64_rsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, false);
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, 0);
}
static int bpf_fill_alu64_arsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, false);
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, 0);
}
static int bpf_fill_alu32_lsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, true);
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, F_ALU32);
}
static int bpf_fill_alu32_rsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, true);
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, F_ALU32);
}
static int bpf_fill_alu32_arsh_imm(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, true);
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, F_ALU32);
}
static int bpf_fill_alu32_lsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, true);
return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, F_ALU32);
}
static int bpf_fill_alu32_rsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, true);
return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, F_ALU32);
}
static int bpf_fill_alu32_arsh_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, true);
return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, F_ALU32);
}
/*
@@ -751,9 +782,9 @@ static int bpf_fill_alu32_arsh_reg(struct bpf_test *self)
* for the case when the source and destination are the same.
*/
static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,
bool alu32)
u32 flags)
{
int bits = alu32 ? 32 : 64;
int bits = (flags & F_ALU32) ? 32 : 64;
int len = 3 + 6 * bits;
struct bpf_insn *insn;
int i = 0;
@@ -770,14 +801,14 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,
/* Perform operation */
insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val);
if (alu32)
if (flags & F_ALU32)
insn[i++] = BPF_ALU32_REG(op, R1, R1);
else
insn[i++] = BPF_ALU64_REG(op, R1, R1);
/* Compute the reference result */
__bpf_alu_result(&res, val, val, op);
if (alu32)
__bpf_alu_result(&res, val, val, op, 0);
if (flags & F_ALU32)
res = (u32)res;
i += __bpf_ld_imm64(&insn[i], R2, res);
@@ -798,32 +829,32 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op,
static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, false);
return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, 0);
}
static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, false);
return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, 0);
}
static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, false);
return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, 0);
}
static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, true);
return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, F_ALU32);
}
static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, true);
return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, F_ALU32);
}
static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self)
{
return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, true);
return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, F_ALU32);
}
/*
@@ -936,17 +967,20 @@ static int __bpf_fill_pattern(struct bpf_test *self, void *arg,
static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg,
struct bpf_insn *insns, s64 dst, s64 imm)
{
int op = *(int *)arg;
int *a = arg;
int op = a[0];
u32 flags = a[1];
s16 off = (flags & F_SIGNED) ? 1 : 0;
int i = 0;
u64 res;
if (!insns)
return 7;
if (__bpf_alu_result(&res, dst, (s32)imm, op)) {
if (__bpf_alu_result(&res, dst, (s32)imm, op, flags)) {
i += __bpf_ld_imm64(&insns[i], R1, dst);
i += __bpf_ld_imm64(&insns[i], R3, res);
insns[i++] = BPF_ALU64_IMM(op, R1, imm);
insns[i++] = BPF_ALU64_IMM_OFF(op, R1, imm, off);
insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
insns[i++] = BPF_EXIT_INSN();
}
@@ -957,17 +991,20 @@ static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg,
static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg,
struct bpf_insn *insns, s64 dst, s64 imm)
{
int op = *(int *)arg;
int *a = arg;
int op = a[0];
u32 flags = a[1];
s16 off = (flags & F_SIGNED) ? 1 : 0;
int i = 0;
u64 res;
if (!insns)
return 7;
if (__bpf_alu_result(&res, (u32)dst, (u32)imm, op)) {
if (__bpf_alu_result(&res, dst, (s32)imm, op, flags | F_ALU32)) {
i += __bpf_ld_imm64(&insns[i], R1, dst);
i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
insns[i++] = BPF_ALU32_IMM(op, R1, imm);
insns[i++] = BPF_ALU32_IMM_OFF(op, R1, imm, off);
insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1);
insns[i++] = BPF_EXIT_INSN();
}
@@ -985,7 +1022,7 @@ static int __bpf_emit_alu64_reg(struct bpf_test *self, void *arg,
if (!insns)
return 9;
if (__bpf_alu_result(&res, dst, src, op)) {
if (__bpf_alu_result(&res, dst, src, op, 0)) {
i += __bpf_ld_imm64(&insns[i], R1, dst);
i += __bpf_ld_imm64(&insns[i], R2, src);
i += __bpf_ld_imm64(&insns[i], R3, res);
@@ -1007,7 +1044,7 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg,
if (!insns)
return 9;
if (__bpf_alu_result(&res, (u32)dst, (u32)src, op)) {
if (__bpf_alu_result(&res, (u32)dst, (u32)src, op, 0)) {
i += __bpf_ld_imm64(&insns[i], R1, dst);
i += __bpf_ld_imm64(&insns[i], R2, src);
i += __bpf_ld_imm64(&insns[i], R3, (u32)res);
@@ -1019,16 +1056,20 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg,
return i;
}
static int __bpf_fill_alu64_imm(struct bpf_test *self, int op)
static int __bpf_fill_alu64_imm(struct bpf_test *self, int op, u32 flags)
{
return __bpf_fill_pattern(self, &op, 64, 32,
int arg[2] = {op, flags};
return __bpf_fill_pattern(self, &arg, 64, 32,
PATTERN_BLOCK1, PATTERN_BLOCK2,
&__bpf_emit_alu64_imm);
}
static int __bpf_fill_alu32_imm(struct bpf_test *self, int op)
static int __bpf_fill_alu32_imm(struct bpf_test *self, int op, u32 flags)
{
return __bpf_fill_pattern(self, &op, 64, 32,
int arg[2] = {op, flags};
return __bpf_fill_pattern(self, &arg, 64, 32,
PATTERN_BLOCK1, PATTERN_BLOCK2,
&__bpf_emit_alu32_imm);
}
@@ -1050,93 +1091,115 @@ static int __bpf_fill_alu32_reg(struct bpf_test *self, int op)
/* ALU64 immediate operations */
static int bpf_fill_alu64_mov_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_MOV);
return __bpf_fill_alu64_imm(self, BPF_MOV, 0);
}
static int bpf_fill_alu64_and_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_AND);
return __bpf_fill_alu64_imm(self, BPF_AND, 0);
}
static int bpf_fill_alu64_or_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_OR);
return __bpf_fill_alu64_imm(self, BPF_OR, 0);
}
static int bpf_fill_alu64_xor_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_XOR);
return __bpf_fill_alu64_imm(self, BPF_XOR, 0);
}
static int bpf_fill_alu64_add_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_ADD);
return __bpf_fill_alu64_imm(self, BPF_ADD, 0);
}
static int bpf_fill_alu64_sub_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_SUB);
return __bpf_fill_alu64_imm(self, BPF_SUB, 0);
}
static int bpf_fill_alu64_mul_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_MUL);
return __bpf_fill_alu64_imm(self, BPF_MUL, 0);
}
static int bpf_fill_alu64_div_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_DIV);
return __bpf_fill_alu64_imm(self, BPF_DIV, 0);
}
static int bpf_fill_alu64_mod_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_MOD);
return __bpf_fill_alu64_imm(self, BPF_MOD, 0);
}
/* Signed ALU64 immediate operations */
static int bpf_fill_alu64_sdiv_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_DIV, F_SIGNED);
}
static int bpf_fill_alu64_smod_imm(struct bpf_test *self)
{
return __bpf_fill_alu64_imm(self, BPF_MOD, F_SIGNED);
}
/* Signed ALU32 immediate operations */
static int bpf_fill_alu32_sdiv_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_DIV, F_SIGNED);
}
static int bpf_fill_alu32_smod_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_MOD, F_SIGNED);
}
/* ALU32 immediate operations */
static int bpf_fill_alu32_mov_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_MOV);
return __bpf_fill_alu32_imm(self, BPF_MOV, 0);
}
static int bpf_fill_alu32_and_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_AND);
return __bpf_fill_alu32_imm(self, BPF_AND, 0);
}
static int bpf_fill_alu32_or_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_OR);
return __bpf_fill_alu32_imm(self, BPF_OR, 0);
}
static int bpf_fill_alu32_xor_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_XOR);
return __bpf_fill_alu32_imm(self, BPF_XOR, 0);
}
static int bpf_fill_alu32_add_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_ADD);
return __bpf_fill_alu32_imm(self, BPF_ADD, 0);
}
static int bpf_fill_alu32_sub_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_SUB);
return __bpf_fill_alu32_imm(self, BPF_SUB, 0);
}
static int bpf_fill_alu32_mul_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_MUL);
return __bpf_fill_alu32_imm(self, BPF_MUL, 0);
}
static int bpf_fill_alu32_div_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_DIV);
return __bpf_fill_alu32_imm(self, BPF_DIV, 0);
}
static int bpf_fill_alu32_mod_imm(struct bpf_test *self)
{
return __bpf_fill_alu32_imm(self, BPF_MOD);
return __bpf_fill_alu32_imm(self, BPF_MOD, 0);
}
/* ALU64 register operations */
@@ -1235,7 +1298,8 @@ static int bpf_fill_alu32_mod_reg(struct bpf_test *self)
* Test JITs that implement complex ALU operations as function
* calls, and must re-arrange operands for argument passing.
*/
static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32)
static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op,
u32 flags)
{
int len = 2 + 10 * 10;
struct bpf_insn *insns;
@@ -1249,28 +1313,37 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32)
return -ENOMEM;
/* Operand and result values according to operation */
if (alu32)
dst = 0x76543210U;
else
dst = 0x7edcba9876543210ULL;
if (flags & F_SIGNED) {
if (flags & F_ALU32)
dst = -76543210;
else
dst = -7654321076543210LL;
} else {
if (flags & F_ALU32)
dst = 0x76543210U;
else
dst = 0x7edcba9876543210ULL;
}
imm = 0x01234567U;
if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
imm &= 31;
__bpf_alu_result(&res, dst, imm, op);
__bpf_alu_result(&res, dst, imm, op, flags);
if (alu32)
if (flags & F_ALU32)
res = (u32)res;
/* Check all operand registers */
for (rd = R0; rd <= R9; rd++) {
i += __bpf_ld_imm64(&insns[i], rd, dst);
if (alu32)
insns[i++] = BPF_ALU32_IMM(op, rd, imm);
s16 off = (flags & F_SIGNED) ? 1 : 0;
if (flags & F_ALU32)
insns[i++] = BPF_ALU32_IMM_OFF(op, rd, imm, off);
else
insns[i++] = BPF_ALU64_IMM(op, rd, imm);
insns[i++] = BPF_ALU64_IMM_OFF(op, rd, imm, off);
insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2);
insns[i++] = BPF_MOV64_IMM(R0, __LINE__);
@@ -1295,123 +1368,145 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32)
/* ALU64 K registers */
static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOV, false);
return __bpf_fill_alu_imm_regs(self, BPF_MOV, 0);
}
static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_AND, false);
return __bpf_fill_alu_imm_regs(self, BPF_AND, 0);
}
static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_OR, false);
return __bpf_fill_alu_imm_regs(self, BPF_OR, 0);
}
static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_XOR, false);
return __bpf_fill_alu_imm_regs(self, BPF_XOR, 0);
}
static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_LSH, false);
return __bpf_fill_alu_imm_regs(self, BPF_LSH, 0);
}
static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_RSH, false);
return __bpf_fill_alu_imm_regs(self, BPF_RSH, 0);
}
static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_ARSH, false);
return __bpf_fill_alu_imm_regs(self, BPF_ARSH, 0);
}
static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_ADD, false);
return __bpf_fill_alu_imm_regs(self, BPF_ADD, 0);
}
static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_SUB, false);
return __bpf_fill_alu_imm_regs(self, BPF_SUB, 0);
}
static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MUL, false);
return __bpf_fill_alu_imm_regs(self, BPF_MUL, 0);
}
static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_DIV, false);
return __bpf_fill_alu_imm_regs(self, BPF_DIV, 0);
}
static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOD, false);
return __bpf_fill_alu_imm_regs(self, BPF_MOD, 0);
}
/* Signed ALU64 K registers */
static int bpf_fill_alu64_sdiv_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_SIGNED);
}
static int bpf_fill_alu64_smod_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_SIGNED);
}
/* ALU32 K registers */
static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOV, true);
return __bpf_fill_alu_imm_regs(self, BPF_MOV, F_ALU32);
}
static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_AND, true);
return __bpf_fill_alu_imm_regs(self, BPF_AND, F_ALU32);
}
static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_OR, true);
return __bpf_fill_alu_imm_regs(self, BPF_OR, F_ALU32);
}
static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_XOR, true);
return __bpf_fill_alu_imm_regs(self, BPF_XOR, F_ALU32);
}
static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_LSH, true);
return __bpf_fill_alu_imm_regs(self, BPF_LSH, F_ALU32);
}
static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_RSH, true);
return __bpf_fill_alu_imm_regs(self, BPF_RSH, F_ALU32);
}
static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_ARSH, true);
return __bpf_fill_alu_imm_regs(self, BPF_ARSH, F_ALU32);
}
static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_ADD, true);
return __bpf_fill_alu_imm_regs(self, BPF_ADD, F_ALU32);
}
static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_SUB, true);
return __bpf_fill_alu_imm_regs(self, BPF_SUB, F_ALU32);
}
static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MUL, true);
return __bpf_fill_alu_imm_regs(self, BPF_MUL, F_ALU32);
}
static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_DIV, true);
return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32);
}
static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOD, true);
return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32);
}
/* Signed ALU32 K registers */
static int bpf_fill_alu32_sdiv_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32 | F_SIGNED);
}
static int bpf_fill_alu32_smod_imm_regs(struct bpf_test *self)
{
return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32 | F_SIGNED);
}
/*
@@ -1442,8 +1537,8 @@ static int __bpf_fill_alu_reg_pairs(struct bpf_test *self, u8 op, bool alu32)
if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH)
src &= 31;
__bpf_alu_result(&res, dst, src, op);
__bpf_alu_result(&same, src, src, op);
__bpf_alu_result(&res, dst, src, op, 0);
__bpf_alu_result(&same, src, src, op, 0);
if (alu32) {
res = (u32)res;
@@ -1626,7 +1721,7 @@ static int __bpf_emit_atomic64(struct bpf_test *self, void *arg,
res = src;
break;
default:
__bpf_alu_result(&res, dst, src, BPF_OP(op));
__bpf_alu_result(&res, dst, src, BPF_OP(op), 0);
}
keep = 0x0123456789abcdefULL;
@@ -1673,7 +1768,7 @@ static int __bpf_emit_atomic32(struct bpf_test *self, void *arg,
res = src;
break;
default:
__bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op));
__bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op), 0);
}
keep = 0x0123456789abcdefULL;
@@ -1939,7 +2034,7 @@ static int __bpf_fill_atomic_reg_pairs(struct bpf_test *self, u8 width, u8 op)
res = mem;
break;
default:
__bpf_alu_result(&res, mem, upd, BPF_OP(op));
__bpf_alu_result(&res, mem, upd, BPF_OP(op), 0);
}
/* Test all operand registers */
@@ -12354,6 +12449,22 @@ static struct bpf_test tests[] = {
{ { 0, 1 } },
.fill_helper = bpf_fill_alu64_mod_imm_regs,
},
{
"ALU64_SDIV_K: registers",
{ },
INTERNAL,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu64_sdiv_imm_regs,
},
{
"ALU64_SMOD_K: registers",
{ },
INTERNAL,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu64_smod_imm_regs,
},
/* ALU32 K registers */
{
"ALU32_MOV_K: registers",
@@ -12451,6 +12562,22 @@ static struct bpf_test tests[] = {
{ { 0, 1 } },
.fill_helper = bpf_fill_alu32_mod_imm_regs,
},
{
"ALU32_SDIV_K: registers",
{ },
INTERNAL,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu32_sdiv_imm_regs,
},
{
"ALU32_SMOD_K: registers",
{ },
INTERNAL,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu32_smod_imm_regs,
},
/* ALU64 X register combinations */
{
"ALU64_MOV_X: register combinations",
@@ -12881,6 +13008,24 @@ static struct bpf_test tests[] = {
.fill_helper = bpf_fill_alu64_mod_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
{
"ALU64_SDIV_K: all immediate value magnitudes",
{ },
INTERNAL | FLAG_NO_DATA,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu64_sdiv_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
{
"ALU64_SMOD_K: all immediate value magnitudes",
{ },
INTERNAL | FLAG_NO_DATA,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu64_smod_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
/* ALU32 immediate magnitudes */
{
"ALU32_MOV_K: all immediate value magnitudes",
@@ -12963,6 +13108,24 @@ static struct bpf_test tests[] = {
.fill_helper = bpf_fill_alu32_mod_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
{
"ALU32_SDIV_K: all immediate value magnitudes",
{ },
INTERNAL | FLAG_NO_DATA,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu32_sdiv_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
{
"ALU32_SMOD_K: all immediate value magnitudes",
{ },
INTERNAL | FLAG_NO_DATA,
{ },
{ { 0, 1 } },
.fill_helper = bpf_fill_alu32_smod_imm,
.nr_testruns = NR_PATTERN_RUNS,
},
/* ALU64 register magnitudes */
{
"ALU64_MOV_X: all register value magnitudes",

75
lib/test_rhashtable.c
View File

@@ -679,6 +679,78 @@ static int threadfunc(void *data)
return err;
}
static int __init test_rhashtable_next_key(void)
{
struct rhashtable_params params = test_rht_params;
struct test_obj_val key_missing = { .id = 99999, .tid = 0 };
struct test_obj_val *prev_key = NULL;
struct rhashtable ht;
struct test_obj *objs, *cur;
int i, count = 0, err;
int visited_keys[8] = { 0 };
const int n = ARRAY_SIZE(visited_keys);
params.nelem_hint = n;
err = rhashtable_init(&ht, &params);
if (err)
return err;
objs = kcalloc(n, sizeof(*objs), GFP_KERNEL);
if (!objs) {
rhashtable_destroy(&ht);
return -ENOMEM;
}
for (i = 0; i < n; i++) {
objs[i].value.id = i;
err = rhashtable_insert_fast(&ht, &objs[i].node, params);
if (err)
goto out;
}
rcu_read_lock();
/* NULL prev_key: walk from the beginning, expect all n elements. */
while ((cur = rhashtable_next_key(&ht, prev_key))) {
if (IS_ERR(cur)) {
err = -EINVAL;
goto unlock;
}
count++;
prev_key = &cur->value;
visited_keys[cur->value.id] = 1;
if (count > n)
break;
}
if (count != n) {
err = -EINVAL;
goto unlock;
}
for (i = 0; i < n; i++) {
if (!visited_keys[i]) {
err = -EINVAL;
goto unlock;
}
}
/* Non-existing prev_key: must return ERR_PTR(-ENOENT). */
cur = rhashtable_next_key(&ht, &key_missing);
if (!IS_ERR(cur) || PTR_ERR(cur) != -ENOENT)
err = -EINVAL;
unlock:
rcu_read_unlock();
out:
for (i = 0; i < n; i++)
rhashtable_remove_fast(&ht, &objs[i].node, params);
kfree(objs);
rhashtable_destroy(&ht);
return err;
}
static int __init test_rht_init(void)
{
unsigned int entries;
@@ -738,6 +810,9 @@ static int __init test_rht_init(void)
test_insert_duplicates_run();
pr_info("Testing rhashtable_next_key: %s\n",
test_rhashtable_next_key() == 0 ? "pass" : "FAIL");
if (!tcount)
return 0;

14
net/bpf/bpf_dummy_struct_ops.c
View File

@@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops;
const struct btf_type *func_proto;
struct bpf_dummy_ops_test_args *args;
struct bpf_tramp_links *tlinks = NULL;
struct bpf_tramp_nodes *tnodes = NULL;
struct bpf_tramp_link *link = NULL;
void *image = NULL;
unsigned int op_idx;
@@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
if (err)
goto out;
tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX);
if (!tlinks) {
tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX);
if (!tnodes) {
err = -ENOMEM;
goto out;
}
@@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
}
/* prog doesn't take the ownership of the reference from caller */
bpf_prog_inc(prog);
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog,
prog->expected_attach_type);
bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops,
prog, prog->expected_attach_type, 0);
op_idx = prog->expected_attach_type;
err = bpf_struct_ops_prepare_trampoline(tlinks, link,
err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node,
&st_ops->func_models[op_idx],
&dummy_ops_test_ret_function,
&image, &image_off,
@@ -198,7 +198,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
bpf_struct_ops_image_free(image);
if (link)
bpf_link_put(&link->link);
kfree(tlinks);
kfree(tnodes);
return err;
}

68
net/bpf/test_run.c
View File

@@ -702,6 +702,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog,
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
if (bpf_fentry_test1(1) != 2 ||
bpf_fentry_test2(2, 3) != 5 ||
bpf_fentry_test3(4, 5, 6) != 15 ||
@@ -747,14 +750,35 @@ static void
__bpf_prog_test_run_raw_tp(void *data)
{
struct bpf_raw_tp_test_run_info *info = data;
struct srcu_ctr __percpu *scp = NULL;
struct bpf_trace_run_ctx run_ctx = {};
struct bpf_run_ctx *old_run_ctx;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
rcu_read_lock();
if (info->prog->sleepable) {
scp = rcu_read_lock_tasks_trace();
migrate_disable();
} else {
rcu_read_lock();
}
if (unlikely(!bpf_prog_get_recursion_context(info->prog))) {
bpf_prog_inc_misses_counter(info->prog);
goto out;
}
info->retval = bpf_prog_run(info->prog, info->ctx);
rcu_read_unlock();
out:
bpf_prog_put_recursion_context(info->prog);
if (info->prog->sleepable) {
migrate_enable();
rcu_read_unlock_tasks_trace(scp);
} else {
rcu_read_unlock();
}
bpf_reset_run_ctx(old_run_ctx);
}
@@ -782,6 +806,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,
if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0)
return -EINVAL;
/*
* Sleepable programs cannot run with preemption disabled or in
* hardirq context (smp_call_function_single), reject the flag.
*/
if (prog->sleepable && (kattr->test.flags & BPF_F_TEST_RUN_ON_CPU))
return -EINVAL;
if (ctx_size_in) {
info.ctx = memdup_user(ctx_in, ctx_size_in);
if (IS_ERR(info.ctx))
@@ -790,24 +821,31 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,
info.ctx = NULL;
}
info.retval = 0;
info.prog = prog;
current_cpu = get_cpu();
if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
cpu == current_cpu) {
if (prog->sleepable) {
__bpf_prog_test_run_raw_tp(&info);
} else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
/* smp_call_function_single() also checks cpu_online()
* after csd_lock(). However, since cpu is from user
* space, let's do an extra quick check to filter out
* invalid value before smp_call_function_single().
*/
err = -ENXIO;
} else {
err = smp_call_function_single(cpu, __bpf_prog_test_run_raw_tp,
&info, 1);
current_cpu = get_cpu();
if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 ||
cpu == current_cpu) {
__bpf_prog_test_run_raw_tp(&info);
} else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
/*
* smp_call_function_single() also checks cpu_online()
* after csd_lock(). However, since cpu is from user
* space, let's do an extra quick check to filter out
* invalid value before smp_call_function_single().
*/
err = -ENXIO;
} else {
err = smp_call_function_single(cpu,
__bpf_prog_test_run_raw_tp,
&info, 1);
}
put_cpu();
}
put_cpu();
if (!err &&
copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32)))

117
net/core/filter.c
View File

@@ -2654,6 +2654,37 @@ static void sk_msg_reset_curr(struct sk_msg *msg)
}
}
static bool sk_msg_elem_is_copy(const struct sk_msg *msg, u32 i)
{
return test_bit(i, msg->sg.copy);
}
static void sk_msg_clear_elem_copy(struct sk_msg *msg, u32 i)
{
__clear_bit(i, msg->sg.copy);
}
static void sk_msg_set_elem_copy(struct sk_msg *msg, u32 i, bool sg_copy)
{
__assign_bit(i, msg->sg.copy, sg_copy);
}
static void sk_msg_clear_copy_range(struct sk_msg *msg, u32 start, u32 end)
{
while (start != end) {
sk_msg_clear_elem_copy(msg, start);
sk_msg_iter_var_next(start);
}
}
static void sk_msg_sg_move(struct sk_msg *msg, u32 dst, u32 src)
{
msg->sg.data[dst] = msg->sg.data[src];
sk_msg_set_elem_copy(msg, dst,
sk_msg_elem_is_copy(msg, src));
}
static const struct bpf_func_proto bpf_msg_cork_bytes_proto = {
.func = bpf_msg_cork_bytes,
.gpl_only = false,
@@ -2692,7 +2723,7 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start,
* account for the headroom.
*/
bytes_sg_total = start - offset + bytes;
if (!test_bit(i, msg->sg.copy) && bytes_sg_total <= len)
if (!sk_msg_elem_is_copy(msg, i) && bytes_sg_total <= len)
goto out;
/* At this point we need to linearize multiple scatterlist
@@ -2733,13 +2764,13 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start,
poffset += len;
sge->length = 0;
put_page(sg_page(sge));
__clear_bit(i, msg->sg.copy);
sk_msg_clear_elem_copy(msg, i);
sk_msg_iter_var_next(i);
} while (i != last_sge);
sg_set_page(&msg->sg.data[first_sge], page, copy, 0);
__clear_bit(first_sge, msg->sg.copy);
sk_msg_clear_elem_copy(msg, first_sge);
/* To repair sg ring we need to shift entries. If we only
* had a single entry though we can just replace it and
@@ -2749,8 +2780,14 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start,
shift = last_sge > first_sge ?
last_sge - first_sge - 1 :
NR_MSG_FRAG_IDS - first_sge + last_sge - 1;
if (!shift)
if (!shift) {
sk_msg_clear_elem_copy(msg, msg->sg.end);
goto out;
}
i = first_sge;
sk_msg_iter_var_next(i);
sk_msg_clear_copy_range(msg, i, last_sge);
i = first_sge;
sk_msg_iter_var_next(i);
@@ -2764,18 +2801,18 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start,
if (move_from == msg->sg.end)
break;
msg->sg.data[i] = msg->sg.data[move_from];
sk_msg_sg_copy_assign(msg, i, msg, move_from);
sk_msg_sg_move(msg, i, move_from);
msg->sg.data[move_from].length = 0;
msg->sg.data[move_from].page_link = 0;
msg->sg.data[move_from].offset = 0;
__clear_bit(move_from, msg->sg.copy);
sk_msg_clear_elem_copy(msg, move_from);
sk_msg_iter_var_next(i);
} while (1);
msg->sg.end = msg->sg.end - shift > msg->sg.end ?
msg->sg.end - shift + NR_MSG_FRAG_IDS :
msg->sg.end - shift;
sk_msg_clear_elem_copy(msg, msg->sg.end);
out:
sk_msg_reset_curr(msg);
msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset;
@@ -2796,9 +2833,10 @@ static const struct bpf_func_proto bpf_msg_pull_data_proto = {
BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
u32, len, u64, flags)
{
bool sge_copy = false, nsge_copy = false, nnsge_copy = false;
struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge;
u32 new, i = 0, l = 0, space, copy = 0, offset = 0;
bool sge_copy, nsge_copy, nnsge_copy, rsge_copy = false;
bool rsge_copy = false;
u8 *raw, *to, *from;
struct page *page;
@@ -2834,6 +2872,9 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
if (!space || (space == 1 && start != offset))
copy = msg->sg.data[i].length;
if (unlikely(copy + len < copy))
return -EINVAL;
page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP,
get_order(copy + len));
if (unlikely(!page))
@@ -2871,7 +2912,7 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
sk_msg_iter_var_prev(i);
psge = sk_msg_elem(msg, i);
rsge = sk_msg_elem_cpy(msg, i);
rsge_copy = test_bit(i, msg->sg.copy);
rsge_copy = sk_msg_elem_is_copy(msg, i);
psge->length = start - offset;
rsge.length -= psge->length;
@@ -2896,21 +2937,21 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
/* Shift one or two slots as needed */
sge = sk_msg_elem_cpy(msg, new);
sge_copy = test_bit(new, msg->sg.copy);
sg_unmark_end(&sge);
sge_copy = sk_msg_elem_is_copy(msg, new);
nsge = sk_msg_elem_cpy(msg, i);
nsge_copy = test_bit(i, msg->sg.copy);
nsge_copy = sk_msg_elem_is_copy(msg, i);
if (rsge.length) {
sk_msg_iter_var_next(i);
nnsge = sk_msg_elem_cpy(msg, i);
nnsge_copy = test_bit(i, msg->sg.copy);
nnsge_copy = sk_msg_elem_is_copy(msg, i);
sk_msg_iter_next(msg, end);
}
while (i != msg->sg.end) {
msg->sg.data[i] = sge;
__assign_bit(i, msg->sg.copy, sge_copy);
sk_msg_set_elem_copy(msg, i, sge_copy);
sge = nsge;
sge_copy = nsge_copy;
sk_msg_iter_var_next(i);
@@ -2918,10 +2959,10 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
nsge = nnsge;
nsge_copy = nnsge_copy;
nnsge = sk_msg_elem_cpy(msg, i);
nnsge_copy = test_bit(i, msg->sg.copy);
nnsge_copy = sk_msg_elem_is_copy(msg, i);
} else {
nsge = sk_msg_elem_cpy(msg, i);
nsge_copy = test_bit(i, msg->sg.copy);
nsge_copy = sk_msg_elem_is_copy(msg, i);
}
}
@@ -2929,14 +2970,15 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start,
/* Place newly allocated data buffer */
sk_mem_charge(msg->sk, len);
msg->sg.size += len;
__clear_bit(new, msg->sg.copy);
sk_msg_clear_elem_copy(msg, new);
sg_set_page(&msg->sg.data[new], page, len + copy, 0);
if (rsge.length) {
get_page(sg_page(&rsge));
sk_msg_iter_var_next(new);
msg->sg.data[new] = rsge;
__assign_bit(new, msg->sg.copy, rsge_copy);
sk_msg_set_elem_copy(msg, new, rsge_copy);
}
sk_msg_clear_elem_copy(msg, msg->sg.end);
sk_msg_reset_curr(msg);
sk_msg_compute_data_pointers(msg);
@@ -2962,12 +3004,11 @@ static void sk_msg_shift_left(struct sk_msg *msg, int i)
do {
prev = i;
sk_msg_iter_var_next(i);
msg->sg.data[prev] = msg->sg.data[i];
sk_msg_sg_copy_assign(msg, prev, msg, i);
sk_msg_sg_move(msg, prev, i);
} while (i != msg->sg.end);
sk_msg_iter_prev(msg, end);
__clear_bit(msg->sg.end, msg->sg.copy);
sk_msg_clear_elem_copy(msg, msg->sg.end);
}
static void sk_msg_shift_right(struct sk_msg *msg, int i)
@@ -2977,28 +3018,29 @@ static void sk_msg_shift_right(struct sk_msg *msg, int i)
sk_msg_iter_next(msg, end);
sge = sk_msg_elem_cpy(msg, i);
sge_copy = test_bit(i, msg->sg.copy);
sge_copy = sk_msg_elem_is_copy(msg, i);
sk_msg_iter_var_next(i);
tmp = sk_msg_elem_cpy(msg, i);
tmp_copy = test_bit(i, msg->sg.copy);
tmp_copy = sk_msg_elem_is_copy(msg, i);
while (i != msg->sg.end) {
msg->sg.data[i] = sge;
__assign_bit(i, msg->sg.copy, sge_copy);
sk_msg_set_elem_copy(msg, i, sge_copy);
sk_msg_iter_var_next(i);
sge = tmp;
sge_copy = tmp_copy;
tmp = sk_msg_elem_cpy(msg, i);
tmp_copy = test_bit(i, msg->sg.copy);
tmp_copy = sk_msg_elem_is_copy(msg, i);
}
sk_msg_clear_elem_copy(msg, msg->sg.end);
}
BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
u32, len, u64, flags)
{
u32 i = 0, l = 0, space, offset = 0;
u64 last = start + len;
int pop;
u64 last = (u64)start + len;
u32 pop;
if (unlikely(flags))
return -EINVAL;
@@ -3047,10 +3089,10 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
*/
if (start != offset) {
struct scatterlist *nsge, *sge = sk_msg_elem(msg, i);
bool sge_copy = sk_msg_elem_is_copy(msg, i);
int a = start - offset;
int b = sge->length - pop - a;
u32 sge_i = i;
bool sge_copy = test_bit(i, msg->sg.copy);
u32 sge_idx = i;
sk_msg_iter_var_next(i);
@@ -3063,7 +3105,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
sg_set_page(nsge,
sg_page(sge),
b, sge->offset + pop + a);
__assign_bit(i, msg->sg.copy, sge_copy);
sk_msg_set_elem_copy(msg, i, sge_copy);
} else {
struct page *page, *orig;
u8 *to, *from;
@@ -3080,7 +3122,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start,
memcpy(to, from, a);
memcpy(to + a, from + a + pop, b);
sg_set_page(sge, page, a + b, 0);
__clear_bit(sge_i, msg->sg.copy);
sk_msg_clear_elem_copy(msg, sge_idx);
put_page(orig);
}
pop = 0;
@@ -5571,11 +5613,24 @@ static int sol_tcp_sockopt(struct sock *sk, int optname,
KERNEL_SOCKPTR(optval), *optlen);
}
static bool sk_allows_sol_ip_sockopt(struct sock *sk)
{
switch (sk->sk_family) {
case AF_INET:
return true;
case AF_INET6:
/* Allow getting/setting sockopt for possible ipv4-mapped ipv6 socket. */
return sk->sk_type != SOCK_RAW && !ipv6_only_sock(sk);
default:
return false;
}
}
static int sol_ip_sockopt(struct sock *sk, int optname,
char *optval, int *optlen,
bool getopt)
{
if (sk->sk_family != AF_INET)
if (!sk_allows_sol_ip_sockopt(sk))
return -EINVAL;
switch (optname) {

12
net/core/lwt_bpf.c
View File

@@ -599,6 +599,7 @@ static int handle_gso_encap(struct sk_buff *skb, bool ipv4, int encap_len)
int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)
{
bool is_udp_tunnel;
struct iphdr *iph;
bool ipv4;
int err;
@@ -612,10 +613,16 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)
ipv4 = true;
if (unlikely(len < iph->ihl * 4))
return -EINVAL;
is_udp_tunnel = iph->protocol == IPPROTO_UDP;
if (unlikely(is_udp_tunnel && len < iph->ihl * 4 + sizeof(struct udphdr)))
return -EINVAL;
} else if (iph->version == 6) {
ipv4 = false;
if (unlikely(len < sizeof(struct ipv6hdr)))
return -EINVAL;
is_udp_tunnel = ((struct ipv6hdr *)iph)->nexthdr == NEXTHDR_UDP;
if (unlikely(is_udp_tunnel && len < sizeof(struct ipv6hdr) + sizeof(struct udphdr)))
return -EINVAL;
} else {
return -EINVAL;
}
@@ -637,6 +644,11 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress)
if (ingress)
skb_postpush_rcsum(skb, iph, len);
skb_reset_network_header(skb);
if (is_udp_tunnel) {
size_t iph_sz = ipv4 ? iph->ihl * 4 : sizeof(struct ipv6hdr);
skb_set_transport_header(skb, skb_network_offset(skb) + iph_sz);
}
memcpy(skb_network_header(skb), hdr, len);
bpf_compute_data_pointers(skb);
skb_clear_hash(skb);

9
net/ipv4/udp_bpf.c
View File

@@ -50,7 +50,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
ret = udp_msg_has_data(sk, psock);
if (!ret) {
release_sock(sk);
wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
lock_sock(sk);
ret = udp_msg_has_data(sk, psock);
}
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -79,6 +81,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
goto out;
}
lock_sock(sk);
msg_bytes_ready:
copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
@@ -90,11 +93,17 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
if (data) {
if (psock_has_data(psock))
goto msg_bytes_ready;
release_sock(sk);
ret = sk_udp_recvmsg(sk, msg, len, flags);
goto out;
}
copied = -EAGAIN;
}
release_sock(sk);
ret = copied;
out:
sk_psock_put(sk, psock);

1
tools/bpf/Makefile
View File

@@ -11,6 +11,7 @@ INSTALL ?= install
CFLAGS += -Wall -O2
CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \
-I$(srctree)/tools/include
CFLAGS += $(EXTRA_CFLAGS)
# This will work when bpf is built in tools env. where srctree
# isn't set and when invoked from selftests build, where srctree

2
tools/bpf/bpftool/Documentation/bpftool-map.rst
View File

@@ -56,7 +56,7 @@ MAP COMMANDS
| | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**
| | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**
| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**
| | **insn_array** }
| | **insn_array** | **rhash** }
DESCRIPTION
===========

20
tools/bpf/bpftool/Makefile
View File

@@ -47,7 +47,8 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_HDRS_
$(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT)
$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \
ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" $@ install_headers
ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" \
CFLAGS="$(LIBBPF_BOOTSTRAP_CFLAGS)" EXTRA_CFLAGS= $@ install_headers
$(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR)
$(call QUIET_INSTALL, $@)
@@ -81,6 +82,13 @@ CFLAGS += -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ \
ifneq ($(BPFTOOL_VERSION),)
CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"'
endif
# This must be done before appending EXTRA_CFLAGS to CFLAGS to avoid
# including flags that are not applicable to the host compiler.
HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
$(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
HOST_CFLAGS += $(HOST_EXTRACFLAGS)
ifneq ($(EXTRA_CFLAGS),)
CFLAGS += $(EXTRA_CFLAGS)
endif
@@ -88,10 +96,11 @@ ifneq ($(EXTRA_LDFLAGS),)
LDFLAGS += $(EXTRA_LDFLAGS)
endif
HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\
$(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS)))
HOST_LDFLAGS := $(LDFLAGS)
# Remove warnings for libbpf bootstrap build
LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS))
INSTALL ?= install
RM ?= rm -f
@@ -106,6 +115,10 @@ ifneq ($(SKIP_CRYPTO),1)
CRYPTO_LIBS := -lcrypto
endif
ifeq ($(MAKECMDGOALS),bootstrap)
FEATURE_TESTS := libelf-zstd
FEATURE_DISPLAY :=
else
FEATURE_TESTS := clang-bpf-co-re
FEATURE_TESTS += llvm
FEATURE_TESTS += libcap
@@ -122,6 +135,7 @@ FEATURE_DISPLAY += libcap
FEATURE_DISPLAY += libbfd
FEATURE_DISPLAY += libbfd-liberty
FEATURE_DISPLAY += libbfd-liberty-z
endif
check_feat := 1
NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall

17
tools/bpf/bpftool/btf.c
View File

@@ -179,8 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
const struct btf_member *m = (const void *)(t + 1);
__u16 vlen = BTF_INFO_VLEN(t->info);
int i;
__u32 i, vlen = BTF_INFO_VLEN(t->info);
if (json_output) {
jsonw_uint_field(w, "size", t->size);
@@ -225,9 +224,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
}
case BTF_KIND_ENUM: {
const struct btf_enum *v = (const void *)(t + 1);
__u16 vlen = BTF_INFO_VLEN(t->info);
__u32 i, vlen = BTF_INFO_VLEN(t->info);
const char *encoding;
int i;
encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
if (json_output) {
@@ -263,9 +261,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
}
case BTF_KIND_ENUM64: {
const struct btf_enum64 *v = btf_enum64(t);
__u16 vlen = btf_vlen(t);
__u32 i, vlen = btf_vlen(t);
const char *encoding;
int i;
encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED";
if (json_output) {
@@ -325,8 +322,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
}
case BTF_KIND_FUNC_PROTO: {
const struct btf_param *p = (const void *)(t + 1);
__u16 vlen = BTF_INFO_VLEN(t->info);
int i;
__u32 i, vlen = BTF_INFO_VLEN(t->info);
if (json_output) {
jsonw_uint_field(w, "ret_type_id", t->type);
@@ -369,8 +365,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id,
case BTF_KIND_DATASEC: {
const struct btf_var_secinfo *v = (const void *)(t + 1);
const struct btf_type *vt;
__u16 vlen = BTF_INFO_VLEN(t->info);
int i;
__u32 i, vlen = BTF_INFO_VLEN(t->info);
if (json_output) {
jsonw_uint_field(w, "size", t->size);
@@ -675,7 +670,7 @@ static __u64 btf_name_hasher(__u64 hash, const struct btf *btf, __u32 name_off)
static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members)
{
const struct btf_type *t = btf__type_by_id(btf, id);
int i;
__u32 i;
size_t hash = 0;
hash = btf_name_hasher(hash, btf, t->name_off);

4
tools/bpf/bpftool/btf_dumper.c
View File

@@ -150,7 +150,7 @@ static int btf_dumper_enum(const struct btf_dumper *d,
{
const struct btf_enum *enums = btf_enum(t);
__s64 value;
__u16 i;
__u32 i;
switch (t->size) {
case 8:
@@ -189,7 +189,7 @@ static int btf_dumper_enum64(const struct btf_dumper *d,
const struct btf_enum64 *enums = btf_enum64(t);
__u32 val_lo32, val_hi32;
__u64 value;
__u16 i;
__u32 i;
value = *(__u64 *)data;
val_lo32 = (__u32)value;

18
tools/bpf/bpftool/gen.c
View File

@@ -1399,7 +1399,7 @@ static int do_skeleton(int argc, char **argv)
continue;
if (use_loader)
printf("t\tint %s_fd;\n", ident);
printf("\t\tint %s_fd;\n", ident);
else
printf("\t\tstruct bpf_link *%s;\n", ident);
}
@@ -2094,7 +2094,8 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi
struct btf_type *cloned_type;
struct btf_param *param;
struct btf_array *array;
int err, i;
__u32 i;
int err;
if (type_id == 0)
return 0;
@@ -2229,7 +2230,8 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool
const struct btf_type *btf_type;
struct btf *btf = info->src_btf;
struct btf_type *cloned_type;
int i, err;
int err;
__u32 i;
if (type_id == 0)
return 0;
@@ -2249,7 +2251,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
struct btf_member *m = btf_members(btf_type);
__u16 vlen = btf_vlen(btf_type);
__u32 vlen = btf_vlen(btf_type);
if (behind_ptr)
break;
@@ -2286,7 +2288,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool
break;
}
case BTF_KIND_FUNC_PROTO: {
__u16 vlen = btf_vlen(btf_type);
__u32 vlen = btf_vlen(btf_type);
struct btf_param *param;
/* mark ret type */
@@ -2492,8 +2494,9 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info)
{
struct btf *btf_new = NULL;
unsigned int *ids = NULL;
unsigned int i, n = btf__type_cnt(info->marked_btf);
unsigned int n = btf__type_cnt(info->marked_btf);
int err = 0;
__u32 i;
btf_new = btf__new_empty();
if (!btf_new) {
@@ -2523,8 +2526,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info)
/* add members for struct and union */
if (btf_is_composite(type)) {
struct btf_member *cloned_m, *m;
unsigned short vlen;
int idx_src;
__u32 vlen, idx_src;
name = btf__str_by_offset(info->src_btf, type->name_off);

2
tools/bpf/bpftool/map.c
View File

@@ -1478,7 +1478,7 @@ static int do_help(int argc, char **argv)
" cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
" queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"
" task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"
" insn_array }\n"
" insn_array | rhash }\n"
" " HELP_SPEC_OPTIONS " |\n"
" {-f|--bpffs} | {-n|--nomount} }\n"
"",

4
tools/bpf/bpftool/net.c
View File

@@ -603,14 +603,14 @@ static int query_flow_dissector(struct bpf_attach_info *attach_info)
&attach_flags, prog_ids, &prog_cnt);
close(fd);
if (err) {
if (errno == EINVAL) {
if (err == -EINVAL) {
/* Older kernel's don't support querying
* flow dissector programs.
*/
errno = 0;
return 0;
}
p_err("can't query prog: %s", strerror(errno));
p_err("can't query prog: %s", strerror(-err));
return -1;
}

2
tools/build/feature/test-bpf.c
View File

@@ -20,6 +20,8 @@
# define __NR_bpf 6319
# elif defined(__mips__) && defined(_ABI64)
# define __NR_bpf 5315
# elif defined(__loongarch__)
# define __NR_bpf 280
# else
# error __NR_bpf not defined. libbpf does not support your arch.
# endif

33
tools/include/uapi/linux/bpf.h
View File

@@ -994,6 +994,7 @@ enum bpf_cmd {
BPF_PROG_STREAM_READ_BY_FD,
BPF_PROG_ASSOC_STRUCT_OPS,
__MAX_BPF_CMD,
BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
};
enum bpf_map_type {
@@ -1046,6 +1047,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_CGRP_STORAGE,
BPF_MAP_TYPE_ARENA,
BPF_MAP_TYPE_INSN_ARRAY,
BPF_MAP_TYPE_RHASH,
__MAX_BPF_MAP_TYPE
};
@@ -1154,6 +1156,9 @@ enum bpf_attach_type {
BPF_TRACE_KPROBE_SESSION,
BPF_TRACE_UPROBE_SESSION,
BPF_TRACE_FSESSION,
BPF_TRACE_FENTRY_MULTI,
BPF_TRACE_FEXIT_MULTI,
BPF_TRACE_FSESSION_MULTI,
__MAX_BPF_ATTACH_TYPE
};
@@ -1178,6 +1183,7 @@ enum bpf_link_type {
BPF_LINK_TYPE_UPROBE_MULTI = 12,
BPF_LINK_TYPE_NETKIT = 13,
BPF_LINK_TYPE_SOCKMAP = 14,
BPF_LINK_TYPE_TRACING_MULTI = 15,
__MAX_BPF_LINK_TYPE,
};
@@ -1321,7 +1327,11 @@ enum {
* BPF_TRACE_UPROBE_MULTI attach type to create return probe.
*/
enum {
BPF_F_UPROBE_MULTI_RETURN = (1U << 0)
/* Get return uprobe. */
BPF_F_UPROBE_MULTI_RETURN = (1U << 0),
/* Get path from provided path_fd. */
BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1),
};
/* link_create.netfilter.flags used in LINK_CREATE command for
@@ -1500,6 +1510,13 @@ struct bpf_stack_build_id {
};
};
struct bpf_common_attr {
__aligned_u64 log_buf;
__u32 log_size;
__u32 log_level;
__u32 log_true_size;
};
#define BPF_OBJ_NAME_LEN 16U
enum {
@@ -1537,6 +1554,11 @@ union bpf_attr {
*
* BPF_MAP_TYPE_ARENA - contains the address where user space
* is going to mmap() the arena. It has to be page aligned.
*
* BPF_MAP_TYPE_RHASH - initial table size hint
* (nelem_hint). 0 = use rhashtable default. Must be
* <= min(max_entries, U16_MAX). Upper 32 bits reserved,
* must be zero.
*/
__u64 map_extra;
@@ -1846,6 +1868,7 @@ union bpf_attr {
__u32 cnt;
__u32 flags;
__u32 pid;
__u32 path_fd;
} uprobe_multi;
struct {
union {
@@ -1861,6 +1884,11 @@ union bpf_attr {
};
__u64 expected_revision;
} cgroup;
struct {
__aligned_u64 ids;
__aligned_u64 cookies;
__u32 cnt;
} tracing_multi;
};
} link_create;
@@ -6698,6 +6726,7 @@ struct bpf_prog_info {
__u32 verified_insns;
__u32 attach_btf_obj_id;
__u32 attach_btf_id;
__u32 :32;
} __attribute__((aligned(8)));
struct bpf_map_info {
@@ -6719,6 +6748,7 @@ struct bpf_map_info {
__u64 map_extra;
__aligned_u64 hash;
__u32 hash_size;
__u32 :32;
} __attribute__((aligned(8)));
struct bpf_btf_info {
@@ -7236,6 +7266,7 @@ enum {
TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */
SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */
SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */
};
enum {

26
tools/include/uapi/linux/btf.h
View File

@@ -33,20 +33,22 @@ struct btf_header {
__u32 layout_len; /* length of layout section */
};
/* Max # of type identifier */
#define BTF_MAX_TYPE 0x000fffff
/* Max offset into the string section */
#define BTF_MAX_NAME_OFFSET 0x00ffffff
/* Max # of struct/union/enum members or func args */
#define BTF_MAX_VLEN 0xffff
enum btf_max {
/* Max possible kind */
BTF_MAX_KIND = 0x0000007f,
/* Max # of type identifier */
BTF_MAX_TYPE = 0x000fffff,
/* Max offset into the string section */
BTF_MAX_NAME_OFFSET = 0x00ffffff,
/* Max # of struct/union/enum members or func args */
BTF_MAX_VLEN = 0x00ffffff,
};
struct btf_type {
__u32 name_off;
/* "info" bits arrangement
* bits 0-15: vlen (e.g. # of struct's members)
* bits 16-23: unused
* bits 24-28: kind (e.g. int, ptr, array...etc)
* bits 29-30: unused
* bits 0-23: vlen (e.g. # of struct's members)
* bits 24-30: kind (e.g. int, ptr, array...etc)
* bit 31: kind_flag, currently used by
* struct, union, enum, fwd, enum64,
* decl_tag and type_tag
@@ -65,8 +67,8 @@ struct btf_type {
};
};
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f)
#define BTF_INFO_VLEN(info) ((info) & 0xffff)
#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f)
#define BTF_INFO_VLEN(info) ((info) & 0xffffff)
#define BTF_INFO_KFLAG(info) ((info) >> 31)
enum {

17
tools/lib/bpf/Makefile
View File

@@ -49,6 +49,14 @@ man_dir_SQ = '$(subst ','\'',$(man_dir))'
export man_dir man_dir_SQ INSTALL
export DESTDIR DESTDIR_SQ
# Defer assigning EXTRA_CFLAGS to CFLAGS until after including
# tools/scripts/Makefile.include, as it may add flags to EXTRA_CFLAGS.
ifdef EXTRA_CFLAGS
CFLAGS :=
else
CFLAGS := -g -O2
endif
include $(srctree)/tools/scripts/Makefile.include
# copy a bit from Linux kbuild
@@ -70,13 +78,6 @@ LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION)
LIB_FILE = libbpf.a libbpf.so*
PC_FILE = libbpf.pc
# Set compile option CFLAGS
ifdef EXTRA_CFLAGS
CFLAGS := $(EXTRA_CFLAGS)
else
CFLAGS := -g -O2
endif
# Append required CFLAGS
override CFLAGS += -std=gnu89
override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum
@@ -84,7 +85,7 @@ override CFLAGS += -Werror -Wall
override CFLAGS += $(INCLUDES)
override CFLAGS += -fvisibility=hidden
override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
override CFLAGS += $(CLANG_CROSS_FLAGS)
override CFLAGS += $(EXTRA_CFLAGS)
# flags specific for shared library
SHLIB_FLAGS := -DSHARED -fPIC

67
tools/lib/bpf/bpf.c
View File

@@ -59,6 +59,8 @@
# define __NR_bpf 6319
# elif defined(__mips__) && defined(_ABI64)
# define __NR_bpf 5315
# elif defined(__loongarch__)
# define __NR_bpf 280
# else
# error __NR_bpf not defined. libbpf does not support your arch.
# endif
@@ -69,6 +71,42 @@ static inline __u64 ptr_to_u64(const void *ptr)
return (__u64) (unsigned long) ptr;
}
static inline int sys_bpf_ext(enum bpf_cmd cmd, union bpf_attr *attr,
unsigned int size,
struct bpf_common_attr *attr_common,
unsigned int size_common)
{
cmd = attr_common ? (cmd | BPF_COMMON_ATTRS) : (cmd & ~BPF_COMMON_ATTRS);
return syscall(__NR_bpf, cmd, attr, size, attr_common, size_common);
}
static inline int sys_bpf_ext_fd(enum bpf_cmd cmd, union bpf_attr *attr,
unsigned int size,
struct bpf_common_attr *attr_common,
unsigned int size_common)
{
int fd;
fd = sys_bpf_ext(cmd, attr, size, attr_common, size_common);
return ensure_good_fd(fd);
}
int probe_sys_bpf_ext(void)
{
const size_t attr_sz = offsetofend(union bpf_attr, prog_token_fd);
union bpf_attr attr;
int fd;
memset(&attr, 0, attr_sz);
fd = syscall(__NR_bpf, BPF_PROG_LOAD | BPF_COMMON_ATTRS, &attr, attr_sz, NULL,
sizeof(struct bpf_common_attr));
if (fd >= 0) {
close(fd);
return -EINVAL;
}
return errno == EFAULT ? 1 : 0;
}
static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
unsigned int size)
{
@@ -173,6 +211,9 @@ int bpf_map_create(enum bpf_map_type map_type,
const struct bpf_map_create_opts *opts)
{
const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size);
const size_t attr_common_sz = sizeof(struct bpf_common_attr);
struct bpf_common_attr attr_common;
struct bpf_log_opts *log_opts;
union bpf_attr attr;
int fd;
@@ -206,7 +247,21 @@ int bpf_map_create(enum bpf_map_type map_type,
attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL));
attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0);
fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);
log_opts = OPTS_GET(opts, log_opts, NULL);
if (!OPTS_VALID(log_opts, bpf_log_opts))
return libbpf_err(-EINVAL);
if (log_opts && feat_supported(NULL, FEAT_BPF_SYSCALL_COMMON_ATTRS)) {
memset(&attr_common, 0, attr_common_sz);
attr_common.log_buf = ptr_to_u64(OPTS_GET(log_opts, buf, NULL));
attr_common.log_size = OPTS_GET(log_opts, size, 0);
attr_common.log_level = OPTS_GET(log_opts, level, 0);
fd = sys_bpf_ext_fd(BPF_MAP_CREATE, &attr, attr_sz, &attr_common, attr_common_sz);
OPTS_SET(log_opts, true_size, attr_common.log_true_size);
} else {
fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz);
OPTS_SET(log_opts, true_size, 0);
}
return libbpf_err_errno(fd);
}
@@ -787,9 +842,19 @@ int bpf_link_create(int prog_fd, int target_fd,
attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
attr.link_create.uprobe_multi.path_fd = OPTS_GET(opts, uprobe_multi.path_fd, 0);
if (!OPTS_ZEROED(opts, uprobe_multi))
return libbpf_err(-EINVAL);
break;
case BPF_TRACE_FENTRY_MULTI:
case BPF_TRACE_FEXIT_MULTI:
case BPF_TRACE_FSESSION_MULTI:
attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0));
attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0));
attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0);
if (!OPTS_ZEROED(opts, tracing_multi))
return libbpf_err(-EINVAL);
break;
case BPF_TRACE_RAW_TP:
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:

25
tools/lib/bpf/bpf.h
View File

@@ -37,6 +37,18 @@ extern "C" {
LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes);
struct bpf_log_opts {
size_t sz; /* size of this struct for forward/backward compatibility */
char *buf;
__u32 size;
__u32 level;
__u32 true_size; /* out parameter set by kernel */
size_t :0;
};
#define bpf_log_opts__last_field true_size
struct bpf_map_create_opts {
size_t sz; /* size of this struct for forward/backward compatibility */
@@ -57,9 +69,12 @@ struct bpf_map_create_opts {
const void *excl_prog_hash;
__u32 excl_prog_hash_size;
struct bpf_log_opts *log_opts;
size_t :0;
};
#define bpf_map_create_opts__last_field excl_prog_hash_size
#define bpf_map_create_opts__last_field log_opts
LIBBPF_API int bpf_map_create(enum bpf_map_type map_type,
const char *map_name,
@@ -429,6 +444,7 @@ struct bpf_link_create_opts {
const unsigned long *ref_ctr_offsets;
const __u64 *cookies;
__u32 pid;
__u32 path_fd;
} uprobe_multi;
struct {
__u64 cookie;
@@ -454,10 +470,15 @@ struct bpf_link_create_opts {
__u32 relative_id;
__u64 expected_revision;
} cgroup;
struct {
const __u32 *ids;
const __u64 *cookies;
__u32 cnt;
} tracing_multi;
};
size_t :0;
};
#define bpf_link_create_opts__last_field uprobe_multi.pid
#define bpf_link_create_opts__last_field uprobe_multi.path_fd
LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,

54
tools/lib/bpf/btf.c
View File

@@ -421,7 +421,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t
{
__u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout);
struct btf_layout *l = btf->layout;
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
__u32 kind = btf_kind(t);
/* Fall back to base BTF if needed as they share layout information */
@@ -454,7 +454,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t
static int btf_type_size(const struct btf *btf, const struct btf_type *t)
{
const int base_size = sizeof(struct btf_type);
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
switch (btf_kind(t)) {
case BTF_KIND_FWD:
@@ -506,7 +506,7 @@ static int btf_bswap_type_rest(struct btf_type *t)
struct btf_array *a;
struct btf_param *p;
struct btf_enum *e;
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
int i;
switch (btf_kind(t)) {
@@ -1007,7 +1007,7 @@ int btf__align_of(const struct btf *btf, __u32 id)
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
const struct btf_member *m = btf_members(t);
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
int i, max_align = 1, align;
for (i = 0; i < vlen; i++, m++) {
@@ -2121,9 +2121,12 @@ static void *btf_add_type_mem(struct btf *btf, size_t add_sz)
btf->hdr.type_len, UINT_MAX, add_sz);
}
static void btf_type_inc_vlen(struct btf_type *t)
static int btf_type_inc_vlen(struct btf_type *t)
{
if (btf_vlen(t) == BTF_MAX_VLEN)
return -ENOSPC;
t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));
return 0;
}
static void btf_hdr_update_type_len(struct btf *btf, int new_len)
@@ -2652,6 +2655,8 @@ int btf__add_field(struct btf *btf, const char *name, int type_id,
t = btf_last_type(btf);
if (!btf_is_composite(t))
return libbpf_err(-EINVAL);
if (btf_vlen(t) == BTF_MAX_VLEN)
return libbpf_err(-ENOSPC);
if (validate_type_id(type_id))
return libbpf_err(-EINVAL);
@@ -2686,6 +2691,7 @@ int btf__add_field(struct btf *btf, const char *name, int type_id,
/* btf_add_type_mem can invalidate t pointer */
t = btf_last_type(btf);
/* update parent type's vlen and kflag */
t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));
@@ -2796,7 +2802,9 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value)
/* update parent type's vlen */
t = btf_last_type(btf);
btf_type_inc_vlen(t);
err = btf_type_inc_vlen(t);
if (err)
return libbpf_err(err);
/* if negative value, set signedness to signed */
if (value < 0)
@@ -2873,7 +2881,9 @@ int btf__add_enum64_value(struct btf *btf, const char *name, __u64 value)
/* update parent type's vlen */
t = btf_last_type(btf);
btf_type_inc_vlen(t);
err = btf_type_inc_vlen(t);
if (err)
return libbpf_err(err);
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
return 0;
@@ -3115,7 +3125,9 @@ int btf__add_func_param(struct btf *btf, const char *name, int type_id)
/* update parent type's vlen */
t = btf_last_type(btf);
btf_type_inc_vlen(t);
err = btf_type_inc_vlen(t);
if (err)
return libbpf_err(err);
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
return 0;
@@ -3257,7 +3269,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __
/* update parent type's vlen */
t = btf_last_type(btf);
btf_type_inc_vlen(t);
err = btf_type_inc_vlen(t);
if (err)
return libbpf_err(err);
btf_hdr_update_type_len(btf, btf->hdr.type_len + sz);
return 0;
@@ -4311,7 +4325,7 @@ static long btf_hash_enum(struct btf_type *t)
static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2)
{
const struct btf_enum *m1, *m2;
__u16 vlen;
__u32 vlen;
int i;
vlen = btf_vlen(t1);
@@ -4329,7 +4343,7 @@ static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2)
static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2)
{
const struct btf_enum64 *m1, *m2;
__u16 vlen;
__u32 vlen;
int i;
vlen = btf_vlen(t1);
@@ -4406,7 +4420,7 @@ static long btf_hash_struct(struct btf_type *t)
static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2)
{
const struct btf_member *m1, *m2;
__u16 vlen;
__u32 vlen;
int i;
if (!btf_equal_common(t1, t2))
@@ -4482,7 +4496,7 @@ static bool btf_compat_array(struct btf_type *t1, struct btf_type *t2)
static long btf_hash_fnproto(struct btf_type *t)
{
const struct btf_param *member = btf_params(t);
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
long h = btf_hash_common(t);
int i;
@@ -4504,7 +4518,7 @@ static long btf_hash_fnproto(struct btf_type *t)
static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2)
{
const struct btf_param *m1, *m2;
__u16 vlen;
__u32 vlen;
int i;
if (!btf_equal_common(t1, t2))
@@ -4530,7 +4544,7 @@ static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2)
static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2)
{
const struct btf_param *m1, *m2;
__u16 vlen;
__u32 vlen;
int i;
/* skip return type ID */
@@ -4578,12 +4592,14 @@ static int btf_dedup_prep(struct btf_dedup *d)
case BTF_KIND_RESTRICT:
case BTF_KIND_PTR:
case BTF_KIND_FWD:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FUNC:
case BTF_KIND_FLOAT:
case BTF_KIND_TYPE_TAG:
h = btf_hash_common(t);
break;
case BTF_KIND_TYPEDEF:
h = btf_hash_typedef(t);
break;
case BTF_KIND_INT:
case BTF_KIND_DECL_TAG:
h = btf_hash_int_decl_tag(t);
@@ -5077,7 +5093,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id,
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
const struct btf_member *cand_m, *canon_m;
__u16 vlen;
__u32 vlen;
if (!btf_shallow_equal_struct(cand_type, canon_type))
return 0;
@@ -5105,7 +5121,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id,
case BTF_KIND_FUNC_PROTO: {
const struct btf_param *cand_p, *canon_p;
__u16 vlen;
__u32 vlen;
if (!btf_compat_fnproto(cand_type, canon_type))
return 0;
@@ -5439,7 +5455,7 @@ static int btf_dedup_ref_type(struct btf_dedup *d, __u32 type_id)
case BTF_KIND_FUNC_PROTO: {
struct btf_param *param;
__u16 vlen;
__u32 vlen;
int i;
ref_type_id = btf_dedup_ref_type(d, t->type);

2
tools/lib/bpf/btf.h
View File

@@ -435,7 +435,7 @@ static inline __u16 btf_kind(const struct btf_type *t)
return BTF_INFO_KIND(t->info);
}
static inline __u16 btf_vlen(const struct btf_type *t)
static inline __u32 btf_vlen(const struct btf_type *t)
{
return BTF_INFO_VLEN(t->info);
}

24
tools/lib/bpf/btf_dump.c
View File

@@ -316,7 +316,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d)
{
int i, j, n = btf__type_cnt(d->btf);
const struct btf_type *t;
__u16 vlen;
__u32 vlen;
for (i = d->last_id + 1; i < n; i++) {
t = btf__type_by_id(d->btf, i);
@@ -485,7 +485,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr)
*/
struct btf_dump_type_aux_state *tstate = &d->type_states[id];
const struct btf_type *t;
__u16 vlen;
__u32 vlen;
int err, i;
/* return true, letting typedefs know that it's ok to be emitted */
@@ -798,7 +798,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id)
*/
if (top_level_def || t->name_off == 0) {
const struct btf_member *m = btf_members(t);
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
int i, new_cont_id;
new_cont_id = t->name_off == 0 ? cont_id : id;
@@ -820,7 +820,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id)
break;
case BTF_KIND_FUNC_PROTO: {
const struct btf_param *p = btf_params(t);
__u16 n = btf_vlen(t);
__u32 n = btf_vlen(t);
int i;
btf_dump_emit_type(d, t->type, cont_id);
@@ -839,7 +839,7 @@ static bool btf_is_struct_packed(const struct btf *btf, __u32 id,
{
const struct btf_member *m;
int max_align = 1, align, i, bit_sz;
__u16 vlen;
__u32 vlen;
m = btf_members(t);
vlen = btf_vlen(t);
@@ -973,7 +973,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d,
bool is_struct = btf_is_struct(t);
bool packed, prev_bitfield = false;
int align, i, off = 0;
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
align = btf__align_of(d->btf, id);
packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0;
@@ -1064,7 +1064,7 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id,
static void btf_dump_emit_enum32_val(struct btf_dump *d,
const struct btf_type *t,
int lvl, __u16 vlen)
int lvl, __u32 vlen)
{
const struct btf_enum *v = btf_enum(t);
bool is_signed = btf_kflag(t);
@@ -1089,7 +1089,7 @@ static void btf_dump_emit_enum32_val(struct btf_dump *d,
static void btf_dump_emit_enum64_val(struct btf_dump *d,
const struct btf_type *t,
int lvl, __u16 vlen)
int lvl, __u32 vlen)
{
const struct btf_enum64 *v = btf_enum64(t);
bool is_signed = btf_kflag(t);
@@ -1122,7 +1122,7 @@ static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id,
const struct btf_type *t,
int lvl)
{
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
btf_dump_printf(d, "enum%s%s",
t->name_off ? " " : "",
@@ -1542,7 +1542,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d,
}
case BTF_KIND_FUNC_PROTO: {
const struct btf_param *p = btf_params(t);
__u16 vlen = btf_vlen(t);
__u32 vlen = btf_vlen(t);
int i;
/*
@@ -2159,7 +2159,7 @@ static int btf_dump_struct_data(struct btf_dump *d,
const void *data)
{
const struct btf_member *m = btf_members(t);
__u16 n = btf_vlen(t);
__u32 n = btf_vlen(t);
int i, err = 0;
/* note that we increment depth before calling btf_dump_print() below;
@@ -2449,7 +2449,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d,
case BTF_KIND_STRUCT:
case BTF_KIND_UNION: {
const struct btf_member *m = btf_members(t);
__u16 n = btf_vlen(t);
__u32 n = btf_vlen(t);
/* if any struct/union member is non-zero, the struct/union
* is considered non-zero and dumped.

8
tools/lib/bpf/features.c
View File

@@ -615,6 +615,11 @@ static int probe_kern_btf_layout(int token_fd)
(char *)layout, token_fd));
}
static int probe_bpf_syscall_common_attrs(int token_fd)
{
return probe_sys_bpf_ext();
}
typedef int (*feature_probe_fn)(int /* token_fd */);
static struct kern_feature_cache feature_cache;
@@ -699,6 +704,9 @@ static struct kern_feature_desc {
[FEAT_BTF_LAYOUT] = {
"kernel supports BTF layout", probe_kern_btf_layout,
},
[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
},
};
bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)

92
tools/lib/bpf/gen_loader.c
View File

@@ -63,6 +63,7 @@ static int realloc_insn_buf(struct bpf_gen *gen, __u32 size)
gen->error = -ENOMEM;
free(gen->insn_start);
gen->insn_start = NULL;
gen->insn_cur = NULL;
return -ENOMEM;
}
gen->insn_start = insn_start;
@@ -86,6 +87,7 @@ static int realloc_data_buf(struct bpf_gen *gen, __u32 size)
gen->error = -ENOMEM;
free(gen->data_start);
gen->data_start = NULL;
gen->data_cur = NULL;
return -ENOMEM;
}
gen->data_start = data_start;
@@ -158,10 +160,16 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps
static int add_data(struct bpf_gen *gen, const void *data, __u32 size)
{
__u32 size8 = roundup(size, 8);
__u64 zero = 0;
__u32 size8;
void *prev;
if (size > INT32_MAX) {
gen->error = -ERANGE;
return 0;
}
size8 = roundup(size, 8);
if (realloc_data_buf(gen, size8))
return 0;
prev = gen->data_cur;
@@ -293,7 +301,6 @@ static void emit_check_err(struct bpf_gen *gen)
emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off));
} else {
gen->error = -ERANGE;
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1));
}
}
@@ -398,13 +405,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
blob_fd_array_off(gen, i));
emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0));
emit(gen, BPF_EXIT_INSN());
if (OPTS_GET(gen->opts, gen_hash, false))
compute_sha_update_offsets(gen);
pr_debug("gen: finish %s\n", errstr(gen->error));
if (!gen->error) {
struct gen_loader_opts *opts = gen->opts;
if (OPTS_GET(opts, gen_hash, false))
compute_sha_update_offsets(gen);
opts->insns = gen->insn_start;
opts->insns_sz = gen->insn_cur - gen->insn_start;
opts->data = gen->data_start;
@@ -419,6 +425,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
bpf_insn_bswap(insn++);
}
}
pr_debug("gen: finish %s\n", errstr(gen->error));
return gen->error;
}
@@ -545,13 +552,22 @@ void bpf_gen__map_create(struct bpf_gen *gen,
default:
break;
}
/* conditionally update max_entries */
if (map_idx >= 0)
/*
* Conditionally update max_entries from the host-supplied loader
* ctx. This sizes the map at runtime, but for a signed loader
* (gen_hash) it would let an untrusted host re-dimension the
* program's maps after emit_signature_match(), outside what the
* signature attests to. Keep the signer-provided max_entries
* baked into the blob in that case.
*/
if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false))
move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4,
sizeof(struct bpf_loader_ctx) +
sizeof(struct bpf_map_desc) * map_idx +
offsetof(struct bpf_map_desc, max_entries),
true /* check that max_entries != 0 */);
/* emit MAP_CREATE command */
emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size);
debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d",
@@ -585,6 +601,23 @@ static void emit_signature_match(struct bpf_gen *gen)
__s64 off;
int i;
/*
* Reject if the metadata map is not exclusive. Without exclusivity
* the cached map->sha[] verified above can be stale: another BPF
* program with map access could have mutated the contents between
* BPF_OBJ_GET_INFO_BY_FD and loader execution.
*/
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
0, 0, 0, 0));
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH));
off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2;
if (is_simm16(off)) {
emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL));
emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off));
} else {
gen->error = -ERANGE;
}
for (i = 0; i < SHA256_DWORD_SIZE; i++) {
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX,
0, 0, 0, 0));
@@ -1053,7 +1086,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
prog_idx, prog_type, insns_off, insn_cnt, license_off);
/* convert blob insns to target endianness */
if (gen->swapped_endian) {
if (gen->swapped_endian && !gen->error) {
struct bpf_insn *insn = gen->data_start + insns_off;
int i;
@@ -1091,7 +1124,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
sizeof(struct bpf_core_relo));
/* convert all info blobs to target endianness */
if (gen->swapped_endian)
if (gen->swapped_endian && !gen->error)
info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
@@ -1169,27 +1202,36 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
value = add_data(gen, pvalue, value_size);
key = add_data(gen, &zero, sizeof(zero));
/* if (map_desc[map_idx].initial_value) {
/*
* if (map_desc[map_idx].initial_value) {
* if (ctx->flags & BPF_SKEL_KERNEL)
* bpf_probe_read_kernel(value, value_size, initial_value);
* else
* bpf_copy_from_user(value, value_size, initial_value);
* }
*
* The runtime initial_value comes from the host-supplied loader
* ctx and would overwrite the blob value after emit_signature_match()
* has already validated map->sha[]. For a signed loader (gen_hash)
* the attested blob value must be authoritative, so skip the override
* and leave the hashed value in place.
*/
emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
sizeof(struct bpf_loader_ctx) +
sizeof(struct bpf_map_desc) * map_idx +
offsetof(struct bpf_map_desc, initial_value)));
emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
0, 0, 0, value));
emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
offsetof(struct bpf_loader_ctx, flags)));
emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
if (!OPTS_GET(gen->opts, gen_hash, false)) {
emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6,
sizeof(struct bpf_loader_ctx) +
sizeof(struct bpf_map_desc) * map_idx +
offsetof(struct bpf_map_desc, initial_value)));
emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8));
emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE,
0, 0, 0, value));
emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size));
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
offsetof(struct bpf_loader_ctx, flags)));
emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user));
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
}
map_update_attr = add_data(gen, &attr, attr_size);
pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n",

473
tools/lib/bpf/libbpf.c
View File

@@ -136,6 +136,9 @@ static const char * const attach_type_name[] = {
[BPF_NETKIT_PEER] = "netkit_peer",
[BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session",
[BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session",
[BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi",
[BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi",
[BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi",
};
static const char * const link_type_name[] = {
@@ -154,6 +157,7 @@ static const char * const link_type_name[] = {
[BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi",
[BPF_LINK_TYPE_NETKIT] = "netkit",
[BPF_LINK_TYPE_SOCKMAP] = "sockmap",
[BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi",
};
static const char * const map_type_name[] = {
@@ -192,6 +196,7 @@ static const char * const map_type_name[] = {
[BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage",
[BPF_MAP_TYPE_ARENA] = "arena",
[BPF_MAP_TYPE_INSN_ARRAY] = "insn_array",
[BPF_MAP_TYPE_RHASH] = "rhash",
};
static const char * const prog_type_name[] = {
@@ -7767,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program
static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name,
int *btf_obj_fd, int *btf_type_id);
static inline bool is_tracing_multi(enum bpf_attach_type type)
{
return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI ||
type == BPF_TRACE_FSESSION_MULTI;
}
static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach)
{
const char *sep, *mod_name = NULL;
int i, mod_len, err;
/*
* We expect attach string in the form of either
* - function_pattern or
* - <module>:function_pattern
*/
sep = strchr(attach, ':');
if (sep) {
mod_name = attach;
mod_len = sep - mod_name;
}
if (!mod_name)
return NULL;
err = load_module_btfs(obj);
if (err)
return NULL;
for (i = 0; i < obj->btf_module_cnt; i++) {
const struct module_btf *mod = &obj->btf_modules[i];
if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0')
return mod;
}
return NULL;
}
static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd)
{
const char *attach_name, *sep;
const struct module_btf *mod;
*btf_obj_fd = 0;
attach_name = strchr(prog->sec_name, '/');
/* Program with no details in spec, using kernel btf. */
if (!attach_name)
return 0;
/* Program with no module section, using kernel btf. */
sep = strchr(++attach_name, ':');
if (!sep)
return 0;
/* Program with module specified, get its btf fd. */
mod = find_attach_module(prog->obj, attach_name);
if (!mod)
return -EINVAL;
*btf_obj_fd = mod->fd;
return 0;
}
/* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */
static int libbpf_prepare_prog_load(struct bpf_program *prog,
struct bpf_prog_load_opts *opts, long cookie)
@@ -7830,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
opts->attach_btf_obj_fd = btf_obj_fd;
opts->attach_btf_id = btf_type_id;
}
if (is_tracing_multi(prog->expected_attach_type)) {
int err, btf_obj_fd = 0;
err = tracing_multi_mod_fd(prog, &btf_obj_fd);
if (err < 0)
return err;
prog->attach_btf_obj_fd = btf_obj_fd;
opts->attach_btf_obj_fd = btf_obj_fd;
}
return 0;
}
@@ -8936,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj)
bpf_map__unpin(&obj->maps[i], NULL);
}
static void bpf_object_post_load_cleanup(struct bpf_object *obj)
static void bpf_object_cleanup_btf(struct bpf_object *obj)
{
int i;
/* clean up fd_array */
zfree(&obj->fd_array);
/* clean up module BTFs */
for (i = 0; i < obj->btf_module_cnt; i++) {
close(obj->btf_modules[i].fd);
@@ -8950,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj)
free(obj->btf_modules[i].name);
}
obj->btf_module_cnt = 0;
obj->btf_module_cap = 0;
obj->btf_modules_loaded = false;
zfree(&obj->btf_modules);
/* clean up vmlinux BTF */
@@ -8957,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj)
obj->btf_vmlinux = NULL;
}
static void bpf_object_post_load_cleanup(struct bpf_object *obj)
{
/* clean up fd_array */
zfree(&obj->fd_array);
/* clean up BTF */
bpf_object_cleanup_btf(obj);
}
static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path)
{
int err;
@@ -9983,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st
static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
static const struct bpf_sec_def section_defs[] = {
SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE),
@@ -10018,11 +10107,16 @@ static const struct bpf_sec_def section_defs[] = {
SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE),
SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp),
SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp),
SEC_DEF("tracepoint.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
SEC_DEF("tp.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp),
SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp),
SEC_DEF("raw_tracepoint.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
SEC_DEF("raw_tp.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp),
SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp),
SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("tp_btf.s+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace),
@@ -10031,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = {
SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace),
SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi),
SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi),
SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi),
SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi),
SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace),
SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm),
SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm),
@@ -12280,7 +12380,7 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
{
DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);
unsigned long offset = 0;
long offset = 0;
const char *func_name;
char *func;
int n;
@@ -12302,6 +12402,13 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
pr_warn("kprobe name is invalid: %s\n", func_name);
return -EINVAL;
}
if (offset < 0) {
free(func);
pr_warn("kprobe offset must be a non-negative integer: %li\n", offset);
return -EINVAL;
}
if (opts.retprobe && offset != 0) {
free(func);
pr_warn("kretprobes do not support offset specification\n");
@@ -12425,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru
return ret;
}
#define MAX_BPF_FUNC_ARGS 12
static bool btf_type_is_modifier(const struct btf_type *t)
{
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_TYPEDEF:
case BTF_KIND_VOLATILE:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_TYPE_TAG:
return true;
default:
return false;
}
}
#define MAX_RESOLVE_DEPTH 32
static int btf_get_type_size(const struct btf *btf, __u32 type_id,
const struct btf_type **ret_type)
{
const struct btf_type *t;
int i;
*ret_type = btf__type_by_id(btf, 0);
if (!type_id)
return 0;
t = btf__type_by_id(btf, type_id);
for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++)
t = btf__type_by_id(btf, t->type);
if (!t || i == MAX_RESOLVE_DEPTH)
return -EINVAL;
*ret_type = t;
if (btf_is_ptr(t))
return btf__pointer_size(btf);
if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t))
return t->size;
return -EINVAL;
}
bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t)
{
const struct btf_param *args;
const struct btf_type *proto;
__u32 i, nargs;
int ret;
if (!btf_is_func(t))
return false;
proto = btf__type_by_id(btf, t->type);
if (!proto || !btf_is_func_proto(proto))
return false;
args = (const struct btf_param *)(proto + 1);
nargs = btf_vlen(proto);
if (nargs > MAX_BPF_FUNC_ARGS)
return false;
/* No support for struct return type. */
ret = btf_get_type_size(btf, proto->type, &t);
if (ret < 0 || btf_is_struct(t) || btf_is_union(t))
return false;
for (i = 0; i < nargs; i++) {
/* No support for variable args. */
if (i == nargs - 1 && args[i].type == 0)
return false;
ret = btf_get_type_size(btf, args[i].type, &t);
/* No support of struct argument size greater than 16 bytes. */
if (ret < 0 || ret > 16)
return false;
/* No support for void argument. */
if (ret == 0)
return false;
}
return true;
}
static int
collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids)
{
__u32 type_id, nr_types = btf__type_cnt(btf);
size_t cap = 0, cnt = 0;
if (!pattern)
return -EINVAL;
for (type_id = 1; type_id < nr_types; type_id++) {
const struct btf_type *t = btf__type_by_id(btf, type_id);
const char *name;
int err;
if (btf_kind(t) != BTF_KIND_FUNC)
continue;
name = btf__name_by_offset(btf, t->name_off);
if (!name)
continue;
if (!glob_match(name, pattern))
continue;
if (!btf_type_is_traceable_func(btf, t))
continue;
err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1);
if (err) {
free(*ids);
return -ENOMEM;
}
(*ids)[cnt++] = type_id;
}
return cnt;
}
static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids)
{
struct bpf_object *obj = prog->obj;
const struct module_btf *mod;
struct btf *btf = NULL;
const char *sep;
int err;
err = bpf_object__load_vmlinux_btf(obj, true);
if (err)
return err;
/* In case we have module specified, we will find its btf and use that. */
sep = strchr(pattern, ':');
if (sep) {
mod = find_attach_module(obj, pattern);
if (!mod) {
err = -EINVAL;
goto cleanup;
}
btf = mod->btf;
pattern = sep + 1;
} else {
/* Program is loaded for kernel module. */
if (prog->attach_btf_obj_fd) {
err = -EINVAL;
goto cleanup;
}
btf = obj->btf_vmlinux;
}
err = collect_btf_func_ids_by_glob(btf, pattern, ids);
cleanup:
bpf_object_cleanup_btf(obj);
return err;
}
struct bpf_link *
bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern,
const struct bpf_tracing_multi_opts *opts)
{
LIBBPF_OPTS(bpf_link_create_opts, lopts);
int prog_fd, link_fd, err, cnt;
__u32 *free_ids = NULL;
struct bpf_link *link;
const __u64 *cookies;
const __u32 *ids;
if (!OPTS_VALID(opts, bpf_tracing_multi_opts))
return libbpf_err_ptr(-EINVAL);
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n",
prog->name);
return libbpf_err_ptr(-EINVAL);
}
cnt = OPTS_GET(opts, cnt, 0);
ids = OPTS_GET(opts, ids, NULL);
cookies = OPTS_GET(opts, cookies, NULL);
if (!!ids != !!cnt)
return libbpf_err_ptr(-EINVAL);
if (pattern && (ids || cookies))
return libbpf_err_ptr(-EINVAL);
if (!pattern && !ids)
return libbpf_err_ptr(-EINVAL);
if (pattern) {
cnt = collect_func_ids_by_glob(prog, pattern, &free_ids);
if (cnt < 0)
return libbpf_err_ptr(cnt);
if (cnt == 0)
return libbpf_err_ptr(-EINVAL);
ids = (const __u32 *) free_ids;
}
lopts.tracing_multi.ids = ids;
lopts.tracing_multi.cookies = cookies;
lopts.tracing_multi.cnt = cnt;
link = calloc(1, sizeof(*link));
if (!link) {
err = -ENOMEM;
goto error;
}
link->detach = &bpf_link__detach_fd;
link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts);
if (link_fd < 0) {
err = -errno;
pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err));
goto error;
}
link->fd = link_fd;
free(free_ids);
return link;
error:
free(link);
free(free_ids);
return libbpf_err_ptr(err);
}
static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
{
static const char *const prefixes[] = {
"fentry.multi",
"fexit.multi",
"fsession.multi",
"fentry.multi.s",
"fexit.multi.s",
"fsession.multi.s",
};
const char *spec = NULL;
char *pattern;
size_t i;
int n;
*link = NULL;
for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
size_t pfx_len;
if (!str_has_pfx(prog->sec_name, prefixes[i]))
continue;
pfx_len = strlen(prefixes[i]);
/* no auto-attach case of, e.g., SEC("fentry.multi") */
if (prog->sec_name[pfx_len] == '\0')
return 0;
if (prog->sec_name[pfx_len] != '/')
continue;
spec = prog->sec_name + pfx_len + 1;
break;
}
if (!spec) {
pr_warn("prog '%s': invalid section name '%s'\n",
prog->name, prog->sec_name);
return -EINVAL;
}
n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern);
if (n < 1) {
pr_warn("tracing multi pattern is invalid: %s\n", spec);
return -EINVAL;
}
*link = bpf_program__attach_tracing_multi(prog, pattern, NULL);
free(pattern);
return libbpf_get_error(*link);
}
static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe,
const char *binary_path, size_t offset)
{
@@ -13145,25 +13525,61 @@ struct bpf_link *bpf_program__attach_tracepoint(const struct bpf_program *prog,
return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL);
}
/*
* Match section name against a prefix array. Returns pointer past
* "prefix/" on match, empty string for bare sections (exact prefix
* match), or NULL if no prefix matches.
*/
static const char *sec_name_match_prefix(const char *sec_name,
const char *const *prefixes,
size_t n)
{
size_t i;
for (i = 0; i < n; i++) {
size_t pfx_len;
if (!str_has_pfx(sec_name, prefixes[i]))
continue;
pfx_len = strlen(prefixes[i]);
if (sec_name[pfx_len] == '\0')
return sec_name + pfx_len;
if (sec_name[pfx_len] != '/' || sec_name[pfx_len + 1] == '\0')
continue;
return sec_name + pfx_len + 1;
}
return NULL;
}
static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link)
{
static const char *const prefixes[] = {
"tp.s",
"tp",
"tracepoint.s",
"tracepoint",
};
char *sec_name, *tp_cat, *tp_name;
const char *match;
*link = NULL;
/* no auto-attach for SEC("tp") or SEC("tracepoint") */
if (strcmp(prog->sec_name, "tp") == 0 || strcmp(prog->sec_name, "tracepoint") == 0)
match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
if (!match) {
pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
return -EINVAL;
}
if (!match[0]) /* bare section name no autoattach */
return 0;
sec_name = strdup(prog->sec_name);
if (!sec_name)
return -ENOMEM;
/* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */
if (str_has_pfx(prog->sec_name, "tp/"))
tp_cat = sec_name + sizeof("tp/") - 1;
else
tp_cat = sec_name + sizeof("tracepoint/") - 1;
tp_cat = sec_name + (match - prog->sec_name);
tp_name = strchr(tp_cat, '/');
if (!tp_name) {
free(sec_name);
@@ -13227,37 +13643,22 @@ static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf
"raw_tracepoint",
"raw_tp.w",
"raw_tracepoint.w",
"raw_tp.s",
"raw_tracepoint.s",
};
size_t i;
const char *tp_name = NULL;
const char *match;
*link = NULL;
for (i = 0; i < ARRAY_SIZE(prefixes); i++) {
size_t pfx_len;
if (!str_has_pfx(prog->sec_name, prefixes[i]))
continue;
pfx_len = strlen(prefixes[i]);
/* no auto-attach case of, e.g., SEC("raw_tp") */
if (prog->sec_name[pfx_len] == '\0')
return 0;
if (prog->sec_name[pfx_len] != '/')
continue;
tp_name = prog->sec_name + pfx_len + 1;
break;
}
if (!tp_name) {
pr_warn("prog '%s': invalid section name '%s'\n",
prog->name, prog->sec_name);
match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes));
if (!match) {
pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name);
return -EINVAL;
}
if (!match[0])
return 0;
*link = bpf_program__attach_raw_tracepoint(prog, tp_name);
*link = bpf_program__attach_raw_tracepoint(prog, match);
return libbpf_get_error(*link);
}

Some files were not shown because too many files have changed in this diff Show More