mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-31 08:15:07 -04:00
Merge branch 'add-arena-argument-support-to-kfuncs-and-struct_ops'
Kumar Kartikeya Dwivedi says: ==================== Add arena argument support to kfuncs and struct_ops This is a continuation of patches in [0], with mostly minor changes and reordering. The motivation is covered in that link. A major change is moving to two tags (__arena and __arena__nullable) and moving the changes to JIT to emit more optimized sequences. Please see commit logs for details. [0]: https://lore.kernel.org/bpf/20260713024414.3759854-1-tj@kernel.org Changelog: ---------- v4 -> v5 v4: https://lore.kernel.org/bpf/20260805210427.3218326-1-memxor@gmail.com * Remove the redundant patch-8 capability comment and duplicate nullable kfunc test coverage. (Eduard) * Introduce the final bpf_tramp_arena_base() interface directly with function-model argument flags, avoiding temporary slot bitmaps and arena_nullable state; simplify struct_ops pointer validation. (Eduard) * Simplify kfunc arena nullability classification by using the common nullable path for both arena suffixes while leaving the function model to distinguish JIT NULL preservation. (Amery) * Keep bpf_prog_has_arena_ctx_arg() in bpf_verifier.h from its introduction so trampoline and verifier users share one inline definition, avoiding BPF_JIT/BPF_SYSCALL link dependencies. (Eduard, BPF CI Bot) * Reject both tracing and extension attachments to struct_ops programs with arena context arguments, and add fentry, fexit, and freplace rejection tests. (Eduard, Sashiko) v3 -> v4 v3: https://lore.kernel.org/bpf/20260803125115.2264733-1-memxor@gmail.com * Rename __arena_nullable to __arena__nullable and prioritize the composite suffix over __nullable during argument classification. (Sashiko, Eduard) * Resolve instructions before collecting subprograms and kfuncs so kfunc prototype validation can use associated arena state. * Move the arena kfunc and JIT-sequence test entry points into prog_tests/verifier.c. (Eduard) * Match the generated L0 target and call in nullable JIT assertions. (Eduard) * Route arena kfunc validation through the common argument-checking path. (Amery) * Reuse btf_func_model argument flags for struct_ops arena arguments instead of maintaining separate trampoline slot metadata. (Eduard) * Check the generic-trampoline arena argument invariant at link time and warn once on violations. (Eduard) * Reject tracing attachments to struct_ops programs with arena context arguments whose indirect trampolines convert the pointers. (Sashiko) v2 -> v3 v2: https://lore.kernel.org/bpf/20260726013105.3689867-1-memxor@gmail.com * Rebase onto current bpf-next to resolve conflicts. v1 -> v2 v1: https://lore.kernel.org/bpf/20260715220052.1590783-1-memxor@gmail.com * Fix documentation to only mention x86 for now. (Sashiko) * Move arg bitmap from insn_aux_data to kfunc descriptor. (Eduard) ==================== Link: https://patch.msgid.link/20260808003938.3486067-1-memxor@gmail.com Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
This commit is contained in:
@@ -278,6 +278,43 @@ An example is given below::
|
||||
...
|
||||
}
|
||||
|
||||
2.3.8 __arena and __arena__nullable Annotations
|
||||
-----------------------------------------------
|
||||
|
||||
Both annotations indicate that the pointer argument points into the
|
||||
calling program's arena. The JIT rebases the value at the call site so
|
||||
the kfunc receives a directly dereferenceable kernel address, subject to
|
||||
the access rules described in :ref:`BPF_kfunc_arena_access` (at most
|
||||
``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access).
|
||||
|
||||
With ``__arena`` the rebase is unconditional and the argument is never
|
||||
NULL: a value whose lower 32 bits are zero arrives as the arena base
|
||||
address (arena offset 0). The kfunc must not check the argument for NULL.
|
||||
With ``__arena__nullable`` such a value arrives as NULL instead and the
|
||||
kfunc must check before dereferencing.
|
||||
|
||||
An example is given below::
|
||||
|
||||
__bpf_kfunc int bpf_process_item(struct item *item__arena)
|
||||
{
|
||||
...
|
||||
}
|
||||
|
||||
Calling such a kfunc requires the program to use an arena map and a JIT with
|
||||
arena argument support (currently x86-64); verification fails otherwise. The
|
||||
program can pass any value without compromising the kernel. A value that does
|
||||
not point into the arena is a program bug.
|
||||
|
||||
The suffixes have the same meaning on the arguments of struct_ops stub
|
||||
functions, with the conversion running in the opposite direction. The
|
||||
kernel caller passes the kernel arena address and the trampoline converts
|
||||
it while saving the arguments, so the callback receives an arena pointer
|
||||
it can dereference directly. With ``__arena`` the kernel caller must not
|
||||
pass NULL. With ``__arena__nullable`` a NULL kernel pointer arrives as NULL.
|
||||
However, there is no obligation to prove to the verifier that such a pointer is
|
||||
non-NULL before use, in-line with existing semantics of arena pointers used in
|
||||
a program (or obtained from any other source).
|
||||
|
||||
.. _BPF_kfunc_nodef:
|
||||
|
||||
2.4 Using an existing kernel function
|
||||
@@ -522,6 +559,8 @@ In order to accommodate such requirements, the verifier will enforce strict
|
||||
PTR_TO_BTF_ID type matching if two types have the exact same name, with one
|
||||
being suffixed with ``___init``.
|
||||
|
||||
.. _BPF_kfunc_arena_access:
|
||||
|
||||
2.8 Accessing arena memory through kfunc arguments
|
||||
--------------------------------------------------
|
||||
|
||||
|
||||
@@ -1678,6 +1678,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Rebase the __arena args of a kfunc call to arena kernel addresses,
|
||||
* rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
|
||||
* arg preserves NULL by skipping the add, tested on the truncated value as
|
||||
* arena NULL is offset 0. Return the number of emitted bytes.
|
||||
*/
|
||||
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
|
||||
const struct bpf_insn *insn, u8 **pprog)
|
||||
{
|
||||
const struct btf_func_model *fm;
|
||||
u8 *prog = *pprog;
|
||||
u8 *start = prog;
|
||||
int i;
|
||||
|
||||
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
|
||||
if (!fm)
|
||||
return -EINVAL;
|
||||
|
||||
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
|
||||
u8 flags = fm->arg_flags[i];
|
||||
u32 reg = BPF_REG_1 + i;
|
||||
|
||||
if (!(flags & BTF_FMODEL_ARENA_ARG))
|
||||
continue;
|
||||
if (WARN_ON_ONCE(!bpf_prog->aux->arena))
|
||||
return -EINVAL;
|
||||
|
||||
/* mov eN, eN: truncate and clear the upper 32 bits */
|
||||
emit_mov_reg(&prog, false, reg, reg);
|
||||
if (flags & BTF_FMODEL_NULLABLE_ARG) {
|
||||
/* test eN, eN; jz over the 3-byte add */
|
||||
maybe_emit_mod(&prog, reg, reg, false);
|
||||
EMIT2(0x85, add_2reg(0xC0, reg, reg));
|
||||
EMIT2(X86_JE, 3);
|
||||
}
|
||||
/* add rN, r12 */
|
||||
maybe_emit_mod(&prog, reg, X86_REG_R12, true);
|
||||
EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
|
||||
}
|
||||
|
||||
*pprog = prog;
|
||||
return prog - start;
|
||||
}
|
||||
|
||||
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
|
||||
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
|
||||
{
|
||||
@@ -2588,6 +2632,12 @@ st: insn_off = insn->off;
|
||||
}
|
||||
if (!imm32)
|
||||
return -EINVAL;
|
||||
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
|
||||
err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
|
||||
if (err < 0)
|
||||
return err;
|
||||
ip += err;
|
||||
}
|
||||
if (priv_frame_ptr) {
|
||||
push_r9(&prog);
|
||||
ip += 2;
|
||||
@@ -2998,11 +3048,39 @@ static int get_nr_used_regs(const struct btf_func_model *m)
|
||||
return nr_used_regs;
|
||||
}
|
||||
|
||||
/*
|
||||
* Convert an arena kernel address into the arena pointer form on its way
|
||||
* into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg
|
||||
* preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit
|
||||
* subtraction both truncates and clears the upper half, so the stored
|
||||
* value satisfies the JIT invariant for arena pointer registers.
|
||||
*/
|
||||
static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
|
||||
{
|
||||
u8 *prog = *pprog;
|
||||
|
||||
if (nullable) {
|
||||
if (src_reg != BPF_REG_0)
|
||||
emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
|
||||
/* test rax, rax; jz over the 5-byte sub */
|
||||
EMIT3(0x48, 0x85, 0xC0);
|
||||
EMIT2(X86_JE, 5);
|
||||
} else if (src_reg != BPF_REG_0) {
|
||||
emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
|
||||
}
|
||||
/* sub eax, base_lo */
|
||||
EMIT1_off32(0x2D, base_lo);
|
||||
|
||||
*pprog = prog;
|
||||
}
|
||||
|
||||
static void save_args(const struct btf_func_model *m, u8 **prog,
|
||||
int stack_size, bool for_call_origin, u32 flags)
|
||||
int stack_size, bool for_call_origin, u32 flags,
|
||||
u64 arena_base)
|
||||
{
|
||||
int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
|
||||
bool use_jmp = bpf_trampoline_use_jmp(flags);
|
||||
int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24;
|
||||
int i, j;
|
||||
|
||||
/* Store function arguments to stack.
|
||||
@@ -3011,6 +3089,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
|
||||
* mov QWORD PTR [rbp-0x8],rsi
|
||||
*/
|
||||
for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) {
|
||||
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
|
||||
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
|
||||
|
||||
arg_regs = (m->arg_size[i] + 7) / 8;
|
||||
|
||||
/* According to the research of Yonghong, struct members
|
||||
@@ -3034,16 +3115,19 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
|
||||
/* copy function arguments from origin stack frame
|
||||
* into current stack frame.
|
||||
*
|
||||
* The starting address of the arguments on-stack
|
||||
* is:
|
||||
* rbp + 8(push rbp) +
|
||||
* 8(return addr of origin call) +
|
||||
* 8(return addr of the caller)
|
||||
* which means: rbp + 24
|
||||
* The arguments on-stack start above the saved rbp
|
||||
* and the return addresses: two return addresses
|
||||
* (origin call and caller) when the trampoline is
|
||||
* entered through the fentry call, so rbp + 24, and
|
||||
* a single one when it is entered with a jmp or
|
||||
* called indirectly, so rbp + 16.
|
||||
*/
|
||||
for (j = 0; j < arg_regs; j++) {
|
||||
emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
|
||||
nr_stack_slots * 8 + 16 + (!use_jmp) * 8);
|
||||
nr_stack_slots * 8 + stack_args_off);
|
||||
if (arena_arg)
|
||||
emit_arena_arg_conv(prog, BPF_REG_0, nullable,
|
||||
(u32)arena_base);
|
||||
emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
|
||||
-stack_size);
|
||||
|
||||
@@ -3064,9 +3148,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
|
||||
|
||||
/* copy the arguments from regs into stack */
|
||||
for (j = 0; j < arg_regs; j++) {
|
||||
emit_stx(prog, BPF_DW, BPF_REG_FP,
|
||||
nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
|
||||
-stack_size);
|
||||
u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
|
||||
|
||||
if (arena_arg) {
|
||||
emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
|
||||
src = BPF_REG_0;
|
||||
}
|
||||
emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
|
||||
stack_size -= 8;
|
||||
nr_regs++;
|
||||
}
|
||||
@@ -3362,6 +3450,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
|
||||
void *orig_call = func_addr;
|
||||
int cookie_off, cookie_cnt;
|
||||
u8 **branches = NULL;
|
||||
u64 arena_base;
|
||||
u64 func_meta;
|
||||
u8 *prog;
|
||||
bool save_ret;
|
||||
@@ -3374,6 +3463,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
|
||||
WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) &&
|
||||
(flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET)));
|
||||
|
||||
arena_base = bpf_tramp_arena_base(m, tnodes, flags);
|
||||
|
||||
for (i = 0; i < m->nr_args; i++)
|
||||
nr_regs += (m->arg_size[i] + 7) / 8 - 1;
|
||||
|
||||
@@ -3508,7 +3599,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
|
||||
emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
|
||||
}
|
||||
|
||||
save_args(m, &prog, regs_off, false, flags);
|
||||
save_args(m, &prog, regs_off, false, flags, arena_base);
|
||||
|
||||
if (flags & BPF_TRAMP_F_CALL_ORIG) {
|
||||
/* arg1: mov rdi, im */
|
||||
@@ -3550,7 +3641,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
|
||||
|
||||
if (flags & BPF_TRAMP_F_CALL_ORIG) {
|
||||
restore_regs(m, &prog, regs_off);
|
||||
save_args(m, &prog, arg_stack_off, true, flags);
|
||||
save_args(m, &prog, arg_stack_off, true, flags, 0);
|
||||
|
||||
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
|
||||
/* Before calling the original function, load the
|
||||
@@ -4051,6 +4142,11 @@ bool bpf_jit_supports_stack_args(void)
|
||||
return true;
|
||||
}
|
||||
|
||||
bool bpf_jit_supports_arena_args(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
|
||||
{
|
||||
if (text_poke_copy(dst, src, len) == NULL)
|
||||
|
||||
@@ -1195,6 +1195,12 @@ struct bpf_prog_offload {
|
||||
/* The argument is signed. */
|
||||
#define BTF_FMODEL_SIGNED_ARG BIT(1)
|
||||
|
||||
/* The argument is an arena pointer. */
|
||||
#define BTF_FMODEL_ARENA_ARG BIT(2)
|
||||
|
||||
/* The argument is nullable. */
|
||||
#define BTF_FMODEL_NULLABLE_ARG BIT(3)
|
||||
|
||||
struct btf_func_model {
|
||||
u8 ret_size;
|
||||
u8 ret_flags;
|
||||
@@ -1268,6 +1274,15 @@ struct bpf_tramp_nodes {
|
||||
int nr_nodes;
|
||||
};
|
||||
|
||||
/*
|
||||
* The arena base against which a struct_ops trampoline converts the
|
||||
* arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF
|
||||
* ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline
|
||||
* converts nothing.
|
||||
*/
|
||||
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
|
||||
struct bpf_tramp_nodes *tnodes, u32 flags);
|
||||
|
||||
struct bpf_tramp_run_ctx;
|
||||
|
||||
/* Different use cases for BPF trampoline:
|
||||
|
||||
@@ -1172,8 +1172,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
|
||||
*btf_id = key & 0x7FFFFFFF;
|
||||
}
|
||||
|
||||
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr, bpfptr_t uattr);
|
||||
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr, bpfptr_t uattr);
|
||||
int bpf_check_btf_info(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr, bpfptr_t uattr);
|
||||
|
||||
@@ -1297,6 +1297,16 @@ static inline u32 type_flag(u32 type)
|
||||
return type & ~BPF_BASE_TYPE_MASK;
|
||||
}
|
||||
|
||||
static inline bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
|
||||
{
|
||||
int i;
|
||||
|
||||
for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
|
||||
if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
static inline enum bpf_prog_type resolve_prog_type(const struct bpf_prog *prog)
|
||||
{
|
||||
return (prog->type == BPF_PROG_TYPE_EXT && prog->aux->saved_dst_prog_type) ?
|
||||
|
||||
@@ -1214,6 +1214,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);
|
||||
bool bpf_jit_supports_percpu_insn(void);
|
||||
bool bpf_jit_supports_kfunc_call(void);
|
||||
bool bpf_jit_supports_stack_args(void);
|
||||
bool bpf_jit_supports_arena_args(void);
|
||||
bool bpf_jit_supports_far_kfunc_call(void);
|
||||
bool bpf_jit_supports_exceptions(void);
|
||||
bool bpf_jit_supports_ptr_xchg(void);
|
||||
|
||||
@@ -147,6 +147,8 @@ void bpf_struct_ops_image_free(void *image)
|
||||
|
||||
#define MAYBE_NULL_SUFFIX "__nullable"
|
||||
#define REFCOUNTED_SUFFIX "__ref"
|
||||
#define ARENA_SUFFIX "__arena"
|
||||
#define ARENA_MAYBE_NULL_SUFFIX "__arena__nullable"
|
||||
|
||||
/* Prepare argument info for every nullable argument of a member of a
|
||||
* struct_ops type.
|
||||
@@ -159,7 +161,7 @@ void bpf_struct_ops_image_free(void *image)
|
||||
* to provide an array of struct bpf_ctx_arg_aux, which in turn provides
|
||||
* the information that used by the verifier to check the arguments of the
|
||||
* BPF struct_ops program assigned to the member. Here, we only care about
|
||||
* the arguments that are marked as __nullable.
|
||||
* the arguments that are marked as __nullable, __ref or __arena.
|
||||
*
|
||||
* The array of struct bpf_ctx_arg_aux is eventually assigned to
|
||||
* prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
|
||||
@@ -172,10 +174,12 @@ static int prepare_arg_info(struct btf *btf,
|
||||
const char *st_ops_name,
|
||||
const char *member_name,
|
||||
const struct btf_type *func_proto, void *stub_func_addr,
|
||||
struct btf_func_model *model,
|
||||
struct bpf_struct_ops_arg_info *arg_info)
|
||||
{
|
||||
const struct btf_type *stub_func_proto, *pointed_type;
|
||||
bool is_nullable = false, is_refcounted = false;
|
||||
bool is_nullable = false, is_refcounted = false, is_arena = false;
|
||||
bool is_arena_nullable = false;
|
||||
const struct btf_param *stub_args, *args;
|
||||
struct bpf_ctx_arg_aux *info, *info_buf;
|
||||
u32 nargs, arg_no, info_cnt = 0;
|
||||
@@ -225,27 +229,39 @@ static int prepare_arg_info(struct btf *btf,
|
||||
/* Prepare info for every nullable argument */
|
||||
info = info_buf;
|
||||
for (arg_no = 0; arg_no < nargs; arg_no++) {
|
||||
/* Skip arguments that is not suffixed with
|
||||
* "__nullable or __ref".
|
||||
bool ptr_to_arena, ptr_to_struct;
|
||||
|
||||
/*
|
||||
* Skip arguments that are not suffixed with "__arena__nullable",
|
||||
* "__arena", "__nullable", or "__ref".
|
||||
*/
|
||||
is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
|
||||
MAYBE_NULL_SUFFIX);
|
||||
is_arena_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
|
||||
ARENA_MAYBE_NULL_SUFFIX);
|
||||
is_arena = btf_param_match_suffix(btf, &stub_args[arg_no], ARENA_SUFFIX);
|
||||
is_nullable = !is_arena_nullable &&
|
||||
btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX);
|
||||
is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no],
|
||||
REFCOUNTED_SUFFIX);
|
||||
|
||||
if (is_nullable)
|
||||
if (is_arena_nullable)
|
||||
suffix = ARENA_MAYBE_NULL_SUFFIX;
|
||||
else if (is_arena)
|
||||
suffix = ARENA_SUFFIX;
|
||||
else if (is_nullable)
|
||||
suffix = MAYBE_NULL_SUFFIX;
|
||||
else if (is_refcounted)
|
||||
suffix = REFCOUNTED_SUFFIX;
|
||||
else
|
||||
continue;
|
||||
|
||||
/* Should be a pointer to struct */
|
||||
pointed_type = btf_type_resolve_ptr(btf,
|
||||
args[arg_no].type,
|
||||
&arg_btf_id);
|
||||
if (!pointed_type ||
|
||||
!btf_type_is_struct(pointed_type)) {
|
||||
/*
|
||||
* Should be a pointer to struct, or any pointer for __arena or
|
||||
* __arena__nullable.
|
||||
*/
|
||||
pointed_type = btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id);
|
||||
ptr_to_arena = pointed_type && (is_arena || is_arena_nullable);
|
||||
ptr_to_struct = pointed_type && btf_type_is_struct(pointed_type);
|
||||
if (!ptr_to_arena && !ptr_to_struct) {
|
||||
pr_warn("stub function %s has %s tagging to an unsupported type\n",
|
||||
stub_fname, suffix);
|
||||
goto err_out;
|
||||
@@ -268,7 +284,18 @@ static int prepare_arg_info(struct btf *btf,
|
||||
info->btf_id = arg_btf_id;
|
||||
info->btf = btf;
|
||||
info->offset = offset;
|
||||
if (is_nullable) {
|
||||
if (is_arena || is_arena_nullable) {
|
||||
/*
|
||||
* Both types get PTR_TO_ARENA. In verifier state,
|
||||
* PTR_TO_ARENA encompasses potential NULL values, but
|
||||
* we do not force the program to check it, or maintain
|
||||
* precision around it, since it has no safety implication.
|
||||
*/
|
||||
info->reg_type = PTR_TO_ARENA;
|
||||
model->arg_flags[arg_no] |= BTF_FMODEL_ARENA_ARG;
|
||||
if (is_arena_nullable)
|
||||
model->arg_flags[arg_no] |= BTF_FMODEL_NULLABLE_ARG;
|
||||
} else if (is_nullable) {
|
||||
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID | PTR_MAYBE_NULL;
|
||||
} else if (is_refcounted) {
|
||||
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID;
|
||||
@@ -460,6 +487,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
|
||||
stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
|
||||
err = prepare_arg_info(btf, st_ops->name, mname,
|
||||
func_proto, stub_func_addr,
|
||||
&st_ops->func_models[i],
|
||||
arg_info + i);
|
||||
if (err)
|
||||
goto errout;
|
||||
|
||||
@@ -6963,15 +6963,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
|
||||
return false;
|
||||
}
|
||||
|
||||
/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */
|
||||
/*
|
||||
* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or
|
||||
* PTR_TO_ARENA (both nullable and non-nullable cases).
|
||||
*/
|
||||
for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
|
||||
const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
|
||||
u32 type, flag;
|
||||
|
||||
type = base_type(ctx_arg_info->reg_type);
|
||||
flag = type_flag(ctx_arg_info->reg_type);
|
||||
if (ctx_arg_info->offset == off && type == PTR_TO_BUF &&
|
||||
(flag & PTR_MAYBE_NULL)) {
|
||||
if (ctx_arg_info->offset == off &&
|
||||
(type == PTR_TO_ARENA ||
|
||||
(type == PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) {
|
||||
info->reg_type = ctx_arg_info->reg_type;
|
||||
return true;
|
||||
}
|
||||
@@ -7539,6 +7543,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)
|
||||
return flags;
|
||||
}
|
||||
|
||||
static u8 __get_arg_fmodel_flags(const struct btf *btf,
|
||||
const struct btf_param *arg,
|
||||
const struct btf_type *t)
|
||||
{
|
||||
u8 flags = __get_type_fmodel_flags(t);
|
||||
|
||||
if (btf_param_match_suffix(btf, arg, "__arena__nullable"))
|
||||
flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG;
|
||||
else if (btf_param_match_suffix(btf, arg, "__arena"))
|
||||
flags |= BTF_FMODEL_ARENA_ARG;
|
||||
else if (btf_param_match_suffix(btf, arg, "__nullable"))
|
||||
flags |= BTF_FMODEL_NULLABLE_ARG;
|
||||
|
||||
return flags;
|
||||
}
|
||||
|
||||
int btf_distill_func_proto(struct bpf_verifier_log *log,
|
||||
struct btf *btf,
|
||||
const struct btf_type *func,
|
||||
@@ -7604,7 +7624,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
|
||||
return -EINVAL;
|
||||
}
|
||||
m->arg_size[i] = ret;
|
||||
m->arg_flags[i] = __get_type_fmodel_flags(t);
|
||||
m->arg_flags[i] = __get_arg_fmodel_flags(btf, &args[i], t);
|
||||
}
|
||||
m->nr_args = nargs;
|
||||
return 0;
|
||||
|
||||
@@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
|
||||
#define MIN_BPF_FUNCINFO_SIZE 8
|
||||
#define MAX_FUNCINFO_REC_SIZE 252
|
||||
|
||||
static int check_btf_func_early(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
bpfptr_t uattr)
|
||||
static int prepare_btf_func(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
bpfptr_t uattr)
|
||||
{
|
||||
u32 krec_size = sizeof(struct bpf_func_info);
|
||||
const struct btf_type *type, *func_proto;
|
||||
@@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,
|
||||
return err;
|
||||
}
|
||||
|
||||
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
bpfptr_t uattr)
|
||||
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
|
||||
const union bpf_attr *attr,
|
||||
bpfptr_t uattr)
|
||||
{
|
||||
struct btf *btf;
|
||||
int err;
|
||||
@@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env,
|
||||
}
|
||||
env->prog->aux->btf = btf;
|
||||
|
||||
err = check_btf_func_early(env, attr, uattr);
|
||||
err = prepare_btf_func(env, attr, uattr);
|
||||
if (err)
|
||||
return err;
|
||||
return 0;
|
||||
|
||||
@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)
|
||||
return false;
|
||||
}
|
||||
|
||||
bool __weak bpf_jit_supports_arena_args(void)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
bool __weak bpf_jit_supports_far_kfunc_call(void)
|
||||
{
|
||||
return false;
|
||||
|
||||
@@ -529,6 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
|
||||
return tnodes;
|
||||
}
|
||||
|
||||
/*
|
||||
* The arena base against which save_args() converts the arguments marked
|
||||
* with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline
|
||||
* converts: it dispatches to a single prog whose arena is known at
|
||||
* generation time. Return 0 when there is nothing to convert.
|
||||
*/
|
||||
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
|
||||
struct bpf_tramp_nodes *tnodes, u32 flags)
|
||||
{
|
||||
const struct bpf_prog *prog;
|
||||
int i;
|
||||
|
||||
if (!(flags & BPF_TRAMP_F_INDIRECT) ||
|
||||
tnodes[BPF_TRAMP_FENTRY].nr_nodes != 1)
|
||||
return 0;
|
||||
|
||||
for (i = 0; i < m->nr_args; i++)
|
||||
if (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG)
|
||||
break;
|
||||
if (i == m->nr_args)
|
||||
return 0;
|
||||
|
||||
/* Verification rejects an arena argument without an arena. */
|
||||
prog = tnodes[BPF_TRAMP_FENTRY].nodes[0]->link->prog;
|
||||
if (WARN_ON_ONCE(!prog->aux->arena))
|
||||
return 0;
|
||||
|
||||
return bpf_arena_get_kern_vm_start(prog->aux->arena);
|
||||
}
|
||||
|
||||
static void bpf_tramp_image_free(struct bpf_tramp_image *im)
|
||||
{
|
||||
bpf_image_ksym_del(&im->ksym);
|
||||
@@ -920,6 +950,13 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
|
||||
int cnt = 0, i;
|
||||
|
||||
kind = bpf_attach_type_to_tramp(node->link->prog);
|
||||
/*
|
||||
* Arena ctx args are converted only by struct_ops indirect
|
||||
* trampolines. They must never be attached to a generic trampoline.
|
||||
*/
|
||||
if (WARN_ON_ONCE(bpf_prog_has_arena_ctx_arg(node->link->prog)))
|
||||
return -ENOTSUPP;
|
||||
|
||||
if (tr->extension_prog)
|
||||
/* cannot attach fentry/fexit if extension prog is attached.
|
||||
* cannot overwrite extension prog either.
|
||||
|
||||
@@ -2837,7 +2837,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
|
||||
static int add_subprogs(struct bpf_verifier_env *env)
|
||||
{
|
||||
struct bpf_subprog_info *subprog = env->subprog_info;
|
||||
int i, ret, insn_cnt = env->prog->len, ex_cb_insn;
|
||||
@@ -2849,8 +2849,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
|
||||
return ret;
|
||||
|
||||
for (i = 0; i < insn_cnt; i++, insn++) {
|
||||
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn) &&
|
||||
!bpf_pseudo_kfunc_call(insn))
|
||||
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
|
||||
continue;
|
||||
|
||||
if (!env->bpf_capable) {
|
||||
@@ -2858,11 +2857,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
|
||||
return -EPERM;
|
||||
}
|
||||
|
||||
if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn))
|
||||
ret = add_subprog(env, i + insn->imm + 1);
|
||||
else
|
||||
ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
|
||||
|
||||
ret = add_subprog(env, i + insn->imm + 1);
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
}
|
||||
@@ -2900,6 +2895,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int add_kfuncs(struct bpf_verifier_env *env)
|
||||
{
|
||||
struct bpf_insn *insn = env->prog->insnsi;
|
||||
int i, ret, insn_cnt = env->prog->len;
|
||||
|
||||
for (i = 0; i < insn_cnt; i++, insn++) {
|
||||
if (!bpf_pseudo_kfunc_call(insn))
|
||||
continue;
|
||||
|
||||
if (!env->bpf_capable) {
|
||||
verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
|
||||
return -EPERM;
|
||||
}
|
||||
|
||||
ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
|
||||
if (ret < 0)
|
||||
return ret;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int check_subprogs(struct bpf_verifier_env *env)
|
||||
{
|
||||
int i, subprog_start, subprog_end, off, cur_subprog = 0;
|
||||
@@ -10760,7 +10777,8 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
|
||||
|
||||
static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
|
||||
{
|
||||
return btf_param_match_suffix(btf, arg, "__nullable");
|
||||
return btf_param_match_suffix(btf, arg, "__nullable") ||
|
||||
btf_param_match_suffix(btf, arg, "__arena");
|
||||
}
|
||||
|
||||
static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
|
||||
@@ -10778,6 +10796,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
|
||||
return btf_param_match_suffix(btf, arg, "__irq_flag");
|
||||
}
|
||||
|
||||
static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
|
||||
{
|
||||
return btf_param_match_suffix(btf, arg, "__arena__nullable") ||
|
||||
btf_param_match_suffix(btf, arg, "__arena");
|
||||
}
|
||||
|
||||
static bool is_kfunc_arg_scalar_with_name(const struct btf *btf,
|
||||
const struct btf_param *arg,
|
||||
const char *name)
|
||||
@@ -10998,6 +11022,7 @@ enum kfunc_ptr_arg_type {
|
||||
KF_ARG_PTR_TO_IRQ_FLAG,
|
||||
KF_ARG_PTR_TO_RES_SPIN_LOCK,
|
||||
KF_ARG_PTR_TO_TASK_WORK,
|
||||
KF_ARG_PTR_TO_ARENA,
|
||||
};
|
||||
|
||||
enum special_kfunc_type {
|
||||
@@ -11283,7 +11308,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
|
||||
reg_arg_name(env, argno), btf_type_str(t));
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
ref_t = btf_type_skip_modifiers(meta->btf, t->type, NULL);
|
||||
ref_tname = btf_name_by_offset(meta->btf, ref_t->name_off);
|
||||
|
||||
@@ -11332,7 +11356,30 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
|
||||
arg_type = KF_ARG_PTR_TO_RES_SPIN_LOCK;
|
||||
else if (is_kfunc_arg_callback(env, meta->btf, &args[arg]))
|
||||
arg_type = KF_ARG_PTR_TO_CALLBACK;
|
||||
else if (arg + 1 < nargs &&
|
||||
else if (is_kfunc_arg_arena(meta->btf, &args[arg])) {
|
||||
if (!bpf_jit_supports_arena_args()) {
|
||||
verbose(env, "JIT does not support kfunc %s() with arena pointer arguments\n",
|
||||
meta->func_name);
|
||||
return -ENOTSUPP;
|
||||
}
|
||||
if (!env->prog->aux->arena) {
|
||||
verbose(env,
|
||||
"%s arena pointer requires a program with an associated arena\n",
|
||||
reg_arg_name(env, argno));
|
||||
return -EINVAL;
|
||||
}
|
||||
if (reg_from_argno(argno) < 0) {
|
||||
verbose(env, "%s arena pointer cannot be a stack argument\n",
|
||||
reg_arg_name(env, argno));
|
||||
return -EINVAL;
|
||||
}
|
||||
/*
|
||||
* Both suffixes accept a constant zero. The function model determines
|
||||
* whether the JIT rebases it to the arena base or preserves NULL.
|
||||
* The common nullable path below records that verifier property.
|
||||
*/
|
||||
arg_type = KF_ARG_PTR_TO_ARENA;
|
||||
} else if (arg + 1 < nargs &&
|
||||
(is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
|
||||
is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
|
||||
if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
|
||||
@@ -12007,7 +12054,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
|
||||
t = btf_type_skip_modifiers(btf, args[i].type, NULL);
|
||||
|
||||
if (btf_type_is_ptr(t) && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) &&
|
||||
!is_kfunc_arg_nullable(meta->btf, &args[i])) {
|
||||
!type_may_be_null(kf_arg_type)) {
|
||||
verbose(env, "Possibly NULL pointer passed to trusted %s\n",
|
||||
reg_arg_name(env, argno));
|
||||
return -EACCES;
|
||||
@@ -12060,6 +12107,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
|
||||
case KF_ARG_PTR_TO_TASK_WORK:
|
||||
case KF_ARG_PTR_TO_IRQ_FLAG:
|
||||
case KF_ARG_PTR_TO_RES_SPIN_LOCK:
|
||||
case KF_ARG_PTR_TO_ARENA:
|
||||
break;
|
||||
case KF_ARG_PTR_TO_DYNPTR:
|
||||
arg_type = ARG_PTR_TO_DYNPTR;
|
||||
@@ -12126,6 +12174,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
|
||||
meta->ret_btf_id = ret;
|
||||
}
|
||||
break;
|
||||
case KF_ARG_PTR_TO_ARENA:
|
||||
if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) {
|
||||
verbose(env, "%s is not a pointer to arena or scalar\n",
|
||||
reg_arg_name(env, argno));
|
||||
return -EINVAL;
|
||||
}
|
||||
break;
|
||||
case KF_ARG_PTR_TO_ALLOC_BTF_ID:
|
||||
if (reg->type == (PTR_TO_BTF_ID | MEM_ALLOC)) {
|
||||
if (!is_bpf_obj_drop_kfunc(meta->func_id)) {
|
||||
@@ -18630,6 +18685,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
|
||||
{
|
||||
const struct btf_type *t, *func_proto;
|
||||
const struct bpf_struct_ops_desc *st_ops_desc;
|
||||
const struct bpf_struct_ops_arg_info *arg_info;
|
||||
const struct bpf_struct_ops *st_ops;
|
||||
const struct btf_member *member;
|
||||
struct bpf_prog *prog = env->prog;
|
||||
@@ -18708,10 +18764,23 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
|
||||
return -EACCES;
|
||||
}
|
||||
|
||||
for (i = 0; i < st_ops_desc->arg_info[member_idx].cnt; i++) {
|
||||
if (st_ops_desc->arg_info[member_idx].info[i].refcounted) {
|
||||
arg_info = &st_ops_desc->arg_info[member_idx];
|
||||
for (i = 0; i < arg_info->cnt; i++) {
|
||||
const struct bpf_ctx_arg_aux *info = &arg_info->info[i];
|
||||
|
||||
if (info->refcounted)
|
||||
has_refcounted_arg = true;
|
||||
break;
|
||||
if (base_type(info->reg_type) == PTR_TO_ARENA) {
|
||||
if (!bpf_jit_supports_arena_args()) {
|
||||
verbose(env, "JIT does not support arena arguments\n");
|
||||
return -ENOTSUPP;
|
||||
}
|
||||
if (!prog->aux->arena) {
|
||||
verbose(env,
|
||||
"arena argument of %s requires a program with an associated arena\n",
|
||||
mname);
|
||||
return -EINVAL;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18732,8 +18801,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
|
||||
prog->aux->attach_func_name = mname;
|
||||
env->ops = st_ops->verifier_ops;
|
||||
|
||||
return bpf_prog_ctx_arg_info_init(prog, st_ops_desc->arg_info[member_idx].info,
|
||||
st_ops_desc->arg_info[member_idx].cnt);
|
||||
return bpf_prog_ctx_arg_info_init(prog, arg_info->info, arg_info->cnt);
|
||||
}
|
||||
#define SECURITY_PREFIX "security_"
|
||||
|
||||
@@ -19000,6 +19068,16 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
|
||||
bpf_log(log, "Subprog %s doesn't exist\n", tname);
|
||||
return -EINVAL;
|
||||
}
|
||||
/*
|
||||
* A struct_ops indirect trampoline converts arena arguments
|
||||
* before invoking its program. A tracing or extension program
|
||||
* attached to the main program would see the converted offset as a
|
||||
* regular BTF pointer.
|
||||
*/
|
||||
if (subprog == 0 && bpf_prog_has_arena_ctx_arg(tgt_prog)) {
|
||||
bpf_log(log, "Cannot attach to a target with arena context arguments\n");
|
||||
return -EOPNOTSUPP;
|
||||
}
|
||||
if (aux->func && aux->func[subprog]->aux->exception_cb) {
|
||||
bpf_log(log,
|
||||
"%s programs cannot attach to exception callback\n",
|
||||
@@ -20135,11 +20213,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
|
||||
INIT_LIST_HEAD(&env->explored_states[i]);
|
||||
INIT_LIST_HEAD(&env->free_list);
|
||||
|
||||
ret = bpf_check_btf_info_early(env, attr, uattr);
|
||||
/* Prepare BTF and func_info needed to discover all subprograms. */
|
||||
ret = bpf_prepare_btf_info(env, attr, uattr);
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
ret = add_subprog_and_kfunc(env);
|
||||
/* Discover all subprograms before validating their layout and BTF. */
|
||||
ret = add_subprogs(env);
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
@@ -20147,14 +20227,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
/* Validate BTF against the complete subprogram layout and apply CO-RE. */
|
||||
ret = bpf_check_btf_info(env, attr, uattr);
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
/* Validate instructions and resolve the program's referenced resources. */
|
||||
ret = check_and_resolve_insns(env);
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
/* Build kfunc prototypes after resolving program resources. */
|
||||
ret = add_kfuncs(env);
|
||||
if (ret < 0)
|
||||
goto skip_full_check;
|
||||
|
||||
if (bpf_prog_is_offloaded(env->prog->aux)) {
|
||||
ret = bpf_prog_offload_verifier_prep(env->prog);
|
||||
if (ret)
|
||||
|
||||
128
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c
Normal file
128
tools/testing/selftests/bpf/prog_tests/test_struct_ops_arena.c
Normal file
@@ -0,0 +1,128 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
#include <test_progs.h>
|
||||
|
||||
#include "struct_ops_arena.skel.h"
|
||||
#include "struct_ops_arena_attach.skel.h"
|
||||
#include "struct_ops_arena_fail.skel.h"
|
||||
|
||||
#if defined(__x86_64__)
|
||||
/*
|
||||
* Attach callbacks with __arena and __arena__nullable arguments and drive
|
||||
* them through the bpf_testmod_ops3_call_test_arena*() kfuncs.
|
||||
*/
|
||||
static void arena_arg(void)
|
||||
{
|
||||
LIBBPF_OPTS(bpf_test_run_opts, topts);
|
||||
struct struct_ops_arena *skel;
|
||||
struct bpf_link *link = NULL;
|
||||
int err;
|
||||
|
||||
skel = struct_ops_arena__open_and_load();
|
||||
if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load"))
|
||||
return;
|
||||
|
||||
link = bpf_map__attach_struct_ops(skel->maps.testmod_arena);
|
||||
if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
|
||||
goto out;
|
||||
|
||||
err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger),
|
||||
&topts);
|
||||
ASSERT_OK(err, "test_run");
|
||||
ASSERT_EQ(topts.retval, 0, "trigger_retval");
|
||||
|
||||
out:
|
||||
bpf_link__destroy(link);
|
||||
struct_ops_arena__destroy(skel);
|
||||
}
|
||||
|
||||
/*
|
||||
* A program with no arena cannot attach to a member with an __arena
|
||||
* argument.
|
||||
*/
|
||||
static void arena_arg_fail(void)
|
||||
{
|
||||
struct struct_ops_arena_fail *skel;
|
||||
|
||||
skel = struct_ops_arena_fail__open_and_load();
|
||||
if (ASSERT_ERR_PTR(skel, "struct_ops_arena_fail__open_and_load"))
|
||||
return;
|
||||
|
||||
struct_ops_arena_fail__destroy(skel);
|
||||
}
|
||||
|
||||
static void arena_arg_attach_one(int target_fd, const char *prog_name)
|
||||
{
|
||||
struct struct_ops_arena_attach *skel;
|
||||
struct bpf_program *prog, *pos;
|
||||
char log_buf[64 * 1024];
|
||||
int err;
|
||||
|
||||
skel = struct_ops_arena_attach__open();
|
||||
if (!ASSERT_OK_PTR(skel, "struct_ops_arena_attach__open"))
|
||||
return;
|
||||
|
||||
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
|
||||
if (!ASSERT_OK_PTR(prog, prog_name))
|
||||
goto out;
|
||||
|
||||
bpf_object__for_each_program(pos, skel->obj)
|
||||
bpf_program__set_autoload(pos, pos == prog);
|
||||
|
||||
err = bpf_program__set_attach_target(prog, target_fd, "test_arena_cb");
|
||||
if (!ASSERT_OK(err, "set_attach_target"))
|
||||
goto out;
|
||||
|
||||
log_buf[0] = '\0';
|
||||
bpf_program__set_log_buf(prog, log_buf, sizeof(log_buf));
|
||||
err = struct_ops_arena_attach__load(skel);
|
||||
|
||||
ASSERT_EQ(err, -EOPNOTSUPP, prog_name);
|
||||
ASSERT_HAS_SUBSTR(log_buf, "Cannot attach to a target with arena context arguments",
|
||||
"verifier_log");
|
||||
|
||||
out:
|
||||
struct_ops_arena_attach__destroy(skel);
|
||||
}
|
||||
|
||||
static void arena_arg_attach(void)
|
||||
{
|
||||
struct struct_ops_arena *skel;
|
||||
int target_fd;
|
||||
|
||||
skel = struct_ops_arena__open_and_load();
|
||||
if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load"))
|
||||
return;
|
||||
|
||||
target_fd = bpf_program__fd(skel->progs.test_arena_cb);
|
||||
arena_arg_attach_one(target_fd, "fentry_test_arena");
|
||||
arena_arg_attach_one(target_fd, "fexit_test_arena");
|
||||
arena_arg_attach_one(target_fd, "freplace_test_arena");
|
||||
|
||||
struct_ops_arena__destroy(skel);
|
||||
}
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Serialized because it attaches the singleton bpf_testmod_ops3, which
|
||||
* test_struct_ops_private_stack also attaches; registering it twice fails
|
||||
* with -EEXIST.
|
||||
*/
|
||||
void serial_test_struct_ops_arena(void)
|
||||
{
|
||||
/*
|
||||
* Arena struct_ops arguments need JIT support, currently x86-64 only.
|
||||
* Elsewhere verification fails with "JIT does not support arena
|
||||
* arguments", so the programs cannot even load.
|
||||
*/
|
||||
#if defined(__x86_64__)
|
||||
if (test__start_subtest("arena_arg"))
|
||||
arena_arg();
|
||||
if (test__start_subtest("arena_arg_fail"))
|
||||
arena_arg_fail();
|
||||
if (test__start_subtest("arena_arg_attach"))
|
||||
arena_arg_attach();
|
||||
#else
|
||||
test__skip();
|
||||
#endif
|
||||
}
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
#include <test_progs.h>
|
||||
|
||||
#include "arena_kfunc.skel.h"
|
||||
#include "arena_kfunc_jit.skel.h"
|
||||
#include "cap_helpers.h"
|
||||
#include "verifier_align.skel.h"
|
||||
#include "verifier_and.skel.h"
|
||||
@@ -162,6 +164,10 @@ static void run_tests_aux(const char *skel_name,
|
||||
|
||||
#define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL)
|
||||
|
||||
void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); }
|
||||
|
||||
void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); }
|
||||
|
||||
void test_verifier_align(void) { RUN(verifier_align); }
|
||||
void test_verifier_and(void) { RUN(verifier_and); }
|
||||
void test_verifier_arena(void) { RUN(verifier_arena); }
|
||||
|
||||
234
tools/testing/selftests/bpf/progs/arena_kfunc.c
Normal file
234
tools/testing/selftests/bpf/progs/arena_kfunc.c
Normal file
@@ -0,0 +1,234 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
|
||||
#define BPF_NO_KFUNC_PROTOTYPES
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include "bpf_misc.h"
|
||||
#include "bpf_experimental.h"
|
||||
#include <bpf_arena_common.h>
|
||||
#include "../test_kmods/bpf_testmod_kfunc.h"
|
||||
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_ARENA);
|
||||
__uint(map_flags, BPF_F_MMAPABLE);
|
||||
/* page 0 hosts the arena global, page 1 is for allocations */
|
||||
__uint(max_entries, 2);
|
||||
} arena SEC(".maps");
|
||||
|
||||
/*
|
||||
* Occupies page 0 so no allocation lands at arena offset 0, which the
|
||||
* nullable tests below must be able to tell apart from NULL.
|
||||
*/
|
||||
u64 __arena arena_pad;
|
||||
|
||||
/* volatile to force the scalar reloads below */
|
||||
volatile u64 stash;
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__success __retval(0)
|
||||
int arena_arg_forms(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
u64 ret;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
/* PTR_TO_ARENA argument */
|
||||
*val = 41;
|
||||
ret = bpf_kfunc_arena_arg_test((u64 *)val);
|
||||
if (ret != 41 || *val != 42)
|
||||
return 2;
|
||||
|
||||
/* the low 32 bits as a scalar */
|
||||
stash = (u32)(u64)val;
|
||||
ret = bpf_kfunc_arena_arg_test((u64 *)stash);
|
||||
if (ret != 42 || *val != 43)
|
||||
return 3;
|
||||
|
||||
/* the full user address as a scalar */
|
||||
stash = (u64)val;
|
||||
bpf_addr_space_cast(stash, 1, 0);
|
||||
ret = bpf_kfunc_arena_arg_test((u64 *)stash);
|
||||
if (ret != 43 || *val != 44)
|
||||
return 4;
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Pin the rebase semantics using the capture kfuncs, which return the raw
|
||||
* argument value: __arena rebases unconditionally, so zero low 32 bits
|
||||
* arrive as the arena kernel base, while __arena__nullable turns them into
|
||||
* NULL.
|
||||
*/
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__success __retval(0)
|
||||
int arena_arg_rebase(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
u64 base, off;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
base = bpf_kfunc_arena_cap_test(NULL);
|
||||
if (!base)
|
||||
return 2;
|
||||
|
||||
/* only the low 32 bits contribute */
|
||||
stash = 0xbadc0ffe00000000;
|
||||
if (bpf_kfunc_arena_cap_test((u64 *)stash) != base)
|
||||
return 3;
|
||||
|
||||
off = (u32)(u64)val;
|
||||
if (bpf_kfunc_arena_cap_test((u64 *)val) != base + off)
|
||||
return 4;
|
||||
|
||||
if (bpf_kfunc_arena_cap_nullable_test(NULL) != 0)
|
||||
return 5;
|
||||
|
||||
stash = 0xbadc0ffe00000000;
|
||||
if (bpf_kfunc_arena_cap_nullable_test((u64 *)stash) != 0)
|
||||
return 6;
|
||||
|
||||
if (bpf_kfunc_arena_cap_nullable_test((u64 *)val) != base + off)
|
||||
return 7;
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__success __retval(0)
|
||||
int arena_args5(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
val[0] = 1;
|
||||
val[1] = 2;
|
||||
val[2] = 4;
|
||||
val[3] = 8;
|
||||
val[4] = 16;
|
||||
|
||||
if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
|
||||
(u64 *)&val[2], (u64 *)&val[3],
|
||||
(u64 *)&val[4]) != 31)
|
||||
return 2;
|
||||
if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
|
||||
(u64 *)&val[2], (u64 *)&val[3], NULL) != 15)
|
||||
return 3;
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__success __retval(0)
|
||||
int arena_arg_mixed(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
val[0] = 7;
|
||||
val[1] = 5;
|
||||
|
||||
if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], NULL) != 7)
|
||||
return 2;
|
||||
|
||||
if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], (u64 *)&val[1]) != 12)
|
||||
return 3;
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* kernel-side faults on unpopulated pages recover via the scratch page */
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__success __retval(0)
|
||||
int arena_arg_unpopulated(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
stash = (u64)val + PAGE_SIZE;
|
||||
bpf_kfunc_arena_arg_test((u64 *)stash);
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__failure __msg("arena pointer requires a program with an associated arena")
|
||||
int arena_arg_no_arena(void *ctx)
|
||||
{
|
||||
bpf_kfunc_arena_arg_test((u64 *)1);
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__failure __msg("is not a pointer to arena or scalar")
|
||||
int arena_arg_bad_reg(void *ctx)
|
||||
{
|
||||
u64 buf = 0;
|
||||
|
||||
/* use the arena so the program passes the arena presence check */
|
||||
bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
bpf_kfunc_arena_arg_test(&buf);
|
||||
return 0;
|
||||
}
|
||||
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \
|
||||
defined(__BPF_FEATURE_STACK_ARGUMENT)
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__failure __msg("arena pointer cannot be a stack argument")
|
||||
int arena_arg_stack(void *ctx)
|
||||
{
|
||||
bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
bpf_kfunc_arena_stack_arg_test(1, 2, 3, 4, 5, (u64 *)1);
|
||||
return 0;
|
||||
}
|
||||
#else
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__description("arena_arg_stack: not supported, dummy test")
|
||||
__success
|
||||
int arena_arg_stack(void *ctx)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
98
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c
Normal file
98
tools/testing/selftests/bpf/progs/arena_kfunc_jit.c
Normal file
@@ -0,0 +1,98 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
|
||||
/*
|
||||
* Verify the JIT-emitted rebase sequences for __arena and __arena__nullable
|
||||
* kfunc arguments. The capture kfuncs take the argument without
|
||||
* dereferencing it, so these tests pin only the emitted code.
|
||||
*/
|
||||
#define BPF_NO_KFUNC_PROTOTYPES
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include "bpf_misc.h"
|
||||
#include "bpf_experimental.h"
|
||||
#include <bpf_arena_common.h>
|
||||
#include "../test_kmods/bpf_testmod_kfunc.h"
|
||||
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_ARENA);
|
||||
__uint(map_flags, BPF_F_MMAPABLE);
|
||||
__uint(max_entries, 1);
|
||||
} arena SEC(".maps");
|
||||
|
||||
/* volatile to force the scalar reloads below */
|
||||
volatile u64 stash;
|
||||
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__jited("...")
|
||||
__jited(" movl %edi, %edi")
|
||||
__jited(" addq %r12, %rdi")
|
||||
__jited("...")
|
||||
__jited(" callq {{.*}}")
|
||||
__success
|
||||
int arena_arg_jit_rebase(void *ctx)
|
||||
{
|
||||
stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
bpf_kfunc_arena_cap_test((u64 *)stash);
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__jited("...")
|
||||
__jited(" movl %edi, %edi")
|
||||
__jited(" testl %edi, %edi")
|
||||
__jited(" je L0")
|
||||
__jited(" addq %r12, %rdi")
|
||||
__jited("L0: callq {{.*}}")
|
||||
__success
|
||||
int arena_arg_jit_nullable(void *ctx)
|
||||
{
|
||||
stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
bpf_kfunc_arena_cap_nullable_test((u64 *)stash);
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("syscall")
|
||||
__arch_x86_64
|
||||
__jited("...")
|
||||
__jited(" movl %edi, %edi")
|
||||
__jited(" addq %r12, %rdi")
|
||||
__jited(" movl %esi, %esi")
|
||||
__jited(" addq %r12, %rsi")
|
||||
__jited(" movl %edx, %edx")
|
||||
__jited(" addq %r12, %rdx")
|
||||
__jited(" movl %ecx, %ecx")
|
||||
__jited(" addq %r12, %rcx")
|
||||
__jited(" movl %r8d, %r8d")
|
||||
__jited(" testl %r8d, %r8d")
|
||||
__jited(" je L0")
|
||||
__jited(" addq %r12, %r8")
|
||||
__jited("L0: callq {{.*}}")
|
||||
__success
|
||||
int arena_arg_jit_args5(void *ctx)
|
||||
{
|
||||
u64 __arena *val;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
val[0] = 1;
|
||||
val[1] = 2;
|
||||
val[2] = 4;
|
||||
val[3] = 8;
|
||||
val[4] = 16;
|
||||
|
||||
bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
|
||||
(u64 *)&val[2], (u64 *)&val[3],
|
||||
(u64 *)&val[4]);
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif /* __BPF_FEATURE_ADDR_SPACE_CAST */
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
115
tools/testing/selftests/bpf/progs/struct_ops_arena.c
Normal file
115
tools/testing/selftests/bpf/progs/struct_ops_arena.c
Normal file
@@ -0,0 +1,115 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
|
||||
#define BPF_NO_KFUNC_PROTOTYPES
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include "bpf_experimental.h"
|
||||
#include <bpf_arena_common.h>
|
||||
#include "../test_kmods/bpf_testmod.h"
|
||||
#include "../test_kmods/bpf_testmod_kfunc.h"
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
|
||||
struct {
|
||||
__uint(type, BPF_MAP_TYPE_ARENA);
|
||||
__uint(map_flags, BPF_F_MMAPABLE);
|
||||
/* page 0 hosts the arena globals, page 1 is for allocations */
|
||||
__uint(max_entries, 2);
|
||||
} arena SEC(".maps");
|
||||
|
||||
/* also associates the callbacks with the arena */
|
||||
u64 __arena arena_touch;
|
||||
/* raw value of the last __arena ctx argument, captured by test_arena_cb */
|
||||
u64 __arena cb_ptr_val;
|
||||
|
||||
SEC("struct_ops/test_arena")
|
||||
int test_arena_cb(unsigned long long *ctx)
|
||||
{
|
||||
u64 __arena *ptr = (u64 __arena *)ctx[0];
|
||||
|
||||
arena_touch++;
|
||||
cb_ptr_val = ctx[0];
|
||||
*ptr += 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("struct_ops/test_arena_nullable")
|
||||
int test_arena_nullable_cb(unsigned long long *ctx)
|
||||
{
|
||||
u64 __arena *ptr = (u64 __arena *)ctx[0];
|
||||
|
||||
arena_touch++;
|
||||
if (!ptr)
|
||||
return 0xbee;
|
||||
*ptr += 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("struct_ops/test_arena_stack")
|
||||
int test_arena_stack_cb(unsigned long long *ctx)
|
||||
{
|
||||
u64 __arena *ptr = (u64 __arena *)ctx[8];
|
||||
|
||||
arena_touch++;
|
||||
/* pin the slot layout: the leading args fill ctx[0]..ctx[7] */
|
||||
if (ctx[0] != 1 || ctx[7] != 8)
|
||||
return 0xbad;
|
||||
*ptr += 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC(".struct_ops.link")
|
||||
struct bpf_testmod_ops3 testmod_arena = {
|
||||
.test_arena = (void *)test_arena_cb,
|
||||
.test_arena_nullable = (void *)test_arena_nullable_cb,
|
||||
.test_arena_stack = (void *)test_arena_stack_cb,
|
||||
};
|
||||
|
||||
SEC("syscall")
|
||||
int trigger(void *ctx)
|
||||
{
|
||||
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
|
||||
u64 __arena *val;
|
||||
int ret;
|
||||
|
||||
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
|
||||
if (!val)
|
||||
return 1;
|
||||
|
||||
*val = 41;
|
||||
ret = bpf_testmod_ops3_call_test_arena((u64 *)val);
|
||||
if (ret)
|
||||
return 2;
|
||||
if (*val != 42)
|
||||
return 3;
|
||||
|
||||
/*
|
||||
* The callback must have seen exactly (u32)(kaddr - kern_vm_start),
|
||||
* which is the arena offset of val with the upper 32 bits clear.
|
||||
*/
|
||||
if (cb_ptr_val != (u32)(u64)val)
|
||||
return 4;
|
||||
|
||||
ret = bpf_testmod_ops3_call_test_arena_nullable((u64 *)val);
|
||||
if (ret)
|
||||
return 5;
|
||||
if (*val != 43)
|
||||
return 6;
|
||||
|
||||
/* NULL survives the nullable kfunc and the trampoline as NULL */
|
||||
ret = bpf_testmod_ops3_call_test_arena_nullable(NULL);
|
||||
if (ret != 0xbee)
|
||||
return 7;
|
||||
|
||||
/* the arena pointer is stack-passed into the trampoline here */
|
||||
ret = bpf_testmod_ops3_call_test_arena_stack((u64 *)val);
|
||||
if (ret)
|
||||
return 8;
|
||||
if (*val != 44)
|
||||
return 9;
|
||||
|
||||
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
|
||||
#endif
|
||||
return 0;
|
||||
}
|
||||
25
tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c
Normal file
25
tools/testing/selftests/bpf/progs/struct_ops_arena_attach.c
Normal file
@@ -0,0 +1,25 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
#include <linux/bpf.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include <bpf/bpf_tracing.h>
|
||||
|
||||
SEC("fentry")
|
||||
int BPF_PROG(fentry_test_arena, unsigned long long *st_ops_ctx)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("fexit")
|
||||
int BPF_PROG(fexit_test_arena, unsigned long long *st_ops_ctx, int ret)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC("freplace")
|
||||
int freplace_test_arena(unsigned long long *st_ops_ctx)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
20
tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
Normal file
20
tools/testing/selftests/bpf/progs/struct_ops_arena_fail.c
Normal file
@@ -0,0 +1,20 @@
|
||||
// SPDX-License-Identifier: GPL-2.0
|
||||
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
|
||||
|
||||
#include <vmlinux.h>
|
||||
#include <bpf/bpf_helpers.h>
|
||||
#include "../test_kmods/bpf_testmod.h"
|
||||
|
||||
char _license[] SEC("license") = "GPL";
|
||||
|
||||
/* No arena in the program: attaching to test_arena must be rejected. */
|
||||
SEC("struct_ops/test_arena")
|
||||
int test_arena_no_arena(unsigned long long *ctx)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
SEC(".struct_ops.link")
|
||||
struct bpf_testmod_ops3 testmod_arena_fail = {
|
||||
.test_arena = (void *)test_arena_no_arena,
|
||||
};
|
||||
@@ -237,6 +237,44 @@ __bpf_kfunc void bpf_kfunc_common_test(void)
|
||||
{
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_arg_test(u64 *val__arena)
|
||||
{
|
||||
u64 old;
|
||||
|
||||
old = *val__arena;
|
||||
*val__arena = old + 1;
|
||||
return old;
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_cap_test(u64 *val__arena)
|
||||
{
|
||||
return (u64)val__arena;
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_cap_nullable_test(u64 *val__arena__nullable)
|
||||
{
|
||||
return (u64)val__arena__nullable;
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_args5_test(u64 *a__arena, u64 *b__arena,
|
||||
u64 *c__arena, u64 *d__arena,
|
||||
u64 *e__arena__nullable)
|
||||
{
|
||||
return *a__arena + *b__arena + *c__arena + *d__arena +
|
||||
(e__arena__nullable ? *e__arena__nullable : 0);
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_stack_arg_test(u64 a, u64 b, u64 c, u64 d, u64 e,
|
||||
u64 *f__arena)
|
||||
{
|
||||
return a + b + c + d + e + *f__arena;
|
||||
}
|
||||
|
||||
__bpf_kfunc u64 bpf_kfunc_arena_mixed_test(u64 *a__arena, u64 *b__arena__nullable)
|
||||
{
|
||||
return *a__arena + (b__arena__nullable ? *b__arena__nullable : 0);
|
||||
}
|
||||
|
||||
__bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr,
|
||||
struct bpf_dynptr *ptr__nullable)
|
||||
{
|
||||
@@ -347,9 +385,29 @@ static int bpf_testmod_test_4(void)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int bpf_testmod_ops3__test_arena(u64 *ptr__arena)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int bpf_testmod_ops3__test_arena_stack(u64 a, u64 b, u64 c, u64 d,
|
||||
u64 e, u64 f, u64 g, u64 h,
|
||||
u64 *ptr__arena)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
static struct bpf_testmod_ops3 __bpf_testmod_ops3 = {
|
||||
.test_1 = bpf_testmod_test_3,
|
||||
.test_2 = bpf_testmod_test_4,
|
||||
.test_arena = bpf_testmod_ops3__test_arena,
|
||||
.test_arena_nullable = bpf_testmod_ops3__test_arena_nullable,
|
||||
.test_arena_stack = bpf_testmod_ops3__test_arena_stack,
|
||||
};
|
||||
|
||||
static void bpf_testmod_test_struct_ops3(void)
|
||||
@@ -368,6 +426,21 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void)
|
||||
st_ops3->test_2();
|
||||
}
|
||||
|
||||
__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena)
|
||||
{
|
||||
return st_ops3->test_arena(ptr__arena);
|
||||
}
|
||||
|
||||
__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nullable)
|
||||
{
|
||||
return st_ops3->test_arena_nullable(ptr__arena__nullable);
|
||||
}
|
||||
|
||||
__bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena)
|
||||
{
|
||||
return st_ops3->test_arena_stack(1, 2, 3, 4, 5, 6, 7, 8, ptr__arena);
|
||||
}
|
||||
|
||||
struct bpf_testmod_btf_type_tag_1 {
|
||||
int a;
|
||||
};
|
||||
@@ -755,6 +828,12 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL)
|
||||
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY)
|
||||
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_value)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_common_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_arg_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_nullable_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_args5_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_stack_arg_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_arena_mixed_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_nested_acquire_nonzero_offset_test, KF_ACQUIRE)
|
||||
@@ -770,6 +849,9 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable)
|
||||
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack)
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test);
|
||||
BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test);
|
||||
BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
|
||||
|
||||
@@ -106,6 +106,12 @@ struct bpf_testmod_ops2 {
|
||||
struct bpf_testmod_ops3 {
|
||||
int (*test_1)(void);
|
||||
int (*test_2)(void);
|
||||
/* Used to test arena pointer arguments. */
|
||||
int (*test_arena)(u64 *ptr);
|
||||
int (*test_arena_nullable)(u64 *ptr);
|
||||
/* enough leading args to force @ptr onto the stack on x86 and arm64 */
|
||||
int (*test_arena_stack)(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f,
|
||||
u64 g, u64 h, u64 *ptr);
|
||||
};
|
||||
|
||||
struct st_ops_args {
|
||||
|
||||
@@ -98,6 +98,15 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym;
|
||||
void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym;
|
||||
|
||||
void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym;
|
||||
__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym;
|
||||
__u64 bpf_kfunc_arena_cap_test(__u64 *val__arena) __ksym;
|
||||
__u64 bpf_kfunc_arena_cap_nullable_test(__u64 *val__arena__nullable) __ksym;
|
||||
__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena,
|
||||
__u64 *c__arena, __u64 *d__arena,
|
||||
__u64 *e__arena__nullable) __ksym;
|
||||
__u64 bpf_kfunc_arena_stack_arg_test(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
|
||||
__u64 *f__arena) __ksym;
|
||||
__u64 bpf_kfunc_arena_mixed_test(__u64 *a__arena, __u64 *b__arena__nullable) __ksym;
|
||||
int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) __ksym;
|
||||
int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
|
||||
int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
|
||||
@@ -111,6 +120,9 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;
|
||||
#endif
|
||||
|
||||
void bpf_testmod_test_mod_kfunc(int i) __ksym;
|
||||
int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym;
|
||||
int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym;
|
||||
int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym;
|
||||
|
||||
__u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
|
||||
__u32 c, __u64 d) __ksym;
|
||||
|
||||
Reference in New Issue
Block a user