Merge branch 'add-arena-argument-support-to-kfuncs-and-struct_ops'

Kumar Kartikeya Dwivedi says:

====================
Add arena argument support to kfuncs and struct_ops

This is a continuation of patches in [0], with mostly minor changes and
reordering. The motivation is covered in that link. A major change is
moving to two tags (__arena and __arena__nullable) and moving the changes
to JIT to emit more optimized sequences.

Please see commit logs for details.

  [0]: https://lore.kernel.org/bpf/20260713024414.3759854-1-tj@kernel.org

Changelog:
----------
v4 -> v5
v4: https://lore.kernel.org/bpf/20260805210427.3218326-1-memxor@gmail.com

 * Remove the redundant patch-8 capability comment and duplicate
   nullable kfunc test coverage. (Eduard)
 * Introduce the final bpf_tramp_arena_base() interface directly with
   function-model argument flags, avoiding temporary slot bitmaps and
   arena_nullable state; simplify struct_ops pointer validation. (Eduard)
 * Simplify kfunc arena nullability classification by using the common
   nullable path for both arena suffixes while leaving the function model
   to distinguish JIT NULL preservation. (Amery)
 * Keep bpf_prog_has_arena_ctx_arg() in bpf_verifier.h from its
   introduction so trampoline and verifier users share one inline
   definition, avoiding BPF_JIT/BPF_SYSCALL link dependencies.
   (Eduard, BPF CI Bot)
 * Reject both tracing and extension attachments to struct_ops programs
   with arena context arguments, and add fentry, fexit, and freplace
   rejection tests. (Eduard, Sashiko)

v3 -> v4
v3: https://lore.kernel.org/bpf/20260803125115.2264733-1-memxor@gmail.com

 * Rename __arena_nullable to __arena__nullable and prioritize the
   composite suffix over __nullable during argument classification.
   (Sashiko, Eduard)
 * Resolve instructions before collecting subprograms and kfuncs so kfunc
   prototype validation can use associated arena state.
 * Move the arena kfunc and JIT-sequence test entry points into
   prog_tests/verifier.c. (Eduard)
 * Match the generated L0 target and call in nullable JIT assertions.
   (Eduard)
 * Route arena kfunc validation through the common argument-checking path.
   (Amery)
 * Reuse btf_func_model argument flags for struct_ops arena arguments
   instead of maintaining separate trampoline slot metadata. (Eduard)
 * Check the generic-trampoline arena argument invariant at link time and
   warn once on violations. (Eduard)
 * Reject tracing attachments to struct_ops programs with arena context
   arguments whose indirect trampolines convert the pointers. (Sashiko)

v2 -> v3
v2: https://lore.kernel.org/bpf/20260726013105.3689867-1-memxor@gmail.com

 * Rebase onto current bpf-next to resolve conflicts.

v1 -> v2
v1: https://lore.kernel.org/bpf/20260715220052.1590783-1-memxor@gmail.com

 * Fix documentation to only mention x86 for now. (Sashiko)
 * Move arg bitmap from insn_aux_data to kfunc descriptor. (Eduard)
====================

Link: https://patch.msgid.link/20260808003938.3486067-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
This commit is contained in:
Eduard Zingerman
2026-08-08 03:03:27 -07:00
21 changed files with 1123 additions and 59 deletions

View File

@@ -278,6 +278,43 @@ An example is given below::
...
}
2.3.8 __arena and __arena__nullable Annotations
-----------------------------------------------
Both annotations indicate that the pointer argument points into the
calling program's arena. The JIT rebases the value at the call site so
the kfunc receives a directly dereferenceable kernel address, subject to
the access rules described in :ref:`BPF_kfunc_arena_access` (at most
``GUARD_SZ / 2``, 32 KiB, past the pointer in a single unchecked access).
With ``__arena`` the rebase is unconditional and the argument is never
NULL: a value whose lower 32 bits are zero arrives as the arena base
address (arena offset 0). The kfunc must not check the argument for NULL.
With ``__arena__nullable`` such a value arrives as NULL instead and the
kfunc must check before dereferencing.
An example is given below::
__bpf_kfunc int bpf_process_item(struct item *item__arena)
{
...
}
Calling such a kfunc requires the program to use an arena map and a JIT with
arena argument support (currently x86-64); verification fails otherwise. The
program can pass any value without compromising the kernel. A value that does
not point into the arena is a program bug.
The suffixes have the same meaning on the arguments of struct_ops stub
functions, with the conversion running in the opposite direction. The
kernel caller passes the kernel arena address and the trampoline converts
it while saving the arguments, so the callback receives an arena pointer
it can dereference directly. With ``__arena`` the kernel caller must not
pass NULL. With ``__arena__nullable`` a NULL kernel pointer arrives as NULL.
However, there is no obligation to prove to the verifier that such a pointer is
non-NULL before use, in-line with existing semantics of arena pointers used in
a program (or obtained from any other source).
.. _BPF_kfunc_nodef:
2.4 Using an existing kernel function
@@ -522,6 +559,8 @@ In order to accommodate such requirements, the verifier will enforce strict
PTR_TO_BTF_ID type matching if two types have the exact same name, with one
being suffixed with ``___init``.
.. _BPF_kfunc_arena_access:
2.8 Accessing arena memory through kfunc arguments
--------------------------------------------------

View File

@@ -1678,6 +1678,50 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
return 0;
}
/*
* Rebase the __arena args of a kfunc call to arena kernel addresses,
* rN = kern_vm_start + (u32)rN, with R12 holding kern_vm_start. A nullable
* arg preserves NULL by skipping the add, tested on the truncated value as
* arena NULL is offset 0. Return the number of emitted bytes.
*/
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
const struct bpf_insn *insn, u8 **pprog)
{
const struct btf_func_model *fm;
u8 *prog = *pprog;
u8 *start = prog;
int i;
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
return -EINVAL;
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
u8 flags = fm->arg_flags[i];
u32 reg = BPF_REG_1 + i;
if (!(flags & BTF_FMODEL_ARENA_ARG))
continue;
if (WARN_ON_ONCE(!bpf_prog->aux->arena))
return -EINVAL;
/* mov eN, eN: truncate and clear the upper 32 bits */
emit_mov_reg(&prog, false, reg, reg);
if (flags & BTF_FMODEL_NULLABLE_ARG) {
/* test eN, eN; jz over the 3-byte add */
maybe_emit_mod(&prog, reg, reg, false);
EMIT2(0x85, add_2reg(0xC0, reg, reg));
EMIT2(X86_JE, 3);
}
/* add rN, r12 */
maybe_emit_mod(&prog, reg, X86_REG_R12, true);
EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
}
*pprog = prog;
return prog - start;
}
static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding)
{
@@ -2588,6 +2632,12 @@ st: insn_off = insn->off;
}
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
if (err < 0)
return err;
ip += err;
}
if (priv_frame_ptr) {
push_r9(&prog);
ip += 2;
@@ -2998,11 +3048,39 @@ static int get_nr_used_regs(const struct btf_func_model *m)
return nr_used_regs;
}
/*
* Convert an arena kernel address into the arena pointer form on its way
* into the BPF ctx, rax = (u32)(src - kern_vm_start). A nullable arg
* preserves NULL, tested on the full 64-bit kernel pointer. The 32-bit
* subtraction both truncates and clears the upper half, so the stored
* value satisfies the JIT invariant for arena pointer registers.
*/
static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
{
u8 *prog = *pprog;
if (nullable) {
if (src_reg != BPF_REG_0)
emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
/* test rax, rax; jz over the 5-byte sub */
EMIT3(0x48, 0x85, 0xC0);
EMIT2(X86_JE, 5);
} else if (src_reg != BPF_REG_0) {
emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
}
/* sub eax, base_lo */
EMIT1_off32(0x2D, base_lo);
*pprog = prog;
}
static void save_args(const struct btf_func_model *m, u8 **prog,
int stack_size, bool for_call_origin, u32 flags)
int stack_size, bool for_call_origin, u32 flags,
u64 arena_base)
{
int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0;
bool use_jmp = bpf_trampoline_use_jmp(flags);
int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24;
int i, j;
/* Store function arguments to stack.
@@ -3011,6 +3089,9 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
* mov QWORD PTR [rbp-0x8],rsi
*/
for (i = 0; i < min_t(int, m->nr_args, MAX_BPF_FUNC_ARGS); i++) {
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
arg_regs = (m->arg_size[i] + 7) / 8;
/* According to the research of Yonghong, struct members
@@ -3034,16 +3115,19 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
/* copy function arguments from origin stack frame
* into current stack frame.
*
* The starting address of the arguments on-stack
* is:
* rbp + 8(push rbp) +
* 8(return addr of origin call) +
* 8(return addr of the caller)
* which means: rbp + 24
* The arguments on-stack start above the saved rbp
* and the return addresses: two return addresses
* (origin call and caller) when the trampoline is
* entered through the fentry call, so rbp + 24, and
* a single one when it is entered with a jmp or
* called indirectly, so rbp + 16.
*/
for (j = 0; j < arg_regs; j++) {
emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
nr_stack_slots * 8 + 16 + (!use_jmp) * 8);
nr_stack_slots * 8 + stack_args_off);
if (arena_arg)
emit_arena_arg_conv(prog, BPF_REG_0, nullable,
(u32)arena_base);
emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
-stack_size);
@@ -3064,9 +3148,13 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
/* copy the arguments from regs into stack */
for (j = 0; j < arg_regs; j++) {
emit_stx(prog, BPF_DW, BPF_REG_FP,
nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
-stack_size);
u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
if (arena_arg) {
emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
src = BPF_REG_0;
}
emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
stack_size -= 8;
nr_regs++;
}
@@ -3362,6 +3450,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
void *orig_call = func_addr;
int cookie_off, cookie_cnt;
u8 **branches = NULL;
u64 arena_base;
u64 func_meta;
u8 *prog;
bool save_ret;
@@ -3374,6 +3463,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) &&
(flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET)));
arena_base = bpf_tramp_arena_base(m, tnodes, flags);
for (i = 0; i < m->nr_args; i++)
nr_regs += (m->arg_size[i] + 7) / 8 - 1;
@@ -3508,7 +3599,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
}
save_args(m, &prog, regs_off, false, flags);
save_args(m, &prog, regs_off, false, flags, arena_base);
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* arg1: mov rdi, im */
@@ -3550,7 +3641,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
if (flags & BPF_TRAMP_F_CALL_ORIG) {
restore_regs(m, &prog, regs_off);
save_args(m, &prog, arg_stack_off, true, flags);
save_args(m, &prog, arg_stack_off, true, flags, 0);
if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
/* Before calling the original function, load the
@@ -4051,6 +4142,11 @@ bool bpf_jit_supports_stack_args(void)
return true;
}
bool bpf_jit_supports_arena_args(void)
{
return true;
}
void *bpf_arch_text_copy(void *dst, void *src, size_t len)
{
if (text_poke_copy(dst, src, len) == NULL)

View File

@@ -1195,6 +1195,12 @@ struct bpf_prog_offload {
/* The argument is signed. */
#define BTF_FMODEL_SIGNED_ARG BIT(1)
/* The argument is an arena pointer. */
#define BTF_FMODEL_ARENA_ARG BIT(2)
/* The argument is nullable. */
#define BTF_FMODEL_NULLABLE_ARG BIT(3)
struct btf_func_model {
u8 ret_size;
u8 ret_flags;
@@ -1268,6 +1274,15 @@ struct bpf_tramp_nodes {
int nr_nodes;
};
/*
* The arena base against which a struct_ops trampoline converts the
* arguments marked with BTF_FMODEL_ARENA_ARG while saving them into the BPF
* ctx, ctx[arg] = (u32)(kaddr - kern_vm_start). Zero when the trampoline
* converts nothing.
*/
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
struct bpf_tramp_nodes *tnodes, u32 flags);
struct bpf_tramp_run_ctx;
/* Different use cases for BPF trampoline:

View File

@@ -1172,8 +1172,8 @@ static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id)
*btf_id = key & 0x7FFFFFFF;
}
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
int bpf_check_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr, bpfptr_t uattr);
@@ -1297,6 +1297,16 @@ static inline u32 type_flag(u32 type)
return type & ~BPF_BASE_TYPE_MASK;
}
static inline bool bpf_prog_has_arena_ctx_arg(const struct bpf_prog *prog)
{
int i;
for (i = 0; i < prog->aux->ctx_arg_info_size; i++)
if (base_type(prog->aux->ctx_arg_info[i].reg_type) == PTR_TO_ARENA)
return true;
return false;
}
static inline enum bpf_prog_type resolve_prog_type(const struct bpf_prog *prog)
{
return (prog->type == BPF_PROG_TYPE_EXT && prog->aux->saved_dst_prog_type) ?

View File

@@ -1214,6 +1214,7 @@ bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);

View File

@@ -147,6 +147,8 @@ void bpf_struct_ops_image_free(void *image)
#define MAYBE_NULL_SUFFIX "__nullable"
#define REFCOUNTED_SUFFIX "__ref"
#define ARENA_SUFFIX "__arena"
#define ARENA_MAYBE_NULL_SUFFIX "__arena__nullable"
/* Prepare argument info for every nullable argument of a member of a
* struct_ops type.
@@ -159,7 +161,7 @@ void bpf_struct_ops_image_free(void *image)
* to provide an array of struct bpf_ctx_arg_aux, which in turn provides
* the information that used by the verifier to check the arguments of the
* BPF struct_ops program assigned to the member. Here, we only care about
* the arguments that are marked as __nullable.
* the arguments that are marked as __nullable, __ref or __arena.
*
* The array of struct bpf_ctx_arg_aux is eventually assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs and passed to the
@@ -172,10 +174,12 @@ static int prepare_arg_info(struct btf *btf,
const char *st_ops_name,
const char *member_name,
const struct btf_type *func_proto, void *stub_func_addr,
struct btf_func_model *model,
struct bpf_struct_ops_arg_info *arg_info)
{
const struct btf_type *stub_func_proto, *pointed_type;
bool is_nullable = false, is_refcounted = false;
bool is_nullable = false, is_refcounted = false, is_arena = false;
bool is_arena_nullable = false;
const struct btf_param *stub_args, *args;
struct bpf_ctx_arg_aux *info, *info_buf;
u32 nargs, arg_no, info_cnt = 0;
@@ -225,27 +229,39 @@ static int prepare_arg_info(struct btf *btf,
/* Prepare info for every nullable argument */
info = info_buf;
for (arg_no = 0; arg_no < nargs; arg_no++) {
/* Skip arguments that is not suffixed with
* "__nullable or __ref".
bool ptr_to_arena, ptr_to_struct;
/*
* Skip arguments that are not suffixed with "__arena__nullable",
* "__arena", "__nullable", or "__ref".
*/
is_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
MAYBE_NULL_SUFFIX);
is_arena_nullable = btf_param_match_suffix(btf, &stub_args[arg_no],
ARENA_MAYBE_NULL_SUFFIX);
is_arena = btf_param_match_suffix(btf, &stub_args[arg_no], ARENA_SUFFIX);
is_nullable = !is_arena_nullable &&
btf_param_match_suffix(btf, &stub_args[arg_no], MAYBE_NULL_SUFFIX);
is_refcounted = btf_param_match_suffix(btf, &stub_args[arg_no],
REFCOUNTED_SUFFIX);
if (is_nullable)
if (is_arena_nullable)
suffix = ARENA_MAYBE_NULL_SUFFIX;
else if (is_arena)
suffix = ARENA_SUFFIX;
else if (is_nullable)
suffix = MAYBE_NULL_SUFFIX;
else if (is_refcounted)
suffix = REFCOUNTED_SUFFIX;
else
continue;
/* Should be a pointer to struct */
pointed_type = btf_type_resolve_ptr(btf,
args[arg_no].type,
&arg_btf_id);
if (!pointed_type ||
!btf_type_is_struct(pointed_type)) {
/*
* Should be a pointer to struct, or any pointer for __arena or
* __arena__nullable.
*/
pointed_type = btf_type_resolve_ptr(btf, args[arg_no].type, &arg_btf_id);
ptr_to_arena = pointed_type && (is_arena || is_arena_nullable);
ptr_to_struct = pointed_type && btf_type_is_struct(pointed_type);
if (!ptr_to_arena && !ptr_to_struct) {
pr_warn("stub function %s has %s tagging to an unsupported type\n",
stub_fname, suffix);
goto err_out;
@@ -268,7 +284,18 @@ static int prepare_arg_info(struct btf *btf,
info->btf_id = arg_btf_id;
info->btf = btf;
info->offset = offset;
if (is_nullable) {
if (is_arena || is_arena_nullable) {
/*
* Both types get PTR_TO_ARENA. In verifier state,
* PTR_TO_ARENA encompasses potential NULL values, but
* we do not force the program to check it, or maintain
* precision around it, since it has no safety implication.
*/
info->reg_type = PTR_TO_ARENA;
model->arg_flags[arg_no] |= BTF_FMODEL_ARENA_ARG;
if (is_arena_nullable)
model->arg_flags[arg_no] |= BTF_FMODEL_NULLABLE_ARG;
} else if (is_nullable) {
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID | PTR_MAYBE_NULL;
} else if (is_refcounted) {
info->reg_type = PTR_TRUSTED | PTR_TO_BTF_ID;
@@ -460,6 +487,7 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
err = prepare_arg_info(btf, st_ops->name, mname,
func_proto, stub_func_addr,
&st_ops->func_models[i],
arg_info + i);
if (err)
goto errout;

View File

@@ -6963,15 +6963,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
return false;
}
/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */
/*
* Check for PTR_TO_RDONLY_BUF_OR_NULL, PTR_TO_RDWR_BUF_OR_NULL or
* PTR_TO_ARENA (both nullable and non-nullable cases).
*/
for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
u32 type, flag;
type = base_type(ctx_arg_info->reg_type);
flag = type_flag(ctx_arg_info->reg_type);
if (ctx_arg_info->offset == off && type == PTR_TO_BUF &&
(flag & PTR_MAYBE_NULL)) {
if (ctx_arg_info->offset == off &&
(type == PTR_TO_ARENA ||
(type == PTR_TO_BUF && (flag & PTR_MAYBE_NULL)))) {
info->reg_type = ctx_arg_info->reg_type;
return true;
}
@@ -7539,6 +7543,22 @@ static u8 __get_type_fmodel_flags(const struct btf_type *t)
return flags;
}
static u8 __get_arg_fmodel_flags(const struct btf *btf,
const struct btf_param *arg,
const struct btf_type *t)
{
u8 flags = __get_type_fmodel_flags(t);
if (btf_param_match_suffix(btf, arg, "__arena__nullable"))
flags |= BTF_FMODEL_ARENA_ARG | BTF_FMODEL_NULLABLE_ARG;
else if (btf_param_match_suffix(btf, arg, "__arena"))
flags |= BTF_FMODEL_ARENA_ARG;
else if (btf_param_match_suffix(btf, arg, "__nullable"))
flags |= BTF_FMODEL_NULLABLE_ARG;
return flags;
}
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *func,
@@ -7604,7 +7624,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
return -EINVAL;
}
m->arg_size[i] = ret;
m->arg_flags[i] = __get_type_fmodel_flags(t);
m->arg_flags[i] = __get_arg_fmodel_flags(btf, &args[i], t);
}
m->nr_args = nargs;
return 0;

View File

@@ -28,9 +28,9 @@ static int check_abnormal_return(struct bpf_verifier_env *env)
#define MIN_BPF_FUNCINFO_SIZE 8
#define MAX_FUNCINFO_REC_SIZE 252
static int check_btf_func_early(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
static int prepare_btf_func(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
{
u32 krec_size = sizeof(struct bpf_func_info);
const struct btf_type *type, *func_proto;
@@ -407,9 +407,9 @@ static int check_core_relo(struct bpf_verifier_env *env,
return err;
}
int bpf_check_btf_info_early(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
{
struct btf *btf;
int err;
@@ -429,7 +429,7 @@ int bpf_check_btf_info_early(struct bpf_verifier_env *env,
}
env->prog->aux->btf = btf;
err = check_btf_func_early(env, attr, uattr);
err = prepare_btf_func(env, attr, uattr);
if (err)
return err;
return 0;

View File

@@ -3308,6 +3308,11 @@ bool __weak bpf_jit_supports_stack_args(void)
return false;
}
bool __weak bpf_jit_supports_arena_args(void)
{
return false;
}
bool __weak bpf_jit_supports_far_kfunc_call(void)
{
return false;

View File

@@ -529,6 +529,36 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a
return tnodes;
}
/*
* The arena base against which save_args() converts the arguments marked
* with BTF_FMODEL_ARENA_ARG. Only the struct_ops indirect trampoline
* converts: it dispatches to a single prog whose arena is known at
* generation time. Return 0 when there is nothing to convert.
*/
u64 bpf_tramp_arena_base(const struct btf_func_model *m,
struct bpf_tramp_nodes *tnodes, u32 flags)
{
const struct bpf_prog *prog;
int i;
if (!(flags & BPF_TRAMP_F_INDIRECT) ||
tnodes[BPF_TRAMP_FENTRY].nr_nodes != 1)
return 0;
for (i = 0; i < m->nr_args; i++)
if (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG)
break;
if (i == m->nr_args)
return 0;
/* Verification rejects an arena argument without an arena. */
prog = tnodes[BPF_TRAMP_FENTRY].nodes[0]->link->prog;
if (WARN_ON_ONCE(!prog->aux->arena))
return 0;
return bpf_arena_get_kern_vm_start(prog->aux->arena);
}
static void bpf_tramp_image_free(struct bpf_tramp_image *im)
{
bpf_image_ksym_del(&im->ksym);
@@ -920,6 +950,13 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node,
int cnt = 0, i;
kind = bpf_attach_type_to_tramp(node->link->prog);
/*
* Arena ctx args are converted only by struct_ops indirect
* trampolines. They must never be attached to a generic trampoline.
*/
if (WARN_ON_ONCE(bpf_prog_has_arena_ctx_arg(node->link->prog)))
return -ENOTSUPP;
if (tr->extension_prog)
/* cannot attach fentry/fexit if extension prog is attached.
* cannot overwrite extension prog either.

View File

@@ -2837,7 +2837,7 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
return 0;
}
static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
static int add_subprogs(struct bpf_verifier_env *env)
{
struct bpf_subprog_info *subprog = env->subprog_info;
int i, ret, insn_cnt = env->prog->len, ex_cb_insn;
@@ -2849,8 +2849,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
return ret;
for (i = 0; i < insn_cnt; i++, insn++) {
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn) &&
!bpf_pseudo_kfunc_call(insn))
if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
continue;
if (!env->bpf_capable) {
@@ -2858,11 +2857,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
return -EPERM;
}
if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn))
ret = add_subprog(env, i + insn->imm + 1);
else
ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
ret = add_subprog(env, i + insn->imm + 1);
if (ret < 0)
return ret;
}
@@ -2900,6 +2895,28 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env)
return 0;
}
static int add_kfuncs(struct bpf_verifier_env *env)
{
struct bpf_insn *insn = env->prog->insnsi;
int i, ret, insn_cnt = env->prog->len;
for (i = 0; i < insn_cnt; i++, insn++) {
if (!bpf_pseudo_kfunc_call(insn))
continue;
if (!env->bpf_capable) {
verbose(env, "loading/calling other bpf or kernel functions are allowed for CAP_BPF and CAP_SYS_ADMIN\n");
return -EPERM;
}
ret = bpf_add_kfunc_call(env, insn->imm, insn->off);
if (ret < 0)
return ret;
}
return 0;
}
static int check_subprogs(struct bpf_verifier_env *env)
{
int i, subprog_start, subprog_end, off, cur_subprog = 0;
@@ -10760,7 +10777,8 @@ static bool is_kfunc_arg_refcounted_kptr(const struct btf *btf, const struct btf
static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param *arg)
{
return btf_param_match_suffix(btf, arg, "__nullable");
return btf_param_match_suffix(btf, arg, "__nullable") ||
btf_param_match_suffix(btf, arg, "__arena");
}
static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg)
@@ -10778,6 +10796,12 @@ static bool is_kfunc_arg_irq_flag(const struct btf *btf, const struct btf_param
return btf_param_match_suffix(btf, arg, "__irq_flag");
}
static bool is_kfunc_arg_arena(const struct btf *btf, const struct btf_param *arg)
{
return btf_param_match_suffix(btf, arg, "__arena__nullable") ||
btf_param_match_suffix(btf, arg, "__arena");
}
static bool is_kfunc_arg_scalar_with_name(const struct btf *btf,
const struct btf_param *arg,
const char *name)
@@ -10998,6 +11022,7 @@ enum kfunc_ptr_arg_type {
KF_ARG_PTR_TO_IRQ_FLAG,
KF_ARG_PTR_TO_RES_SPIN_LOCK,
KF_ARG_PTR_TO_TASK_WORK,
KF_ARG_PTR_TO_ARENA,
};
enum special_kfunc_type {
@@ -11283,7 +11308,6 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
reg_arg_name(env, argno), btf_type_str(t));
return -EINVAL;
}
ref_t = btf_type_skip_modifiers(meta->btf, t->type, NULL);
ref_tname = btf_name_by_offset(meta->btf, ref_t->name_off);
@@ -11332,7 +11356,30 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
arg_type = KF_ARG_PTR_TO_RES_SPIN_LOCK;
else if (is_kfunc_arg_callback(env, meta->btf, &args[arg]))
arg_type = KF_ARG_PTR_TO_CALLBACK;
else if (arg + 1 < nargs &&
else if (is_kfunc_arg_arena(meta->btf, &args[arg])) {
if (!bpf_jit_supports_arena_args()) {
verbose(env, "JIT does not support kfunc %s() with arena pointer arguments\n",
meta->func_name);
return -ENOTSUPP;
}
if (!env->prog->aux->arena) {
verbose(env,
"%s arena pointer requires a program with an associated arena\n",
reg_arg_name(env, argno));
return -EINVAL;
}
if (reg_from_argno(argno) < 0) {
verbose(env, "%s arena pointer cannot be a stack argument\n",
reg_arg_name(env, argno));
return -EINVAL;
}
/*
* Both suffixes accept a constant zero. The function model determines
* whether the JIT rebases it to the arena base or preserves NULL.
* The common nullable path below records that verifier property.
*/
arg_type = KF_ARG_PTR_TO_ARENA;
} else if (arg + 1 < nargs &&
(is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
@@ -12007,7 +12054,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
t = btf_type_skip_modifiers(btf, args[i].type, NULL);
if (btf_type_is_ptr(t) && (bpf_register_is_null(reg) || type_may_be_null(reg->type)) &&
!is_kfunc_arg_nullable(meta->btf, &args[i])) {
!type_may_be_null(kf_arg_type)) {
verbose(env, "Possibly NULL pointer passed to trusted %s\n",
reg_arg_name(env, argno));
return -EACCES;
@@ -12060,6 +12107,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
case KF_ARG_PTR_TO_TASK_WORK:
case KF_ARG_PTR_TO_IRQ_FLAG:
case KF_ARG_PTR_TO_RES_SPIN_LOCK:
case KF_ARG_PTR_TO_ARENA:
break;
case KF_ARG_PTR_TO_DYNPTR:
arg_type = ARG_PTR_TO_DYNPTR;
@@ -12126,6 +12174,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
meta->ret_btf_id = ret;
}
break;
case KF_ARG_PTR_TO_ARENA:
if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) {
verbose(env, "%s is not a pointer to arena or scalar\n",
reg_arg_name(env, argno));
return -EINVAL;
}
break;
case KF_ARG_PTR_TO_ALLOC_BTF_ID:
if (reg->type == (PTR_TO_BTF_ID | MEM_ALLOC)) {
if (!is_bpf_obj_drop_kfunc(meta->func_id)) {
@@ -18630,6 +18685,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
{
const struct btf_type *t, *func_proto;
const struct bpf_struct_ops_desc *st_ops_desc;
const struct bpf_struct_ops_arg_info *arg_info;
const struct bpf_struct_ops *st_ops;
const struct btf_member *member;
struct bpf_prog *prog = env->prog;
@@ -18708,10 +18764,23 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
return -EACCES;
}
for (i = 0; i < st_ops_desc->arg_info[member_idx].cnt; i++) {
if (st_ops_desc->arg_info[member_idx].info[i].refcounted) {
arg_info = &st_ops_desc->arg_info[member_idx];
for (i = 0; i < arg_info->cnt; i++) {
const struct bpf_ctx_arg_aux *info = &arg_info->info[i];
if (info->refcounted)
has_refcounted_arg = true;
break;
if (base_type(info->reg_type) == PTR_TO_ARENA) {
if (!bpf_jit_supports_arena_args()) {
verbose(env, "JIT does not support arena arguments\n");
return -ENOTSUPP;
}
if (!prog->aux->arena) {
verbose(env,
"arena argument of %s requires a program with an associated arena\n",
mname);
return -EINVAL;
}
}
}
@@ -18732,8 +18801,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env)
prog->aux->attach_func_name = mname;
env->ops = st_ops->verifier_ops;
return bpf_prog_ctx_arg_info_init(prog, st_ops_desc->arg_info[member_idx].info,
st_ops_desc->arg_info[member_idx].cnt);
return bpf_prog_ctx_arg_info_init(prog, arg_info->info, arg_info->cnt);
}
#define SECURITY_PREFIX "security_"
@@ -19000,6 +19068,16 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
bpf_log(log, "Subprog %s doesn't exist\n", tname);
return -EINVAL;
}
/*
* A struct_ops indirect trampoline converts arena arguments
* before invoking its program. A tracing or extension program
* attached to the main program would see the converted offset as a
* regular BTF pointer.
*/
if (subprog == 0 && bpf_prog_has_arena_ctx_arg(tgt_prog)) {
bpf_log(log, "Cannot attach to a target with arena context arguments\n");
return -EOPNOTSUPP;
}
if (aux->func && aux->func[subprog]->aux->exception_cb) {
bpf_log(log,
"%s programs cannot attach to exception callback\n",
@@ -20135,11 +20213,13 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
INIT_LIST_HEAD(&env->explored_states[i]);
INIT_LIST_HEAD(&env->free_list);
ret = bpf_check_btf_info_early(env, attr, uattr);
/* Prepare BTF and func_info needed to discover all subprograms. */
ret = bpf_prepare_btf_info(env, attr, uattr);
if (ret < 0)
goto skip_full_check;
ret = add_subprog_and_kfunc(env);
/* Discover all subprograms before validating their layout and BTF. */
ret = add_subprogs(env);
if (ret < 0)
goto skip_full_check;
@@ -20147,14 +20227,21 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
/* Validate BTF against the complete subprogram layout and apply CO-RE. */
ret = bpf_check_btf_info(env, attr, uattr);
if (ret < 0)
goto skip_full_check;
/* Validate instructions and resolve the program's referenced resources. */
ret = check_and_resolve_insns(env);
if (ret < 0)
goto skip_full_check;
/* Build kfunc prototypes after resolving program resources. */
ret = add_kfuncs(env);
if (ret < 0)
goto skip_full_check;
if (bpf_prog_is_offloaded(env->prog->aux)) {
ret = bpf_prog_offload_verifier_prep(env->prog);
if (ret)

View File

@@ -0,0 +1,128 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <test_progs.h>
#include "struct_ops_arena.skel.h"
#include "struct_ops_arena_attach.skel.h"
#include "struct_ops_arena_fail.skel.h"
#if defined(__x86_64__)
/*
* Attach callbacks with __arena and __arena__nullable arguments and drive
* them through the bpf_testmod_ops3_call_test_arena*() kfuncs.
*/
static void arena_arg(void)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
struct struct_ops_arena *skel;
struct bpf_link *link = NULL;
int err;
skel = struct_ops_arena__open_and_load();
if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load"))
return;
link = bpf_map__attach_struct_ops(skel->maps.testmod_arena);
if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
goto out;
err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.trigger),
&topts);
ASSERT_OK(err, "test_run");
ASSERT_EQ(topts.retval, 0, "trigger_retval");
out:
bpf_link__destroy(link);
struct_ops_arena__destroy(skel);
}
/*
* A program with no arena cannot attach to a member with an __arena
* argument.
*/
static void arena_arg_fail(void)
{
struct struct_ops_arena_fail *skel;
skel = struct_ops_arena_fail__open_and_load();
if (ASSERT_ERR_PTR(skel, "struct_ops_arena_fail__open_and_load"))
return;
struct_ops_arena_fail__destroy(skel);
}
static void arena_arg_attach_one(int target_fd, const char *prog_name)
{
struct struct_ops_arena_attach *skel;
struct bpf_program *prog, *pos;
char log_buf[64 * 1024];
int err;
skel = struct_ops_arena_attach__open();
if (!ASSERT_OK_PTR(skel, "struct_ops_arena_attach__open"))
return;
prog = bpf_object__find_program_by_name(skel->obj, prog_name);
if (!ASSERT_OK_PTR(prog, prog_name))
goto out;
bpf_object__for_each_program(pos, skel->obj)
bpf_program__set_autoload(pos, pos == prog);
err = bpf_program__set_attach_target(prog, target_fd, "test_arena_cb");
if (!ASSERT_OK(err, "set_attach_target"))
goto out;
log_buf[0] = '\0';
bpf_program__set_log_buf(prog, log_buf, sizeof(log_buf));
err = struct_ops_arena_attach__load(skel);
ASSERT_EQ(err, -EOPNOTSUPP, prog_name);
ASSERT_HAS_SUBSTR(log_buf, "Cannot attach to a target with arena context arguments",
"verifier_log");
out:
struct_ops_arena_attach__destroy(skel);
}
static void arena_arg_attach(void)
{
struct struct_ops_arena *skel;
int target_fd;
skel = struct_ops_arena__open_and_load();
if (!ASSERT_OK_PTR(skel, "struct_ops_arena__open_and_load"))
return;
target_fd = bpf_program__fd(skel->progs.test_arena_cb);
arena_arg_attach_one(target_fd, "fentry_test_arena");
arena_arg_attach_one(target_fd, "fexit_test_arena");
arena_arg_attach_one(target_fd, "freplace_test_arena");
struct_ops_arena__destroy(skel);
}
#endif
/*
* Serialized because it attaches the singleton bpf_testmod_ops3, which
* test_struct_ops_private_stack also attaches; registering it twice fails
* with -EEXIST.
*/
void serial_test_struct_ops_arena(void)
{
/*
* Arena struct_ops arguments need JIT support, currently x86-64 only.
* Elsewhere verification fails with "JIT does not support arena
* arguments", so the programs cannot even load.
*/
#if defined(__x86_64__)
if (test__start_subtest("arena_arg"))
arena_arg();
if (test__start_subtest("arena_arg_fail"))
arena_arg_fail();
if (test__start_subtest("arena_arg_attach"))
arena_arg_attach();
#else
test__skip();
#endif
}

View File

@@ -2,6 +2,8 @@
#include <test_progs.h>
#include "arena_kfunc.skel.h"
#include "arena_kfunc_jit.skel.h"
#include "cap_helpers.h"
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
@@ -162,6 +164,10 @@ static void run_tests_aux(const char *skel_name,
#define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL)
void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); }
void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); }
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }

View File

@@ -0,0 +1,234 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"
#include "bpf_experimental.h"
#include <bpf_arena_common.h>
#include "../test_kmods/bpf_testmod_kfunc.h"
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
/* page 0 hosts the arena global, page 1 is for allocations */
__uint(max_entries, 2);
} arena SEC(".maps");
/*
* Occupies page 0 so no allocation lands at arena offset 0, which the
* nullable tests below must be able to tell apart from NULL.
*/
u64 __arena arena_pad;
/* volatile to force the scalar reloads below */
volatile u64 stash;
SEC("syscall")
__arch_x86_64
__success __retval(0)
int arena_arg_forms(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
u64 ret;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
/* PTR_TO_ARENA argument */
*val = 41;
ret = bpf_kfunc_arena_arg_test((u64 *)val);
if (ret != 41 || *val != 42)
return 2;
/* the low 32 bits as a scalar */
stash = (u32)(u64)val;
ret = bpf_kfunc_arena_arg_test((u64 *)stash);
if (ret != 42 || *val != 43)
return 3;
/* the full user address as a scalar */
stash = (u64)val;
bpf_addr_space_cast(stash, 1, 0);
ret = bpf_kfunc_arena_arg_test((u64 *)stash);
if (ret != 43 || *val != 44)
return 4;
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}
/*
* Pin the rebase semantics using the capture kfuncs, which return the raw
* argument value: __arena rebases unconditionally, so zero low 32 bits
* arrive as the arena kernel base, while __arena__nullable turns them into
* NULL.
*/
SEC("syscall")
__arch_x86_64
__success __retval(0)
int arena_arg_rebase(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
u64 base, off;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
base = bpf_kfunc_arena_cap_test(NULL);
if (!base)
return 2;
/* only the low 32 bits contribute */
stash = 0xbadc0ffe00000000;
if (bpf_kfunc_arena_cap_test((u64 *)stash) != base)
return 3;
off = (u32)(u64)val;
if (bpf_kfunc_arena_cap_test((u64 *)val) != base + off)
return 4;
if (bpf_kfunc_arena_cap_nullable_test(NULL) != 0)
return 5;
stash = 0xbadc0ffe00000000;
if (bpf_kfunc_arena_cap_nullable_test((u64 *)stash) != 0)
return 6;
if (bpf_kfunc_arena_cap_nullable_test((u64 *)val) != base + off)
return 7;
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}
SEC("syscall")
__arch_x86_64
__success __retval(0)
int arena_args5(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
val[0] = 1;
val[1] = 2;
val[2] = 4;
val[3] = 8;
val[4] = 16;
if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
(u64 *)&val[2], (u64 *)&val[3],
(u64 *)&val[4]) != 31)
return 2;
if (bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
(u64 *)&val[2], (u64 *)&val[3], NULL) != 15)
return 3;
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}
SEC("syscall")
__arch_x86_64
__success __retval(0)
int arena_arg_mixed(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
val[0] = 7;
val[1] = 5;
if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], NULL) != 7)
return 2;
if (bpf_kfunc_arena_mixed_test((u64 *)&val[0], (u64 *)&val[1]) != 12)
return 3;
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}
/* kernel-side faults on unpopulated pages recover via the scratch page */
SEC("syscall")
__arch_x86_64
__success __retval(0)
int arena_arg_unpopulated(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
stash = (u64)val + PAGE_SIZE;
bpf_kfunc_arena_arg_test((u64 *)stash);
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}
SEC("syscall")
__arch_x86_64
__failure __msg("arena pointer requires a program with an associated arena")
int arena_arg_no_arena(void *ctx)
{
bpf_kfunc_arena_arg_test((u64 *)1);
return 0;
}
SEC("syscall")
__arch_x86_64
__failure __msg("is not a pointer to arena or scalar")
int arena_arg_bad_reg(void *ctx)
{
u64 buf = 0;
/* use the arena so the program passes the arena presence check */
bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_kfunc_arena_arg_test(&buf);
return 0;
}
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && \
defined(__BPF_FEATURE_STACK_ARGUMENT)
SEC("syscall")
__arch_x86_64
__failure __msg("arena pointer cannot be a stack argument")
int arena_arg_stack(void *ctx)
{
bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_kfunc_arena_stack_arg_test(1, 2, 3, 4, 5, (u64 *)1);
return 0;
}
#else
SEC("syscall")
__arch_x86_64
__description("arena_arg_stack: not supported, dummy test")
__success
int arena_arg_stack(void *ctx)
{
return 0;
}
#endif
char _license[] SEC("license") = "GPL";

View File

@@ -0,0 +1,98 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
/*
* Verify the JIT-emitted rebase sequences for __arena and __arena__nullable
* kfunc arguments. The capture kfuncs take the argument without
* dereferencing it, so these tests pin only the emitted code.
*/
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"
#include "bpf_experimental.h"
#include <bpf_arena_common.h>
#include "../test_kmods/bpf_testmod_kfunc.h"
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
__uint(max_entries, 1);
} arena SEC(".maps");
/* volatile to force the scalar reloads below */
volatile u64 stash;
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
SEC("syscall")
__arch_x86_64
__jited("...")
__jited(" movl %edi, %edi")
__jited(" addq %r12, %rdi")
__jited("...")
__jited(" callq {{.*}}")
__success
int arena_arg_jit_rebase(void *ctx)
{
stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_kfunc_arena_cap_test((u64 *)stash);
return 0;
}
SEC("syscall")
__arch_x86_64
__jited("...")
__jited(" movl %edi, %edi")
__jited(" testl %edi, %edi")
__jited(" je L0")
__jited(" addq %r12, %rdi")
__jited("L0: callq {{.*}}")
__success
int arena_arg_jit_nullable(void *ctx)
{
stash = (u64)bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_kfunc_arena_cap_nullable_test((u64 *)stash);
return 0;
}
SEC("syscall")
__arch_x86_64
__jited("...")
__jited(" movl %edi, %edi")
__jited(" addq %r12, %rdi")
__jited(" movl %esi, %esi")
__jited(" addq %r12, %rsi")
__jited(" movl %edx, %edx")
__jited(" addq %r12, %rdx")
__jited(" movl %ecx, %ecx")
__jited(" addq %r12, %rcx")
__jited(" movl %r8d, %r8d")
__jited(" testl %r8d, %r8d")
__jited(" je L0")
__jited(" addq %r12, %r8")
__jited("L0: callq {{.*}}")
__success
int arena_arg_jit_args5(void *ctx)
{
u64 __arena *val;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
val[0] = 1;
val[1] = 2;
val[2] = 4;
val[3] = 8;
val[4] = 16;
bpf_kfunc_arena_args5_test((u64 *)&val[0], (u64 *)&val[1],
(u64 *)&val[2], (u64 *)&val[3],
(u64 *)&val[4]);
return 0;
}
#endif /* __BPF_FEATURE_ADDR_SPACE_CAST */
char _license[] SEC("license") = "GPL";

View File

@@ -0,0 +1,115 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "bpf_experimental.h"
#include <bpf_arena_common.h>
#include "../test_kmods/bpf_testmod.h"
#include "../test_kmods/bpf_testmod_kfunc.h"
char _license[] SEC("license") = "GPL";
struct {
__uint(type, BPF_MAP_TYPE_ARENA);
__uint(map_flags, BPF_F_MMAPABLE);
/* page 0 hosts the arena globals, page 1 is for allocations */
__uint(max_entries, 2);
} arena SEC(".maps");
/* also associates the callbacks with the arena */
u64 __arena arena_touch;
/* raw value of the last __arena ctx argument, captured by test_arena_cb */
u64 __arena cb_ptr_val;
SEC("struct_ops/test_arena")
int test_arena_cb(unsigned long long *ctx)
{
u64 __arena *ptr = (u64 __arena *)ctx[0];
arena_touch++;
cb_ptr_val = ctx[0];
*ptr += 1;
return 0;
}
SEC("struct_ops/test_arena_nullable")
int test_arena_nullable_cb(unsigned long long *ctx)
{
u64 __arena *ptr = (u64 __arena *)ctx[0];
arena_touch++;
if (!ptr)
return 0xbee;
*ptr += 1;
return 0;
}
SEC("struct_ops/test_arena_stack")
int test_arena_stack_cb(unsigned long long *ctx)
{
u64 __arena *ptr = (u64 __arena *)ctx[8];
arena_touch++;
/* pin the slot layout: the leading args fill ctx[0]..ctx[7] */
if (ctx[0] != 1 || ctx[7] != 8)
return 0xbad;
*ptr += 1;
return 0;
}
SEC(".struct_ops.link")
struct bpf_testmod_ops3 testmod_arena = {
.test_arena = (void *)test_arena_cb,
.test_arena_nullable = (void *)test_arena_nullable_cb,
.test_arena_stack = (void *)test_arena_stack_cb,
};
SEC("syscall")
int trigger(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
u64 __arena *val;
int ret;
val = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
if (!val)
return 1;
*val = 41;
ret = bpf_testmod_ops3_call_test_arena((u64 *)val);
if (ret)
return 2;
if (*val != 42)
return 3;
/*
* The callback must have seen exactly (u32)(kaddr - kern_vm_start),
* which is the arena offset of val with the upper 32 bits clear.
*/
if (cb_ptr_val != (u32)(u64)val)
return 4;
ret = bpf_testmod_ops3_call_test_arena_nullable((u64 *)val);
if (ret)
return 5;
if (*val != 43)
return 6;
/* NULL survives the nullable kfunc and the trampoline as NULL */
ret = bpf_testmod_ops3_call_test_arena_nullable(NULL);
if (ret != 0xbee)
return 7;
/* the arena pointer is stack-passed into the trampoline here */
ret = bpf_testmod_ops3_call_test_arena_stack((u64 *)val);
if (ret)
return 8;
if (*val != 44)
return 9;
bpf_arena_free_pages(&arena, (void __arena *)val, 1);
#endif
return 0;
}

View File

@@ -0,0 +1,25 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
SEC("fentry")
int BPF_PROG(fentry_test_arena, unsigned long long *st_ops_ctx)
{
return 0;
}
SEC("fexit")
int BPF_PROG(fexit_test_arena, unsigned long long *st_ops_ctx, int ret)
{
return 0;
}
SEC("freplace")
int freplace_test_arena(unsigned long long *st_ops_ctx)
{
return 0;
}
char _license[] SEC("license") = "GPL";

View File

@@ -0,0 +1,20 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include "../test_kmods/bpf_testmod.h"
char _license[] SEC("license") = "GPL";
/* No arena in the program: attaching to test_arena must be rejected. */
SEC("struct_ops/test_arena")
int test_arena_no_arena(unsigned long long *ctx)
{
return 0;
}
SEC(".struct_ops.link")
struct bpf_testmod_ops3 testmod_arena_fail = {
.test_arena = (void *)test_arena_no_arena,
};

View File

@@ -237,6 +237,44 @@ __bpf_kfunc void bpf_kfunc_common_test(void)
{
}
__bpf_kfunc u64 bpf_kfunc_arena_arg_test(u64 *val__arena)
{
u64 old;
old = *val__arena;
*val__arena = old + 1;
return old;
}
__bpf_kfunc u64 bpf_kfunc_arena_cap_test(u64 *val__arena)
{
return (u64)val__arena;
}
__bpf_kfunc u64 bpf_kfunc_arena_cap_nullable_test(u64 *val__arena__nullable)
{
return (u64)val__arena__nullable;
}
__bpf_kfunc u64 bpf_kfunc_arena_args5_test(u64 *a__arena, u64 *b__arena,
u64 *c__arena, u64 *d__arena,
u64 *e__arena__nullable)
{
return *a__arena + *b__arena + *c__arena + *d__arena +
(e__arena__nullable ? *e__arena__nullable : 0);
}
__bpf_kfunc u64 bpf_kfunc_arena_stack_arg_test(u64 a, u64 b, u64 c, u64 d, u64 e,
u64 *f__arena)
{
return a + b + c + d + e + *f__arena;
}
__bpf_kfunc u64 bpf_kfunc_arena_mixed_test(u64 *a__arena, u64 *b__arena__nullable)
{
return *a__arena + (b__arena__nullable ? *b__arena__nullable : 0);
}
__bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr,
struct bpf_dynptr *ptr__nullable)
{
@@ -347,9 +385,29 @@ static int bpf_testmod_test_4(void)
return 0;
}
static int bpf_testmod_ops3__test_arena(u64 *ptr__arena)
{
return 0;
}
static int bpf_testmod_ops3__test_arena_nullable(u64 *ptr__arena__nullable)
{
return 0;
}
static int bpf_testmod_ops3__test_arena_stack(u64 a, u64 b, u64 c, u64 d,
u64 e, u64 f, u64 g, u64 h,
u64 *ptr__arena)
{
return 0;
}
static struct bpf_testmod_ops3 __bpf_testmod_ops3 = {
.test_1 = bpf_testmod_test_3,
.test_2 = bpf_testmod_test_4,
.test_arena = bpf_testmod_ops3__test_arena,
.test_arena_nullable = bpf_testmod_ops3__test_arena_nullable,
.test_arena_stack = bpf_testmod_ops3__test_arena_stack,
};
static void bpf_testmod_test_struct_ops3(void)
@@ -368,6 +426,21 @@ __bpf_kfunc void bpf_testmod_ops3_call_test_2(void)
st_ops3->test_2();
}
__bpf_kfunc int bpf_testmod_ops3_call_test_arena(u64 *ptr__arena)
{
return st_ops3->test_arena(ptr__arena);
}
__bpf_kfunc int bpf_testmod_ops3_call_test_arena_nullable(u64 *ptr__arena__nullable)
{
return st_ops3->test_arena_nullable(ptr__arena__nullable);
}
__bpf_kfunc int bpf_testmod_ops3_call_test_arena_stack(u64 *ptr__arena)
{
return st_ops3->test_arena_stack(1, 2, 3, 4, 5, 6, 7, 8, ptr__arena);
}
struct bpf_testmod_btf_type_tag_1 {
int a;
};
@@ -755,6 +828,12 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_value)
BTF_ID_FLAGS(func, bpf_kfunc_common_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_arg_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_cap_nullable_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_args5_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_stack_arg_test)
BTF_ID_FLAGS(func, bpf_kfunc_arena_mixed_test)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_mem_len_pass1)
BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test)
BTF_ID_FLAGS(func, bpf_kfunc_nested_acquire_nonzero_offset_test, KF_ACQUIRE)
@@ -770,6 +849,9 @@ BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_1)
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_2)
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena)
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_nullable)
BTF_ID_FLAGS(func, bpf_testmod_ops3_call_test_arena_stack)
BTF_ID_FLAGS(func, bpf_kfunc_get_default_trusted_ptr_test);
BTF_ID_FLAGS(func, bpf_kfunc_put_default_trusted_ptr_test);
BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)

View File

@@ -106,6 +106,12 @@ struct bpf_testmod_ops2 {
struct bpf_testmod_ops3 {
int (*test_1)(void);
int (*test_2)(void);
/* Used to test arena pointer arguments. */
int (*test_arena)(u64 *ptr);
int (*test_arena_nullable)(u64 *ptr);
/* enough leading args to force @ptr onto the stack on x86 and arm64 */
int (*test_arena_stack)(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f,
u64 g, u64 h, u64 *ptr);
};
struct st_ops_args {

View File

@@ -98,6 +98,15 @@ void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym;
void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) __ksym;
void bpf_kfunc_call_test_mem_len_pass1(void *mem, int len) __ksym;
__u64 bpf_kfunc_arena_arg_test(__u64 *val__arena) __ksym;
__u64 bpf_kfunc_arena_cap_test(__u64 *val__arena) __ksym;
__u64 bpf_kfunc_arena_cap_nullable_test(__u64 *val__arena__nullable) __ksym;
__u64 bpf_kfunc_arena_args5_test(__u64 *a__arena, __u64 *b__arena,
__u64 *c__arena, __u64 *d__arena,
__u64 *e__arena__nullable) __ksym;
__u64 bpf_kfunc_arena_stack_arg_test(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e,
__u64 *f__arena) __ksym;
__u64 bpf_kfunc_arena_mixed_test(__u64 *a__arena, __u64 *b__arena__nullable) __ksym;
int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) __ksym;
int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) __ksym;
@@ -111,6 +120,9 @@ u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused) __ksym;
#endif
void bpf_testmod_test_mod_kfunc(int i) __ksym;
int bpf_testmod_ops3_call_test_arena(__u64 *ptr__arena) __ksym;
int bpf_testmod_ops3_call_test_arena_nullable(__u64 *ptr__arena__nullable) __ksym;
int bpf_testmod_ops3_call_test_arena_stack(__u64 *ptr__arena) __ksym;
__u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
__u32 c, __u64 d) __ksym;