Kuniyuki Iwashima 38163af068
bpf: Introduce SK_BPF_BYPASS_PROT_MEM. ...
If a socket has sk->sk_bypass_prot_mem flagged, the socket opts out
of the global protocol memory accounting.
This is easily controlled by net.core.bypass_prot_mem sysctl, but it
lacks flexibility.
Let's support flagging (and clearing) sk->sk_bypass_prot_mem via
bpf_setsockopt() at the BPF_CGROUP_INET_SOCK_CREATE hook.
int val = 1;
bpf_setsockopt(ctx, SOL_SOCKET, SK_BPF_BYPASS_PROT_MEM,
&val, sizeof(val));
As with net.core.bypass_prot_mem, this is inherited to child sockets,
and BPF always takes precedence over sysctl at socket(2) and accept(2).
SK_BPF_BYPASS_PROT_MEM is only supported at BPF_CGROUP_INET_SOCK_CREATE
and not supported on other hooks for some reasons:
1. UDP charges memory under sk->sk_receive_queue.lock instead
of lock_sock()
2. Modifying the flag after skb is charged to sk requires such
adjustment during bpf_setsockopt() and complicates the logic
unnecessarily
We can support other hooks later if a real use case justifies that.
Most changes are inline and hard to trace, but a microbenchmark on
__sk_mem_raise_allocated() during neper/tcp_stream showed that more
samples completed faster with sk->sk_bypass_prot_mem == 1. This will
be more visible under tcp_mem pressure (but it's not a fair comparison).
# bpftrace -e 'kprobe:__sk_mem_raise_allocated { @start[tid] = nsecs; }
kretprobe:__sk_mem_raise_allocated /@start[tid]/
{ @end[tid] = nsecs - @start[tid]; @times = hist(@end[tid]); delete(@start[tid]); }'
# tcp_stream -6 -F 1000 -N -T 256
Without bpf prog:
[128, 256) 3846 | |
[256, 512) 1505326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K) 1371006 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 198207 |@@@@@@ |
[2K, 4K) 31199 |@ |
With bpf prog in the next patch:
(must be attached before tcp_stream)
# bpftool prog load sk_bypass_prot_mem.bpf.o /sys/fs/bpf/test type cgroup/sock_create
# bpftool cgroup attach /sys/fs/cgroup/test cgroup_inet_sock_create pinned /sys/fs/bpf/test
[128, 256) 6413 | |
[256, 512) 1868425 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K) 1101697 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 117031 |@@@@ |
[2K, 4K) 11773 | |
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com >
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org >
Acked-by: Roman Gushchin <roman.gushchin@linux.dev >
Link: https://patch.msgid.link/20251014235604.3057003-6-kuniyu@google.com
2025-10-16 12:04:47 -07:00
2025-09-19 09:40:46 +02:00
2021-12-21 15:09:46 -08:00
2025-09-24 17:09:44 +02:00
2021-09-13 14:51:10 -05:00
2025-03-08 08:57:18 +09:00
2024-04-15 13:42:38 +02:00
2022-11-17 22:49:39 +01:00
2025-09-13 13:47:19 +01:00
2025-09-08 08:06:37 -06:00
2025-07-03 09:25:01 +02:00
2022-06-28 21:26:05 +02:00
2025-09-11 15:40:55 +02:00
2022-06-28 21:26:05 +02:00
2023-08-22 15:13:20 +02:00
2022-06-28 21:26:05 +02:00
2022-08-09 19:51:05 +02:00
2023-04-27 18:49:24 -04:00
2025-08-16 08:47:38 +08:00
2024-09-11 11:23:56 +02:00
2024-07-29 01:19:51 +01:00
2024-02-19 10:58:24 +00:00
2025-04-02 18:23:31 -07:00
2022-04-29 14:38:03 -07:00
2022-11-23 19:55:22 +01:00
2023-06-20 14:28:17 -06:00
2022-05-13 10:56:10 +02:00
2021-11-01 13:36:08 +00:00
2025-08-29 11:04:02 +02:00
2022-08-11 10:31:19 -07:00
2022-12-22 18:18:37 -08:00
2023-03-10 21:05:16 +01:00
2025-08-30 10:15:30 -04:00
2023-05-30 16:42:00 -07:00
2024-08-30 08:22:36 +02:00
2024-02-16 12:42:59 +11:00
2024-10-05 18:23:38 +02:00
2025-02-22 11:36:22 +01:00
2025-07-08 10:23:13 -04:00
2025-02-10 09:54:19 -07:00
2024-09-11 10:45:28 -06:00
2025-05-05 13:22:39 -06:00
2023-07-24 20:11:54 -06:00
2025-10-16 12:04:47 -07:00
2025-02-05 16:17:59 -08:00
2024-07-11 15:33:28 +02:00
2025-07-22 01:13:03 +02:00
2022-05-18 00:11:18 +08:00
2025-02-19 11:10:54 +01:00
2025-07-04 19:21:53 -05:00
2022-05-15 18:31:28 -06:00
2025-04-30 08:16:07 +02:00
2024-08-08 15:23:21 +02:00
2023-07-14 08:43:49 -10:00
2024-05-24 10:36:55 +01:00
2021-11-26 16:48:59 +01:00
2022-06-28 21:26:05 +02:00
2025-03-05 12:00:03 -05:00
2025-06-12 14:00:18 +02:00
2025-02-05 13:22:04 +09:00
2024-10-05 13:22:04 +08:00
2024-04-30 08:48:10 -07:00
2022-01-26 14:49:46 +01:00
2022-06-28 21:26:05 +02:00
2023-01-20 09:33:22 +00:00
2025-08-26 17:24:16 -07:00
2021-10-18 12:28:53 -07:00
2023-07-20 17:24:57 -05:00
2024-06-11 12:57:49 -05:00
2023-03-06 15:49:07 -06:00
2025-05-04 11:35:06 +02:00
2022-06-28 21:26:05 +02:00
2022-06-20 18:21:25 +02:00
2024-06-05 14:52:15 +05:30
2025-09-29 18:57:41 -07:00
2022-08-29 15:32:13 +02:00
2022-06-03 20:09:27 +08:00
2023-08-23 14:17:42 -07:00
2025-03-25 11:30:42 -07:00
2025-03-25 12:48:05 -04:00
2025-10-01 09:48:21 +02:00
2025-06-25 15:24:13 -07:00
2025-08-14 11:40:13 +02:00
2023-06-15 14:55:15 +02:00
2024-02-14 11:01:01 +00:00
2024-09-18 07:40:56 +09:00
2025-09-26 08:36:08 -04:00
2025-02-12 02:29:30 +00:00
2025-06-23 12:45:13 +02:00
2025-02-05 17:21:07 +01:00
2024-01-27 18:08:54 -08:00
2025-09-19 14:26:16 +02:00
2025-04-17 19:04:50 -07:00
2024-12-22 11:29:50 +01:00
2025-03-04 09:47:36 -07:00
2023-03-16 21:20:32 -07:00
2025-08-29 15:08:07 +02:00
2025-04-08 19:32:11 -07:00
2023-08-09 15:43:27 +09:30
2022-02-17 09:09:37 +01:00
2025-09-05 09:11:28 +02:00
2025-05-03 12:02:09 +02:00
2022-08-10 13:49:50 +01:00
2024-02-15 08:44:57 +01:00
2023-10-27 09:21:13 +02:00
2024-05-07 01:35:57 +02:00
2025-07-08 15:31:44 +02:00
2023-10-27 18:04:30 +08:00
2022-09-20 11:53:32 +01:00
2025-09-17 11:37:23 +02:00
2023-03-10 21:05:16 +01:00
2022-02-18 13:08:18 +00:00
2021-11-15 07:53:10 -08:00
2025-07-14 09:15:58 +02:00
2025-09-24 07:50:09 -07:00
2024-05-06 11:13:56 +02:00
2023-10-04 13:00:34 +05:30
2025-04-17 19:04:50 -07:00
2025-04-17 19:04:50 -07:00
2025-04-17 19:04:50 -07:00
2025-04-17 19:04:50 -07:00
2021-07-29 15:06:50 +01:00
2025-04-17 19:04:50 -07:00
2025-09-11 19:02:50 -07:00
2022-09-15 09:08:09 +02:00
2025-04-17 19:04:50 -07:00
2025-04-17 19:04:50 -07:00
2025-09-19 17:21:55 -07:00
2022-08-18 20:37:35 -07:00
2025-04-17 19:04:50 -07:00
2025-04-17 19:04:50 -07:00
2022-06-28 21:26:05 +02:00
2025-04-17 19:04:50 -07:00
2024-04-02 18:24:33 -07:00
2025-07-08 18:07:26 +02:00
2024-04-01 10:49:28 +01:00
2025-04-17 19:04:50 -07:00
2025-07-10 14:48:29 +02:00
2022-06-28 21:26:05 +02:00
2025-06-10 14:41:14 -07:00
2024-09-04 16:40:33 -07:00
2024-12-05 10:01:09 +01:00
2024-08-26 09:37:23 -07:00
2024-03-06 13:13:00 +01:00
2025-09-15 14:32:54 +02:00
2025-09-15 14:32:54 +02:00
2025-10-07 08:04:09 -06:00
2024-02-28 11:19:41 +00:00
2024-08-22 10:45:12 +02:00
2023-01-06 19:22:53 -08:00
2025-07-18 17:33:08 -03:00
2023-08-15 10:06:49 -06:00
2025-04-17 19:04:50 -07:00
2022-06-28 21:26:05 +02:00
2024-12-05 10:01:09 +01:00
2022-10-17 09:51:26 -05:00
2021-10-05 06:54:16 -05:00
2024-12-05 10:01:28 +01:00
2025-07-25 13:06:19 -07:00
2022-06-28 21:26:05 +02:00
2025-05-08 16:04:06 +03:00
2025-09-09 15:58:56 +02:00
2022-06-28 21:26:05 +02:00
2022-06-28 21:26:05 +02:00
2024-06-01 07:28:21 +02:00
2024-09-01 20:26:05 -07:00
2025-09-13 17:32:43 -07:00
2025-10-07 14:09:19 -04:00
2025-03-05 10:47:33 -05:00
2025-08-27 04:37:03 -04:00
2022-09-20 09:13:38 +02:00
2025-04-17 11:09:10 +02:00
2024-09-12 20:28:46 -07:00
2022-05-25 09:51:36 -07:00
2022-07-08 10:46:53 -07:00
2025-03-10 09:10:53 -04:00
2022-05-10 06:30:05 -06:00
2024-08-19 22:36:26 -04:00
2022-08-29 10:44:08 +02:00
2024-05-08 17:29:24 -04:00
2021-10-21 23:36:28 +02:00
2025-07-08 12:39:24 +02:00
2025-01-16 17:22:59 -08:00
2024-11-05 13:26:42 +02:00
2023-05-25 16:21:22 +02:00
2023-03-23 17:25:46 +01:00
2023-12-15 17:01:30 +01:00
2023-01-07 11:29:29 +01:00
2023-01-18 17:12:37 -08:00
2025-09-13 16:54:41 -07:00
2022-06-28 21:26:05 +02:00
2024-07-19 20:22:12 +02:00
2022-01-11 18:45:02 -08:00
2025-02-12 12:12:34 +01:00
2025-09-22 11:51:24 -07:00
2025-09-26 17:44:05 -07:00
2022-02-19 16:05:54 +00:00
2025-03-21 18:24:22 +00:00
2024-09-06 08:31:40 -06:00
2022-06-28 21:26:05 +02:00
2025-06-30 18:14:23 -07:00
2025-06-25 15:35:20 -07:00
2024-10-10 08:30:22 -07:00
2025-04-17 19:04:50 -07:00
2025-07-25 13:06:19 -07:00
2025-07-24 18:34:55 -07:00
2022-09-07 16:46:03 +02:00
2025-04-17 19:04:50 -07:00
2023-10-20 11:43:35 +01:00
2024-08-12 17:50:34 -07:00
2024-11-07 10:21:58 +01:00
2025-01-21 15:30:01 -05:00
2022-03-13 12:59:35 -04:00
2024-05-06 12:47:24 -04:00
2024-07-08 14:10:05 -04:00
2025-02-28 11:51:32 -08:00
2021-09-14 11:11:20 +02:00
2021-10-21 17:25:17 +02:00
2025-09-19 11:55:56 +02:00
2023-09-27 09:40:02 +02:00
2025-09-19 14:26:18 +02:00
2023-11-28 19:05:16 +00:00
2025-01-08 13:18:11 +01:00
2022-05-20 06:17:33 -06:00
2022-02-27 10:58:04 -08:00
2025-07-07 14:30:39 -07:00
2025-04-17 12:30:02 +02:00
2024-04-25 12:37:12 -07:00
2023-03-23 12:22:19 +09:00
2025-10-03 12:13:14 -05:00
2025-07-24 16:51:46 -05:00
2025-05-22 11:03:41 +02:00
2021-12-23 09:32:51 +01:00
2025-08-20 15:30:36 +02:00
2025-06-24 15:50:06 +02:00
2025-05-15 11:44:29 -07:00
2025-07-23 17:52:07 -07:00
2023-06-07 14:26:09 -06:00
2025-01-08 13:18:09 +01:00
2022-12-01 03:22:20 +00:00
2025-09-13 16:55:05 -07:00
2024-07-05 17:45:47 -07:00
2024-10-24 16:30:33 -07:00
2023-07-20 22:14:21 +12:00
2025-08-16 17:20:23 +08:00
2025-09-17 12:17:05 +02:00
2025-09-18 12:32:06 +02:00
2025-09-25 11:00:59 -07:00
2025-05-11 17:48:15 -07:00
2025-07-07 08:39:33 +02:00
2023-08-09 09:15:39 +02:00
2024-07-19 20:22:48 +02:00
2023-12-20 19:26:31 -05:00
2022-05-09 14:00:07 +02:00
2025-06-18 09:21:58 +02:00
2022-06-28 21:26:05 +02:00
2023-01-06 19:28:01 -08:00
2023-07-15 11:35:02 -07:00
2022-12-27 12:52:12 +01:00
2022-03-23 19:58:40 +01:00
2025-02-25 18:14:03 -08:00
2025-04-14 17:36:41 -07:00
2024-06-18 10:09:17 -10:00
2023-03-09 11:31:44 +01:00
2023-07-17 16:08:08 -07:00
2024-12-18 17:00:29 -08:00
2024-10-22 08:16:40 -06:00
2022-08-03 16:29:08 -07:00
2022-09-20 12:33:22 +02:00
2022-06-28 21:26:05 +02:00
2023-10-10 08:51:10 +02:00
2023-09-18 11:14:42 +02:00
2024-01-27 18:15:24 -08:00
2024-09-05 20:11:43 +02:00
2024-07-16 11:44:00 -04:00
2023-12-26 20:24:33 +00:00
2024-08-20 11:38:23 +02:00
2025-07-14 18:41:42 -07:00
2022-06-01 16:48:05 -07:00
2022-01-08 17:41:00 +09:00
2025-05-07 14:25:30 -07:00
2025-08-28 13:06:48 +02:00
2022-06-28 21:26:05 +02:00
2022-11-09 14:06:51 +01:00
2023-12-02 01:17:48 +02:00
2025-07-25 13:06:19 -07:00
2023-03-24 16:59:09 -04:00
2025-05-20 22:49:39 -07:00
2024-07-01 09:44:27 +01:00
2025-09-18 08:47:52 +02:00
2022-11-17 11:04:23 -08:00
2025-09-15 17:34:06 +02:00
2025-02-11 20:53:14 +01:00
2023-12-04 10:37:46 +01:00
2025-06-19 14:28:22 +02:00
2025-05-21 13:41:03 +02:00
2022-04-06 13:48:02 +01:00
2022-09-27 17:29:09 -07:00
2023-06-15 13:41:53 +02:00
2024-07-03 16:40:45 -04:00
2022-06-27 16:25:41 +02:00
2024-09-30 09:20:59 +01:00
2025-06-30 20:13:42 -06:00
2025-04-17 12:30:02 +02:00
2024-09-11 20:44:32 -07:00
2024-07-03 12:24:54 +02:00
2022-06-28 21:26:05 +02:00
2022-08-31 09:07:53 +02:00
2024-03-18 10:13:03 -04:00
2023-12-29 11:58:24 -08:00
2023-03-23 17:25:46 +01:00
2025-03-03 18:23:36 +01:00
2022-12-07 17:58:46 +01:00
2025-09-04 10:37:05 +02:00
2025-09-09 15:58:44 +02:00
2024-04-29 14:56:36 +02:00
2024-04-29 14:56:37 +02:00
2024-08-22 16:56:24 -07:00
2024-09-10 02:51:48 -04:00
2025-09-21 17:44:20 -04:00
2024-01-27 18:08:54 -08:00
2022-07-11 09:54:37 +02:00
2025-08-05 15:41:14 -06:00
2025-07-08 18:05:23 +02:00
2025-08-26 03:38:19 -04:00
2025-09-09 15:58:56 +02:00
2022-06-28 21:26:05 +02:00
2024-09-10 02:51:48 -04:00
2023-04-04 11:01:57 -04:00
2024-05-14 10:54:49 -04:00
2023-12-19 11:51:31 -07:00
2024-10-05 13:22:04 +08:00
2021-11-04 16:36:54 +01:00
2025-03-28 04:53:51 +03:00
2021-11-01 05:26:48 -04:00
2025-09-22 09:29:28 +01:00
2021-12-06 15:03:05 +01:00
2024-05-22 08:31:15 -04:00
2025-07-08 18:05:47 +02:00
2025-04-14 09:51:31 -04:00
2021-08-26 22:28:03 +02:00
2024-01-10 13:01:37 -05:00
2022-08-16 01:40:24 -04:00
2025-05-27 10:27:54 -04:00
2021-08-05 10:24:08 +01:00
2025-02-14 12:58:02 +01:00
2025-09-22 09:29:29 +01:00
2021-09-05 16:23:09 -04:00
2025-06-24 17:12:59 -07:00
2024-10-09 12:16:18 +01:00
2025-08-01 10:42:22 +02:00
2025-05-27 09:06:19 +02:00
2023-05-28 19:07:48 -06:00
2025-02-13 19:35:31 -08:00
2024-12-05 10:01:15 +01:00
2024-06-04 08:59:58 +02:00
38163af068