Eric Dumazet
d4438ce68b
inet: call inet6_ehashfn() once from inet6_hash_connect()
inet6_ehashfn() being called from __inet6_check_established()
has a big impact on performance, as shown in the Tested section.
After prior patch, we can compute the hash for port 0
from inet6_hash_connect(), and derive each hash in
__inet_hash_connect() from this initial hash:
hash(saddr, lport, daddr, dport) == hash(saddr, 0, daddr, dport) + lport
Apply the same principle for __inet_check_established(),
although inet_ehashfn() has a smaller cost.
Tested:
Server: ulimit -n 40000; neper/tcp_crr -T 200 -F 30000 -6 --nolog
Client: ulimit -n 40000; neper/tcp_crr -T 200 -F 30000 -6 --nolog -c -H server
Before this patch:
utime_start=0.286131
utime_end=4.378886
stime_start=11.952556
stime_end=1991.655533
num_transactions=1446830
latency_min=0.001061085
latency_max=12.075275028
latency_mean=0.376375302
latency_stddev=1.361969596
num_samples=306383
throughput=151866.56
perf top:
50.01% [kernel] [k] __inet6_check_established
20.65% [kernel] [k] __inet_hash_connect
15.81% [kernel] [k] inet6_ehashfn
2.92% [kernel] [k] rcu_all_qs
2.34% [kernel] [k] __cond_resched
0.50% [kernel] [k] _raw_spin_lock
0.34% [kernel] [k] sched_balance_trigger
0.24% [kernel] [k] queued_spin_lock_slowpath
After this patch:
utime_start=0.315047
utime_end=9.257617
stime_start=7.041489
stime_end=1923.688387
num_transactions=3057968
latency_min=0.003041375
latency_max=7.056589232
latency_mean=0.141075048 # Better latency metrics
latency_stddev=0.526900516
num_samples=312996
throughput=320677.21 # 111 % increase, and 229 % for the series
perf top: inet6_ehashfn is no longer seen.
39.67% [kernel] [k] __inet_hash_connect
37.06% [kernel] [k] __inet6_check_established
4.79% [kernel] [k] rcu_all_qs
3.82% [kernel] [k] __cond_resched
1.76% [kernel] [k] sched_balance_domains
0.82% [kernel] [k] _raw_spin_lock
0.81% [kernel] [k] sched_balance_rq
0.81% [kernel] [k] sched_balance_trigger
0.76% [kernel] [k] queued_spin_lock_slowpath
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@amazon.com>
Tested-by: Jason Xing <kerneljasonxing@gmail.com>
Reviewed-by: Jason Xing <kerneljasonxing@gmail.com>
Link: https://patch.msgid.link/20250305034550.879255-3-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-03-06 15:26:02 -08:00
..
2024-05-01 18:07:37 +01:00
2025-02-13 11:15:37 -05:00
2024-10-08 15:33:49 -07:00
2024-12-09 14:44:59 -08:00
2025-02-14 10:58:08 -08:00
2024-10-09 13:42:04 +01:00
2025-01-19 16:41:56 +01:00
2025-03-03 15:04:10 -08:00
2024-11-07 10:21:58 +01:00
2025-02-06 16:27:31 -08:00
2024-10-24 16:03:40 +02:00
2024-08-26 09:37:23 -07:00
2024-01-02 12:41:16 +00:00
2024-10-23 11:43:47 +02:00
2025-01-20 12:16:04 -08:00
2023-12-24 15:22:50 +00:00
2024-06-25 11:10:18 +02:00
2024-08-15 17:18:52 -07:00
2025-01-06 15:57:01 -08:00
2024-08-26 09:37:22 -07:00
2024-08-26 09:37:22 -07:00
2024-11-14 11:16:28 +01:00
2024-08-27 13:11:37 -07:00
2025-02-27 14:03:14 +01:00
2024-10-02 17:23:23 -04:00
2025-02-28 14:08:59 +01:00
2024-03-06 21:01:26 +01:00
2024-10-30 15:29:59 +01:00
2024-10-02 17:23:23 -04:00
2023-07-14 20:39:29 -07:00
2023-07-27 17:17:32 -07:00
2025-01-16 13:04:58 -08:00
2025-03-03 15:44:19 -08:00
2025-02-11 11:58:17 +01:00
2023-10-24 13:08:14 -07:00
2025-01-17 20:01:09 -08:00
2024-05-08 10:35:09 +01:00
2024-08-26 09:37:23 -07:00
2025-02-20 13:17:16 -08:00
2024-05-29 17:34:49 -07:00
2024-12-09 14:44:59 -08:00
2024-11-13 18:49:50 -08:00
2024-08-26 09:37:23 -07:00
2024-04-01 21:27:08 -07:00
2024-10-11 15:35:05 -07:00
2025-02-19 18:43:38 -08:00
2024-07-15 09:14:39 -07:00
2024-10-30 17:33:53 -07:00
2023-11-02 09:31:02 +01:00
2023-04-12 16:40:39 -07:00
2023-08-08 15:58:23 -07:00
2024-12-09 14:44:59 -08:00
2024-04-01 10:49:28 +01:00
2025-02-27 14:03:14 +01:00
2023-06-10 00:11:41 -07:00
2024-05-07 01:35:55 +02:00
2023-07-28 14:07:59 -07:00
2024-04-30 18:46:52 -07:00
2024-08-26 09:37:23 -07:00
2024-05-08 10:35:09 +01:00
2024-10-25 10:44:41 +01:00
2023-11-20 11:43:03 +01:00
2024-02-26 18:40:34 -08:00
2025-03-03 15:44:19 -08:00
2024-05-13 18:19:09 -06:00
2025-02-11 13:08:00 +01:00
2024-05-23 14:14:23 -07:00
2025-03-06 15:26:02 -08:00
2025-01-06 13:49:38 -08:00
2024-12-11 20:17:33 -08:00
2024-12-17 19:37:00 -08:00
2024-02-28 11:19:41 +00:00
2024-10-11 15:35:05 -07:00
2024-07-14 07:34:16 -07:00
2024-04-01 10:49:28 +01:00
2025-03-03 15:04:09 -08:00
2025-02-21 15:28:02 -08:00
2023-04-22 01:39:41 +02:00
2025-03-06 15:26:02 -08:00
2024-08-05 16:27:26 -07:00
2025-02-18 18:27:20 -08:00
2024-10-09 08:53:01 +02:00
2024-08-19 18:36:12 -07:00
2025-02-10 18:09:09 -08:00
2024-12-06 17:43:08 -08:00
2023-08-04 15:33:17 -07:00
2023-08-04 15:33:17 -07:00
2024-07-15 08:51:01 -07:00
2023-07-20 10:46:28 +02:00
2024-08-26 09:37:23 -07:00
2024-07-15 08:51:19 -07:00
2025-03-05 19:16:56 -08:00
2025-02-26 15:48:47 +01:00
2024-10-02 17:23:23 -04:00
2025-01-20 12:20:42 +00:00
2024-11-09 09:04:54 -08:00
2024-11-09 09:04:54 -08:00
2025-02-24 18:23:44 -08:00
2024-08-12 17:23:57 -07:00
2024-11-09 13:22:57 -08:00
2024-11-09 13:22:57 -08:00
2024-11-30 13:41:50 -08:00
2025-02-18 18:33:29 -08:00
2024-10-10 08:30:22 -07:00
2025-02-12 16:37:35 -08:00
2025-02-05 17:49:08 -08:00
2023-11-20 10:15:16 -08:00
2024-11-21 08:28:08 -08:00
2025-02-17 16:46:03 -08:00
2025-02-06 16:27:30 -08:00
2024-08-12 17:50:34 -07:00
2024-08-30 22:30:55 +02:00
2023-07-14 20:39:29 -07:00
2023-08-04 15:33:50 -07:00
2024-04-01 10:49:28 +01:00
2023-07-14 20:39:30 -07:00
2023-06-02 09:55:22 +01:00
2025-01-20 09:21:27 +00:00
2024-02-02 10:57:55 -08:00
2024-04-30 18:46:52 -07:00
2024-03-07 21:12:43 -08:00
2024-07-11 18:11:31 -07:00
2023-04-06 12:01:20 -07:00
2024-08-26 09:37:23 -07:00
2024-08-26 09:37:23 -07:00
2024-06-06 15:18:04 +02:00
2025-02-06 16:14:14 -08:00
2023-06-19 11:32:58 -07:00
2024-04-01 11:28:32 +01:00
2023-07-14 20:39:30 -07:00
2024-08-07 10:24:46 +01:00
2025-02-21 15:28:03 -08:00
2025-02-05 18:32:06 -08:00
2024-12-04 14:42:31 -05:00
2024-05-30 18:29:38 -07:00
2024-06-24 16:41:23 -07:00
2024-05-30 18:29:38 -07:00
2024-04-30 13:24:48 +02:00
2024-08-02 17:16:59 -07:00
2025-03-03 17:16:34 -08:00
2023-10-04 11:49:20 -07:00
2025-01-29 13:32:08 -08:00
2024-02-16 09:36:37 +00:00
2024-02-14 10:49:37 +01:00
2024-11-03 12:10:11 -08:00
2023-12-04 14:45:26 -08:00
2025-03-03 15:44:19 -08:00
2024-07-08 14:07:31 -07:00
2024-05-09 20:25:55 -07:00
2023-07-28 14:07:59 -07:00
2024-12-16 12:47:29 +00:00
2023-08-01 15:06:27 -07:00
2024-04-01 10:49:28 +01:00
2024-11-18 11:56:21 +00:00
2023-09-14 16:16:36 +02:00
2024-12-09 14:47:04 -08:00
2024-08-26 09:37:23 -07:00
2025-02-20 15:13:45 -08:00
2025-02-20 15:13:45 -08:00
2025-02-27 14:03:52 +01:00
2025-01-27 15:15:12 -08:00
2025-02-10 17:54:43 -08:00