mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-31 14:04:27 -04:00
Code Issues Packages Projects Releases Wiki Activity

NFSD: Eliminate percpu counter contention in IO byte accounting

Browse Source 
nfsd_stats_io_write_add() and nfsd_stats_io_read_add() accumulate
byte counts in per-net-namespace and per-export percpu_counters
using percpu_counter_add(), which applies the default batch
threshold of max(32, 2*nr_cpus).

For a 4 KB NFS WRITE, the amount (4096) always exceeds this
threshold, so percpu_counter_add_batch() acquires the counter's
global spinlock on every update. Each WRITE RPC updates two
counters (per-net and per-export), producing two global lock
acquisitions per operation. Profiling on a 10-CPU RDMA NFS
server shows 0.44% of total CPU cycles spent contending on
these locks during a small random write workload.

Switch to percpu_counter_add_local(), which batches with
INT_MAX so that updates always remain on the per-CPU fast
path regardless of the amount. All readers of these counters
already use percpu_counter_sum_positive(), which sums the
per-CPU deltas under the global lock, so read accuracy is
unaffected.

Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: NeilBrown <neil@brown.name>
Link: https://patch.msgid.link/20260717001232.438792-5-cel@kernel.org
Signed-off-by: Chuck Lever <cel@kernel.org>
This commit is contained in:
Chuck Lever
2026-07-16 20:12:31 -04:00
parent ee987730d6
commit 264226677d
1 changed files with 26 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
fs/nfsd/stats.h
View File

@@ -59,20 +59,42 @@ static inline void nfsd_stats_fh_stale_inc(struct nfsd_net *nn,
percpu_counter_inc(&exp->ex_stats->counter[EXP_STATS_FH_STALE]);
}
/**
* nfsd_stats_io_read_add - Count number of bytes for an NFS READ
* @nn: target network namespace
* @exp: target export
* @amount: byte count
*
* These counters are updated on every READ request. Readers use
* percpu_counter_sum_positive(), so local batching does not affect
* read accuracy.
*/
static inline void nfsd_stats_io_read_add(struct nfsd_net *nn,
struct svc_export *exp, s64 amount)
{
percpu_counter_add(&nn->counter[NFSD_STATS_IO_READ], amount);
percpu_counter_add_local(&nn->counter[NFSD_STATS_IO_READ], amount);
if (exp && exp->ex_stats)
percpu_counter_add(&exp->ex_stats->counter[EXP_STATS_IO_READ], amount);
percpu_counter_add_local(&exp->ex_stats->counter[EXP_STATS_IO_READ],
amount);
}
/**
* nfsd_stats_io_write_add - Count number of bytes for an NFS WRITE
* @nn: target network namespace
* @exp: target export
* @amount: byte count
*
* These counters are updated on every WRITE request. Readers use
* percpu_counter_sum_positive(), so local batching does not affect
* read accuracy.
*/
static inline void nfsd_stats_io_write_add(struct nfsd_net *nn,
struct svc_export *exp, s64 amount)
{
percpu_counter_add(&nn->counter[NFSD_STATS_IO_WRITE], amount);
percpu_counter_add_local(&nn->counter[NFSD_STATS_IO_WRITE], amount);
if (exp && exp->ex_stats)
percpu_counter_add(&exp->ex_stats->counter[EXP_STATS_IO_WRITE], amount);
percpu_counter_add_local(&exp->ex_stats->counter[EXP_STATS_IO_WRITE],
amount);
}
static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)