mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-31 09:20:13 -04:00
f6f47a9ca8
squashfs_cache_get() puts a task to sleep when its block is not cached and
every cache entry is busy. Those sleeps are non-exclusive, so the
nr_exclusive == 1 budget squashfs_cache_put() has always passed to
wake_up() is inert and one release makes every waiter runnable. A wakee
only returns to squashfs_cache_get() if it observes cache->unused before
the entry is reclaimed; later wakees see zero and re-queue inside
wait_event() without rescanning. One freed entry satisfies exactly one
capacity waiter, so waking the rest is waste.
On a Meta production host serving a Python web application from a packaged
squashfs image, a 30-second trace caught 1,045,132 cache-release wake
calls and 19,511,556 wakeups: 18.7 per release, although each release
added only one reusable cache entry. This was causing significant spikes
in CPU usage.
Make the waits exclusive, enqueueing while still holding cache->lock so
that a concurrent lookup either sees the waiter queued or the waiter sees
the block that lookup publishes. Two things follow.
A wakee cannot be assumed to consume the entry it was woken for: it may
find its own block published meanwhile, share that entry, and leave the
freed one unclaimed. So a wakee which shares hands its wakeup on to the
next waiter, as commit 0ddad21d3e ("pipe: use exclusive waits when
reading or writing") does with wake_next_reader.
And a waiter can now sleep through a publication of the very block it
wants, which the old broadcast gave it repeated chances to notice. So
waiters are keyed by block: publishing wakes every waiter for that block
(nr_exclusive == 0), freeing an entry wakes one. That needs a custom wake
callback, like wake_page_function() in mm/filemap.c, which also records
which wakeup arrived so the handoff only fires for a capacity wakee.
Broadcast is kept where more than one task can proceed - every waiter for
a published block, and the wake_up_all() on entry->wait_queue - at the
cost of walking the queue under wait_queue.lock to test the key. Waiters
are now served FIFO with a scheduling round trip per handoff hop, so
per-waiter latency changes; the filebench run below is 4x oversubscribed,
where that should hurt most.
Measured on a 32-CPU VM against a read-only squashfs (gzip,
DECOMP_MULTI_PERCPU, FILE_DIRECT, default 8 metadata / 3 fragment cache
entries) staged in tmpfs, page cache dropped each iteration to force cold
decompression:
elbencho, 64 threads
metadata stat 700 -> 1320 files/s 1.9x
small-file read 40 -> 60 MiB/s 1.5x
filebench, 128 threads, open+read+stat+close (mean of 3x 30s)
throughput 11,314 -> 25,186 ops/s 2.2x
sched:sched_wakeup 27.0 -> 4.55 per op 5.9x fewer
context switches 37.2 -> 7.64 per op 4.9x fewer
Wakeups and context switches are per operation, since the two runs did
2.2x different amounts of work. Workloads which never queue for a cache
entry gain no wakeups.
Link: https://lore.kernel.org/20260807172421.3875982-1-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Phillip Lougher <phillip@squashfs.org.uk>
Cc: Boris Burkov <boris@bur.io>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>