kasan_quarantine_remove_cache() first invokes per_cpu_remove_cache() on
all online CPUs. Each callback moves objects belonging to the cache from
cpu_quarantine to the CPU's shrink_qlist, where they can later be freed
from task context.
kmem_cache_destroy() invokes the quarantine removal path while holding
cpus_read_lock(), but kmem_cache_shrink() does not. The latter can
therefore race with CPU offlining as follows:
kmem_cache_shrink() CPU hotplug
------------------- -----------
on_each_cpu()
CPU1 moves objects to
CPU1's shrink_qlist
on_each_cpu() returns
CPU1 goes offline
kasan_cpu_offline()
drains cpu_quarantine
leaves shrink_qlist untouched
for_each_online_cpu()
skips CPU1
The objects left on CPU1's shrink_qlist are not returned to the slab
allocator. This may prevent kmem_cache_shrink() from releasing slabs that
would otherwise become empty. If CPU1 remains offline, a later
kmem_cache_destroy() also skips the list and can report that the cache
still contains objects.
An intermittent occurrence was observed with a virtio-9p filesystem. The
mount and umount commands both returned 0, but the kernel logged the
following during the userspace-triggered teardown:
[ 2994.380134][ T111] BUG 9p-fcall-cache-1 (Tainted: G B ): Objects remaining on __kmem_cache_shutdown()
[ 2994.381140][ T111] Object 0xff11000004361118 @offset=4376
[ 2994.381607][ T111] Allocated in p9_fcall_init+0x201/0x400 age=19564 cpu=1 pid=104
[ 2994.382591][ T111] p9_fcall_init+0x201/0x400
[ 2994.382810][ T111] p9_tag_alloc+0x12f/0x700
[ 2994.382982][ T111] p9_client_prepare_req+0x102/0x3e0
[ 2994.383165][ T111] p9_client_rpc+0x1ab/0xa50
[ 2994.383334][ T111] p9_client_getattr_dotl+0xb0/0x1a0
[ 2994.383515][ T111] v9fs_vfs_getattr_dotl+0x115/0x360
[ 2994.383719][ T111] vfs_getattr_nosec+0x22c/0x3a0
[ 2994.383910][ T111] vfs_statx+0xd7/0x170
[ 2994.384062][ T111] vfs_fstatat+0x45/0x80
[ 2994.384215][ T111] __do_sys_newfstatat+0x84/0xe0
[ 2994.384386][ T111] do_syscall_64+0x115/0x6a0
[ 2994.384566][ T111] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 2994.399720][ T111] WARNING: mm/slub.c:1244 at __kmem_cache_shutdown+0x363/0x500, CPU#0: busybox/111
[ 2994.405655][ T111] Call Trace:
[ 2994.406325][ T111] kmem_cache_destroy+0x73/0x1b0
[ 2994.406630][ T111] p9_client_destroy+0x271/0x3c0
[ 2994.407210][ T111] v9fs_session_close+0x3c/0x260
[ 2994.407409][ T111] v9fs_kill_super+0x48/0x90
[ 2994.407584][ T111] deactivate_locked_super+0xa3/0x160
[ 2994.407778][ T111] cleanup_mnt+0x1dd/0x3e0
Thus, a successful umount left objects in the 9p fcall cache and prevented
the cache from being destroyed cleanly.
Per-CPU shrink_qlist storage exists for every possible CPU, and each list
is protected by its own raw spinlock. Iterate over possible CPUs so that
a list populated before its CPU went offline is drained as well.
for_each_possible_cpu() can do more work than for_each_online_cpu(), but
this change only affects CONFIG_KASAN_GENERIC kernels. The extra work is
limited to cache shrink and cache destruction paths and does not affect
the normal allocation/free fast path. It adds one raw-spinlock-protected
scan of each possible CPU's shrink list. These lists are normally empty;
a non-empty list is traversed to remove objects belonging to the cache
being shrunk or destroyed.
Link: https://lore.kernel.org/20260808031459.3032812-1-sh_def@163.com
Fixes: 07d067e4f2 ("kasan: fix sleeping function called from invalid context on RT kernel")
Signed-off-by: Hui Su <sh_def@163.com>
Reviewed-by: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: "Zhang, Qiang1" <qiang1.zhang@intel.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
zswap_shrinker_count() calls mem_cgroup_flush_stats(), which takes the
global cgroup rstat lock synchronously. On machines with many CPUs and
NUMA nodes, this creates severe lock contention in the kswapd reclaim
path:
- Multiple kswapd threads (one per NUMA node) run concurrently.
- do_shrink_slab() invokes zswap_shrinker_count() for each
memcg-aware shrinker pass.
- Each call flushes the full cgroup rstat hierarchy under the global
lock.
On AMD EPYC 9684X machines (96 cores, 192 threads, 12 NUMA nodes) running
production workloads with zswap enabled, perf shows 2.88% of kernel cycles
in osq_lock contention from this path:
2.88% [k] osq_lock
--__mutex_lock.constprop.0
--__cgroup_rstat_lock
--cgroup_rstat_flush_locked
--cgroup_rstat_flush
--zswap_shrinker_count
do_shrink_slab
shrink_slab
shrink_node
balance_pgdat
kswapd
84% of kswapd kernel cycles are spent in
shrink_slab -> zswap_shrinker_count -> cgroup_rstat_flush, not in actual
page reclaim (shrink_lruvec).
Controlled A/B on identical hardware and workload:
shrinker=Y: 2.88% osq_lock, memory PSI 1.58%
shrinker=N: 0.00% osq_lock, memory PSI 0.57%
eBPF-based rstat lock wait measurement across 8 production metals
confirms the contention splits cleanly along shrinker enablement:
shrinker=Y: 50-250x more contended lock acquisitions (248/s vs 1.1/s)
shrinker=N: baseline lock wait (0.0017 s/s vs 1.04 s/s)
zswap_shrinker_count() only produces a heuristic estimate, scaled by
compression ratio via mult_frac(). The actual writeback happens in
zswap_shrinker_scan(). Slightly stale stats are acceptable here.
Switch to mem_cgroup_flush_stats_ratelimited(), which only flushes if
the periodic 2-second flusher is one full cycle late. This matches the
approach already used in prepare_scan_control() (mm/vmscan.c) for the
same reclaim path.
After applying this patch, rstat flush latency and lock wait time on
shrinker=Y machines dropped to the same level as shrinker=N controls,
while the zswap shrinker continues to function (pool size remains
bounded under the max_pool_percent cap).
Previously discussed:
- Chengming Zhou (Dec 2023): rstat contention from
zswap_shrinker_count [1]
- Shakeel Butt (Aug 2024): zswap_shrinker_count still uses sync
flush [2]
- Yosry Ahmed (Aug 2024): suggested eliminating in-kernel
flushers [3]
- Jesper Dangaard Brouer (Sep 2024): cgroup/rstat V11 patch [4]
Link: https://lore.kernel.org/20260702180908.150136-1-yunzhao@cloudflare.com
Link: https://lore.kernel.org/linux-mm/20231206103935.3440502-1-zhouchengming@bytedance.com/ [1]
Link: https://lore.kernel.org/linux-mm/CALvZod7LFxLCxVpOFH8b2Ppm8T40HPGMKQwX_=NPCWB_mFW+oQ@mail.gmail.com/ [2]
Link: https://lore.kernel.org/linux-mm/CAJD7tkYvFyOSX+rP_FKGBhxvZiCDxtpsNp-c5CGOA-4Bq9oXSg@mail.gmail.com/ [3]
Link: https://lore.kernel.org/linux-mm/172616070094.2055617.17676042522679701515.stgit@firesoul/ [4]
Suggested-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org>
Signed-off-by: Yunzhao Li <yunzhao@cloudflare.com>
Tested-by: Yunzhao Li <yunzhao@cloudflare.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Jesper Dangaard Brouer <hawk@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Yosry Ahmed <yosry@kernel.org>
Cc: Yunzhao Li <yunzhao@cloudflare.com>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Block I/O benefits from batching just as much as remote file systems.
Extend struct swap_iocb to support building a bio on the fly as well, and
rewrite the block based swap code for it. This especially benefits
submit_bio based drivers that do not have the block plugging available,
but also saves allocating extra bios for blk-mq drivers.
Add a pre-allocated bio to struct swap_iocb in a union with kiocb used for
file system based swap so that struct swap_iocb can be used for all swap
I/O, and initialize the pool for it unconditionally.
Various low-level bdev and fs functions are now replaced with a unified
can_merge/add/submit scheme.
Note that the block based swap code now uses the same memcg-based check
previously added for file system based swap as well.
Link: https://lore.kernel.org/20260713093350.2154226-4-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Generalize the context currently provided by double pointers to struct
swap_iocb to an on-stack context. This cleans up the code and prepares
for adding more fields and supporting batching multiple folios into a
single bio for block-based swap as well.
This new swap_io_ctx is required for all functions using it, the old way
of allowing a NULL iocb for some callers is removed to keep the interface
consistent. To reduce code duplication caused by this, a new
swap_cache_read_folio_sync helper is added to consolidate the code to call
swap_cache_read_folio with a local swap_io_ctx.
The unpug helpers are renamed to use the submit wording as they are
generalized.
Link: https://lore.kernel.org/20260713093350.2154226-3-hch@lst.de
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
shmem_fallocate() validates offset + len with inode_newsize_ok(), but then
rounds that end offset up to a page boundary before entering the
preallocation loop.
For a valid request ending at MAX_LFS_FILESIZE, such as offset = 0 and len
= LLONG_MAX, adding PAGE_SIZE - 1 to the validated end can overflow the
signed loff_t used for the rounded end calculation. If that wrapped value
is then converted into a page index, shmem_fallocate() can enter the folio
allocation loop with an invalid range.
Use check_add_overflow() when calculating the page-aligned end, and fail
before entering the allocation loop if the rounded end cannot be
represented.
Link: https://lore.kernel.org/1929a466735dcbb9438936ff50b7a4fc2332a8a4.1785377919.git.zhilinz@nebusec.ai
Fixes: e2d12e22c5 ("tmpfs: support fallocate preallocation")
Signed-off-by: Zhiling Zou <zhilinz@nebusec.ai>
Reported-by: Vega <vega@nebusec.ai>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Add the following 2 scenarios to the allocinfo ioctl kselftest:
1. Validate size based filtering
2. Validate lineno based filtering
The first test uses "do_init_module" as the candidate function for the
test. This is because the associated site will only allocate memory when
a kernel module is loaded. The return value of get_content_id() changes
every time modules are loaded or unloaded. Hence, as long as
get_content_id() values at the start and the end of the test are the same,
the memory allocated by the do_init_module call site should also remain
the same. Consequently, the test can assume consistency between the value
returned by the ioctl and the procfs resulting in less flakiness.
Link: https://lore.kernel.org/e5171926b48802531284c1cb5f04734017141341.1783532853.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Tested-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Hao Ge <hao.ge@linux.dev>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "alloc_tag: introduce IOCTL-based filtering for MAP", v8.
Currently, memory allocation profiling data is primarily exposed through
/proc/allocinfo. While useful for manual inspection, this text-based
interface poses challenges for production monitoring and large-scale
analysis:
1. Userspace must parse large amounts of text to extract specific
fields.
2. To find specific tags, userspace must read the entire dataset,
requiring many context switches and high data copying.
3. The kernel currently aggregates per-CPU counters for every allocation
size, even those the user intends to filter out immediately.
This series introduces a new IOCTL-based binary interface for allocinfo
that supports kernel-side filtering. By allowing the user to specify a
filter mask, we significantly reduce the work performed in-kernel and the
amount of data transferred to userspace. The IOCTL mechanism was chosen
for allocinfo to address the per-CPU counter aggregation bottleneck. A
traditional read() operation must report the total allocation count and
sizes for every code tag in the system. Doing so requires iterating
across all CPUs to sum their per-CPU counters for thousands of tags, which
introduces substantial runtime overhead.
The IOCTL interface allows userspace to push selective filtering criteria
directly into the kernel before the per-CPU counter aggregation. The
kernel aggregates per-CPU counters only for a small subset of tags that
match the filter. This results in significant performance improvement.
Beyond fast filtered retrieval, the IOCTL foundation allows introducing a
context capture mechanism in the future to capture the context for
specific allocations.
Performance measurements were conducted on an Intel Xeon Platinum 8481C
(224 CPUs) with caches dropped before each run.
The IOCTL mechanism shows a ~20x performance improvement for
filtered queries. The kernel avoids the expensive per-CPU counter
aggregation (alloc_tag_read) for any tags that fail the initial string
or location filters.
Scenario 1: Specific File Filtering (arch/x86/events/rapl.c)
1. Traditional (cat /proc/allocinfo | grep): 22ms (sys)
2. IOCTL Interface: 1ms (sys)
Scenario 2: Compound Filtering (Filename + Size)
1. Traditional: (cat ... | grep | awk): 21ms (sys)
2. IOCTL Interface: 1ms (sys)
Scenario 3: Size-Based Filtering (min_size = 1MB)
1. Traditional: (cat ... | awk): 21ms (sys)
2. IOCTL Interface: 14ms (sys)
This patch (of 6):
Add the following ioctl commands for /proc/allocinfo file:
ALLOCINFO_IOC_CONTENT_ID - gets content identifier which can be used to
check whether the file content has changed specifically due to module
load/unload. Every time a module is loaded / unloaded, the returned value
will be different. By comparing the identifier value at the beginning and
at the end of the content retrieval operation, users can validate
retrieved information for consistency.
ALLOCINFO_IOC_GET_AT - gets the record at the specified position. This is
the position of a record in /proc/allocinfo.
ALLOCINFO_IOC_GET_NEXT - gets the record next to the last retrieved one.
If no records were previously retrieved, returns the first record.
Note, function file and module names often have the same prefixes,
therefore when filtering for them, we compare the last 64 characters to
minimize the chances of name collisions.
[akpm@linux-foundation.org: include compat.h, per Suren]
Closes: https://lore.kernel.org/oe-kbuild-all/202607091820.qbjlGhKK-lkp@intel.com/
Link: https://lore.kernel.org/cover.1783532853.git.abhishekbapat@google.com
Link: https://lore.kernel.org/15596de2607ef13e7c77c6d74763f4ae992ec475.1783532853.git.abhishekbapat@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Hao Ge <hao.ge@linux.dev>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Sourav Panda <souravpanda@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit 2b6a3f061f ("mm: declare VMA flags by bit") significantly
refactored the header file include/linux/mm.h. In that step, it
introduced a typo in an ifdef, referring to a non-existing config option
STACK_GROWS_UP, whereas the actual config option is called STACK_GROWSUP.
Commit 40a4af52e0 ("mm: fix CONFIG_STACK_GROWSUP typo in mm.h") fixed
this typo in the mm.h header file, but did not update the copy of the code
in tools/testing/vma/include/dup.h. Update this copy as well.
Commit message adapted from the above-referenced fix to mm.h.
Link: https://lore.kernel.org/20260611012258.432043-1-enelsonmoore@gmail.com
Signed-off-by: Ethan Nelson-Moore <enelsonmoore@gmail.com>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Alice Ryhl <aliceryhl@google.com>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Currently, hugetlb_cma reservation only supports absolute sizes (e.g.,
hugetlb_cma=2G or hugetlb_cma=0:1G,1:1G). This can be restrictive in
heterogeneous environments or when deploying common kernel command lines
across machines with different memory capacities.
Add support for percentage-based hugetlb_cma reservation (e.g.,
hugetlb_cma=20% or hugetlb_cma=0:20%,1:10%).
The percentage is calculated against the total memory (for global
settings) or against the node-specific memory (for node-specific settings)
using memblock APIs during early boot.
Link: https://lore.kernel.org/20260807040003.2156630-1-souravpanda@google.com
Signed-off-by: Sourav Panda <souravpanda@google.com>
Acked-by: Usama Arif <usama.arif@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Currently, shrink_memcg() writes back at most one entry per-node during
its traversal. This makes shrink_worker() inefficient, as it must
repeatedly re-enter shrink_memcg() to make any substantial progress.
Under high memory pressure, this can cause the writeback speed to be too
slow to keep up with refaults, leading to zswap store failures and forcing
pages to skip zswap and go directly to disk, which results in an LRU
inversion.
To address this, extend the per-node scan budget in shrink_memcg() from a
single entry to up to SWAP_CLUSTER_MAX pages, enabling batch writeback for
both the shrink_worker() and zswap_store() paths.
Test Setup:
- Total memory: 32 GB, 1 NUMA node.
- zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
Test Case 1:
Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
with random data (to avoid compression). Then, use cgroup memory.reclaim
to force a large amount of anonymous pages into zswap. At an interval of
2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
and the rest are zeros, and then trigger reclamation of this 4K page through
cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
will be triggered.
The test data after running for 120s is as follows:
Baseline Patched
shrink_worker wakeups 5,363 169
shrink_memcg calls 11,373,201 350,703
written_back pages 40,212 40,241
zswap_store calls 161,190 163,753
store succeeded (ret=1) 102,743 117,183
store rejected (ret=0) 58,447 46,570
store reject rate ~36% ~28%
pool_limit_hit delta 55,826 33,760
pswpout 98,659 86,811
pswpin 2 0
Test Case 2:
We evaluated the following two sub-configurations using stress-ng inside
a cgroup capped at memory.max=1G for 120 seconds:
Test Case 2a (max_pool_percent=1): Continuously triggers the global
zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
shrinking.
Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
Command executed for both setups:
bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
120s -q'
Test Case 2a (max_pool_percent=1):
Baseline Patched
shrink_worker wakeups 5,640 1,308
shrink_memcg calls 8,481,500 3,140,972
written_back pages 260 468,216
zswap_store calls 2,742,756 2,011,269
store succeeded (ret=1) 934,640 947,988
store rejected (ret=0) 1,808,116 1,063,281
store reject rate ~66% ~52%
pool_limit_hit delta 1,181,310 196,882
pswpout 1,808,376 1,531,497
pswpin 4,288,497 3,635,365
Test Case 2b (zswap.max=320M, max_pool_percent=50):
Baseline Patched
shrink_worker wakeups 0 0
shrink_memcg calls 687,608 54,002
written_back pages 639,176 846,663
zswap_store calls 1,224,222 1,228,548
store succeeded (ret=1) 992,816 1,208,123
store rejected (ret=0) 231,431 20,425
store reject rate ~19% ~2%
pool_limit_hit delta 0 0
pswpout 870,745 867,360
pswpin 1,707,823 1,216,814
Under identical workloads and runtimes, batched zswap shrinking exhibits a
significant reduction in both shrink_worker() wakeups and shrink_memcg()
calls. Furthermore, the sharp drop in both pswpin and zswap_store()
rejections demonstrates that batching zswap shrink operations effectively
mitigates zswap_store() failures caused by hitting the pool limit. This
significantly prevents pages from bypassing zswap and falling back
directly to disk, thereby reducing LRU inversion.
Link: https://lore.kernel.org/20260806070943.95542-3-jiahao.kernel@gmail.com
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
Suggested-by: Yosry Ahmed <yosry@kernel.org>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/zswap: Fixes and improves the zswap shrink", v4.
This series fixes and improves the zswap global shrinker
(shrink_worker()): Patch 1: Fix missing global shrinker when memory cgroup
is disabled. Patch 2: Extend shrink_memcg() to support batch writeback
and thereby improving the writeback efficiency in the shrink_worker() and
zswap_store() paths.
This patch (of 2):
Zswap writeback when the global pool limit is hit fails when memory cgroup
is disabled. The pool remains full until it is organically drained by
swapins or memory freeing, leading to zswap store failures and pages
bypassing getting written directly to the backing swap device, causing LRU
inversion (hotter pages with higher fault latency).
This happens because mem_cgroup_iter() always returns NULL when memory
cgroups are disabled. As a result, the global shrinker shrink_worker()
repeatedly takes empty walks. After MAX_RECLAIM_RETRIES failed attempts,
the worker gives up without writing back any pages.
Therefore, when memory cgroup is disabled, fall through with the !memcg
branch and shrink the root memcg directly.
With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU
is empty, which means the total pages are already below thr. In the
absence of heavy concurrent zswap stores, the loop then safely bails out
via the zswap_total_pages() <= thr check; otherwise, it will resume
shrinking the memcg after processing the reschedule check. For any other
return value from shrink_memcg(), the loop is guaranteed to terminate,
either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
This is a potential performance regression for people using zswap
without memcg that was introduced by the commit in "Fixes".
Link: https://lore.kernel.org/20260806070943.95542-1-jiahao.kernel@gmail.com
Link: https://lore.kernel.org/20260806070943.95542-2-jiahao.kernel@gmail.com
Fixes: a65b0e7607 ("zswap: make shrinking memcg-aware")
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Reported-by: Yosry Ahmed <yosry@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Tejun Heo <tj@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "selftests/mm: Handle unsupported and transient test
conditions", v3.
Several MM selftests report failures when the test environment lacks
an underlying prerequisite, such as fallocate() support, MADV_REMOVE,
local page-cache semantics, or swap.
This series converts those unsupported cases to SKIP while preserving
failures for unexpected errors. It also allows migration tests to retry
transient move_pages() failures.
This patch (of 4):
The tmpfile-backed COW cases allocate a one-page file with fallocate()
before exercising private and shared mappings. When the filesystem
backing tmpfile() does not implement fallocate(), setup fails with
EOPNOTSUPP and no COW behavior is exercised.
This occurs when the temporary directory resides on a filesystem with
limited allocation support, such as NFSv3. Reporting a failure adds noise
because the test prerequisite is absent rather than the COW implementation
being broken.
Report EOPNOTSUPP as a skip. Continue treating every other fallocate()
error as a failure so unexpected setup regressions remain visible.
Link: https://lore.kernel.org/20260727095225.372655-1-usama.anjum@arm.com
Link: https://lore.kernel.org/20260727095225.372655-2-usama.anjum@arm.com
Fixes: f8664f3c4a ("selftests/vm: cow: basic COW tests for non-anonymous pages")
Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
Tested-by: Sarthak Sharma <sarthak.sharma@arm.com>
Acked-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
If the kernel command line includes "debug_guardpage_minorder" without an
equals sign (i.e., no value is provided), the early parameter parser
passes a NULL buf pointer to the setup function.
kstrtouint() does not perform a NULL check on its input and calls directly
into kstrtoull() which dereferences s[0] unconditionally, leading to a
NULL pointer dereference and early boot crash.
Additionally, the error path's pr_err("%s", buf) would also crash with a
NULL format argument.
Link: https://lore.kernel.org/20260806004556.2633049-1-ye.liu@linux.dev
Fixes: c0a32fc5a2 ("mm: more intensive memory corruption debugging")
Signed-off-by: Ye Liu <liuye@kylinos.cn>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Fix five format string issues in show_free_areas() and __show_mem():
1-2. reserved_highatomic and free_highatomic: %luKB -> %lukB
The uppercase "KB" is inconsistent with all other fields in the
same output block and with /proc/meminfo convention.
3. local_pcp: %ukB -> %lukB with explicit (unsigned long) cast
per_cpu_pages.count is int, so K(count) yields int. Using %u
was a signed/unsigned mismatch. Cast to unsigned long and use
%lu for consistency with all other K() usages in the file.
4. total pagecache pages: %ld -> %lu
global_node_page_state() returns unsigned long. Using %ld is a
signedness mismatch caught by gcc -Wformat-signedness.
5. hwpoisoned pages: %lu -> %ld
atomic_long_read() returns long (signed). Using %lu is a
signedness mismatch caught by gcc -Wformat-signedness.
Verified with: make KCFLAGS="-Wformat -Wformat-signedness" mm/show_mem.o
Link: https://lore.kernel.org/20260805021556.1908807-1-ye.liu@linux.dev
Signed-off-by: Ye Liu <liuye@kylinos.cn>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mem_section_usage_size() is only needed by sparsemem implementation code
after commit ae751d567b ("mm/bootmem_info: stop marking
mem_section_usage as MIX_SECTION_INFO"), so keeping the declaration in
mmzone.h now exposes the helper to all mmzone.h users for no reason.
Move the helper to sparse.h so sparse.c and sparse-vmemmap.c can share it
through the internal header. While doing so, calculate the allocation
size with struct_size_t(), which ties the expression to the
pageblock_flags trailing array instead of open-coding the struct header
plus bitmap size.
Link: https://lore.kernel.org/20260805022536.1206575-1-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Currently, userspace has limited visibility into the exact active runtime
state of memory allocation profiling and its page extension compression
('sysctl.vm.mem_profiling={0|1|never}[,compressed]').
While reading the sysctl provides basic on/off status, it is currently
impossible for userspace to natively determine whether page-tag
compression was successfully enabled without scraping dmesg boot logs.
Add a new read-only sysctl representing how compression was configured
at boot time.
Link: https://lore.kernel.org/c795f8089f82841e8a6e00d7ca286da2b23aeb7b.1785950530.git.abhishekbapat@google.com
Signed-off-by: Abhishek Bapat <abhishekbapat@google.com>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Cc: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
read_block_state() calculates nr_pages before taking dev_lock. If the
device is reset and reinitialized with a smaller disksize before lock
acquisition, nr_pages still describes the old table. The subsequent loop
can then call slot_lock() past the end of the newly allocated table.
Read disksize after acquiring dev_lock and checking that the device is
initialized. The read lock then keeps the table and its bound stable for
the duration of the scan.
Link: https://lore.kernel.org/20260804065919.3970386-3-xialonglong2025@163.com
Fixes: c0265342bf ("zram: introduce zram memory tracking")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "zram: fix stale scan bounds after reinitialization".
Both writeback_store() and read_block_state() derive their table scan
bounds from zram->disksize before acquiring dev_lock. If the device is
reset and reinitialized with a smaller disksize between that read and lock
acquisition, the bound can describe the old table while the scan operates
on the new one. This can lead to out-of-bounds slot accesses.
Move both bound calculations under dev_lock so each bound remains
consistent with the table throughout its scan. Keep the fixes separate
because the affected interfaces originate from different commits and can
be backported independently.
This patch (of 2):
writeback_store() calculates the table scan bounds before taking dev_lock.
A reset followed by reconfiguration with a smaller disksize can therefore
replace zram->table while writeback_store() is waiting for the lock. Once
it acquires the lock, it sees an initialized device but scans the new
table using the old upper bound, resulting in an out-of-bounds access.
Calculate the number of pages while holding dev_lock so the scan bound
matches the table protected by the lock.
Link: https://lore.kernel.org/20260804065919.3970386-1-xialonglong2025@163.com
Link: https://lore.kernel.org/20260804065919.3970386-2-xialonglong2025@163.com
Fixes: a939888ec3 ("zram: support idle/huge page writeback")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Minchan Kim <minchan@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "zram: fix zstd error paths and add parameter validation", v6,
Patch 1 removes zstd_release_params() from both zstd_create() and
zstd_setup_params() error paths -- the former is a layering violation
in a per-CPU callback, the latter is redundant as zcomp_init() already
calls release_params() on setup failure.
Patch 2 rejects zero-size dictionaries and prints distinct error
messages for sz < 0 (returns the original error code) and sz == 0
("empty file"). Currently errors are silently swallowed.
Patch 3 adds pr_fmt to each backend file so that pr_err() messages
are auto-prefixed with the algorithm name.
Patch 4 validates dict and level parameters in each backend's
.setup_params(), rejecting unsupported combinations and out-of-range
levels.
Patch 5 resets per-priority params on algorithm change before init.
This patch (of 5):
zstd_setup_params() creates global cdict and ddict stored in
params->drv_data, shared across all per-CPU contexts. The per-CPU
zstd_create() error path called zstd_release_params(), which freed those
globally-shared objects. This is a layering violation: a per-CPU callback
should only clean up its own context, not release resources owned by the
compression lifecycle.
zstd_setup_params() called zstd_release_params() on its own error path as
well, but zcomp_init() already calls release_params() when setup fails, so
this is redundant.
Remove zstd_release_params() from both error paths.
Link: https://lore.kernel.org/20260804093841.67920-1-haoqinhuang7@gmail.com
Link: https://lore.kernel.org/20260804093841.67920-2-haoqinhuang7@gmail.com
Signed-off-by: Haoqin Huang <haoqinhuang@tencent.com>
Signed-off-by: Rongwei Wang <zigiwang@tencent.com>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Tested-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: David Sterba <dsterba@suse.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Nick Terrell <terrelln@fb.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
