Patch series "selftests/damon: misc fixes for test bugs", v3.
This series fixes several bugs in the DAMON selftests. Most are trivial
but makes test output wrong or even silently pass the one test case for
'avail_operation' file existence check.
Patch 1 fixes mutable default arguments in DamonCtx.__init__() that cause
state to leak between test instances.
Patch 2 fixes wrong operator precedence and join TypeError in
damos_tried_regions.py.
Patch 3 fixes several wrong strings that produce dead elif branches,
skipped file existence checks, and broken dict key lookups.
This patch (of 3):
DamonCtx.__init__() uses mutable default values for monitoring_attrs,
targets, and schemes. In Python these are evaluated once at function
definition time, so multiple DamonCtx instances can unintentionally share
the same lists and DamonAttrs instance.
Replace the mutable defaults with None sentinels and initialize the
objects when needed.
Link: https://lore.kernel.org/20260629144648.134092-1-sj@kernel.org
Link: https://lore.kernel.org/20260601032314.424013-2-kunwu.chan@linux.dev
Link: https://lore.kernel.org/20260629144648.134092-2-sj@kernel.org
Co-developed-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Kunwu Chan <chentao@kylinos.cn>
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Kunwu Chan <chentao@kylinos.cn>
Cc: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: cleanup clear_not_present_full_ptes()", v2.
While doing some review, I stumbled over clear_not_present_full_ptes() and
concluded that it needs some love.
Let's remove pte_clear_not_present_full() and cleanup
clear_not_present_full_ptes(), renaming it to clear_non_present_ptes().
This patch (of 3):
On sparc64, pte_clear_not_present_full() nowadays does a simple
__set_pte_at(). In __set_pte_at() -> maybe_tlb_batch_add(), we check
pte_accessible() to see whether to call tlb_batch_add().
However, non-present PTEs are surely not accessible, so tlb_batch_add() is
never called and the "full" parameter is irrelevant.
Let's drop the helper and just let common code do a pte_clear().
pte_clear() on sparc64 maps to set_pte_at()->set_ptes()->__set_pte_at()
... so it ends up calling the same function, just with "full=0".
Given that "full" is irrelevant, there is no change.
We added pte_clear_not_present_full() for sparc64 in commit 90f08e399d
("sparc: mmu_gather rework"), and I suspect that it was already not
required back then.
Link: https://lore.kernel.org/20260629-clear_not_present_full_ptes-v2-0-96089871a1e7@kernel.org
Link: https://lore.kernel.org/20260629-clear_not_present_full_ptes-v2-1-96089871a1e7@kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Andreas Larsson <andreas@gaisler.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Jann Horn <jannh@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Oscar Salvador (SUSE) <osalvador@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
the issue for tried region directories in the error handling path of
damon_sysfs_populate_region_dir() by adding a kobject_del() call.
Link: https://lore.kernel.org/20260628220121.97360-12-sj@kernel.org
Fixes: b574a82d10 ("mm/damon/sysfs-schemes: implement tried_regions/<r>/probes/")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
the issue for data attribute probe filter directories in the error
handling path of damon_sysfs_probes_add_dirs() by adding a kobject_del()
call.
Link: https://lore.kernel.org/20260628220121.97360-11-sj@kernel.org
Fixes: af7cb41af9 ("mm/damon/sysfs: implement filters directory")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
those issues for data attribute probe filter directories by adding
kobject_del() calls.
Link: https://lore.kernel.org/20260628220121.97360-10-sj@kernel.org
Fixes: 82e66aef7714 ("mm/damon/sysfs: implement filter dir")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
those issues for data attribute probe directories by adding kobject_del()
calls.
Link: https://lore.kernel.org/20260628220121.97360-9-sj@kernel.org
Fixes: bf3ea3d30880 ("mm/damon/sysfs: implement probe dir")
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
those issues for scheme action destination directories by adding
kobject_del() calls.
Link: https://lore.kernel.org/20260628220121.97360-8-sj@kernel.org
Fixes: 2cd0bf85a2 ("mm/damon/sysfs-schemes: implement DAMOS action destinations directory")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 6.17.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
those issues for the normal creation path of region directories and the
error path of target directories, by adding kobject_del() calls.
Link: https://lore.kernel.org/20260628220121.97360-3-sj@kernel.org
Fixes: 2031b14ea7 ("mm/damon/sysfs: support the physical address space monitoring")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 5.18.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon/sysfs: kobject_del() directories that users can
create/remove".
DAMON sysfs interface allows users to create and remove arbitrary number
of directories on sysfs, using a few files having 'nr_' prefix. For
example, 'nr_kdamonds'. When the user writes a number 'N' to the files,
directories having name starting from '0' to 'N - 1' are created in the
same directory. The pre-existing number-named directories are removed
before creating the new directories.
For the removal of the existing directories, DAMON sysfs interface use
only kobject_put(). Because DAMON sysfs interface is the only kernel
component that manages the directories, there is no problem in normal
situations. However, if CONFIG_DEBUG_KOBJECT_RELEASE is enabled, the
removal of dirs are delayed. Let's suppose a user writes a non-zero
number to the 'nr_*' files while there are pre-existing number-named
directories, on the config enabled kernel. DAMON sysfs interface
decreases the reference counts of the existing directories and immediately
creates new directories. Because the removal of the sysfs directories is
delayed, it shows some pre-existing directories of the same names when it
tries to create the new directories, and fails.
For example, the issue can be triggered like below:
# grep DEBUG_KOBJECT_RELEASE /boot/config-$(uname -r)
CONFIG_DEBUG_KOBJECT_RELEASE=y
# ls
nr_kdamonds
# echo 1 > nr_kdamonds
# echo 1 > nr_kdamonds
bash: echo: write error: File exists
# dmesg
[...]
[ 300.880458] kobject: kobject_add_internal failed for 0 with -EEXIST, don't try to register things with the same name in the same directory.
[...]
Some of the error handling paths of the directories also lack the
kobject_del() call. If the user uses nr_* file right after the errors,
similar issues can happen.
This doesn't cause catastrophic issues like kernel panics or memory
corruptions. Users can work around by removing all directories first
(write 0 to the nr_* files) and then create new directories after
confirming the old directories are gone. But, this is definitely a bug
that causes a bad user experience.
Fix the issues by calling kobject_del() before creating new directories.
This patch (of 11)
On CONFIG_DEBUG_KOBJECT_RELEASE enabled kernel, lack of kobject_del()
could cause directories creation failures due to the name conflicts. Fix
those issues for normal creation paths of target, context and kdamond
directories, and error paths of context and kdamond directories by adding
kobject_del() calls.
Note that this fix for target directories is not complete since it has a
similar issue in the damon_sysfs_targets_add_dirs() error path. Because
the normal path issue and the error path issue are introduced by different
commits, this commit is fixing only the normal path issue. A commit for
the error path will be added next.
Link: https://lore.kernel.org/20260628220121.97360-1-sj@kernel.org
Link: https://lore.kernel.org/20260628220121.97360-2-sj@kernel.org
Fixes: c951cd3b89 ("mm/damon: implement a minimal stub for sysfs-based DAMON interface")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 5.18.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_sample_prcl_start() calls damon_call() right after damon_start() is
succeeded. The kdamond that has started by the damon_start() could be
terminated by itself before or in the middle of the damon_call()
execution. There could be multiple reasons for such a stop including
monitoring target process termination and kdamond_fn() internal memory
allocation failures. In the case, damon_call() will fail and return an
error without cleaning up the DAMON context object. The
damon_sample_prcl_start() caller assumes it would clean up the object,
though. When the user requests to start DAMON again,
damon_sample_prcl_start() is called again, allocates a new DAMON context
object and overwrites the pointer for the previous object. As a result,
the previous context object is leaked.
Safely stop the kdamond and deallocate the context object when the failure
is returned. Note that the kdamond should be stopped first, because
damon_call() failure means not complete termination of the kdamond but
only the fact that the termination process has started.
The user impact shouldn't be that significant because the race is not easy
to happen, and only up to one DAMON context object can be leaked per race.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-7-sj@kernel.org
Link: https://lore.kernel.org/20260610035214.4850-1-sj@kernel.org [1]
Fixes: a6c33f1054 ("samples/damon/prcl: use damon_call() repeat mode instead of damon_callback")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.17.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_sample_wsse_start() calls damon_call() right after damon_start() is
succeeded. The kdamond that has started by the damon_start() could be
terminated by itself before or in the middle of the damon_call()
execution. There could be multiple reasons for such a stop including
monitoring target process termination and kdamond_fn() internal memory
allocation failures. In the case, damon_call() will fail and return an
error without cleaning up the DAMON context object. The
damon_sample_wsse_start() caller assumes it would clean up the object,
though. When the user requests to start DAMON again,
damon_sample_wsse_start() is called again, allocates a new DAMON context
object and overwrites the pointer for the previous object. As a result,
the previous context object is leaked.
Safely stop the kdamond and deallocate the context object when the failure
is returned. Note that the kdamond should be stopped first, because
damon_call() failure means not complete termination of the kdamond but
only the fact that the termination process has started.
The user impact shouldn't be that significant because the race is not easy
to happen, and only up to one DAMON context object can be leaked per race.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-6-sj@kernel.org
Link: https://lore.kernel.org/20260610034828.4632-1-sj@kernel.org [1]
Fixes: cc9c1b8c20 ("samples/damon/wsse: use damon_call() repeat mode instead of damon_callback")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.17.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_sample_mtier_stop() assumes its damon_stop() call will always
successfully stops the two DAMON contexts. Hence it deallocates the two
DAMON contexts after the damon_stop() call. However, if a given context
is already stopped, damon_stop() fails and returns an error while letting
the DAMON contexts that have not yet stopped keep running. This kind of
unexpected early DAMON context stops could happen due to memory allocation
failures in kdamond_fn(). Because damon_sample_mtier_stop() just
deallocates all DAMON contexts with damon_target and damon_region objects
that are linked to the contexts, the execution of the unstopped DAMON
context (kdamond) ends up using the memory that freed (use-after-free).
Fix the issue by separating the damon_stop() to be invoked per context.
Note that DAMON_SYSFS also allows multiple DAMON contexts execution. But,
it calls damon_stop() for each context one by one. Hence this issue is
only in mtier.
For the long term, it would be better to refactor damon_stop() to always
ensure stopping all contexts regardless of the failures in the middle.
Make this fix in the current way, though, to keep it simple and easy to
backport. I will do the refactoring later.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-5-sj@kernel.org
Link: https://lore.kernel.org/20260609014219.3013-1-sj@kernel.org [1]
Fixes: 82a08bde3c ("samples/damon: implement a DAMON module for memory tiering")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.16.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_sample_mtier_start() callers assume it will clean up resources when
it fails. And the function does the cleanup for context buildup failures.
However, it is not doing the cleanup for damon_start() failure.
As a result, when damon_start() fails, it could leak the memory for DAMON
context. Also, if damon_start() fails for only the second context, the
first context will indefinitely run, and avoid starting other DAMON
contexts since it is running in the exclusive mode. Stop possibly started
DAMON context and free the contexts in case of the failure to fix the
issues.
Note that the issue can reliably be reproduced because the module calls
damon_start() in the exclusive mode. For example,
$ sudo damo start
$ echo Y | sudo tee /sys/module/damon_sample_mtier/parameters/enabled
$ sudo cat /proc/allocinfo | grep damon_new_ctx
Because the first command is running another DAMON instance, the second
command fails the damon_start() call because the new DAMON instance cannot
exclusively run. And without this fix, by repeating the second and the
third commands above, we can show the memory consumption is only
increasing due to the leaks. It requires the sudo permission though.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-4-sj@kernel.org
Link: https://lore.kernel.org/20260608112455.274231F00893@smtp.kernel.org [1]
Fixes: 82a08bde3c ("samples/damon: implement a DAMON module for memory tiering")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.16.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_sample_prcl_start() callers assume it will clean up resources when
it fails. And the function does the cleanup for context buildup failures.
However, it is not doing the cleanup for damon_start() failure. As a
result, when damon_start() fails, it leaks the memory for DAMON context.
Free the context in case of the failure to fix the issues.
Note that the issue can reliably be reproduced because the module calls
damon_start() in the exclusive mode. For example,
$ sudo damo start
$ echo $$ | sudo tee /sys/module/damon_sample_prcl/parameters/target_pid
$ echo Y | sudo tee /sys/module/damon_sample_prcl/parameters/enabled
$ sudo cat /proc/allocinfo | grep damon_new_ctx
Because the first command is running another DAMON instance, the third
command fails the damon_start() call because the new DAMON instance cannot
exclusively run. And without this fix, by repeating the third and the
fourth commands above, we can show the memory consumption is only
increasing due to the leaks. It requires the sudo permission though.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-3-sj@kernel.org
Link: https://lore.kernel.org/20260609145814.70163-1-sj@kernel.org [1]
Fixes: 2aca254620 ("samples/damon: introduce a skeleton of a smaple DAMON module for proactive reclamation")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.14.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "samples/damon: handle damon_{start,stop}() failures".
All DAMON sample modules are not correctly handling failures from
damon_start(). Among those, mtier also has an additional problem for
handling of damon_stop() failures. wsse and prcl also have a problem in
their damon_call() failure handling. As a result, memory leaks, next
DAMON operation disruptions, and use-after-free can happen. Fix those.
Note that only the damon_start() failure caused issues can reliably be
reproduced. Reproducing those issues require the admin permission,
though.
This patch (of 6):
damon_sample_wsse_start() callers assume it will clean up resources when
it fails. And the function does the cleanup for context buildup failures.
However, it is not doing the cleanup for damon_start() failure. As a
result, when damon_start() fails, it leaks the memory for DAMON context.
Free the context in case of the failure to fix the issues.
Note that the issue can reliably be reproduced because the module calls
damon_start() in the exclusive mode. For example,
$ sudo damo start
$ echo $$ | sudo tee /sys/module/damon_sample_wsse/parameters/target_pid
$ echo Y | sudo tee /sys/module/damon_sample_wsse/parameters/enabled
$ sudo cat /proc/allocinfo | grep damon_new_ctx
Because the first command is running another DAMON instance, the third
command fails the damon_start() call because the new DAMON instance cannot
exclusively run. And without this fix, by repeating the third and the
fourth commands above, we can show the memory consumption is only
increasing due to the leaks. It requires the sudo permission though.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260628215447.96166-2-sj@kernel.org
Link: https://lore.kernel.org/20260609145814.70163-1-sj@kernel.org [1]
Fixes: b757c6cfc6 ("samples/damon/wsse: start and stop DAMON as the user requests")
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: Zenghui Yu <zenghui.yu@linux.dev>
Cc: <stable@vger.kernel.org> # 6.14.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
swap_cluster_readahead() and swap_vma_readahead() end the readahead loop
with an explicit lru_add_drain() call. That drain is a leftover from
2.6.12 era code and serves no functional purpose for the callers:
- do_swap_page() ignores LRU residency for the readahead folios;
it only needs the target folio it called swapin_readahead() for,
and if the write-fault path needs the target folio on the LRU to count
references accurately, it runs its own lru_add_drain() at the
wp_can_reuse_anon_folio() and do_swap_page() sites.
- shmem_swapin_cluster() immediately locks the returned folio, waits
for writeback, then operates on it - LRU residency of either the target
or the readahead folios is irrelevant.
- try_to_unuse() likewise locks the folio and calls unuse_pte() without
depending on LRU presence.
Folios newly added to the swap cache by the readahead loop sit in the
per-CPU LRU folio_batch and will be drained naturally as the batch fills
(FOLIO_BATCH_SIZE),by the next reclaim/compaction lru_add_drain_all() and
so on. The unconditional drain only synchronously flushes a partial batch
and forces contention on lruvec_lock.
On a 176-CPU production host running a memory-pressured workload, this
path was observed to call folio_batch_move_lru() from
swap_cluster_readahead() ~28K/min, a very large source of LRU lock
traffic.
This is a direct continuation of the cleanup started in commit
1aa43598c0 ("mm: remove unnecessary calls to lru_add_drain") which
removed the equivalent drain from free_pages_and_swap_cache() with the
same rationale. A detailed reasoning for this is present in [1].
Remove both drains.
Link: https://lore.kernel.org/20260608143242.2869392-1-usama.arif@linux.dev
Link: https://lore.kernel.org/all/dca2824e8e88e826c6b260a831d79089b5b9c79d.camel@surriel.com/T/#u [1]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Rik van Riel <riel@surriel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The main thread function has recently grown to the point of exceeding
stack frame size warning limits in some configurations. This is what I
hit on s390 with clang and CONFIG_KASAN:
mm/damon/core.c:3440:31: error: stack frame size (1352) exceeds limit (1280) in 'kdamond_fn' [-Werror,-Wframe-larger-than]
3440 | static int kdamond_fn(struct damon_ctx *ctx)
The largest stack usage here is inside of the kdamond_tune_intervals(), so
by marking that one as noinline_for_stack, the functions individually stay
below the warning limit, though kdamond_fn() itself still uses hundreds of
kilobytes for some reason.
Link: https://lore.kernel.org/20260611125704.3386176-1-arnd@kernel.org
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Bill Wendling <morbo@google.com>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Quanmin Yan <yanquanmin1@huawei.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Currently in zs_free(), the class->lock is held until the zspage is
completely freed and the counters are updated. However, freeing pages
back to the buddy allocator requires acquiring the zone lock.
Under heavy memory pressure, zone lock contention can be severe. When
this happens, the CPU holding the class->lock will stall waiting for the
zone lock, thereby blocking all other CPUs attempting to acquire the same
class->lock.
This patch shrinks the critical section of the class->lock to reduce lock
contention. By moving the actual page freeing process outside the
class->lock, we can improve the concurrency performance of zs_free().
Testing on the RADXA O6 platform shows that with 12 CPUs concurrently
performing zs_free() operations, the execution time is reduced by 20%.
Link: https://lore.kernel.org/20260626015003.2965881-4-haowenchao22@gmail.com
Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
With class_idx encoded in obj, zs_free() can locate the size_class without
holding pool->lock on 64-bit systems. Page migration also takes
class->lock and only rewrites the PFN field of obj, so:
1. read obj locklessly,
2. lock the size_class derived from obj's class_idx,
3. re-read obj under class->lock to get a stable PFN.
This eliminates the rwlock read-side cacheline bouncing between zs_free()
and migration/compaction on multi-core systems.
Annotate handle_to_obj()/record_obj() with READ_ONCE()/WRITE_ONCE() to
prevent load/store tearing on the lockless read path and silence KCSAN
data race reports.
When ZS_OBJ_CLASS_BITS == 0 (32-bit, or 64-bit with obj too narrow to hold
class_idx), zs_free() keeps pool->lock.
[akpm@linux-foundation.org: build fix]
[akpm@linux-foundation.org: fix obj_to_class_idx() warning yet again]
[baohua@kernel.org: update the comment about pool lock]
Link: https://lore.kernel.org/20260725035733.53241-1-baohua@kernel.org
Link: https://lore.kernel.org/20260626015003.2965881-3-haowenchao22@gmail.com
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Xueyuan Chen <xueyuan.chen21@gmail.com>
Cc: kernel test robot <lkp@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/zsmalloc: reduce lock contention in zs_free()", v6.
This series reduces lock contention in zs_free(), which dominates the
unmap path under memory pressure on Android (LMK kills) and on x86 servers
running zswap-heavy workloads.
The current zs_free() takes pool->lock (rwlock, read side) just to look up
the size_class for a handle, then takes class->lock and holds it across
__free_zspage() which can call into the buddy allocator and acquire
zone->lock. Two costs follow:
* pool->lock reader-counter cacheline bouncing among concurrent
zs_free() callers.
* class->lock held across folio_put(), so any zone->lock wait
fans out to every other zs_free() on the same class.
The series tackles both:
Patch 1: encode size_class index into obj alongside PFN and obj_idx,
so zs_free() can locate the class without pool->lock.
Patch 2: drop pool->lock from zs_free() on 64-bit; 32-bit unchanged.
Patch 3: move zspage page-freeing out of class->lock.
Patch 4: document the three free_zspage helper variants that result
from the split in patch 3.
Performance results:
Test: each process independently mmap 256MB, write data, madvise
MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap.
Raspberry Pi 4B (4-core ARM64 Cortex-A72):
mode Base Patched Speedup
single 59.0ms 56.0ms 1.05x
multi 2p 94.6ms 66.7ms 1.42x
multi 4p 202.9ms 110.6ms 1.83x
x86 (20-core Intel i7-12700, 16 concurrent processes):
mode Base Patched Speedup
single 11.7ms 9.8ms 1.19x
multi 2p 24.1ms 17.2ms 1.40x
multi 4p 63.0ms 45.3ms 1.39x
This patch (of 4):
Encode the size_class index (class_idx) into the obj value so that
zs_free() can determine the correct size_class without dereferencing the
handle->obj->PFN->zpdesc->zspage->class chain under pool->lock. class_idx
is invariant across page migration (only PFN is rewritten), so a lockless
read of obj always yields a valid class_idx.
Where obj has more bits below the PFN field than obj_idx alone needs,
split that space into class_idx and obj_idx subfields:
|<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
+-----------------+-------------------------+-----------------------+
| PFN | class_idx | obj_idx |
+-----------------+-------------------------+-----------------------+
MSB ^ LSB
|
+-- ZS_OBJ_PFN_SHIFT
The macro layout changes as follows:
Before After Meaning
---------------- ------------------ ----------------------------
OBJ_INDEX_BITS ZS_OBJ_IDX_BITS width of obj_idx subfield
OBJ_INDEX_MASK ZS_OBJ_IDX_MASK mask of obj_idx subfield
(n/a) ZS_OBJ_CLASS_BITS width of class_idx subfield
(n/a) ZS_OBJ_CLASS_MASK mask of class_idx subfield
(n/a) ZS_OBJ_PFN_SHIFT bit offset of PFN in obj
ZS_OBJ_CLASS_BITS folds to 0 (and the layout collapses to [PFN | obj_idx])
when obj has no spare bits, i.e. on 32-bit or on 64-bit fallback paths
where MAX_POSSIBLE_PHYSMEM_BITS == BITS_PER_LONG (e.g. UML); zs_free()
then falls back to pool->lock.
[akpm@linux-foundation.org: fix obj_to_class_idx() defined but not used, remove duplicated #ifdef]
Link: https://lore.kernel.org/20260626015003.2965881-1-haowenchao22@gmail.com
Link: https://lore.kernel.org/20260626015003.2965881-2-haowenchao22@gmail.com
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Xueyuan Chen <xueyuan.chen21@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Correct some kernel-doc issues in radix-tree.h:
- use "DOC:" so that a kernel-doc comment is parsed correctly
(or we could just use "/*" for that comment)
- add one function parameter description
- add one function parameter name inside the prototype
to fix these warnings:
Warning: include/linux/radix-tree.h:164 Incorrect use of kernel-doc format:
* radix_tree_deref_slot - dereference a slot
Warning: include/linux/radix-tree.h:177 cannot understand function
prototype: '* @slot: slot pointer, returned by radix_tree_lookup_slot
Warning: include/linux/radix-tree.h:192 function parameter 'treelock'
not described in 'radix_tree_deref_slot_protected'
Warning: include/linux/radix-tree.h:309 function parameter '' not
described in 'radix_tree_next_chunk'
Link: https://lore.kernel.org/20260627185859.1632928-1-rdunlap@infradead.org
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
kmemleak_scan() scans the per-cpu sections, the struct page ranges and the
task stacks in sequence. Each loop now bails out once scan_block()
reports the scan was interrupted, but the later phases are still entered
and only bail on their first scan_block() call.
Jump straight to the gray list scan once a phase reports an interrupted
scan, so the remaining scan phases are not entered at all. This does not
change the scan results, it only avoids the pointless re-entry.
Link: https://lore.kernel.org/20260626-kmemleak_improve-v1-1-d40c7616f64f@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: Oleg Nesterov <oleg@redhat.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The early boot gigantic hugepage allocation helpers currently mix
allocation with huge_bootmem_page setup, and leave part of the
initialization flow in architecture code.
Refactor the interface to return the allocated huge page pointer and move
the huge_bootmem_page setup into the generic hugetlb code. This makes the
architecture-specific paths focus only on finding memory, while the common
code handles node placement and early page metadata setup in one place.
This also lets powerpc benefit from memblock_reserved_mark_noinit(), which
it did not enable before.
In addition, upcoming cross-zone validation for boot-time gigantic hugetlb
reservation is common logic. With this refactoring, that logic can stay
in the generic code instead of being duplicated in architecture-specific
paths.
Link: https://lore.kernel.org/20260612035903.2468601-14-songmuchun@bytedance.com
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Oscar Salvador (SUSE) <osalvador@suse.de>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oscar Salvador (SUSE) <osalvador@kernel.org>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
