damon_stop() has no reason to fail. It returns an error code only for
possible future changes that can make it fail. Such a change has not been
made yet, and this only makes the error handling complicated and
confusing. Ensure it returns no error.
Link: https://lore.kernel.org/20260706140628.87414-4-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon: refactor damon_{start,stop,commit}() for simple
error handling".
damon_start() and damon_stop() could leave a few of requested DAMON
contexts running when they return an error. damon_commit() failure stops
the DAMON context, but in an asynchronous way. Callers should stop the
left-over DAMON contexts. It is easy to make mistakes, and indeed a few
bugs from such mistakes were found and fixed.
Refactor the core API functions to guarantee contexts are completely
stopped under failures. Remove the caller side error handlers that are no
longer needed due to the refactoring.
Patches Sequence
================
Patch 1 refactors damon_start() to ensure all contexts are stopped for
failures. Patch 2 updates unnecessary damon_start() error handling from
mtier sample module.
Patch 3 refactors damon_stop() to always success. Patches 4-6 updates
callers (damon_{sysfs,reclaim lru_sort}) to ignore the return value.
Patch 7 update damon_stop() return value to void. Patch 8 simplifies
damon_stop() error handling in mtier sample module.
Patch 9 refactors damon_call() to return errors only after the context
is completely stopped. Patches 10 and 11 remove unnecessary error
handlings from callers (wsse and prcl sample modules).
This patch (of 11):
When multiple contexts are passed to damon_start(), the function starts
the contexts one by one. If any of the operations fails, it immediately
returns an error. Contexts that successfully started before the failure
keep running. The caller should catch this and stop the contexts. It is
complicated and easy to make mistakes. Stop all contexts in damon_start()
under the failures.
Link: https://lore.kernel.org/20260706140628.87414-1-sj@kernel.org
Link: https://lore.kernel.org/20260706140628.87414-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
In the ensure_file() function, the "not_exist" code path checks whether
$dir exists as a regular file. However, the intent is to verify that the
target file ($file) does not exist, not the $dir. Testing $dir makes the
existence check effectively useless -- it tests the wrong path and thus
never catches the case where the file is unexpectedly present.
Replace $dir with $file so the not_exist verification targets the correct
path.
Link: https://lore.kernel.org/20260706134305.5224-1-a929244872@163.com
Signed-off-by: wang wei <a929244872@163.com>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a macro that
only gets defined by the Makefiles in the kernel. This can be very
confusing when switching between userspace and kernelspace coding, or when
dealing with uapi headers that rather should use __ASSEMBLER__ instead.
So let's standardize now on the __ASSEMBLER__ macro that is provided by
the compilers.
This is a completely mechanical patch (done with a simple "sed -i"
statement.
Link: https://lore.kernel.org/20260619154704.41293-1-thuth@redhat.com
Signed-off-by: Thomas Huth <thuth@redhat.com>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Add a sysctl panic_on_unrecoverable_memory_failure (disabled by default)
that triggers a kernel panic when memory_failure() encounters pages that
cannot be recovered. This provides a clean crash with useful debug
information rather than allowing silent data corruption or a delayed crash
at an unrelated code path.
Panic eligibility is intentionally narrow: only MF_MSG_KERNEL with result
== MF_IGNORED panics. After the previous patch, MF_MSG_KERNEL covers
PG_reserved pages and the kernel-owned pages promoted from
get_hwpoison_page() via -ENOTRECOVERABLE (slab, page tables,
large-kmalloc).
All other action types are excluded:
- MF_MSG_GET_HWPOISON and MF_MSG_KERNEL_HIGH_ORDER can be reached by
transient refcount races with the page allocator (an in-flight buddy
allocation has refcount 0 and is no longer on the buddy free list,
briefly), and panicking on them would risk killing the box for what
is actually a recoverable userspace page.
- MF_MSG_UNKNOWN means identify_page_state() could not classify the
page; that is precisely the wrong basis for a panic decision.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-4-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The previous patch teaches get_any_page() to return -ENOTRECOVERABLE for
stable unhandlable kernel pages (PG_reserved, slab, page tables,
large-kmalloc). memory_failure() still folds every negative return into
MF_MSG_GET_HWPOISON, so callers that want to react to the unrecoverable
cases (a panic option, smarter logging) cannot tell them apart from
transient page-allocator races.
Turn the post-call branch into a switch over the get_hwpoison_page()
return code: map -ENOTRECOVERABLE to MF_MSG_KERNEL and any other negative
return to MF_MSG_GET_HWPOISON. case 0 keeps the existing free-buddy /
kernel-high-order handling and case 1 falls through to the rest of
memory_failure() unchanged.
The MF_MSG_KERNEL label and tracepoint string are kept as "reserved kernel
page" to avoid breaking userspace tools that match on those literals; the
enum value still adequately tags the failure even though it now also
covers slab, page tables and large-kmalloc pages.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-3-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
get_any_page() collapses every HWPoisonHandlable() rejection into a
single -EIO via the __get_hwpoison_page() -> -EBUSY -> shake_page()
-> retry path. That is correct for the transient case (a userspace
folio briefly off LRU during migration or compaction, which a later
shake can drag back), but wrong for stable kernel-owned pages: slab,
page-table, large-kmalloc and PG_reserved pages will never become
HWPoisonHandlable(), so the retry loop is wasted work and the final
-EIO loses the "this is structurally unrecoverable" information.
memory_failure() then maps -EIO into MF_MSG_GET_HWPOISON, which the
panic-on-unrecoverable sysctl deliberately does not act on.
Introduce is_kernel_owned_page(), a small predicate that positively
identifies pages the hwpoison handler cannot recover from:
is_kernel_owned_page(p) :=
PageReserved(p) ||
PageSlab(head) || PageTable(head) || PageLargeKmalloc(head)
where head = compound_head(p).
PG_reserved is a per-page flag (PF_NO_COMPOUND) and is tested on the
page directly. The slab, page-table and large-kmalloc page-type bits
are only stored on the head page, so those tests resolve the compound
head first, then re-read compound_head(page) afterwards: a concurrent
split or compound free that moves head invalidates the just-read flags
and the loop retries. The lookup still takes no refcount, mirroring
the rest of get_any_page(); the recheck closes the common split race,
and a residual free->alloc->free in the same window can only mis-tag
a genuinely poisoned page, never reclassify a handlable one.
No MF_SOFT_OFFLINE / page_has_movable_ops() opt-out is needed: a
movable_ops page is always PageOffline or PageZsmalloc, whose page_type is
mutually exclusive with slab, page-table and large-kmalloc, and it never
carries PG_reserved, so it can never match any of the checks above.
The list is intentionally not exhaustive. vmalloc and kernel-stack pages,
for example, do not carry a page_type bit and would need a different
oracle; they keep going through the existing retry path unchanged. This
is the smallest set we can identify with certainty by page type.
Wire the helper into the top of get_any_page() to short-circuit those
pages before the retry loop runs. On a hit, drop the caller's
MF_COUNT_INCREASED reference (if any) and return -ENOTRECOVERABLE straight
away. Pages outside the helper's positive list still take the existing
retry path and return -EIO, leaving operator-visible behaviour for those
cases unchanged.
Extend the unhandlable-page pr_err() to fire for either errno and update
the get_hwpoison_page() kerneldoc to document the new return.
memory_failure() still folds every negative return into
MF_MSG_GET_HWPOISON via its existing "else if (res < 0)" branch, so this
patch on its own only changes the errno that soft_offline_page() can
propagate to its callers. A follow-up wires -ENOTRECOVERABLE through
memory_failure() and reports MF_MSG_KERNEL for the unrecoverable cases,
which is what the panic_on_unrecoverable_memory_failure sysctl observes.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-2-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Suggested-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/memory-failure: add panic option for unrecoverable
pages", v10.
A multi-bit ECC error on a kernel-owned page that the memory failure
handler cannot recover is currently swallowed: PG_hwpoison is set, the
event is logged, and the kernel keeps running. The corrupted memory
remains accessible to the kernel and either drives silent data corruption
or surfaces seconds-to-minutes later as an apparently unrelated crash. In
a large fleet that delayed, unattributable crash turns into significant
engineering effort to root-cause; in a kdump configuration, by the time
the crash happens the original error context (faulting PFN, MCE/GHES
record, page state) is long gone.
This series adds an opt-in sysctl,
vm.panic_on_unrecoverable_memory_failure, that converts an unrecoverable
kernel-page hwpoison event into an immediate panic with a clean
dmesg/vmcore that still contains the original failure context. The
default is disabled so existing workloads see no change.
There is a selftest that test different cases, and I tested it using
the following variants:
┌─────────┬──────────┬───────────────────────────────────────────────────────────┐
│ Variant │ PFN │ Result │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ rodata │ 0x2600 │ Panic with "Memory failure: 0x2600: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ slab │ 0x100032 │ Panic with "Memory failure: 0x100032: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ pgtable │ 0x100000 │ Panic with "Memory failure: 0x100000: unrecoverable page" │
└─────────┴──────────┴───────────────────────────────────────────────────────────┘
Each one shows the same call trace, exactly the path the series builds:
hard_offline_page_store
→ memory_failure
→ action_result
→ panic("Memory failure: %#lx: unrecoverable page")
This patch (of 5):
The first entry of error_states[],
{ reserved, reserved, MF_MSG_KERNEL, me_kernel },
is unreachable. identify_page_state() has two callers, and neither
one can dispatch a PG_reserved page to me_kernel():
* memory_failure() reaches identify_page_state() only after
get_hwpoison_page() returned 1. get_any_page() reaches that
return only via __get_hwpoison_page(), which only takes a
refcount when the page is HWPoisonHandlable().
HWPoisonHandlable() is an allowlist for LRU, free-buddy, and
(for soft-offline) movable_ops pages -- PG_reserved pages do
not satisfy any of these, so they fail with -EBUSY/-EIO long
before identify_page_state() runs.
* try_memory_failure_hugetlb() reaches identify_page_state() only
via the MF_HUGETLB_IN_USED branch, where the page is necessarily
a hugetlb folio. hugetlb folios don't carry PG_reserved at that
point: hugetlb_folio_init_vmemmap() calls __folio_clear_reserved()
during init, so the reserved entry would not match even if it
were still present.
me_kernel() never executes and the entry exists only to be matched
against by code that cannot see it.
Drop the entry, the me_kernel() helper, and the now-unused
"reserved" macro. Leave the MF_MSG_KERNEL enum value in place: it
remains part of the tracepoint and pr_err() string tables, and
follow-on work to classify unrecoverable kernel pages can reuse it
without churning the user-visible enum.
No functional change.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-0-c6ed5b62eea2@debian.org
Link: https://lore.kernel.org/20260630-ecc_panic-v10-1-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The legacy memory.limit_in_bytes and memory.memsw.limit_in_bytes writers
retry page_counter_set_max() by reclaiming synchronously in the writer
context. memory.force_empty similarly loops in synchronous reclaim until
the cgroup is empty or reclaim stops making progress.
These writes hold a kernfs active reference on the file. If cgroup
removal starts in parallel, the remover sets CSS_DYING and then waits in
kernfs_drain() under cgroup_mutex for the active reference to drain.
Continuing reclaim after the memcg is dying can therefore delay cgroup
removal and keep cgroup_mutex held for a long time.
Stop the v1 reclaim loops once the memcg is dying. For limit resizing,
keep the existing -EBUSY semantics when the new limit could not be
installed. For memory.force_empty, keep the existing best-effort success
semantics.
Link: https://lore.kernel.org/20260702120235.376752-5-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jiayuan Chen <jiayuan.chen@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "memcg: bail out reclaim when memcg is dying", v3.
This series mitigates a system-wide stall we hit when a cgroup is
removed while one of its memory control files is doing synchronous
reclaim.
Problem Description
===================
Writing to memory.high, memory.max or memory.reclaim runs reclaim
synchronously in the writer's context, looping until the usage drops below
the target (or, for memory.reclaim, until the requested amount has been
reclaimed). On a large cgroup this can take a long time. The latency is
especially bad when reclaim has to perform swap I/O, where it is bound by
the swap device write bandwidth, and under thrashing it is effectively
unbounded - each round reclaims a few pages that the workload immediately
faults back in, so the loop keeps making "progress" and never converges.
The legacy (v1) reclaim loops in memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty share the same pattern.
These writes go through cgroup_file_write(), which does not take
cgroup_mutex and does not pin the css. Instead, kernfs guarantees the
node (and thus the css) stays alive for the duration of the operation by
holding an active reference. So while the reclaim loop runs, the active
reference on the file is held.
If another task removes the same cgroup in parallel, cgroup_rmdir() takes
cgroup_mutex and then blocks in kernfs_drain() waiting for that active
reference to drain. Because cgroup_mutex is held throughout the wait,
every other task that needs it piles up behind the remover - in our case
the whole machine ground to a halt, with hung_task reports for the remover
and for unrelated tasks merely reading /proc/<pid>/cgroup:
INFO: task cgdelete:366634 blocked for more than 159 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
kernfs_drain+0xe6/0x150
__kernfs_remove.part.0+0xd0/0x200
kernfs_remove_by_name_ns+0x75/0xd0
cgroup_addrm_files+0x325/0x410
css_clear_dir+0x50/0xf0
cgroup_destroy_locked+0xdf/0x1e0
cgroup_rmdir+0x2d/0xd0
kernfs_iop_rmdir+0x53/0x90
vfs_rmdir+0x98/0x240
do_rmdir+0x172/0x1b0
__x64_sys_rmdir+0x42/0x70
x64_sys_call+0xeb0/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
INFO: task systemd-journal:2352 blocked for more than 182 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
schedule_preempt_disabled+0xe/0x20
__mutex_lock.constprop.0+0x3bb/0x640
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x3c/0x50
proc_cgroup_show+0x4d/0x380
proc_single_show+0x53/0xe0
seq_read_iter+0x12f/0x4b0
seq_read+0xcd/0x110
vfs_read+0xb1/0x360
? __seccomp_filter+0x368/0x590
ksys_read+0x73/0x100
__x64_sys_read+0x19/0x30
x64_sys_call+0x18d3/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
The system recovers only once the reclaim finally finishes and releases
the active reference. The reclaim itself is pointless here: the cgroup is
being torn down and its remaining pages will be reparented to the parent
anyway.
Even though we check signal_pending(current) in the reclaim loop, the
typical symptom is that cat /proc/<pid>/cgroup gets stuck. By the time
someone looks for which task is actually stuck in reclaim, the hung task
timeout has already been hit. This makes the problem particularly nasty
to debug from a hung-task report alone, because the blocked tasks shown
are often the victims, not the reclaim writer itself.
Our Mitigation
==============
cgroup destruction sets CSS_DYING in kill_css_sync() *before*
css_clear_dir() triggers the kernfs_drain() that blocks the remover. The
in-flight reclaim loop is therefore guaranteed to observe it before
starting another reclaim iteration. This series checks memcg_is_dying()
in the v2 reclaim loops (memory.high, memory.max and proactive reclaim)
and the v1 reclaim loops (memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty), and bails out early,
so the writer drops the active reference promptly and the remover can make
progress.
Unlike the no-progress guard (MAX_RECLAIM_RETRIES), which only fires when
reclaim makes zero progress, the dying check also covers the slow swap I/O
and thrashing cases, where reclaim keeps succeeding a little and the loop
would otherwise never converge.
For memory.reclaim, bailing out because the memcg is dying means the
requested reclaim amount was not satisfied, so the write returns -EAGAIN.
This is orthogonal to commit c8e6002bd6 ("memcg: introduce non-blocking
limit setting option"): O_NONBLOCK lets a caller avoid the synchronous
reclaim up front, while this series handles the case where reclaim is
already running when the cgroup starts being removed.
This patch (of 4):
memory.high reclaims synchronously in the writer's context, and the
latency can be very high - especially when reclaim performs swap I/O, or
under thrashing where the loop may not converge for a long time.
While this runs the kernfs active reference on the file is held, so a
concurrent removal of the same cgroup blocks in kernfs_drain() under
cgroup_mutex until it finishes. Reclaiming a dying cgroup is pointless,
as its pages are reparented to the parent anyway.
Mitigate this by bailing out of the reclaim loop once memcg_is_dying().
Link: https://lore.kernel.org/20260702120235.376752-1-jiayuan.chen@linux.dev
Link: https://lore.kernel.org/20260702120235.376752-2-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mem_cgroup_css_reset() is called when the memory controller is disabled on
a cgroup but the memcg cannot be destroyed because it is pinned by a
subsystem dependency -- for example, the io controller declares
.depends_on = 1 << memory_cgrp_id, so memory remains in the cgroup_ss_mask
and the css is hidden rather than killed.
The purpose of css_reset is to revert the memcg to its vanilla state so
that no policies are applied and the css can be safely made visible again
later. Currently, all page counters (memory.max, swap.max, kmem.max,
tcpmem.max) and other limits (soft_limit, memory.high, swap.high) are
reset to their defaults, but zswap_max and zswap_writeback are not.
These fields are initialized in css_alloc (zswap_max = PAGE_COUNTER_MAX,
zswap_writeback inherited from parent) but were missing from css_reset.
As a result, stale zswap policies remain in effect after css_reset: the
zswap charge path (obj_cgroup_may_zswap) continues to enforce the old
zswap_max limit, and the writeback path continues to honor the old
zswap_writeback setting, even though the memory controller has been
"disabled" on this cgroup.
Reset zswap_max to PAGE_COUNTER_MAX and zswap_writeback to true, matching
their defaults in css_alloc.
Test:
echo "+memory +io" > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/test
mkdir /sys/fs/cgroup/test/child
echo "+memory +io" > /sys/fs/cgroup/test/cgroup.subtree_control
echo 10000 > /sys/fs/cgroup/test/child/memory.zswap.max
# child/memory.swap.max and child/memory.zswam.max disappear
echo "-memory" > /sys/fs/cgroup/test/cgroup.subtree_control
# re-enable memory control
echo "+memory" > /sys/fs/cgroup/test/cgroup.subtree_control
# before this patch
cat /sys/fs/cgroup/test/child/memory.zswap.max
8192
# after this patch, same as memory.swap.max
cat /sys/fs/cgroup/test/child/memory.zswap.max
max
Link: https://lore.kernel.org/20260703063826.306878-1-jiayuan.chen@linux.dev
Link: https://lore.kernel.org/20260702024827.353185-1-jiayuan.chen@linux.dev
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
DAMON_SYSFS validates user inputs for monitoring target regions to
disallow negative size regions. DAMON core assumes only positive size
regions, though. The validation is incomplete. Fortunately
damon_set_regions(), which is eventually used by DAMON_SYSFS, does the
complete validation. Remove the incomplete and unnecessary validation.
Link: https://lore.kernel.org/20260705155600.96555-9-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_set_region_system_rams() validates user inputs to avoid creating a
negative size region. But DAMON core avoids zero size, too. The check is
incomplete. The complete check is done inside damon_set_regions(), which
is eventually called from damon_set_region_system_rams_default(). Drop
the incomplete and unnecessary check.
Link: https://lore.kernel.org/20260705155600.96555-8-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The kernel doc comment of damon_region clearly specifies every region
should have positive size. But it is unclear who should verify it.
damon_set_regions() is the recommended DAMON core function for setting
regions from the callers, and has the verification. Update the comment to
clarify the callers should be ok to pass any values for region addresses,
as long as they use damon_set_regions().
Link: https://lore.kernel.org/20260705155600.96555-7-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
DAMON sysfs interface does parameters validation-purpose
damon_commit_ctx() calls for parameters update. Now the same logic is
embedded inside damon_commit_ctx() itself. Hence, the validation in DAMON
sysfs interface is just an unnecessary duplicate. Remove it.
Link: https://lore.kernel.org/20260705155600.96555-4-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_start() and damon_commit_ctx() are two main DAMON core API functions
for setting whole DAMON parameters. While damon_commit_ctx() does
holistic parameters testing, damon_start() just believes the caller
validated the whole thing. Embed the holistic parameter check that is
already in damon_commit_ctx() into damon_start(). After this change, the
callers can safely call damon_start() without validating the parameters.
Link: https://lore.kernel.org/20260705155600.96555-3-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon: validate all parameters in the core".
DAMON has a number of parameters. Some of the parameters are validated by
DAMON, while some are validated by DAMON API callers. Each caller has
their own set of parameters that are exposed to users. Hence each caller
has their own ways to do the validation. There is no clear lines for the
responsibilities. It is confusing and easy to make mistakes at
validations. Actually we found a few bugs in the class.
Define DAMON core as the place to do all the validations and implement
those. damon_set_regions(), damon_start() and damon_commit_ctx() are the
three main DAMON core API functions for setup of DAMON parameters. Make
the three functions to do safe and holistic parameter checks. The first
one is already providing the validation, so changes are only for the last
two functions.
This might add unnecessary validations for some use cases. The overhead
should be negligible as parameters update is expected to only rarely
happen. It reduces the number of places to check and fix for bugs of the
class from all callers to the single component. The maintenance
efficiency gain is obvious.
Further cleanup documents and caller code. Make the lines of validation
duties clearer. Remove validations that are no more needed owing to the
core validations.
Patches Sequence
================
Patch 1 implements the core holistic parameters validation for
damon_commit_ctx(). Patch 2 extends the validation to damon_start().
Patch 3 removes the duplicated holistic parameters validation in DAMON
sysfs interface, which is now embedded into the core layer. Patches 4 and
5 remove duplicated min_region_sz validation in DAMON modules. Patches 6
updates kernel doc to clarify damon_set_regions() is doing the range
validation. Patches 7 and 8 remove monitoring target range validations
that doesn't necessary thanks to the validation in damon_set_regions().
This patch (of 8):
damon_commit_ctx() does its holistic parameter set validation while
applying the new parameter in the set one by one. If it finds a parameter
is invalid, because some invalid parameters may already be committed (it
is called "commit" but not atomic and irreversable), it stops the running
DAMON context.
The callers of the function therefore have to validate the parameters
before calling it. Because the function already embeds holistic
validation, DAMON_SYSFS reuses it in a safe way. It creates a
test-purpose context that is not running but mimics the running one, and
calls damon_commit_ctx() against the test purpose context. If it
succeeds, the parameters are considered valid, and a real
damon_commit_ctx() call against the running context is made with those.
Other callers such as DAMON_RECLAIM and DAMON_LRU_SORT do not expose full
parameters to users. For efficiency, they validate only the known set of
parameters. The efficiency gain is arguably small and doubtful, though.
Meanwhile the maintenance overhead of the multiple different validations
is clearly high. We actually found and fixed a few bugs in the class.
Update damon_commit_ctx() to embed DAMON_SYSFS' safe and holistic
validation approach. Callers can simply call damon_commit_ctx() without
worrying if their parameters are invalid.
Note that damon_commit_ctx() can still cause an unexpected stop of the
running context, if internal memory allocation fails. It is arguably
unlikely since those internal allocations are too small to fail, but
theoretically possible. It should also be better addressed, but not
necessarily a blocker of this small and incremental improvement effort.
Link: https://lore.kernel.org/20260705155600.96555-1-sj@kernel.org
Link: https://lore.kernel.org/20260705155600.96555-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Now that alloc_pages has an entrypoint that allows passing alloc_flags, we
can take advantage of this to start removing GFP flags that are only used
for mm-internal stuff.
This requires also plumbing the alloc_flags into some more of the
allocator code, in particular __alloc_pages[_noprof]() gets an alloc_flags
arg to go along with its callees, and we now need to pass those flags
deeper into the allocator so they can reach the alloc_tag code.
While moving the flag definition into page_alloc.h, also update the
comment per Hao's suggestion.
No functional change intended.
Link: https://lore.kernel.org/all/b4916118-3537-4e19-8bc8-1d103dd0d225@linux.dev/
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-15-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Tested-by: Hao Ge <hao.ge@linux.dev>
Acked-by: Hao Ge <hao.ge@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
It's no longer used outside of mm/.
Since this means __alloc_pages_noprof() is no longer visible from gfp.h,
this also means moving the definition of alloc_pages_node_noprof into
the .c file.
Also remove references to this API from the documentation tree -
referring to the specific function name was already questionable but
now the function is not even public it definitely seems wrong.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-14-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
There were only a few users, which have been removed. The only advantage
of this API over alloc_pages_node() is avoiding a single conditional
branch. The disadvantages are:
1. More API surface, more sources of confusion, more maintenance.
2. Worse impact of CPU hotplug bugs: most users of __alloc_pages_node()
were using the result of cpu_to_node(); if the CPU gets hotplugged
out this will return NUMA_NO_NODE. If one of these paths fails to
protect against a concurrent hotplug then page_alloc.c will use
NUMA_NO_NODE as an index into NODE_DATA() and cause some horrible
memory corruption or other. With alloc_pages_node(), the code might
just work fine.
Ulterior motive: this frees up the __* variants of the allocator APIs to
serve specifically for use as mm-internal API.
Link: https://lore.kernel.org/20260703-alloc-trylock-v5-13-c87b714e19d3@google.com
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
