Patch series "mm/damon: optimize out nr_accesses_bp".
TLDR: Replace damon_region->nr_accesses_bp, which is easy to be wrong,
with a simpler on-demand moving sum function, damon_nr_accesses_mvsum().
Background
==========
DAMON's monitoring output (access pattern snapshot, or more technically
speaking, damon_region->nr_accesses) is completed once per aggregation
interval, which is 100 ms by default. Users can arbitrarily increase the
interval for demand. Under the suggested intervals auto-tuning setup, it
can span up to 200 seconds. If the aggregation interval is too long, the
snapshot users cannot use it in reasonable time. To mitigate this, we
introduced a new field of damon_region, namely nr_accesses_bp. It
contains a pseudo moving sum of nr_accesses in bp units and is updated for
each sampling interval.
It turned out keeping it correctly updated every sampling interval is not
that easy. From online parameter update feature development and more
experimental hacks, we found it is easy to be corrupted. Once it is
corrupted, DAMON's monitoring outputs become quite insane. Hence we added
a few validation checks. It is easy to be corrupted because it requires
every update per sampling interval to be correct.
Solution
========
There is no real reason to keep it updated every sampling interval. Due
to the simple pseudo-moving sum mechanism and existing helper field
(last_nr_accesses), we can also calculate the pseudo moving sum on demand
in a much simpler way.
Implement a function for getting the pseudo moving sum on demand, and
replace nr_accessses_bp uses with the new function. Also remove no more
needed tests for nr_accesses_bp and the per-sampling interval update
functions. Finally, remove the nr_accesses_bp. The new function is quite
simple.
Discussion
==========
Depending on the use case, multiple nr_accesses readers could be executed
in the same kdamond_fn() main loop iteration, which is executed once per
sampling interval. Such readers include DAMON region exporting
tracepoints (damon_[region_]aggregated and damos_before_apply), DAMOS, and
DAMON sysfs interface logic for update_schemes_tried_regions command. In
this case, the new function will be called multiple times and this could
be overhead compared to the old logic, which simply reads the field
without any additional work. Nonetheless, the new function is quite
simple. And the new approach does nothing while there is no need to read.
The old approach had to execute its update function for each region for
every sampling interval. Hence the new approach is believed to be even
more lightweight in common case, and the overhead is anyway negligible.
One more advantage of this change is that one field from the damon_region
struct is removed. On setups that uses a high number of DAMON regions,
this could be a potential memory space benefit.
Patches Sequence
================
Patch 1 introduces the new function for getting the pseudo moving sum of
nr_accesses on demands. Patch 2 implements a unit test for the new
function's internal logic. Patch 3 and 4 update monitoring logic and the
new function to ready for safe use on the existing logic. Patches 5-7
replace uses of nr_accesses_bp in DAMOS, tracepoints and DAMON sysfs
interface with the new function, respectively. Patches 8-10 removes
nr_accesses_bp validation functions in DAMON core, one by one. Patches 11
and 12 further remove tests and test helper for nr_accesses_bp,
respectively. Patches 13 removes the setups and updates or nr_accesses_bp
field. Patches 14-16 cleans up function parameters that are no more being
used due to the previous patch. Patch 17 removes the function that was
used for updating nr_accesses_bp field with its unit test, which is the
single remaining caller of the function. Finally, patch 18 removes
damon_region->nr_accesses_bp field.
This patch (of 18):
Introduce a new DAMON core function, damon_nr_accesses_mvsum(). It
returns a pseudo moving sum value of a given region's nr_accesses for the
last aggregation interval. The internal logic is the same to
nr_accesses_bp. The difference is that nr_accesses_bp is updated for each
sampling interval, while the new function needs to be executed only when
requested. Hence the function's return value is the same as the value of
nr_accesses_bp.
Link: https://lore.kernel.org/20260630040812.149729-1-sj@kernel.org
Link: https://lore.kernel.org/20260630040812.149729-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Non-cooperarive uffd events are inherently racy and can happen in parallel
with other userfaultfd operations.
During event tests in uffd-unit-tests, the uffd monitor calls
UFFDIO_UNREGISTER upon receiving UFFD_EVENT_REMOVE.
In parallel, the faulting_process() verifies that the removed memory is
actually zeroed.
If a verification read wins the race with UFFDIO_UNREGISTER, it causes a
missing fault that uffd monitor would receive after UFFDIO_UNREGISTER is
complete. The monitor resolves the fault using UFFDIO_COPY that fails
with -ENOENT which means that VMA has been changed (see commit
27d02568f5 ("userfaultfd: mcopy_atomic: return -ENOENT when no
compatible VMA found")).
Treat -ENOENT returned by UFFDIO_COPY as non-fatal, the same way -EEXIST
is treated for concurrent faults, and don't fail the test.
Link: https://lore.kernel.org/20260701200932.1470525-1-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
dmirror_fault() is called from the dmirror_read() and dmirror_write()
retry loops after dmirror_do_read() or dmirror_do_write() finds a missing
device page table entry.
If the mirrored mm has already exited, mmget_not_zero() fails. The
current code returns 0 in that case, which tells the caller that faulting
succeeded even though no page was faulted and no device page table entry
was installed. The caller then retries the same address, hits -ENOENT
again, and can loop forever without making progress.
Return -EFAULT instead, so the ioctl fails when the mirrored mm is no
longer faultable.
Link: https://lore.kernel.org/178294308408.327222.3319445682023999403.stgit@skinsburskii
Fixes: b2ef9f5a5c ("mm/hmm/test: add selftest driver for HMM")
Signed-off-by: Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Ralph Campbell <rcampbell@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
migrate_vma_collect_pmd() can detect a large folio while holding the PTE
lock, then drop the PTE lock before calling migrate_vma_split_folio().
The split helper took its own reference, but only after the lock had
already been dropped.
One way to hit this is device migration over a range that contains a large
folio. The walker reads the PTE while holding the PTE lock and derives
the folio either from a present PTE via vm_normal_page(), or from a
non-present PTE that encodes a device-private softleaf entry. It then has
to drop the PTE lock because split_folio() can block. Before
migrate_vma_split_folio() gets a folio reference, concurrent reclaim,
migration, or truncation can replace or clear the entry and drop the last
reference to the folio. The split helper would then take a reference and
lock on a stale folio pointer.
Take a temporary reference before dropping the PTE lock and pass that
reference into migrate_vma_split_folio(). The helper consumes the
reference, so split_folio() still sees only the expected caller pin
instead of an extra pin that could make the split fail.
[usama.arif@linux.dev: condense comment about folio reference]
Link: https://lore.kernel.org/87bbf335-648f-4065-abc8-3eaab5a3beeb@linux.dev
Link: https://lore.kernel.org/20260701140638.840773-1-usama.arif@linux.dev
Fixes: 022a12deda ("mm/migrate_device: handle partially mapped folios during collection")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260630164143.1595669-1-usama.arif%40linux.dev
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
KCSAN reports a data race between page_to_nid()/folio_pgdat() reading
page->flags and folio_trylock()/folio_lock() concurrently doing
test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.:
BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp
The race is benign: nid/zone bits are set once at page init and never
overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside
memdesc_nid/zonenum() was checking a by-value copy of the flags word, not
the live page->flags, so it failed to annotate the real access.
Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and
memdesc_is_zone_device() to take a const memdesc_flags_t * and update all
callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS()
operates on the actual shared word.
Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under
ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone
field is absent. memdesc_section() needs no such guard, since
SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined.
Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead
of reading page->flags, since NODES_MASK is 0 and the check can never
fire; page_to_nid()/folio_nid() now just call memdesc_nid()
unconditionally and rely on that stub, instead of duplicating the
CONFIG_NUMA split at each call site.
[zhuhui@kylinos.cn: v11]
Link: https://lore.kernel.org/20260708083308.747930-1-hui.zhu@linux.dev
[ziy@nvidia.com: build fix]
Link: https://lore.kernel.org/DJSGLP9VFLRI.355AAWR2VHIEL@nvidia.com
Link: https://lore.kernel.org/20260630070810.470763-1-hui.zhu@linux.dev
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Clean up mm/vmpressure.c by separating the cgroup v1 userspace eventfd
interface from the shared and v2 in-kernel code.
Currently, almost half of mm/vmpressure.c exists to serve tree=true:
struct vmpressure_event, the events list and its mutex, the work_struct
and vmpressure_work_fn that drains tree_scanned/tree_reclaimed, the parent
walk, vmpressure_event(), vmpressure_register_event(),
vmpressure_unregister_event(), and vmpressure_prio() (which always calls
vmpressure() with tree=true).
Move it all into mm/memcontrol-v1.c (built only when CONFIG_MEMCG_V1=y) as
a single contiguous block, following the per-component layout already used
by that file. Keeping the v1 vmpressure code with the rest of the
deprecated cgroup v1 memory controller makes the full footprint of the
CONFIG_MEMCG_V1 option easy to see in one place, which matters more than
component-level file separation for code that has no active development.
vmpressure.c keeps the shared bits (constants, vmpressure_calc_level, the
runtime hierarchy check, the tree=false body, init/cleanup plumbing) and
calls into three small v1 hooks for the tree=true accumulator and the v1
portions of init/cleanup. The hooks have static-inline no-op stubs in
include/linux/vmpressure.h for the !MEMCG_V1 case, so callers don't need
ifdefs. vmpressure_prio() gets the same treatment, which means vmscan.c's
call site disappears at compile time on v2-only kernels.
The only #ifdef CONFIG_MEMCG_V1 in source remains around the v1-only
fields inside struct vmpressure itself.
Memory savings on CONFIG_MEMCG_V1=n (measured with pahole):
struct vmpressure : 112B -> 24B
struct mem_cgroup : 1664B -> 1536B
This split is the first step toward eventually making vmpressure
CONFIG_MEMCG_V1 only. The v2 in-kernel socket pressure path (tree=false)
cannot be removed today immediately: PSI is not an exact replacement for
vmpressure, and switching networking socket-buffer back-off to PSI may
regress networking performance or increase memory pressure in workloads
that today rely on vmpressure's hysteresis. The medium-term plan is to
introduce a PSI-based socket-pressure path, keep vmpressure available for
v2 behind a defconfig as an opt-out for several releases, and only then
drop the tree=false path entirely, at which point everything that remains
of the vmpressure block in mm/memcontrol-v1.c is the whole subsystem.
Link: https://lore.kernel.org/20260630112617.1198623-3-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/vmpressure: reduce CPU, memory and code overhead on
cgroup v2", v3.
The vmpressure subsystem has two distinct consumers, gated by the @tree
argument:
tree=false : in-kernel socket pressure, consumed by TCP/SCTP. This
is cgroup v2 only; v1 sockets read memcg->tcpmem_pressure
instead.
tree=true : cgroup v1 userspace eventfd notifications via the
memory.pressure_level / cgroup.event_control interface.
v2 has no equivalent (userspace gets reclaim signals
through memory.pressure / PSI, which doesn't touch
vmpressure).
So of the four (hierarchy, tree) combinations, only two carry data that
anyone reads. The existing early return in vmpressure() covered v1 +
tree=false; the symmetric v2 + tree=true case was falling through and
doing the full lock / accumulate / schedule_work / parent-walk dance, even
though the events list it eventually iterates is empty on cgroup v2
(vmpressure_register_event() is wired up only through the v1 cftype
"memory.pressure_level" and can't be reached from a v2 memcg).
Patch 1 extends the existing early return to also skip v2 + tree=true. On
a v2-only host this eliminates a contended path where reclaimers can
serialize on a single global sr_lock. bpftrace on a 176-core production
host (cgroup v2, 285 memcgs, sustained reclaim) showed ~16,200 such calls
per minute with tree = true.
Patch 2 follows up with a cleanup: it splits the v1 userspace eventfd
interface (struct vmpressure_event, the events list and its mutex, the
work_struct and its handler, the parent walk, vmpressure_register_event /
unregister_event, and vmpressure_prio) into a new mm/memcontrol-v1.c built
only when CONFIG_MEMCG_V1=y, behind small no-op stubs in the header.
mm/vmpressure.c keeps the shared bits and the tree=false socket-pressure
path. The size of vmpressure.c goes down to half and the code is much
more simpler. The only #ifdef CONFIG_MEMCG_V1 remaining in source is
around the v1-only fields inside struct vmpressure itself. Memory savings
on CONFIG_MEMCG_V1=n:
struct vmpressure : 112B -> 24B
struct mem_cgroup : 1664B -> 1536B
This split is the first step toward eventually making vmpressure
CONFIG_MEMCG_V1 only. The v2 in-kernel socket pressure path (tree=false)
cannot be removed today immediately: PSI is not an exact replacement for
vmpressure, and switching networking socket-buffer back-off to PSI may
regress networking performance or increase memory pressure in workloads
that today rely on vmpressure's hysteresis. The medium-term plan is to
introduce a PSI-based socket-pressure path, keep vmpressure available for
v2 behind a defconfig as an opt-out for several releases, and only then
drop the tree=false path entirely, at which point everything that remains
in mm/memcontrol-v1.c is the whole subsystem.
This patch (of 2):
vmpressure() has two outputs gated by the @tree argument:
@tree=false drives in-kernel socket pressure (mem_cgroup_set_
socket_pressure), consumed by TCP/SCTP. This only
applies on cgroup v2; on v1 socket memory is charged
separately via tcpmem and the consumer reads
memcg->tcpmem_pressure instead.
@tree=true drives userspace eventfd notifications via the v1
memory.pressure_level / cgroup.event_control interface.
v2 has no equivalent: userspace gets reclaim signals
through memory.pressure (PSI), which does not touch
vmpressure.
The existing early return covered v1 + @tree=false. The symmetric v2 +
@tree=true case was falling through and doing the full lock / accumulate /
schedule_work / parent-walk dance for an events list that can never be
populated. bpftrace on a 176-core production host (cgroup v2,
CONFIG_MEMCG_V1=n, 285 memcgs, sustained reclaim) showed ~16,200
@tree=true vmpressure() calls per minute. Add an early return that skips
cgroup v2 + tree = true which avoids us doing all this work. On a v2-only
host this also eliminates a lock contention path that can serialise
reclaimers on a single global sr_lock.
[usama.arif@linux.dev: simplify the guard]
Link: https://lore.kernel.org/e8e1a409-48d8-4fa7-ae98-49485a1607f6@linux.dev
Link: https://lore.kernel.org/20260630112617.1198623-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260630112617.1198623-2-usama.arif@linux.dev
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Since 79359d6d24 ("hugetlb: perform vmemmap optimization on a list of
pages") __hugetlb_vmemmap_optimize_folios() unconditionally issues a final
flush_tlb_all() in its out path. However, a TLB flush must be paired with
an actual page table modification.
When the input folio list is empty, neither PMD splitting nor PTE
remapping takes place, so no page tables are modified and the flush is
pure overhead. An empty list is reached in common paths such as
gather_bootmem_prealloc_node() on nodes without bootmem gigantic pages,
hugetlb_pages_alloc_boot_node() when no pages were allocated, and runtime
allocation failure paths in set_max_huge_pages().
Add an early return for empty lists. This restores the basic invariant
that TLB flushes are only issued when page tables have been modified, and
it also makes the NULL hstate passed by gather_bootmem_prealloc_node() on
an empty list harmless.
Assisted-by: kimi-cli:kimi-k2.7 code
Assisted-by: Github Copilot:gpt-5.2 #Reported-by
Link: https://lore.kernel.org/20260701053422.3664813-1-guanwentao@uniontech.com
Link: https://lore.kernel.org/20260630113857.3319612-1-guanwentao@uniontech.com
Fixes: 79359d6d24 ("hugetlb: perform vmemmap optimization on a list of pages")
Signed-off-by: Wentao Guan <guanwentao@uniontech.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Guan Wentao <guanwentao@uniontech.com>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
ALLOC_HIGHATOMIC currently provides both access to MIGRATE_HIGHATOMIC free
pages and permission to create new highatomic pageblock reserves. This
makes it unsuitable for the fastpath.
However, the fastpath can reach rmqueue_buddy() while MIGRATE_HIGHATOMIC
reserves have free pages available. In this situation, the allocation can
fall back to other migratetypes without trying those reserves first.
Allow high-priority non-blocking allocations to use existing
MIGRATE_HIGHATOMIC reserves on the buddy fastpath without growing them.
First tighten the criteria for reserving pageblocks so that growth may
only occur in the slowpath. Then allow fastpath usage by enabling
ALLOC_HIGHATOMIC when the GFP mask describes a non-blocking high-priority
allocation. This logic has been factored out from gfp_to_alloc_flags() to
a new function gfp_to_alloc_flags_nonblocking().
A UDP receive workload was run with free MIGRATE_HIGHATOMIC pageblocks
available in the target zone. Before this patch, the workload did not
consume these blocks. With this patch, eligible order-1 allocations
reaching the buddy path consumed existing MIGRATE_HIGHATOMIC pageblocks,
with no highatomic misses observed. The workload did not grow highatomic
reserves and NAPI page-frag allocations remained healthy with no failures
or order-0 fallbacks.
Link: https://lore.kernel.org/20260623004600.113347-1-jp.kobryn@linux.dev
Signed-off-by: JP Kobryn <jp.kobryn@linux.dev>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_commit_target_regions() calls damon_set_regions() for updating the
destination target's monitoring target region boundaries. It sets the
boundaries same to source target's monitoring regions, even if they are
adjacent. Meanwhile, damon_set_region() sets the destination target
regions exactly the same to the source, only when the target regions are
empty. When there are existing target regions, only a few regions are
expanded or shrunk to fit on only the boundaries for disjoint regions in
the source. Hence the adjacent source ranges mean nothing in common
cases. When there are many regions, such adjacent range setup is only a
waste of time and space. We recently found [1] it is actually causing
memory overhead. Setup the ranges for only distinct ranges.
Link: https://lore.kernel.org/20260630141726.92246-9-sj@kernel.org
Link: https://lore.kernel.org/20260603112306.58490-1-akinobu.mita@gmail.com [1]
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: preparatory patches for PMD level swap entries", v2.
This is the preparatory part of the PMD page table swapin work. The full
PMD swap entry series has been split into two parts:
1. this preparatory series, which contains the first 6 patches. Zi [1]
and Lance [2] suggested to separate this out from the core series.
2. the PMD swap entry core series, which depends on this one. I will
send this once the preparatory series is merged in mm-new as v3
as the combined is currently at v2 [1].
I have not marked this prep series as v3, as its not really adding
support for PMD swap entries.
This series does not introduce PMD swap entries and does not install any
new page-table entry type. It only cleans up existing PMD softleaf
helpers and call sites so the follow-up PMD swap entry series can be
smaller and easier to review.
It should be safe to merge independently. The patches are either helper
additions, refactors of existing open-coded logic, defensive checks that
preserve current migration/device-private behavior, or a mechanical rename
of the PMD softleaf Kconfig gate. The follow-up series depends on these
helpers, but this series does not depend on the follow-up series.
Patch breakdown:
1. mm: add softleaf_to_pmd() and convert existing callers
Add the PMD counterpart to softleaf_to_pte() and convert existing
swp_entry_to_pmd() users that are constructing PMD softleaf
entries.
2. mm: extract mm_prepare_for_swap_entries() helper
Hoist the "register mm with swapoff" double-checked-locking
pattern out of try_to_unmap_one() and copy_nonpresent_pte() so
future PMD-level users do not need another open-coded copy.
3. fs/proc: use softleaf_has_pfn() in pagemap PMD walker
Avoid assuming every non-present PMD softleaf entry encodes a PFN.
Existing migration/device-private behavior is preserved.
4. mm/huge_memory: move softleaf_to_folio() inside migration branch
Keep the folio lookup in change_non_present_huge_pmd() scoped to
the migration-entry branch that actually needs it.
5. mm/migrate_device: move softleaf_to_folio() inside device-private
branch
Apply the same ordering cleanup to migrate_vma_collect_pmd(): only
derive a folio after confirming the PMD entry is device-private.
6. mm: rename ARCH_ENABLE_THP_MIGRATION to ARCH_HAS_PMD_SOFTLEAVES
Rename the architecture gate to describe what it actually enables:
PMD softleaf entries. Migration remains the only current user in
this series; the follow-up series adds PMD swap entries.
This patch (of 6):
Add softleaf_to_pmd() as the PMD counterpart to softleaf_to_pte(),
completing the symmetry of the softleaf abstraction for page table leaf
entries.
The upcoming PMD swap entry support needs to construct PMD entries from
swap entries. Converting existing swp_entry_to_pmd() callers to
softleaf_to_pmd() in a prep patch keeps the feature patches focused on new
functionality rather than mixing refactoring with new code.
Link: https://lore.kernel.org/20260706114320.1643046-1-usama.arif@linux.dev
Link: https://lore.kernel.org/20260706114320.1643046-2-usama.arif@linux.dev
Link: https://lore.kernel.org/all/6E99CC4E-A026-4DE3-8A5A-34216771F521@nvidia.com/ [1]
Link: https://lore.kernel.org/all/b08cafbb-a4b7-4609-84ae-dbb2cfcfc8be@linux.dev/#t [2]
Link: https://lore.kernel.org/all/20260602142537.198755-1-usama.arif@linux.dev/ [3]
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
folio_check_references() will return FOLIOREF_RECLAIM_CLEAN for referenced
file folios, indicating that we can proceed to reclaim clean file folios
or keep them if they are dirty file folios. However, after commit
6b0dfabb35 ("fs: Remove aops->writepage"), we no longer attempt to write
back filesystem folios through reclaim. Instead, we always activate dirty
file folios and wakeup the flush workers to write them back. As a result,
the FOLIOREF_RECLAIM_CLEAN logic is now redundant: for dirty file folios,
we will no longer reach the 'references == FOLIOREF_RECLAIM_CLEAN' branch
in shrink_folio_list().
Additionally, lazyfree folios are also placed on the file LRU list, but if
a lazyfree folio becomes dirty, try_to_unmap() will fail and thus prevent
reclaim of the re-dirtied lazyfree folios.
Therefore, we can drop the FOLIOREF_RECLAIM_CLEAN-related logic.
Link: https://lore.kernel.org/def70a713e10bcbdf3b9fccc2139ecc07b64f2cb.1782715791.git.baolin.wang@linux.alibaba.com
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon/core: detect internal variation above
max_nr_regions/2", v3.
kdamond_split_regions() bails out early when nr_regions is already above
max_nr_regions / 2. A large region that picks up new internal variation
after that point never gets split, so we lose visibility into its hot/cold
structure.
We hit this with damon-paddr on hugepage workloads and damon-vaddr on
processes that mmap a large anonymous range.
Example with max_nr_regions == 1500. A target ends up with 799 small
hot/cold regions plus one big region (an earlier merge collapsed a
uniformly-accessed range into a single piece):
H:hot
C:cold
r1 r2 r3 r800
HHHHHH|CCCCCC|HHHHHH|...|HHHHHH..........................|
nr_regions = 800 > max_nr_regions / 2 = 750
Now a cold subarea shows up inside r800:
r1 r2 r3 r800
HHHHHH|CCCCCC|HHHHHH|...|HHHHHH........CCCCCC.............|
The small regions can't merge with each other (their access counts
differ), so budget never frees up. r800 can't be split because nr_regions
> max_nr_regions / 2 returns early. The cold subarea stays invisible.
Patch 1 keeps refining on this path: when nr_regions is above
max_nr_regions / 2 but still under the maximum, it splits a fraction of
the regions instead of returning. The fraction shrinks as the remaining
budget shrinks, so the count approaches max_nr_regions smoothly. A
useless split is undone by the next merge cycle.
Patch 2 adds a KUnit test for the case where nr_regions is already above
max_nr_regions / 2.
Thanks to SJ for the suggestion to drive the split fraction from the
remaining budget rather than an age-based filter.
This patch (of 2):
kdamond_split_regions() returns early when nr_regions is above
max_nr_regions / 2, leaving internal access variation inside a large
region undetected.
Such a layout is common with damon-paddr on hugepage workloads or
damon-vaddr on processes with a large anonymous mmap.
For example, with max_nr_regions == 1500, a target may end up with 799
small alternating-temperature regions plus one large region that absorbed
a uniformly-accessed range during an earlier merge:
H:hot
C:cold
r1 r2 r3 r800
HHHHHH|CCCCCC|HHHHHH|...|HHHHHH..........................|
nr_regions = 800 > max_nr_regions / 2 = 750
If a cold subarea later emerges inside r800:
r1 r2 r3 r800
HHHHHH|CCCCCC|HHHHHH|...|HHHHHH........CCCCCC.............|
The small regions cannot merge with each other (different access counts),
so the budget stays full. r800 cannot be split because nr_regions >
max_nr_regions / 2 causes an early return. The cold subarea is never
discovered.
When nr_regions is above max_nr_regions / 2 but still under the maximum,
split only a fraction of the regions instead of returning. One region in
every 'max_nr_regions / budget' regions is split, where budget is the
remaining room (max_nr_regions - nr_regions), starting from a rotating
offset so different regions get picked over time. The fraction shrinks as
the budget shrinks, so the region count keeps refining while approaching
max_nr_regions smoothly rather than overshooting it. An unnecessary split
is reverted by the next kdamond_merge_regions().
Link: https://lore.kernel.org/20260629145630.134891-1-sj@kernel.org
Link: https://lore.kernel.org/20260626085851.70754-2-jiayuan.chen@linux.dev
Link: https://lore.kernel.org/20260629145630.134891-2-sj@kernel.org
Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com>
Signed-off-by: SJ Park <sj@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Shu Anzai <shu17az@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Writing a non-zero value to a kdamond's 'refresh_ms' sysfs file should
make DAMON periodically update the read-only sysfs files on its own,
without the user writing update keywords such as 'update_schemes_stats' to
the 'state' file. This behavior has no test coverage.
Add a test that starts a kdamond with refresh_ms set and a 'stat' scheme
whose default access pattern matches every monitored region, then polls
the scheme's 'nr_tried' stats file directly, without requesting an update.
The value can become non-zero only via the periodic refresh, so the test
confirms refresh_ms works; with refresh_ms disabled the stat stays zero
and the test fails.
Link: https://lore.kernel.org/20260602131217.2210912-3-linuxoid@gmail.com
Link: https://lore.kernel.org/20260629144927.134237-3-sj@kernel.org
Signed-off-by: Ruslan Valiyev <linuxoid@gmail.com>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "selftests/damon: test kdamond refresh_ms", v2.
The kdamond 'refresh_ms' sysfs file makes DAMON periodically update its
read-only sysfs files (DAMOS stats, tuned monitoring intervals and the
kdamond pid) on its own, so users don't have to write update keywords such
as 'update_schemes_stats' to the 'state' file. It has no selftest
coverage.
The first patch adds refresh_ms support to the _damon_sysfs.py test
control module. The second adds a test that sets refresh_ms and confirms
a scheme's stats are updated under sysfs without an explicit update
request; the test skips on kernels that predate the refresh_ms file.
Tested on current mainline under a DAMON-enabled kernel: the new test
passes and the existing DAMON selftests show no new failures.
This patch (of 2):
The Kdamond class has no way to set the kdamond-level 'refresh_ms' sysfs
file, which makes DAMON periodically update the read-only sysfs files
(DAMOS stats, tuned monitoring intervals and the kdamond pid) on its own.
Add a 'refresh_ms' parameter to Kdamond. When it is set (including to
zero, to disable the periodic update), write it before turning the kdamond
on, so tests can exercise the auto-update behavior. Leaving it unset
keeps the previous behavior of not touching the file, so callers running
against kernels without the feature are unaffected.
Link: https://lore.kernel.org/20260602131217.2210912-2-linuxoid@gmail.com
Link: https://lore.kernel.org/20260629144927.134237-2-sj@kernel.org
Signed-off-by: Ruslan Valiyev <linuxoid@gmail.com>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
