mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-31 02:21:39 -04:00
0d1daaed8beaaa97c59f9c5e3e4dfeb7674892c7
1464152 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
SJ Park
|
0d1daaed8b |
mm/damon/core: extend merge function to work with probe hits
When probe weights are set, users may want DAMON monitoring results to be optimized for the weights. For that, regions adjustment should work for the weighted sum of probe hits. Extend damon_merge_regions_of() to detect if the weights are set, and work with probe hits in the case. The weights setup detection function is incomplete. It always returns false. It is intentional, so that more changes to completely support weights can be made in an incremental but safe way. Until the function is completed, all changes depend on it is no-op, so DAMON works in the current mode. Link: https://lore.kernel.org/20260710134651.18084-9-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
e1f150d415 |
mm/damon/core: use abs_diff() instead of abs()
Use of abs() in damon_merge_regions_of() could cause a silent integer overflow since the macro casts unsigned int to signed int. It is unlikely to have such a large value for nr_accesses. Even though it happens, the user impact is just degraded monitoring results. Users showing bad monitoring results for weird setup is quite trivial. But the code is obviously wrong. Use abs_diff() instead. The issue was discovered [1] by Sashiko. Link: https://lore.kernel.org/20260710134651.18084-8-sj@kernel.org Link: https://lore.kernel.org/20260705213817.100841-1-sj@kernel.org/ [1] Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
3858025f48 |
mm/damon/paddr: respect return_max_wsum
apply_probes() ops implementation in DAMON_PADDR is ignoring return_max_wsum. Respect it. Link: https://lore.kernel.org/20260710134651.18084-7-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
cfe9e8c738 |
mm/damon/core: implement damon_probe_hits_wsum()
When damon_probe->weight is set, the weighted sum of probe hits will be useful. It will be useful for not only the users but also DAMON internal logics like regions merging. Implement a function for calculating it. Link: https://lore.kernel.org/20260710134651.18084-6-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
d235513a7c |
mm/damon/core: ask apply_probe() to return max probe hits weighted sum
check_accesses() DAMON ops callback returns the maximum nr_accesses of regions. DAMON core uses it to calculate a reasonable region merge threshold. The core will need to adjust regions for not nr_accesses but probe hits weighted sum in future. For that, the core needs to know the maximum weighted sum of the regions. Update the protocol for the task. Link: https://lore.kernel.org/20260710134651.18084-5-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
a75de62a5f |
mm/damon/paddr: set samples in apply_probes() if requested
apply_probe() callback implementation in DAMON_PADDR is ignoring set_samples parameter. Respect it. Link: https://lore.kernel.org/20260710134651.18084-4-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
1138ec78dc |
mm/damon/core: ask apply_probes() ops callback to set sampling address
prepare_access_checks() DAMON ops callback sets the monitoring sampling address per region. In future, DAMON will be able to call only apply_probes(). In this case, applyy_probes() may need to do the sampling address setup, to minimize unnecessary regions iteration. Update the protocol for the request. Link: https://lore.kernel.org/20260710134651.18084-3-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
37aae41c10 |
mm/damon/core: introduce damon_probe->weight
Patch series "mm/damon: introduce data attributes only monitoring". TL;DR: Introduce a way to get DAMON's best effort accuracy monitoring of user-demanding non-access data attributes. Background ========== DAMON was initially designed for only access monitoring. It turned out users want to get the information together with more data attributes. For example, some users want to know how much of a hot memory region belongs to huge pages or specific cgroups. Page level properties based monitoring was introduced with commit |
||
|
Christoph Hellwig
|
747beac679 |
mm: remove wb_writeout_inc
Remove this entirely unused but exported function. Link: https://lore.kernel.org/20260710051052.1839523-1-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Jan Kara <jack@suse.cz> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Acked-by: SJ Park <sj@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
John Hubbard
|
c494788faf |
mm/gup: fix GUP-fast fallback for NULL-mapping order-0 folios
Since commit |
||
|
Sayali Patil
|
4e1fbffb33 |
selftests/mm: fix ternary operator precedence in ksm_tests
The KSM selftest uses conditional expressions to skip accesses to
merge_across_nodes on systems without NUMA support. However, the ternary
operator is combined with logical OR without parentheses:
a || numa_available() ? 0 : b || c
Due to operator precedence rules, this is parsed as:
(a || numa_available()) ? 0 : (b || c)
instead of the intended:
a || (numa_available() ? 0 : b) || c
Add parentheses around the conditional expressions to ensure the
correct evaluation order.
Link: https://lore.kernel.org/ce859430287ed2642848c933a90eb9a69da361f0.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Sayali Patil
|
15828a150c |
selftests/mm: fix ksm NUMA merge test for systems with memoryless NUMA nodes
The KSM NUMA merge test allocates identical pages on different NUMA nodes
and verifies KSM behavior with merge_across_nodes enabled and disabled.
On systems with memoryless NUMA nodes, for example:
#numactl -H
available: 2 nodes (0,4)
.....
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 14825 MB
node 0 free: 1382 MB
node 4 cpus:
node 4 size: 0 MB
node 4 free: 0 MB
the test may attempt to allocate memory on a node without memory, causing
numa_alloc_onnode() to fail and resulting in a spurious test failure.
The test currently checks numa_num_configured_nodes() to determine whether
sufficient NUMA nodes are available. However, configured nodes do not
necessarily have memory.
Reuse the existing get_first_mem_node() and get_next_mem_node() helpers to
locate NUMA nodes that actually contain memory, and skip the test when
fewer than two such nodes are available.
Before patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
mbind: Invalid argument
ok 1 KSM NUMA merging
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
mbind: Invalid argument
not ok 1 KSM NUMA merging
Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
[FAIL]
not ok 2 ksm_tests -N -m 0 # exit=1
After patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 2 ksm_tests -N -m 0
Link: https://lore.kernel.org/78a3b0e3fb94004c0710872c5bab6f7381b7d63c.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Sayali Patil
|
5c4c48ff5a |
selftests/mm: handle EINVAL when configuring gigantic hugepages
Patch series "selftests/mm: avoid false failures in hugetlb and KSM
tests", v3.
This series fixes issues in the hugetlb and KSM MM selftest categories
that can report failures when the prerequisites for the tests are not
satisfied.
Patch 1 updates the hugetlb selftest helpers to handle -EINVAL when
attempting to configure gigantic HugeTLB pages via nr_hugepages. PowerPC
hash MMU pSeries systems expose gigantic hugepage sizes but do not allow
runtime allocation of such pages, causing the sysfs write to fail. Handle
this case gracefully and continue running the test instead of aborting.
Patch 2 fixes the KSM NUMA merge test on systems with memoryless NUMA
nodes. The test currently relies on the number of configured NUMA nodes
and may attempt allocations on nodes that have no memory, resulting in
spurious failures. Use the existing helpers to identify NUMA nodes that
contain memory and skip the test when fewer than two such nodes are
available.
Patch 3 fixes a pre-existing operator precedence issue in ksm_tests, where
a ternary expression combined with logical OR operators could be evaluated
differently than intended. Added parentheses to ensure the correct
evaluation order.
These changes improve handling of unsupported test configurations and
unmet test prerequisites, avoiding spurious failures.
This patch (of 3):
Some MM selftests attempt to configure the amount of HugeTLB pages of
different sizes by writing to nr_hugepages.
PowerPC hash MMU pSeries systems advertise gigantic hugepage sizes but do
not support runtime allocation of such pages, writes to the corresponding
nr_hugepages file fail with -EINVAL. This causes the test to bail out
even though the failure is due to a platform limitation rather than the
functionality being tested.
Ignore -EINVAL when configuring nr_hugepages so that tests continue to run
on systems where gigantic hugepage allocation is unsupported.
Before patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
Bail out! /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages
write(0) failed: Invalid argument
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
[FAIL]
After patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
Link: https://lore.kernel.org/cover.1783446924.git.sayalip@linux.ibm.com
Link: https://lore.kernel.org/2e3b585cbb30b2fc495dcd49d75de6f6da61861c.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Hongfu Li
|
f421d67d2c |
selftests/mm: fix memleak in migration benchmark
Several early return paths in run_migration_benchmark() skip
hmm_buffer_free(), leaking the buffer. Replace with a single cleanup
label.
Link: https://lore.kernel.org/20260709081843.1451202-1-lihongfu@kylinos.cn
Fixes:
|
||
|
xu xin
|
0471cade0a |
ksm: use precise linear_page_index instead of the whole address space
Since we now have linear_page_index available that we can use here, allowing for optimizing the RMAP walk, we can also use it to locate more precisely all related-processes when error hits the KSM page, which will decrease a lot of invalid iterations. Link: https://lore.kernel.org/20260709173312403qgj1Af6pRkFMDSsmc19sM@zte.com.cn Signed-off-by: xu xin <xu.xin16@zte.com.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
887311e5cd |
mm/ksm: initialize the addr only once in collect_procs_ksm
Patch series "KSM: use linear_page_index in collect_procs_ksm()", v2.
In collect_procs_ksm() which is used to collect processes when the error
hit an ksm page, there is the same issue with rmap_walk_ksm (see the
previous discussion at [1]). So we apply the similar logic changes to the
collect_procs_ksm().
The patch [1/2] move the initializaion of addr from the position inside
loop to the position before the loop, since the variable will not change
in the loop.
The patch [2/2] optimize collect_procs_ksm by passing a suitable page
offset range to the anon_vma_interval_tree_foreach loop to reduce
ineffective checks.
This patch (of 2):
Similar to
|
||
|
Mike Rapoport (Microsoft)
|
ef79e0f5e3 |
mm: split out vmalloc declarations from internal.h
mm/internal.h becomes more and more bloated. Move declarations related to vmalloc to a new mm/vmalloc.h header. No functional changes. Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
2b65a42a08 |
mm: split out sparse declarations from internal.h
mm/internal.h becomes more and more bloated. Move declarations related to SPARSE and SPARSE_VMEMMAP memory models to a new mm/sparse.h header. No functional changes. Link: https://lore.kernel.org/20260709-internal-h-v2-2-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
55ed40abb2 |
mm: split out mm_init and memblock declarations from internal.h
Patch series "mm: split a couple of headers from internal.h", v2. mm/internal.h becomes more and more bloated. Split declarations related to mm_init, memblock, vmalloc and sparse into new headers. This patch (of 3): mm/internal.h becomes more and more bloated. Move declarations for related to mm/mm_init.c and mm/memblock.c to a new mm/mm_init.h header. No functional changes. [rppt@kernel.org: split stubfs from internal.h to mm_init.h] Link: https://lore.kernel.org/alJd1BLypyK9Mpaw@kernel.org Link: https://lore.kernel.org/20260709-internal-h-v2-0-695631425968@kernel.org Link: https://lore.kernel.org/20260709-internal-h-v2-1-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
1695841621 |
ksm: add mremap selftests for ksm_rmap_walk
The existing tools/testing/selftests/mm/rmap.c has already one testcase
for ksm_rmap_walk in TEST_F(migrate, ksm), which takes use of migration of
page from one NUMA node to another NUMA node. However, it just lacks the
scenario of mremapped VMAs.
We add the calling of mremap() and then trigger KSM to merge pages before
migrating, which is specifically to test an optimization which is
introduced by this patch ("ksm: Optimize rmap_walk_ksm by passing a
suitable address pgoff").
This test can reproduce the issue that Hugh points out at
https://lore.kernel.org/all/02e1b8df-d568-8cbb-b8f6-46d5476d9d75@google.com/
Link: https://lore.kernel.org/20260703162637070FU4ekl58Hw_Z7OSuJryZB@zte.com.cn
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
xu xin
|
96d2d9acef |
ksm: optimize rmap_walk_ksm by passing a suitable page index
User impact / Why this matters to Linux users ============================================= When a system runs with KSM enabled and memory becomes tight, KSM pages may be swapped out or migrated. The kernel then performs a reverse map walk by rmap_walk_ksm to locate all page table entries that reference these pages. If A large number of unrelated VMAs can attach to a single anon_vma related with this KSM page, then rmap_walk might be severe performance bottleneck. In our embedded test environment, we observed ~20,000 VMAs sharing one anon_vma without any fork purely from VMA splits which cause 200~700ms duration of rmap_walk_ksm. When one of those VMAs mapped a KSM page, then this KSM page's rmapping will become bottleneck with hold its anon_vma lock for a long time. The anon_vma lock is not only used by KSM; it is a core lock protecting the VMA interval tree and is acquired by many critical memory operations: ' Page faults: do_anonymous_page(), do_wp_page() (during COW) ' Memory reclaim: try_to_unmap() ' Page migration & compaction: migrate_pages(), compact_zone() ' mlock / munlock: mlock_fixup() ' Process exit: exit_mmap() (tearing down VMAs) ' Cgroup memory accounting: mem_cgroup_move_charge() If one thread holds the anon_vma lock for hundreds of milliseconds because of an inefficient KSM rmap walk, any other thread that tries to acquire the same lock (e.g., an application taking a page fault, kswapd reclaiming pages, or a migration thread) will block. This leads to stalled application threads, increased latency spikes, and in extreme cases container timeouts or watchdog triggers. This patch reduces the worst-case anon_vma lock hold time during ksm_rmap_walk from >500 ms to <1 ms, thereby almost eliminating this source of lock contention and improving system responsiveness under memory pressure. Real-world examples: ==================== - JVM / Go runtime: These use mmap for heap regions and later call mprotect(PROT_NONE) for garbage collection barriers or guard pages, splitting the original VMA into thousands of small pieces over time. - Database engines (MySQL, PostgreSQL): Large shared memory buffers or anonymous mappings are managed with madvise(MADV_DONTNEED) to release specific pages, which also splits VMAs. Root Cause ========== Through local debugging trace analysis, we found that most of the latency of rmap_walk_ksm occurs within anon_vma_interval_tree_foreach, leading to an excessively long hold time on the anon_vma lock (even reaching 500ms or more), which in turn causes upper-layer applications (waiting for the anon_vma lock) to be blocked for extended periods. Further investigation revealed that 99.9% of iterations inside the anon_vma_interval_tree_foreach loop are skipped due to the first check "if (addr < vma->vm_start || addr >= vma->vm_end)), indicating that a large number of loop iterations are ineffective. This inefficiency arises because the start page index and the end page index parameters passed to anon_vma_interval_tree_foreach span the entire address space from 0 to ULONG_MAX, resulting in very poor loop efficiency. Solution ======== We cannot rely solely on anon_vma to locate all PTEs mapping this page but also need to have the original page's linear_page_index. Since the implementation of anon_vma_interval_tree_foreach it essentially iterates to find a suitable VMA such that the provided page index falls within the candidate's vm_pgoff range. vm_pgoff <= original linear page offset <= (vm_pgoff + vma_pages(v) - 1) Fortunately, an earlier commit introduced the linear_page_index to struct ksm_rmap_item, allowing for optimizing the RMAP walk. Test results ============ A rmap testbench can be obtained with two Out-Of-Tree patches at [1][2]. After applying the OOT patches and building rmap_benchmark from: tools/testing/rmap/rmap_benchmark.c, we can start the performance test. The testing result in QEMU is shown as follows: KSM rmapping Maximum duration Average duration Before: 705.12 ms (705119858 ns) 532.04 ms (532041586 ns) After: 1.67 ms (1665917 ns) 1.44 ms (1443784 ns) The benchmark numbers are realistic, since we observed ~20,000 VMAs sharing one anon_vma on a production system running a Java application with KSM enabled. The lock hold time before the patch was measured at 228ms (max) during rmap walks triggered by memory compaction and page migration. The benchmark reproduces that VMA count and lockhold behavior in a controlled environment. Link: https://lore.kernel.org/20260703162510242nxmjbcLy5ccp1dbZSK3EU@zte.com.cn Link: https://lore.kernel.org/all/202605301703094695zmVgcSC27BNR0rH0N8_x@zte.com.cn [1] Link: https://lore.kernel.org/all/20260530170404509QpJmBtpSjn3uQHeVKA2iA@zte.com.cn/ [2] Co-developed-by: Wang Yaxin <wang.yaxin@zte.com.cn> Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> Signed-off-by: xu xin <xu.xin16@zte.com.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Hugh Dickins <hughd@google.com> Cc: "Liam R. Howlett" <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
a5650de041 |
ksm: add linear_page_index into ksm_rmap_item
Patch series "KSM: performance optimizations for rmap_walk_ksm", v11.
This series fixes a severe KSM reverse-mapping performance problem that
can freeze applications for hundreds of milliseconds under memory pressure
especially when a lot of unrelated VMAs sharing a single anon_vma.
Two key highlights:
1. Lock hold time drops from >500ms to <2ms
- In our benchmark (20,000 VMAs sharing an anon_vma), worst-case
anon_vma lock hold time during KSM rmap walk went from 705ms
down to 1.67ms (max) and 1.44ms (avg).
2. Real user impact
- The anon_vma lock is also acquired by page faults, reclaim,
migration, compaction, mlock, exit_mmap, and cgroup accounting.
- A long hold due to inefficient rmap walks stalls application
threads, causing latency spikes, reduced throughput, or even
container timeouts.
- The problem occurs even without fork() – VMA splitting (e.g.,
via mprotect or madvise over time) can create tens of thousands
of VMAs all attached to the same anon_vma.
Real-world examples:
- JVM / Go runtime: These use mmap for heap regions and later call
mprotect(PROT_NONE) for garbage collection barriers or guard pages,
splitting the original VMA into thousands of small pieces over time.
- Database engines (MySQL, PostgreSQL): Large shared memory buffers or
anonymous mappings are managed with madvise(MADV_DONTNEED) to release
specific pages, which also splits VMAs.
Why the benchmark numbers are realistic: We observed ~20,000 VMAs sharing
one anon_vma on a production system running a Java application with KSM
enabled. The lock hold time before the patch was measured at 228 ms
(max) during rmap walks triggered by memory compaction and page migration.
The benchmark reproduces that VMA count and lock‑hold behavior in a
controlled environment.
For systems that do not have thousands of VMAs per anon_vma, the patch
adds negligible overhead (a single pgoff comparison). For systems that do
suffer from this issue, the improvement is dramatic: 1) Worst‑case
anon_vma lock hold time drops from hundreds of milliseconds to under
2 ms.2)This directly reduces blocking of parallel operations that need
the same lock – page faults, reclaim, migration, compaction, mlock, and
exit_mmap.
End‑users will see lower tail latency (fewer application stalls), higher
throughput under memory pressure, and no more spurious lockup warnings or
container timeouts caused by excessive lock hold times.
In short: workloads that do not hit this pathological pattern are
unaffected; those that do will see a 100x to 500x reduction in lock hold
times, which translates directly into a more responsive system.
This patch (of 3):
As preparation for KSM rmap optimizations, let's track the original
linear_page_index() of a de-duplicated page in its ksm_rmap_item, so we
can efficiently search for the page in an address space, avoiding scanning
the entire address space. This was previously discussed in [1, 2].
To avoid growing ksm_rmap_item, let's squeeze it into the existing
structure by overlying some members (oldchecksum, age, remaining_skips)
that are only relevant while on the unstable tree. The new entry will
only be relevant for entries in the stable tree.
However, as the age information is read by should_skip_rmap_item() with
the smart-scanning approach even while we have an entry in the stable
tree, but the page changes (no longer a KSM page, for example due to COW),
we have to change the handling there a bit.
We'll calculate the linear page index in try_to_merge_with_ksm_page(),
when adding it to the stable tree, and reset the index (to reset overlayed
data) when removing an item from the stable tree -- in
remove_rmap_item_from_tree(), remove_node_from_stable_tree() and
break_cow().
To be specially clarified, the reason for resetting the stored index at
break_cow() is:
- When a page successfully becomes a KSM page (i.e., after
stable_tree_append() sets STABLE_FLAG), both anon_vma and the index are
stored and remain valid.
- However, during the merging process, there are several failure paths
where we already prepared an rmap item to be added to the stable tree,
but must revert that as some part of the merge process failed. Examples
include:
1 The second call to try_to_merge_with_ksm_page() fails in
try_to_merge_two_pages().
2 stable_tree_insert() fails in cmp_and_merge_page().
In such cases, break_cow() is invoked to break the COW mapping and
discard the KSM state.
Currently, break_cow() already contains a
put_anon_vma(rmap_item->anon_vma) to release the reference taken during
the aborted merge. Because the index is logically paired with anon_vma
(both are only meaningful when the rmap_item is in a stable state), it
must also be cleared (or reset) in break_cow() to avoid leaving stale
linear_page_index values that could confuse subsequent rmap walks or
scanning logic.
Link: https://lore.kernel.org/20260703162253688u8Str9eFLR8TGCmo7nIOF@zte.com.cn
Link: https://lore.kernel.org/20260703162357853iIa-RP7if9hRlAIuTh5La@zte.com.cn
Link: https://lore.kernel.org/all/adTPQSb-qSSHviJN@lucifer/ [1]
Link: https://lore.kernel.org/all/202604091806051535BJWZ_FTtdIm3Snk24ei_@zte.com.cn/ [2]
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
SJ Park
|
d59bf2653c |
mm/damon/core: handle unreset probe_hits in probe_hits_mvsum()
If damon_update_monitoring_result() is called at the end of the aggregation interval, probe_hits is not reset. That's because the value will be exposed to the user via damon_region_aggregated trace event. Meanwhile, damon_probe_hits_mvsum() can be called in this state. Due to its logic, it will return a value that is incorrectly high. This could happen if the user requested DAMOS schemes applied regions sysfs files update exactly in the time sequence. The impact is minor, but better to avoid. Check the timing and simply return the fully aggregated last_probe_hits, like damon_nr_accesses_mvsum() also does. It is not 100% accurate since it is the last interval's aggregation. But better than the value that is completely reset. Link: https://lore.kernel.org/20260708135359.122587-8-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
e06b7f0cf8 |
mm/damon/core: update probe hits for new parameter commit
Users can update DAMON parameters at runtime. If the samples and/or aggregation intervals are updated in this way, monitoring results depending on the intervals should also be updated for a more accurate snapshot. The age and nr_accesses are properly updated, while probe_hits are not updated in the way. Do the update. Link: https://lore.kernel.org/20260708135359.122587-7-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
84113a30a8 |
mm/damon/core: s/nr_accesses_for_new_attrs/nr_samples_for_new_attrs/
damon_nr_accesses_for_new_attrs() can be used for not only nr_accesses but also any positive sample count, like probe_hits. Rename to be able to be used for such general uses without confusion. Link: https://lore.kernel.org/20260708135359.122587-6-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
f9088c9845 |
mm/damon/core: s/nr_accesses_to_accesses_bp/sample_count_to_bp/
damon_nr_accesses_to_accesses_bp() actually converts a positive sample count to the ratio. Rename it to better describe what it really does and not confusing for more general uses. Link: https://lore.kernel.org/20260708135359.122587-5-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
71599dc253 |
mm/damon/core: s/accesses_bp_to_nr_accesses/sample_bp_to_count/
accesses_bp_to_nr_accesses() actually converts a positive samples ratio to the count. Rename it to better describe what it really does and not confusing for more general uses. Link: https://lore.kernel.org/20260708135359.122587-4-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
571cc9a34e |
mm/damon/core: s/damon_max_nr_accesses()/damon_nr_samples_per_aggr()/
damon_max_nr_accesses() actually returns the number of samples DAMON checks for each region per each aggregation interval. Rename it to better describe what it really does and not confusing for more general uses. Link: https://lore.kernel.org/20260708135359.122587-3-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
f3ab162a92 |
mm/damon/core: remove comment and test for nr_to_bp() divide-by-zero
Patch series "mm/damon: update probe hits for runtime parameter commits".
DAMON users can update DAMON parameters such as sampling and aggregation
intervals at runtime. For such changes, monitoring results that depend on
the intervals should be properly updated for better accuracy. For
example, the access frequency counter (nr_accesses) is updated. The data
attribute monitoring counter (probe_hits) is not being updated, though.
Do the updates for new parameters.
Patch 1 removes obsolete comments and test code for a function that this
series will touch. Patches 2-5 rename functions that are being used for
nr_accesses update, to be able to be used for probe_hits without
confusion. Patch 6 does the probe_hits update. Patch 7 update
damon_probe_hits_mvsum() to cover a corner case from the update for better
accuracy.
This patch (of 7):
The comments on damon_nr_accesses_to_accesses_bp() and its unit test warn
it can trigger division-by-zero when the aggregation interval is zero.
Commit
|
||
|
David Hildenbrand (Arm)
|
3b7cee2e9a |
mm/bootmem_info: remove CONFIG_HAVE_BOOTMEM_INFO_NODE
The whole infrastructure is unused now. Let's remove the config option along with mm/bootmem_info. + include/linux/bootmem_info.h. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-10-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
4f9ec35df5 |
mm/sparse: remove bootmem_info.h include
No longer required, so let's remove it. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-9-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
788a0efb95 |
mm/hugetlb_vmemmap: remove bootmem_info leftovers
We never set CONFIG_HAVE_BOOTMEM_INFO_NODE, so we can just switch to free_reserved_page() and drop the register_page_bootmem_memmap() call. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-8-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
ea963eab11 |
x86/mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE
CONFIG_HAVE_BOOTMEM_INFO_NODE now essentially doesn't do anything. So let's remove support for CONFIG_HAVE_BOOTMEM_INFO_NODE. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-7-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
f20e92095f |
x86/mm: stop marking page tables as MIX_SECTION_INFO
There is no good reason to mark boot page tables as MIX_SECTION_INFO: we only free boot page tables when they are completely empty, and memory offlining/hotunplug doesn't benefit from it in any way. So just stop marking page tables as MIX_SECTION_INFO. In free_pagetable(), we can now simply free reserved pages directly. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-6-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
7bfc04d6bd |
x86/mm: stop marking vmemmap as SECTION_INFO
We added the whole bootmem registration machinery in commit
|
||
|
David Hildenbrand (Arm)
|
f4436442e5 |
mm/bootmem_info: allow calling free_bootmem_page() on pages without a bootmem_type
As preparation for further changes, let's temporarily allow freeing pages that were not previously registered. This will allow freeing unregistered vmemmap pages allocated during boot through free_bootmem_page() from hugetlb code, until we fully rip all of that out. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-4-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Acked-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
cbbd561893 |
s390/mm: use free_reserved_pages() in vmem_free_pages()
Let's use our new generic helper. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-3-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Heiko Carstens <hca@linux.ibm.com> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
d79db3f396 |
mm: provide free_reserved_pages(), removing x86 variant
Let's extend free_reserved_page() in page_alloc.c to free_reserved_pages(), dropping the custom x86 variant. The common-code variant will consume an order, so adjust the x86 callers accordingly. Make free_reserved_pages() assume that we are freeing ordinary high-order pages, just with the special "reserved" flavor. The target use case for now is freeing vmemmap PMD pages. Set the refcount directly to 0 (instead of 1) and call __free_frozen_pages(). Set the page count to 0 before clearing PG_reserved, so someone checking PG_reserved (and not finding it set) to then try grabbing a ref would not suddenly have that ref be dropped. That is arguably cleaner and safer than the old way of doing it. Add some kerneldoc. Use a single adjust_managed_page_count() call. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-2-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
David Hildenbrand (Arm)
|
5e83b4944d |
x86/mm: drop order parameter from free_pagetable()
Patch series "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)", v2. Let's remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE, performing some smaller cleanups around freeing of reserved vmemmap pages on the way. This patch (of 10): All callers pass 0, so let's drop the parameter. Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-0-4afc76c73d61@kernel.org Link: https://lore.kernel.org/20260716-bootmem_info_part2-v2-1-4afc76c73d61@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: "Borislav Petkov (AMD)" <bp@alien8.de> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Hajime Tazaki
|
324853ce8d |
mm: nommu: fix the error path when vma_iter_prealloc() fails
When vma_iter_prealloc() fails in do_mmap(), it jumps to error_just_free
as a error path of this function, but there are several possible issues.
1) It jumps to error_just_free without updating ret to -ENOMEM, meaning
do_mmap() will return 0 on failure.
2) The error path unconditionally frees the region struct. Since the
region was already added to the global nommu_region_tree via
add_nommu_region(), leaving it makes a potential dangling pointer in
the tree and may cause a use-after-free on the next tree walk.
3) If do_mmap() finds an existing overlapping shared region, it
increments its usage, sets region to this existing pregion, and jumps
to share:
region = pregion;
result = start;
goto share;
When vma_iter_prealloc() fails and jumps to error_just_free, the
error path unconditionally frees the region:
error:
...
if (region->vm_file)
fput(region->vm_file);
kmem_cache_free(vm_region_jar, region);
This potentially leaves a dangling pointer in nommu_region_tree and
causes RB-tree corruption.
4) When establishing a new private mapping, do_mmap_private() allocates
physical pages and assigns them to region->vm_start:
base = alloc_pages_exact(total << PAGE_SHIFT, GFP_KERNEL);
...
region->vm_start = (unsigned long) base;
If we later fail at vma_iter_prealloc() and jump to error_just_free,
the region struct is freed, but the backing physical memory isn't freed
via free_page_series().
5) In the error label of do_mmap(), the vm_area_struct allocated is
freed by vm_area_free(vma) but not called after vma_close(), leaving
potential memory leak which should be handled by a custom .close
handler of vm_ops.
This commit fixes those issues by introducing new jump label,
error_vma_iter_prealloc, to correctly handle the error case of
vma_iter_prealloc(), updating ret value (1), and move the region updates
after the place that the allocation is finished (2).
Additionally, the commit removes the existing goto label, error, and
consolidates to error_just_free as existing `goto error;` code blocks
always release nommu_region_sem.
Moreover, it only frees region allocated in this request to avoid
freeing the shared, existing region shared by other processes (3), and
free physical memory when do_mmap_private() allocates (4). It also add
vma_close() before vm_area_free() to fix the potential leak (5).
Those issues are discovered by Sashiko, linked below.
Link: https://lore.kernel.org/20260708083829.576036-1-thehajime@gmail.com
Link: https://sashiko.dev/#/patchset/20260702012830.667205-1-thehajime%40gmail.com
Link: https://sashiko.dev/#/patchset/c8513ee5aa8444ec9bf6c276043c9f833016a2fa.1783304131.git.thehajime%40gmail.com
Link: https://sashiko.dev/#/patchset/20260707235137.498738-1-thehajime%40gmail.com
Fixes:
|
||
|
Kiryl Shutsemau (Meta)
|
a137d9f8f5 |
Documentation/userfaultfd: document RWP working set tracking
Add an admin-guide section covering UFFDIO_REGISTER_MODE_RWP: - sync and async fault models; - UFFDIO_RWPROTECT semantics; - UFFD_FEATURE_RWP_ASYNC; - UFFDIO_SET_MODE runtime mode flips. It also covers typical VMM working-set-tracking workflow from detection loop through sync-mode eviction and back to async. Link: https://lore.kernel.org/20260708111417.173443-16-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Cc: kernel test robot <lkp@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
3628215f7f |
selftests/mm: add userfaultfd RWP tests
Coverage for UFFDIO_REGISTER_MODE_RWP and UFFDIO_RWPROTECT:
rwp-async async mode — touch pages, verify permissions are
auto-restored without a message
rwp-sync sync mode — access blocks, handler resolves via
UFFDIO_RWPROTECT
rwp-pagemap PAGEMAP_SCAN reports still-cold pages via
inverted PAGE_IS_ACCESSED
rwp-mprotect RWP survives mprotect(PROT_NONE) ->
mprotect(PROT_READ|PROT_WRITE) round-trip
rwp-gup GUP walks through a protnone RWP PTE (pipe
write/read drives the GUP path)
rwp-async-toggle UFFDIO_SET_MODE flips between sync and async
without re-registering
rwp-close closing the uffd restores page permissions
rwp-fork RWP survives fork() with EVENT_FORK; child's
PTEs keep the uffd bit
rwp-fork-pin RWP survives fork() on an RO-longterm-pinned
anon page (forces copy_present_page()); child
read auto-resolves and clears the bit, proving
PAGE_NONE was in place
rwp-wp-exclusive register with MODE_WP|MODE_RWP returns -EINVAL
All tests run against anon, shmem, shmem-private, hugetlb, and
hugetlb-private memory, except rwp-fork-pin which is anon-only —
copy_present_page() is the private-anon pinned-exclusive fork path.
Snapshot the RWP additions into tools/include/uapi/linux/userfaultfd.h so
the selftest builds without requiring "make headers" first, matching the
mechanism established by commit
|
||
|
Kiryl Shutsemau (Meta)
|
39a7c34ea5 |
userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle
Add an ioctl to toggle async mode at runtime without re-registering the userfaultfd. This allows a VMM to switch between sync and async RWP modes on-the-fly -- for example, starting in async mode for working set scanning, then switching to sync mode to intercept faults during page eviction. UFFDIO_SET_MODE takes an enable/disable bitmask of UFFD_FEATURE_* flags. Only UFFD_FEATURE_RWP_ASYNC is toggleable today; the ioctl rejects any other bit with -EINVAL. Enabling RWP_ASYNC also requires RWP to have been negotiated at UFFDIO_API time, mirroring the UFFDIO_API invariant. Fault-path readers of ctx->features run under mmap_read_lock or a per-VMA lock; the RMW takes mmap_write_lock and calls vma_start_write() on every UFFD-armed VMA, so those readers are fully excluded. userfaultfd_show_fdinfo(), however, reads ctx->features without any lock, so the RMW is written as a single WRITE_ONCE and fdinfo reads it with READ_ONCE. That keeps the lockless observer from seeing a mid-RMW intermediate and removes the audit burden when new toggleable bits are added later. When switching to async, pending sync waiters are woken so they retry and auto-resolve under the new mode. Link: https://lore.kernel.org/20260708111417.173443-14-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
45347e3d32 |
userfaultfd: add UFFD_FEATURE_RWP_ASYNC for async fault resolution
Sync RWP delivers a message and blocks the faulting thread until the handler resolves the fault. For working-set tracking the VMM does not need the message: it just needs to know, at scan time, which pages were touched. Async RWP serves that use case — the kernel restores access in-place and the faulting thread continues without blocking. The VMM reconstructs the access pattern after the fact via PAGEMAP_SCAN: pages whose uffd bit is still set (inverted PAGE_IS_ACCESSED) were not re-accessed since the last RWP cycle. Worth calling out: async resolution upgrades writable private anon PTEs via pte_mkwrite() when can_change_pte_writable() allows, mirroring do_numa_page(). Without it, every re-access of an RWP'd writable page would COW-fault a second time. UFFD_FEATURE_RWP_ASYNC requires UFFD_FEATURE_RWP. Link: https://lore.kernel.org/20260708111417.173443-13-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
3389648819 |
mm/pagemap: add PAGE_IS_ACCESSED for RWP tracking
PAGEMAP_SCAN already reports PAGE_IS_WRITTEN from the inverted uffd PTE bit, targeting the UFFDIO_WRITEPROTECT workflow. UFFDIO_RWPROTECT reuses the same PTE bit as a marker for read-write protection, but "has been written" and "has been accessed" are distinct semantic signals — they happen to share one PTE bit today only because the two implementations share infrastructure. Give RWP its own pagemap category so the UAPI does not conflate them: PAGE_IS_WRITTEN reported on VM_UFFD_WP VMAs, !pte_uffd(pte) PAGE_IS_ACCESSED reported on VM_UFFD_RWP VMAs, !pte_uffd(pte) Both still read the same PTE bit today, but each is scoped to the VMA whose registered mode makes the bit meaningful. If a future implementation moves RWP to a separate PTE bit, only PAGE_IS_ACCESSED switches over. This is a UAPI narrowing. Outside VM_UFFD_WP VMAs the uffd bit is always clear, so PAGEMAP_SCAN used to flag PAGE_IS_WRITTEN on every present PTE there — a meaningless duplicate of PAGE_IS_PRESENT. Now PAGE_IS_WRITTEN fires only inside VM_UFFD_WP VMAs. pagemap_hugetlb_category() now takes the vma like its PTE/PMD peers. Link: https://lore.kernel.org/20260708111417.173443-12-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
2d427bb016 |
mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP
Wire the fault side of read-write protection tracking and turn the
userspace interface on.
An RWP-protected PTE is PAGE_NONE with the uffd bit set. The PROT_NONE
triggers a fault on any access; the uffd bit distinguishes it from plain
mprotect(PROT_NONE) or NUMA hinting.
Fault dispatch, per level:
PTE handle_pte_fault() -> do_uffd_rwp()
PMD __handle_mm_fault() -> do_huge_pmd_uffd_rwp()
hugetlb hugetlb_fault() -> hugetlb_handle_userfault()
The RWP branches gate on userfaultfd_pte_rwp() /
userfaultfd_huge_pmd_rwp() (VM_UFFD_RWP plus the uffd bit) and fall
through to do_numa_page() / do_huge_pmd_numa_page() otherwise. Each
delivers a UFFD_PAGEFAULT_FLAG_RWP message through handle_userfault(); the
handler resolves it with UFFDIO_RWPROTECT clearing MODE_RWP.
userfaultfd_must_wait() and userfaultfd_huge_must_wait() add matching
protnone+uffd waiters so sync-mode fault handlers block correctly.
Expose the UAPI:
UFFDIO_REGISTER_MODE_RWP -> UFFD_API_REGISTER_MODES
UFFD_FEATURE_RWP -> UFFD_API_FEATURES
_UFFDIO_RWPROTECT -> UFFD_API_RANGE_IOCTLS
UFFD_API_RANGE_IOCTLS_BASIC
UFFD_FEATURE_RWP is masked out at UFFDIO_API time when PROT_NONE is not
available or VM_UFFD_RWP aliases VM_NONE (32-bit), so userspace never sees
an advertised-but-broken feature.
Works on anonymous, shmem, and hugetlb memory.
Link: https://lore.kernel.org/20260708111417.173443-11-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Kiryl Shutsemau (Meta)
|
6eab8f2cc6 |
userfaultfd: add UFFDIO_REGISTER_MODE_RWP and UFFDIO_RWPROTECT plumbing
Add the userspace interface for read-write protection tracking:
- UFFDIO_REGISTER_MODE_RWP register a range for RWP tracking
- UFFD_FEATURE_RWP capability bit
- UFFDIO_RWPROTECT install / remove RWP on a range
Introduce CONFIG_USERFAULTFD_RWP, auto-selected on 64-bit kernels with
ARCH_HAS_PTE_PROTNONE and HAVE_ARCH_USERFAULTFD_WP. The symbol gates
VM_UFFD_RWP (previously aliased to VM_NONE) and the smaps/trace-flag hooks
added in the preparatory patches; without it the UAPI bits added here have
nothing to drive and would be unreachable.
Registration sets VM_UFFD_RWP on the VMA. Combining MODE_WP with MODE_RWP
is rejected because both modes claim the uffd PTE bit.
UFFDIO_RWPROTECT is the bidirectional counterpart of
UFFDIO_WRITEPROTECT:
- MODE_RWP change_protection() with MM_CP_UFFD_RWP
installs PAGE_NONE and sets the uffd bit on
present PTEs
- !MODE_RWP change_protection() with MM_CP_UFFD_RWP_RESOLVE
restores vma->vm_page_prot and clears the bit
userfaultfd_clear_vma() runs the same resolve pass on unregister so RWP
state cannot outlive the uffd.
Re-registering a range must not drop a mode that installs per-PTE markers
(WP or RWP); doing so returns -EBUSY. This also closes a pre-existing
window where re-registering without MODE_WP would strand uffd-wp markers:
before, those caused extra write-faults but were otherwise benign; with
RWP preservation in place, a subsequent mprotect() on a VM_UFFD_RWP VMA
would silently promote the stale markers to RWP.
The feature is not yet advertised. UFFDIO_REGISTER_MODE_RWP,
UFFD_FEATURE_RWP, and _UFFDIO_RWPROTECT are intentionally absent from
UFFD_API_REGISTER_MODES, UFFD_API_FEATURES, and UFFD_API_RANGE_IOCTLS, so
UFFDIO_API masks them out and the register-mode validator rejects the bit.
The follow-up patch adds fault dispatch and exposes the UAPI.
Link: https://lore.kernel.org/20260708111417.173443-10-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Kiryl Shutsemau (Meta)
|
7974c23853 |
mm: handle VM_UFFD_RWP in khugepaged, rmap, and GUP
Three mm paths outside the fault handler gate on the uffd PTE bit today: khugepaged (skip collapse on ranges carrying markers), rmap (cap unmap batching), and GUP (force a fault through gup_can_follow_protnone). Extend each to treat VM_UFFD_RWP the same as VM_UFFD_WP; otherwise per-PTE RWP state is silently destroyed or bypassed. khugepaged: try_collapse_pte_mapped_thp() and file_backed_vma_is_retractable() already refuse to collapse or retract page tables on ranges carrying the uffd PTE bit. Broaden the VMA predicate from userfaultfd_wp() to userfaultfd_protected() so VM_UFFD_RWP ranges get the same protection. hpage_collapse_scan_pmd() needs no change — its existing pte_uffd() check already catches an RWP PTE because it carries the uffd bit. rmap: folio_unmap_pte_batch() caps batching at 1 for VM_UFFD_RWP so the restore path handles each PTE with its own marker. GUP: gup_can_follow_protnone() forces a fault on VM_UFFD_RWP VMAs regardless of FOLL_HONOR_NUMA_FAULT. RWP uses protnone as an access-tracking marker, not for NUMA hinting, so any GUP — read or write — must go through the userfaultfd fault path. Link: https://lore.kernel.org/20260708111417.173443-9-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
763f43865f |
mm: preserve RWP marker across PTE rewrites
The uffd PTE bit must survive any kernel path that rewrites a PTE on a VM_UFFD_RWP VMA, otherwise the marker that carries PAGE_NONE semantics is silently dropped and the next access leaks past RWP tracking. Wire the preservation through every path that rewrites a VM_UFFD_RWP PTE. Swap and device-exclusive: do_swap_page(), restore_exclusive_pte(), and unuse_pte() (swapoff()) re-apply PAGE_NONE when the swap PTE carries the uffd bit and the VMA has VM_UFFD_RWP. Migration: remove_migration_pte() and remove_migration_pmd() do the same after the migration entry is replaced with a real PTE/PMD. Fork: __copy_present_ptes(), copy_present_page(), copy_nonpresent_pte(), copy_huge_pmd(), copy_huge_non_present_pmd(), and copy_hugetlb_page_range() keep the uffd bit on the child when the destination VMA has VM_UFFD_RWP, matching the existing VM_UFFD_WP handling. Add VM_UFFD_RWP to VM_COPY_ON_FORK so the flag itself propagates. mprotect(): change_pte_range() and change_huge_pmd() restore PAGE_NONE after pte_modify()/pmd_modify() have recomputed the base protection from a (possibly user-changed) vm_page_prot. pte_modify() preserves _PAGE_UFFD, so the bit stays; we just have to force PAGE_NONE back on top. Link: https://lore.kernel.org/20260708111417.173443-8-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
9cf3c554ac |
mm: add MM_CP_UFFD_RWP change_protection() flag
Preparatory patch. Add the change_protection() primitive that userfaultfd RWP will use. An RWP-protected PTE is PAGE_NONE with the uffd PTE bit set. The PROT_NONE half makes the CPU fault on any access; the uffd bit distinguishes an RWP fault from a plain mprotect(PROT_NONE) or NUMA hinting fault. MM_CP_UFFD_WP and MM_CP_UFFD_RWP share the same PTE bit, so the two cannot be used together on the same range. Two new change_protection() flags: MM_CP_UFFD_RWP install PAGE_NONE and set the uffd bit MM_CP_UFFD_RWP_RESOLVE restore vma->vm_page_prot, clear the uffd bit Both are wired through change_pte_range(), change_huge_pmd(), and hugetlb_change_protection() so anon, shmem, THP, and hugetlb all share the same semantics. Link: https://lore.kernel.org/20260708111417.173443-7-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |