mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-08-31 03:35:32 -04:00
5ec02309d774e46bcf92d4262dfbdcf6ee2998b3
1464179 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lorenzo Stoakes
|
5ec02309d7 |
mm/rmap: elide unnecessary static inline's in interval_tree.c
It's not necessary to declare these functions static inline as they are contained within a single compilation unit. This makes the anonymous interval tree code consistent with the newly updated file-backed interval tree code. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-7-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
7a67b96af0 |
mm/rmap: parameterise vma_interval_tree_*() by address_space
The file-backed mapping interval tree functions vma_interval_tree_*() accept a raw rb_root_cached pointer to determine the tree in which they are operating. However, in each case, this is always associated with an address_space data type. So simply pass a pointer to that instead to simplify the code, and more clearly differentiate between these operations and those concerning anonymous mappings. While we're here, make the generated interval tree functions static as they do not need to be used externally (any previously existing external users have now been removed). We also rename VMA parameters from 'node' to 'vma' as calling this a node is simply confusing, update the input index types to pgoff_t since they reference page offsets and rename the parameters to pgoff_start and pgoff_last. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-6-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
dbde17041c |
mm/rmap: update mm/interval_tree.c comments
Update the file comment to clarify that both file-backed and anonymous interval trees are provided, referencing the relevant data types for clarity. Also add comments to indicate which parts of the file apply to each. While we're here, convert the VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA(). Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-5-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
1219731c49 |
mm: introduce and use vma_end_pgoff()
We already have vma_last_pgoff() which retrieves the last page offset within a VMA. However, code often wishes to span a page offset range, which requires the exclusive end of this range. So provide this in vma_end_pgoff() and update vma_last_pgoff() to use this function. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-4-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
3542bc07f4 |
tools/testing/vma: use vma_start_pgoff() in merge tests
Now we have the vma_start_pgoff() helper, update the merge tests to make use of it for consistency. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-3-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
f9711438df |
mm: add kdoc comments for vma_start/last_pgoff()
Describe what vma_start_pgoff() and vma_last_pgoff() actually provide in detail. This is in order that we can differentiate this between functions that will be added in a subsequent patch which provide a different page offset. We go to lengths to describe the edge cases that can be run into here. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-2-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
995f0eed5e |
mm: move vma_start_pgoff() into mm.h and clean up
Patch series "mm: make VMA page offset handling more consistent", v2. This series performs a series of cleanups and improvements around how the vma->vm_pgoff field is used. Folios belonging to file-backed mappings are simply indexed by the page offset within the file they map (excepting pfnmap and driver mappings which sometimes do things different). However, anonymous folios belonging to pure anonymous mappings are indexed by their "virtual" page offset, which is equal to addr >> PAGE_SHIFT at the time at which the VMA was first faulted in. The page offset of a VMA is stored in vma->vm_pgoff and indicates the page offset of the start of the VMA range, whether it be file-backed or anonymous. The work here both cleans up how we reference this field, as well as laying the foundations for a future series which addresses the inconsistency of CoW'd folios in MAP_PRIVATE-file backed mappings, which are indexed as if they were file-backed but behave as if they were anonymous. This future series will make it such that all anonymous folios are indexed by virtual page offset whether belonging to VMAs who satisfy vma_is_anonymous() or MAP_PRIVATE-mapped file-backed mappings. This series: * Exposes vma_start_pgoff() and updates the kernel to use it consistently. * Adds and uses the useful vma_end_pgoff() helper. * Parameterises the file-backed mapping helpers vma_interval_tree_*() by address_space rather than rb_root_cached. * Renames: the misleadingly-named vma_interval_tree_*() helpers to mapping_rmap_tree_*() to be consistent with the renamed anon_rmap_tree_*(). * Parameterises anon_rmap_tree_*() by anon_vma. * Moves mm/interval_tree.c to the rmap section. * Adds vmg_*() helpers for page offset. * Clarifies the confusing vmg_adjust_set_range() function. * Introduces linear_page_delta() to provide relative pgoff within a VMA. * Replaces open-coded versions of linear_page_delta() and linear_page_index() with invocations of these functions. * Introduces and uses vma_assert_can_modify() to account for whether a VMA can be modified (detached or write locked). * Adds and uses vma_[add,sub]_pgoff() to adjust VMA page offset. * Moves __install_special_mapping() to vma.c. * Makes vma_set_range() static and internal to vma.c. * Introduces and makes use of vma_set_pgoff(). * Fixes incorrect vma.h header inclusion. * Defaults VMA userland tests to 64-bit vma flags size. * Updates VMA userland tests to give better output on failure. * Various smaller cleanups. * Updated the vma_set_pgoff() assert to account for MAP_PRIVATE /dev/zero correctly. This patch (of 33): vma_last_pgoff() already lives there, so it's a bit odd to keep vma_start_pgoff() in mm/interval_tree.c. Move them together. These each return unsigned long, which pgoff_t is typedef'd to. Make this consistent and have these functions return pgoff_t instead. Additionally, express vma_last_pgoff() in terms of vma_start_pgoff(), since we wrap the vma->vm_pgoff access, we may as well use it here. Also while we're here, const-ify the VMA and cleanup a bit. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-0-2a5aa403d977@kernel.org Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-1-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
aa8543e662 |
drm/gpusvm: use hmm_range_fault_unlocked_timeout() for range faults
Several GPU SVM paths take mmap_read_lock() only to call hmm_range_fault() and open-code mmu interval sequence setup before each HMM walk. They also retry -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires. Use hmm_range_fault_unlocked_timeout() for those faults. The HMM helper now owns mmap_lock acquisition and refreshes range->notifier_seq for its internal retries, while GPU SVM keeps its existing driver-lock validation with mmu_interval_read_retry() after a successful fault. drm_gpusvm_scan_mm() and drm_gpusvm_range_evict() pass HMM_RANGE_DEFAULT_TIMEOUT as the helper retry budget for each HMM fault attempt. drm_gpusvm_get_pages() keeps its existing absolute outer deadline because it can be reached from GPU page-fault workers, where fatal signals from the faulting process cannot stop an endless invalidation retry loop. It passes the remaining time from that deadline to HMM for each fault attempt. Leave drm_gpusvm_check_pages() on hmm_range_fault() because that path is called with the mmap lock already held by its caller. Link: https://lore.kernel.org/20260723-hmm-v10-v11-8-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
a1b25d7caf |
accel/amdxdna: use hmm_range_fault_unlocked_timeout() for range population
aie2_populate_range() takes mmap_read_lock() only around hmm_range_fault(). It also open-codes the mmu interval sequence setup before each HMM walk and retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires. Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns the mmap lock and refreshes mapp->range.notifier_seq for its internal retries, so the driver only needs to call the helper and then validate the sequence before marking the mapping populated. Pass HMM_RANGE_DEFAULT_TIMEOUT as the helper retry budget for each HMM population attempt. This scopes the timeout to repeated HMM notifier retries while preserving the existing outer loop that moves between invalid mappings and restarts when the interval is invalidated before the driver updates its mapping state. Keep returning -ETIME when the HMM retry budget expires, matching the driver's existing timeout error convention. Link: https://lore.kernel.org/20260723-hmm-v10-v11-7-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
e269fb946b |
RDMA/umem: use hmm_range_fault_unlocked_timeout() for ODP faults
ib_umem_odp_map_dma_and_lock() takes mmap_read_lock() only around hmm_range_fault(), then retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires. Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns the mmap lock and refreshes range->notifier_seq for its internal retries. ODP keeps using HMM_RANGE_DEFAULT_TIMEOUT for each HMM fault attempt, while interval invalidation retries continue to be handled by the existing outer loop. ODP still validates the interval notifier sequence while holding umem_mutex before DMA mapping pages. Link: https://lore.kernel.org/20260723-hmm-v10-v11-6-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
74682a7774 |
drm/nouveau: use hmm_range_fault_unlocked_timeout() for SVM faults
nouveau_range_fault() takes mmap_read_lock() only to call hmm_range_fault(). It also keeps a single HMM_RANGE_DEFAULT_TIMEOUT deadline across both HMM -EBUSY retries and post-fault mmu_interval_read_retry() retries. Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns the mmap lock and refreshes range->notifier_seq for its internal retries. Nouveau keeps its existing absolute deadline in the outer loop and passes the remaining jiffies to the helper for each fault attempt, so retries caused by mmu_interval_read_retry() do not reset the overall retry budget. Nouveau still validates the interval notifier sequence while holding svmm->mutex before programming the GPU mapping. Link: https://lore.kernel.org/20260723-hmm-v10-v11-5-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
39899823ad |
mshv: use hmm_range_fault_unlocked_timeout() for region faults
MSHV currently faults movable memory regions by taking mmap_read_lock() around hmm_range_fault(). That prevents the fault path from handling VMAs whose fault handlers need to drop mmap_lock, such as userfaultfd-backed mappings. Use hmm_range_fault_unlocked_timeout() instead. Passing a timeout of 0 preserves MSHV's existing unbounded retry behavior while letting the HMM helper own mmap_lock acquisition and refresh range->notifier_seq internally before walking the range. After the fault succeeds, MSHV still takes mreg_mutex and checks mmu_interval_read_retry() before installing the pages into the region, so the existing invalidation synchronization is preserved. Fold the small fault-and-lock helper into mshv_region_range_fault(), since the remaining retry path is just the standard "fault, take the driver lock, check the interval notifier sequence" pattern. Link: https://lore.kernel.org/20260723-hmm-v10-v11-4-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
9365813cad |
selftests/mm: add HMM test for mmap lock-dropping faults
Add test_hmm coverage for the HMM lock-dropping fault path. The test module gets a new HMM_DMIRROR_READ_UNLOCKED ioctl that calls hmm_range_fault_unlocked_timeout() with a timeout of 0, exercising the unbounded retry mode while allowing the mmap lock to be dropped during fault handling. Add a userfaultfd_read selftest that registers an anonymous mapping with UFFDIO_REGISTER_MODE_MISSING, services the faults from a handler thread with UFFDIO_COPY, and verifies that HMM can read back the data supplied by the handler. This exercises the path where handle_mm_fault() drops mmap_lock and hmm_range_fault_unlocked_timeout() restarts the walk internally. Assisted-by: GitHub-Copilot:claude-opus-4.6 Link: https://lore.kernel.org/20260723-hmm-v10-v11-3-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jason Gunthorpe <jgg@nvidia.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
1211708312 |
mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
hmm_range_fault() requires the caller to hold the mmap read lock for the duration of the call. This is incompatible with mappings whose fault handler may release the mmap lock, notably userfaultfd-managed regions, where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED after dropping the lock. Drivers that need to populate device page tables for such mappings have no way to do so today. Add hmm_range_fault_unlocked_timeout() for callers that do not need to hold mmap_lock across any work outside the HMM fault itself. The helper takes mmap_read_lock_killable() internally, calls the common HMM fault implementation, and releases the lock before returning if it is still held. The timeout is specified in jiffies; passing 0 retries indefinitely, while a non-zero timeout makes the helper return -EBUSY when the retry budget expires. The retry deadline is set before refreshing the notifier sequence and acquiring mmap_lock, so contended mmap_lock acquisition is included in the retry budget. After acquiring mmap_lock, the helper also rejects unstable address spaces before walking page tables. When handle_mm_fault() drops mmap_lock, or when the range is invalidated, hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and retries the walk internally. If the lock was dropped, the retry deadline is also restarted because a lock-dropping fault handler made progress. Ordinary -EBUSY retries keep the existing deadline, preserving the caller's timeout policy for repeated mmu-notifier invalidations. The caller only needs to perform the usual post-success mmu_interval_read_retry() check while holding its update lock before consuming the pfns. If mmap_lock acquisition is interrupted or a fatal signal is pending during retry handling, -EINTR is returned instead. The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The existing hmm_range_fault() path still passes no locked state, does not allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper preserving the existing API contract for current callers. The previous refactor that moved page fault handling out of the page-table walk callbacks is what makes this change small. Faults now run after walk_page_range() has unwound, with only mmap_lock held, so dropping it does not interact with the walker's pte spinlock or hugetlb_vma_lock. Hugetlb regions therefore participate in the unlocked path uniformly with PTE- and PMD-level mappings; no special case is required. Documentation/mm/hmm.rst is updated with a description of the new API and the recommended caller pattern. Link: https://lore.kernel.org/20260723-hmm-v10-v11-2-c55b003a4b61@gmail.com Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dexuan Cui <decui@microsoft.com> Cc: Haiyang Zhang <haiyangz@microsoft.com> Cc: Jason Gunthorpe <jgg@nvidia.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: K. Y. Srinivasan <kys@microsoft.com> Cc: Leon Romanovsky <leon@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lizhi Hou <lizhi.hou@amd.com> Cc: Long Li <longli@microsoft.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Lyude <lyude@redhat.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Maxime Ripard <mripard@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Oded Gabbay <ogabbay@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Liu <wei.liu@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Stanislav Kinsburskii
|
1ef12d3b30 |
mm/hmm: move page fault handling out of walk callbacks
Patch series "mm/hmm: Add mmap lock-drop support for userfaultfd-backed
mappings", v11.
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers most notably userfaultfd require the mmap lock to
be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the mmap
lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
This patch (of 8):
hmm_range_fault() currently triggers page faults from inside the
page-table walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the
walk framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out
of walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Link: https://lore.kernel.org/20260723-hmm-v10-v11-0-c55b003a4b61@gmail.com
Link: https://lore.kernel.org/20260723-hmm-v10-v11-1-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Long Li <longli@microsoft.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Lyude <lyude@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Andrew Morton
|
cf6daf2493 |
mm/hmm: make CONFIG_HMM_MIRROR select CONFIG_MMU_NOTIFIER
hmm doesn't currently build well with MMU_NOTIFIER=n: With microblaze https://download.01.org/0day-ci/archive/20260702/202607021433.DYT5fDqE-lkp@intel.com/config (CONFIG_MMU_NOTIFIER=n): mm/hmm.c: In function 'hmm_range_fault_unlocked_timeout': mm/hmm.c:804:25: error: implicit declaration of function 'mmu_interval_read_begin'; did you mean 'mmu_interval_check_retry'? [-Wimplicit-function-declaration] 804 | mmu_interval_read_begin(range->notifier); | ^~~~~~~~~~~~~~~~~~~~~~~ | mmu_interval_check_retry Quoting Stanislav: : Documentation/mm/hmm.rst explicitly states: : : "Address space mirroring's main objective is to allow duplication of a : range of CPU page table into a device page table; HMM helps keep both : synchronized. A device driver that wants to mirror a process address : space must start with the registration of a mmu_interval_notifier" : : I think making CONFIG_HMM_MIRROR dependent on the CONFIG_MMU_NOTIFIER : is the right thing to do. So select MMU_NOTIFIER if hmm.c is to be compiled. Cc: David Hildenbrand <david@kernel.org> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Leon Romanovsky <leon@kernel.org> Cc: Stanislav Kinsburskii <skinsburskii@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
ac59b45efd |
Docs/ABI/damon: document probe weight file
Update DAMON ABI document for the newly added probe weight sysfs file. Link: https://lore.kernel.org/20260710134651.18084-20-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
3baf78e04b |
Docs/admin-guide/mm/damon/usage: document weight sysfs file
Update DAMON usage document for newly added probe weight file. Link: https://lore.kernel.org/20260710134651.18084-19-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
f10489d2ef |
Docs/mm/damon/design: document attrs-only monitoring
Update DAMON design document for the attributes only monitoring mode. Link: https://lore.kernel.org/20260710134651.18084-18-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
2430a8d517 |
mm/damon/sysfs: implement probe/weight file
Implement a new sysfs file, 'weight', under probe directory. Users will be able to set the probe weight and enable the attributes only monitoring using it. Link: https://lore.kernel.org/20260710134651.18084-17-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
8d5e0ebac9 |
mm/damon/core: implement damon_has_probe_weight()
DAMON monitors only data attributes when probe weights are set. However, the function for detecting it is intentionally returning false always. The intention was to make changes for complete support of the mode in a safe and incremental way. Now all the building blocks are made. Complete the mode detection function so that the mode can really start working. Link: https://lore.kernel.org/20260710134651.18084-16-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
db697312bb |
mm/damon/core: get merge threshold from probe hits when weights are set
When probe weights are set, DAMON merges regions based on their probe hits weighted sum. But the merge threshold is calculated based on the access frequency. Update it to retrieve the maximum probe hits weighted sum in the snapshot from apply_probes() ops callback, and generate the threshold based on it. Link: https://lore.kernel.org/20260710134651.18084-15-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
b9bc678bfb |
mm/damon/core: s/max_nr_accesses/max_merge_score/ in kdamond_fn()
When probe weights are set, DAMON merge logic works with the probe hits weighted sum. The core logic is setting the merge threshold based on the access frequency, though. Before using probe hits based threshold, rename the variables to look more general. Link: https://lore.kernel.org/20260710134651.18084-14-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
a8ba965a4c |
mm/damon/core: set samples in apply_probes() if probe weights are set
When probe weights are set, the prepare_access_checks() ops callback is not invoked, so sampling addresses of regions are unset. Ask apply_probes() to set those together in the case. Link: https://lore.kernel.org/20260710134651.18084-13-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
05c978da7f |
mm/damon/core: disable access monitoring when probe weights are set
When users set probe weights, they are interested in the data attributes. Not necessarily they are interested in data access patterns. Disable the data access monitoring in the case. Link: https://lore.kernel.org/20260710134651.18084-12-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
23c0623f80 |
mm/damon/core: validate params for probe hits weighted sum overflow
damon_probe_hits_wsum() could overflow in weird setups. Users could set the weight unreasonably high. They could also set the aggregation interval unreasonably high compared to the sampling interval. Such user setup is unlikely. Even if such setup is used, damon_has_probe_weights() always returns false, so the overflow cannot happen. The function may be completed in future, though. Even if the overflow happens, the consequence is degraded monitoring results for the unreasonable setup. It is just a trivial user experience issue. It is still better to be prevented unless the cost is expensive. Avoid the overflow by adding the parameter validation in the core layer parameters validation function. Link: https://lore.kernel.org/20260710134651.18084-11-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
93508425db |
mm/damon/core: disallow probe_hits overflow on attrs only monitoring
When any damon_probe->weight is set, DAMON will do only probe monitoring. probe_hits is 'unsigned char'. It could overflow when the aggregation interval is larger than the sampling interval times 256. damon_as_probe_weights() always return false, so such overflow cannot happen. Even if it happens, it only degrades the monitoring results. That said, the overflow is not intentional. It is better to be prevented as long as the cost is not expensive. Disallow the overflow by adding a validation logic on the core layer parameters validation function. Link: https://lore.kernel.org/20260710134651.18084-10-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
0d1daaed8b |
mm/damon/core: extend merge function to work with probe hits
When probe weights are set, users may want DAMON monitoring results to be optimized for the weights. For that, regions adjustment should work for the weighted sum of probe hits. Extend damon_merge_regions_of() to detect if the weights are set, and work with probe hits in the case. The weights setup detection function is incomplete. It always returns false. It is intentional, so that more changes to completely support weights can be made in an incremental but safe way. Until the function is completed, all changes depend on it is no-op, so DAMON works in the current mode. Link: https://lore.kernel.org/20260710134651.18084-9-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
e1f150d415 |
mm/damon/core: use abs_diff() instead of abs()
Use of abs() in damon_merge_regions_of() could cause a silent integer overflow since the macro casts unsigned int to signed int. It is unlikely to have such a large value for nr_accesses. Even though it happens, the user impact is just degraded monitoring results. Users showing bad monitoring results for weird setup is quite trivial. But the code is obviously wrong. Use abs_diff() instead. The issue was discovered [1] by Sashiko. Link: https://lore.kernel.org/20260710134651.18084-8-sj@kernel.org Link: https://lore.kernel.org/20260705213817.100841-1-sj@kernel.org/ [1] Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
3858025f48 |
mm/damon/paddr: respect return_max_wsum
apply_probes() ops implementation in DAMON_PADDR is ignoring return_max_wsum. Respect it. Link: https://lore.kernel.org/20260710134651.18084-7-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
cfe9e8c738 |
mm/damon/core: implement damon_probe_hits_wsum()
When damon_probe->weight is set, the weighted sum of probe hits will be useful. It will be useful for not only the users but also DAMON internal logics like regions merging. Implement a function for calculating it. Link: https://lore.kernel.org/20260710134651.18084-6-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
d235513a7c |
mm/damon/core: ask apply_probe() to return max probe hits weighted sum
check_accesses() DAMON ops callback returns the maximum nr_accesses of regions. DAMON core uses it to calculate a reasonable region merge threshold. The core will need to adjust regions for not nr_accesses but probe hits weighted sum in future. For that, the core needs to know the maximum weighted sum of the regions. Update the protocol for the task. Link: https://lore.kernel.org/20260710134651.18084-5-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
a75de62a5f |
mm/damon/paddr: set samples in apply_probes() if requested
apply_probe() callback implementation in DAMON_PADDR is ignoring set_samples parameter. Respect it. Link: https://lore.kernel.org/20260710134651.18084-4-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
1138ec78dc |
mm/damon/core: ask apply_probes() ops callback to set sampling address
prepare_access_checks() DAMON ops callback sets the monitoring sampling address per region. In future, DAMON will be able to call only apply_probes(). In this case, applyy_probes() may need to do the sampling address setup, to minimize unnecessary regions iteration. Update the protocol for the request. Link: https://lore.kernel.org/20260710134651.18084-3-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
37aae41c10 |
mm/damon/core: introduce damon_probe->weight
Patch series "mm/damon: introduce data attributes only monitoring". TL;DR: Introduce a way to get DAMON's best effort accuracy monitoring of user-demanding non-access data attributes. Background ========== DAMON was initially designed for only access monitoring. It turned out users want to get the information together with more data attributes. For example, some users want to know how much of a hot memory region belongs to huge pages or specific cgroups. Page level properties based monitoring was introduced with commit |
||
|
Christoph Hellwig
|
747beac679 |
mm: remove wb_writeout_inc
Remove this entirely unused but exported function. Link: https://lore.kernel.org/20260710051052.1839523-1-hch@lst.de Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Jan Kara <jack@suse.cz> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Acked-by: SJ Park <sj@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
John Hubbard
|
c494788faf |
mm/gup: fix GUP-fast fallback for NULL-mapping order-0 folios
Since commit |
||
|
Sayali Patil
|
4e1fbffb33 |
selftests/mm: fix ternary operator precedence in ksm_tests
The KSM selftest uses conditional expressions to skip accesses to
merge_across_nodes on systems without NUMA support. However, the ternary
operator is combined with logical OR without parentheses:
a || numa_available() ? 0 : b || c
Due to operator precedence rules, this is parsed as:
(a || numa_available()) ? 0 : (b || c)
instead of the intended:
a || (numa_available() ? 0 : b) || c
Add parentheses around the conditional expressions to ensure the
correct evaluation order.
Link: https://lore.kernel.org/ce859430287ed2642848c933a90eb9a69da361f0.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Sayali Patil
|
15828a150c |
selftests/mm: fix ksm NUMA merge test for systems with memoryless NUMA nodes
The KSM NUMA merge test allocates identical pages on different NUMA nodes
and verifies KSM behavior with merge_across_nodes enabled and disabled.
On systems with memoryless NUMA nodes, for example:
#numactl -H
available: 2 nodes (0,4)
.....
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 14825 MB
node 0 free: 1382 MB
node 4 cpus:
node 4 size: 0 MB
node 4 free: 0 MB
the test may attempt to allocate memory on a node without memory, causing
numa_alloc_onnode() to fail and resulting in a spurious test failure.
The test currently checks numa_num_configured_nodes() to determine whether
sufficient NUMA nodes are available. However, configured nodes do not
necessarily have memory.
Reuse the existing get_first_mem_node() and get_next_mem_node() helpers to
locate NUMA nodes that actually contain memory, and skip the test when
fewer than two such nodes are available.
Before patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
mbind: Invalid argument
ok 1 KSM NUMA merging
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
mbind: Invalid argument
not ok 1 KSM NUMA merging
Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
[FAIL]
not ok 2 ksm_tests -N -m 0 # exit=1
After patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 2 ksm_tests -N -m 0
Link: https://lore.kernel.org/78a3b0e3fb94004c0710872c5bab6f7381b7d63c.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Sayali Patil
|
5c4c48ff5a |
selftests/mm: handle EINVAL when configuring gigantic hugepages
Patch series "selftests/mm: avoid false failures in hugetlb and KSM
tests", v3.
This series fixes issues in the hugetlb and KSM MM selftest categories
that can report failures when the prerequisites for the tests are not
satisfied.
Patch 1 updates the hugetlb selftest helpers to handle -EINVAL when
attempting to configure gigantic HugeTLB pages via nr_hugepages. PowerPC
hash MMU pSeries systems expose gigantic hugepage sizes but do not allow
runtime allocation of such pages, causing the sysfs write to fail. Handle
this case gracefully and continue running the test instead of aborting.
Patch 2 fixes the KSM NUMA merge test on systems with memoryless NUMA
nodes. The test currently relies on the number of configured NUMA nodes
and may attempt allocations on nodes that have no memory, resulting in
spurious failures. Use the existing helpers to identify NUMA nodes that
contain memory and skip the test when fewer than two such nodes are
available.
Patch 3 fixes a pre-existing operator precedence issue in ksm_tests, where
a ternary expression combined with logical OR operators could be evaluated
differently than intended. Added parentheses to ensure the correct
evaluation order.
These changes improve handling of unsupported test configurations and
unmet test prerequisites, avoiding spurious failures.
This patch (of 3):
Some MM selftests attempt to configure the amount of HugeTLB pages of
different sizes by writing to nr_hugepages.
PowerPC hash MMU pSeries systems advertise gigantic hugepage sizes but do
not support runtime allocation of such pages, writes to the corresponding
nr_hugepages file fail with -EINVAL. This causes the test to bail out
even though the failure is due to a platform limitation rather than the
functionality being tested.
Ignore -EINVAL when configuring nr_hugepages so that tests continue to run
on systems where gigantic hugepage allocation is unsupported.
Before patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
Bail out! /sys/kernel/mm/hugepages/hugepages-16777216kB/nr_hugepages
write(0) failed: Invalid argument
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
[FAIL]
After patch:
-------------------------
running ./hugetlb-madvise
-------------------------
TAP version 13
1..1
[INFO] detected hugetlb page size: 16777216 KiB
[INFO] detected hugetlb page size: 16384 KiB
ok 1 MADV_DONTNEED and MADV_REMOVE on hugetlb
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
Link: https://lore.kernel.org/cover.1783446924.git.sayalip@linux.ibm.com
Link: https://lore.kernel.org/2e3b585cbb30b2fc495dcd49d75de6f6da61861c.1783446924.git.sayalip@linux.ibm.com
Fixes:
|
||
|
Hongfu Li
|
f421d67d2c |
selftests/mm: fix memleak in migration benchmark
Several early return paths in run_migration_benchmark() skip
hmm_buffer_free(), leaking the buffer. Replace with a single cleanup
label.
Link: https://lore.kernel.org/20260709081843.1451202-1-lihongfu@kylinos.cn
Fixes:
|
||
|
xu xin
|
0471cade0a |
ksm: use precise linear_page_index instead of the whole address space
Since we now have linear_page_index available that we can use here, allowing for optimizing the RMAP walk, we can also use it to locate more precisely all related-processes when error hits the KSM page, which will decrease a lot of invalid iterations. Link: https://lore.kernel.org/20260709173312403qgj1Af6pRkFMDSsmc19sM@zte.com.cn Signed-off-by: xu xin <xu.xin16@zte.com.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
887311e5cd |
mm/ksm: initialize the addr only once in collect_procs_ksm
Patch series "KSM: use linear_page_index in collect_procs_ksm()", v2.
In collect_procs_ksm() which is used to collect processes when the error
hit an ksm page, there is the same issue with rmap_walk_ksm (see the
previous discussion at [1]). So we apply the similar logic changes to the
collect_procs_ksm().
The patch [1/2] move the initializaion of addr from the position inside
loop to the position before the loop, since the variable will not change
in the loop.
The patch [2/2] optimize collect_procs_ksm by passing a suitable page
offset range to the anon_vma_interval_tree_foreach loop to reduce
ineffective checks.
This patch (of 2):
Similar to
|
||
|
Mike Rapoport (Microsoft)
|
ef79e0f5e3 |
mm: split out vmalloc declarations from internal.h
mm/internal.h becomes more and more bloated. Move declarations related to vmalloc to a new mm/vmalloc.h header. No functional changes. Link: https://lore.kernel.org/20260709-internal-h-v2-3-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
2b65a42a08 |
mm: split out sparse declarations from internal.h
mm/internal.h becomes more and more bloated. Move declarations related to SPARSE and SPARSE_VMEMMAP memory models to a new mm/sparse.h header. No functional changes. Link: https://lore.kernel.org/20260709-internal-h-v2-2-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Mike Rapoport (Microsoft)
|
55ed40abb2 |
mm: split out mm_init and memblock declarations from internal.h
Patch series "mm: split a couple of headers from internal.h", v2. mm/internal.h becomes more and more bloated. Split declarations related to mm_init, memblock, vmalloc and sparse into new headers. This patch (of 3): mm/internal.h becomes more and more bloated. Move declarations for related to mm/mm_init.c and mm/memblock.c to a new mm/mm_init.h header. No functional changes. [rppt@kernel.org: split stubfs from internal.h to mm_init.h] Link: https://lore.kernel.org/alJd1BLypyK9Mpaw@kernel.org Link: https://lore.kernel.org/20260709-internal-h-v2-0-695631425968@kernel.org Link: https://lore.kernel.org/20260709-internal-h-v2-1-695631425968@kernel.org Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Muchun Song <muchun.song@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pratyush Yadav <pratyush@kernel.org> Acked-by: SJ Park <sj@kernel.org> Cc: Alexander Graf <graf@amazon.com> Cc: Alexander Potapenko <glider@google.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Marco Elver <elver@google.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
1695841621 |
ksm: add mremap selftests for ksm_rmap_walk
The existing tools/testing/selftests/mm/rmap.c has already one testcase
for ksm_rmap_walk in TEST_F(migrate, ksm), which takes use of migration of
page from one NUMA node to another NUMA node. However, it just lacks the
scenario of mremapped VMAs.
We add the calling of mremap() and then trigger KSM to merge pages before
migrating, which is specifically to test an optimization which is
introduced by this patch ("ksm: Optimize rmap_walk_ksm by passing a
suitable address pgoff").
This test can reproduce the issue that Hugh points out at
https://lore.kernel.org/all/02e1b8df-d568-8cbb-b8f6-46d5476d9d75@google.com/
Link: https://lore.kernel.org/20260703162637070FU4ekl58Hw_Z7OSuJryZB@zte.com.cn
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
xu xin
|
96d2d9acef |
ksm: optimize rmap_walk_ksm by passing a suitable page index
User impact / Why this matters to Linux users ============================================= When a system runs with KSM enabled and memory becomes tight, KSM pages may be swapped out or migrated. The kernel then performs a reverse map walk by rmap_walk_ksm to locate all page table entries that reference these pages. If A large number of unrelated VMAs can attach to a single anon_vma related with this KSM page, then rmap_walk might be severe performance bottleneck. In our embedded test environment, we observed ~20,000 VMAs sharing one anon_vma without any fork purely from VMA splits which cause 200~700ms duration of rmap_walk_ksm. When one of those VMAs mapped a KSM page, then this KSM page's rmapping will become bottleneck with hold its anon_vma lock for a long time. The anon_vma lock is not only used by KSM; it is a core lock protecting the VMA interval tree and is acquired by many critical memory operations: ' Page faults: do_anonymous_page(), do_wp_page() (during COW) ' Memory reclaim: try_to_unmap() ' Page migration & compaction: migrate_pages(), compact_zone() ' mlock / munlock: mlock_fixup() ' Process exit: exit_mmap() (tearing down VMAs) ' Cgroup memory accounting: mem_cgroup_move_charge() If one thread holds the anon_vma lock for hundreds of milliseconds because of an inefficient KSM rmap walk, any other thread that tries to acquire the same lock (e.g., an application taking a page fault, kswapd reclaiming pages, or a migration thread) will block. This leads to stalled application threads, increased latency spikes, and in extreme cases container timeouts or watchdog triggers. This patch reduces the worst-case anon_vma lock hold time during ksm_rmap_walk from >500 ms to <1 ms, thereby almost eliminating this source of lock contention and improving system responsiveness under memory pressure. Real-world examples: ==================== - JVM / Go runtime: These use mmap for heap regions and later call mprotect(PROT_NONE) for garbage collection barriers or guard pages, splitting the original VMA into thousands of small pieces over time. - Database engines (MySQL, PostgreSQL): Large shared memory buffers or anonymous mappings are managed with madvise(MADV_DONTNEED) to release specific pages, which also splits VMAs. Root Cause ========== Through local debugging trace analysis, we found that most of the latency of rmap_walk_ksm occurs within anon_vma_interval_tree_foreach, leading to an excessively long hold time on the anon_vma lock (even reaching 500ms or more), which in turn causes upper-layer applications (waiting for the anon_vma lock) to be blocked for extended periods. Further investigation revealed that 99.9% of iterations inside the anon_vma_interval_tree_foreach loop are skipped due to the first check "if (addr < vma->vm_start || addr >= vma->vm_end)), indicating that a large number of loop iterations are ineffective. This inefficiency arises because the start page index and the end page index parameters passed to anon_vma_interval_tree_foreach span the entire address space from 0 to ULONG_MAX, resulting in very poor loop efficiency. Solution ======== We cannot rely solely on anon_vma to locate all PTEs mapping this page but also need to have the original page's linear_page_index. Since the implementation of anon_vma_interval_tree_foreach it essentially iterates to find a suitable VMA such that the provided page index falls within the candidate's vm_pgoff range. vm_pgoff <= original linear page offset <= (vm_pgoff + vma_pages(v) - 1) Fortunately, an earlier commit introduced the linear_page_index to struct ksm_rmap_item, allowing for optimizing the RMAP walk. Test results ============ A rmap testbench can be obtained with two Out-Of-Tree patches at [1][2]. After applying the OOT patches and building rmap_benchmark from: tools/testing/rmap/rmap_benchmark.c, we can start the performance test. The testing result in QEMU is shown as follows: KSM rmapping Maximum duration Average duration Before: 705.12 ms (705119858 ns) 532.04 ms (532041586 ns) After: 1.67 ms (1665917 ns) 1.44 ms (1443784 ns) The benchmark numbers are realistic, since we observed ~20,000 VMAs sharing one anon_vma on a production system running a Java application with KSM enabled. The lock hold time before the patch was measured at 228ms (max) during rmap walks triggered by memory compaction and page migration. The benchmark reproduces that VMA count and lockhold behavior in a controlled environment. Link: https://lore.kernel.org/20260703162510242nxmjbcLy5ccp1dbZSK3EU@zte.com.cn Link: https://lore.kernel.org/all/202605301703094695zmVgcSC27BNR0rH0N8_x@zte.com.cn [1] Link: https://lore.kernel.org/all/20260530170404509QpJmBtpSjn3uQHeVKA2iA@zte.com.cn/ [2] Co-developed-by: Wang Yaxin <wang.yaxin@zte.com.cn> Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> Signed-off-by: xu xin <xu.xin16@zte.com.cn> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Hugh Dickins <hughd@google.com> Cc: "Liam R. Howlett" <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
xu xin
|
a5650de041 |
ksm: add linear_page_index into ksm_rmap_item
Patch series "KSM: performance optimizations for rmap_walk_ksm", v11.
This series fixes a severe KSM reverse-mapping performance problem that
can freeze applications for hundreds of milliseconds under memory pressure
especially when a lot of unrelated VMAs sharing a single anon_vma.
Two key highlights:
1. Lock hold time drops from >500ms to <2ms
- In our benchmark (20,000 VMAs sharing an anon_vma), worst-case
anon_vma lock hold time during KSM rmap walk went from 705ms
down to 1.67ms (max) and 1.44ms (avg).
2. Real user impact
- The anon_vma lock is also acquired by page faults, reclaim,
migration, compaction, mlock, exit_mmap, and cgroup accounting.
- A long hold due to inefficient rmap walks stalls application
threads, causing latency spikes, reduced throughput, or even
container timeouts.
- The problem occurs even without fork() – VMA splitting (e.g.,
via mprotect or madvise over time) can create tens of thousands
of VMAs all attached to the same anon_vma.
Real-world examples:
- JVM / Go runtime: These use mmap for heap regions and later call
mprotect(PROT_NONE) for garbage collection barriers or guard pages,
splitting the original VMA into thousands of small pieces over time.
- Database engines (MySQL, PostgreSQL): Large shared memory buffers or
anonymous mappings are managed with madvise(MADV_DONTNEED) to release
specific pages, which also splits VMAs.
Why the benchmark numbers are realistic: We observed ~20,000 VMAs sharing
one anon_vma on a production system running a Java application with KSM
enabled. The lock hold time before the patch was measured at 228 ms
(max) during rmap walks triggered by memory compaction and page migration.
The benchmark reproduces that VMA count and lock‑hold behavior in a
controlled environment.
For systems that do not have thousands of VMAs per anon_vma, the patch
adds negligible overhead (a single pgoff comparison). For systems that do
suffer from this issue, the improvement is dramatic: 1) Worst‑case
anon_vma lock hold time drops from hundreds of milliseconds to under
2 ms.2)This directly reduces blocking of parallel operations that need
the same lock – page faults, reclaim, migration, compaction, mlock, and
exit_mmap.
End‑users will see lower tail latency (fewer application stalls), higher
throughput under memory pressure, and no more spurious lockup warnings or
container timeouts caused by excessive lock hold times.
In short: workloads that do not hit this pathological pattern are
unaffected; those that do will see a 100x to 500x reduction in lock hold
times, which translates directly into a more responsive system.
This patch (of 3):
As preparation for KSM rmap optimizations, let's track the original
linear_page_index() of a de-duplicated page in its ksm_rmap_item, so we
can efficiently search for the page in an address space, avoiding scanning
the entire address space. This was previously discussed in [1, 2].
To avoid growing ksm_rmap_item, let's squeeze it into the existing
structure by overlying some members (oldchecksum, age, remaining_skips)
that are only relevant while on the unstable tree. The new entry will
only be relevant for entries in the stable tree.
However, as the age information is read by should_skip_rmap_item() with
the smart-scanning approach even while we have an entry in the stable
tree, but the page changes (no longer a KSM page, for example due to COW),
we have to change the handling there a bit.
We'll calculate the linear page index in try_to_merge_with_ksm_page(),
when adding it to the stable tree, and reset the index (to reset overlayed
data) when removing an item from the stable tree -- in
remove_rmap_item_from_tree(), remove_node_from_stable_tree() and
break_cow().
To be specially clarified, the reason for resetting the stored index at
break_cow() is:
- When a page successfully becomes a KSM page (i.e., after
stable_tree_append() sets STABLE_FLAG), both anon_vma and the index are
stored and remain valid.
- However, during the merging process, there are several failure paths
where we already prepared an rmap item to be added to the stable tree,
but must revert that as some part of the merge process failed. Examples
include:
1 The second call to try_to_merge_with_ksm_page() fails in
try_to_merge_two_pages().
2 stable_tree_insert() fails in cmp_and_merge_page().
In such cases, break_cow() is invoked to break the COW mapping and
discard the KSM state.
Currently, break_cow() already contains a
put_anon_vma(rmap_item->anon_vma) to release the reference taken during
the aborted merge. Because the index is logically paired with anon_vma
(both are only meaningful when the rmap_item is in a stable state), it
must also be cleared (or reset) in break_cow() to avoid leaving stale
linear_page_index values that could confuse subsequent rmap walks or
scanning logic.
Link: https://lore.kernel.org/20260703162253688u8Str9eFLR8TGCmo7nIOF@zte.com.cn
Link: https://lore.kernel.org/20260703162357853iIa-RP7if9hRlAIuTh5La@zte.com.cn
Link: https://lore.kernel.org/all/adTPQSb-qSSHviJN@lucifer/ [1]
Link: https://lore.kernel.org/all/202604091806051535BJWZ_FTtdIm3Snk24ei_@zte.com.cn/ [2]
Signed-off-by: xu xin <xu.xin16@zte.com.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
SJ Park
|
d59bf2653c |
mm/damon/core: handle unreset probe_hits in probe_hits_mvsum()
If damon_update_monitoring_result() is called at the end of the aggregation interval, probe_hits is not reset. That's because the value will be exposed to the user via damon_region_aggregated trace event. Meanwhile, damon_probe_hits_mvsum() can be called in this state. Due to its logic, it will return a value that is incorrectly high. This could happen if the user requested DAMOS schemes applied regions sysfs files update exactly in the time sequence. The impact is minor, but better to avoid. Check the timing and simply return the fully aggregated last_probe_hits, like damon_nr_accesses_mvsum() also does. It is not 100% accurate since it is the last interval's aggregation. But better than the value that is completely reset. Link: https://lore.kernel.org/20260708135359.122587-8-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |