There are many instances in which linear_page_index() (as well as
linear_page_delta()) is open-coded, which is confusing and inconsistent.
Additionally, vma->vm_pgoff doesn't necessarily make it clear that this is
the page offset of the start of the VMA range.
Doing so also aids greppability.
So use vma_start_pgoff() in favour of directly accessing vma->vm_pgoff, and
linear_page_index() where we can.
This also lays the ground for future changes which will add an anonymous
page offset in order to be able to index MAP_PRIVATE-file backed anon
folios in terms of their virtual page offset.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-18-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: SJ Park <sj@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The family of vma_interval_tree_() functions manipulate the
address_space (which, of course, is generally referred to as 'mapping')
reverse mapping, but are named the 'VMA' interval tree.
VMAs may be mapped by an anon_vma, an address_space, or both. Therefore
calling the mapping interval tree a 'VMA' interval tree is rather
confusing.
This is also inconsistent with the anon_vma_interval_tree_*() functions
which explicitly reference the rmap object to which they pertain.
Rename the vma_interval_tree_*() functions to mapping_rmap_tree_*() to
correct this.
We will rename the anon rmap functions similarly in a subsequent patch.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-8-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The file-backed mapping interval tree functions vma_interval_tree_*()
accept a raw rb_root_cached pointer to determine the tree in which they are
operating.
However, in each case, this is always associated with an address_space data
type.
So simply pass a pointer to that instead to simplify the code, and more
clearly differentiate between these operations and those concerning
anonymous mappings.
While we're here, make the generated interval tree functions static as they
do not need to be used externally (any previously existing external users
have now been removed).
We also rename VMA parameters from 'node' to 'vma' as calling this a node
is simply confusing, update the input index types to pgoff_t since they
reference page offsets and rename the parameters to pgoff_start and
pgoff_last.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-6-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Cc: Liam R. Howlett (Oracle) <liam@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: make VMA page offset handling more consistent", v2.
This series performs a series of cleanups and improvements around how the
vma->vm_pgoff field is used.
Folios belonging to file-backed mappings are simply indexed by the page
offset within the file they map (excepting pfnmap and driver mappings
which sometimes do things different).
However, anonymous folios belonging to pure anonymous mappings are indexed
by their "virtual" page offset, which is equal to addr >> PAGE_SHIFT at
the time at which the VMA was first faulted in.
The page offset of a VMA is stored in vma->vm_pgoff and indicates the page
offset of the start of the VMA range, whether it be file-backed or
anonymous.
The work here both cleans up how we reference this field, as well as
laying the foundations for a future series which addresses the
inconsistency of CoW'd folios in MAP_PRIVATE-file backed mappings, which
are indexed as if they were file-backed but behave as if they were
anonymous.
This future series will make it such that all anonymous folios are indexed
by virtual page offset whether belonging to VMAs who satisfy
vma_is_anonymous() or MAP_PRIVATE-mapped file-backed mappings.
This series:
* Exposes vma_start_pgoff() and updates the kernel to use it consistently.
* Adds and uses the useful vma_end_pgoff() helper.
* Parameterises the file-backed mapping helpers vma_interval_tree_*()
by address_space rather than rb_root_cached.
* Renames: the misleadingly-named vma_interval_tree_*() helpers to
mapping_rmap_tree_*() to be consistent with the renamed
anon_rmap_tree_*().
* Parameterises anon_rmap_tree_*() by anon_vma.
* Moves mm/interval_tree.c to the rmap section.
* Adds vmg_*() helpers for page offset.
* Clarifies the confusing vmg_adjust_set_range() function.
* Introduces linear_page_delta() to provide relative pgoff within a VMA.
* Replaces open-coded versions of linear_page_delta() and
linear_page_index() with invocations of these functions.
* Introduces and uses vma_assert_can_modify() to account for whether a VMA
can be modified (detached or write locked).
* Adds and uses vma_[add,sub]_pgoff() to adjust VMA page offset.
* Moves __install_special_mapping() to vma.c.
* Makes vma_set_range() static and internal to vma.c.
* Introduces and makes use of vma_set_pgoff().
* Fixes incorrect vma.h header inclusion.
* Defaults VMA userland tests to 64-bit vma flags size.
* Updates VMA userland tests to give better output on failure.
* Various smaller cleanups.
* Updated the vma_set_pgoff() assert to account for MAP_PRIVATE /dev/zero
correctly.
This patch (of 33):
vma_last_pgoff() already lives there, so it's a bit odd to keep
vma_start_pgoff() in mm/interval_tree.c. Move them together.
These each return unsigned long, which pgoff_t is typedef'd to. Make this
consistent and have these functions return pgoff_t instead.
Additionally, express vma_last_pgoff() in terms of vma_start_pgoff(), since
we wrap the vma->vm_pgoff access, we may as well use it here.
Also while we're here, const-ify the VMA and cleanup a bit.
Also update the VMA userland tests to reflect the change.
No functional change intended.
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-0-2a5aa403d977@kernel.org
Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-1-2a5aa403d977@kernel.org
Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Cc: Kai Huang <kai.huang@intel.com>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
Cc: SJ Park <sj@kernel.org>
Cc: Thomas Zimmermann <tzimmermann@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Several GPU SVM paths take mmap_read_lock() only to call hmm_range_fault()
and open-code mmu interval sequence setup before each HMM walk. They also
retry -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires.
Use hmm_range_fault_unlocked_timeout() for those faults. The HMM helper
now owns mmap_lock acquisition and refreshes range->notifier_seq for its
internal retries, while GPU SVM keeps its existing driver-lock validation
with mmu_interval_read_retry() after a successful fault.
drm_gpusvm_scan_mm() and drm_gpusvm_range_evict() pass
HMM_RANGE_DEFAULT_TIMEOUT as the helper retry budget for each HMM fault
attempt. drm_gpusvm_get_pages() keeps its existing absolute outer
deadline because it can be reached from GPU page-fault workers, where
fatal signals from the faulting process cannot stop an endless
invalidation retry loop. It passes the remaining time from that deadline
to HMM for each fault attempt.
Leave drm_gpusvm_check_pages() on hmm_range_fault() because that path is
called with the mmap lock already held by its caller.
Link: https://lore.kernel.org/20260723-hmm-v10-v11-8-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Long Li <longli@microsoft.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Lyude <lyude@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call. This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock. Drivers that need to populate device page tables
for such mappings have no way to do so today.
Add hmm_range_fault_unlocked_timeout() for callers that do not need to
hold mmap_lock across any work outside the HMM fault itself. The helper
takes mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still
held. The timeout is specified in jiffies; passing 0 retries
indefinitely, while a non-zero timeout makes the helper return -EBUSY when
the retry budget expires. The retry deadline is set before refreshing the
notifier sequence and acquiring mmap_lock, so contended mmap_lock
acquisition is included in the retry budget. After acquiring mmap_lock,
the helper also rejects unstable address spaces before walking page
tables.
When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally. If the lock was dropped, the retry deadline
is also restarted because a lock-dropping fault handler made progress.
Ordinary -EBUSY retries keep the existing deadline, preserving the
caller's timeout policy for repeated mmu-notifier invalidations.
The caller only needs to perform the usual post-success
mmu_interval_read_retry() check while holding its update lock before
consuming the pfns. If mmap_lock acquisition is interrupted or a fatal
signal is pending during retry handling, -EINTR is returned instead.
The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.
The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small. Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock.
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.
Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.
Link: https://lore.kernel.org/20260723-hmm-v10-v11-2-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Long Li <longli@microsoft.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Lyude <lyude@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/hmm: Add mmap lock-drop support for userfaultfd-backed
mappings", v11.
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers most notably userfaultfd require the mmap lock to
be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the mmap
lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
This patch (of 8):
hmm_range_fault() currently triggers page faults from inside the
page-table walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the
walk framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out
of walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Link: https://lore.kernel.org/20260723-hmm-v10-v11-0-c55b003a4b61@gmail.com
Link: https://lore.kernel.org/20260723-hmm-v10-v11-1-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Long Li <longli@microsoft.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Lyude <lyude@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
hmm doesn't currently build well with MMU_NOTIFIER=n:
With microblaze
https://download.01.org/0day-ci/archive/20260702/202607021433.DYT5fDqE-lkp@intel.com/config
(CONFIG_MMU_NOTIFIER=n):
mm/hmm.c: In function 'hmm_range_fault_unlocked_timeout':
mm/hmm.c:804:25: error: implicit declaration of function 'mmu_interval_read_begin'; did you mean 'mmu_interval_check_retry'? [-Wimplicit-function-declaration]
804 | mmu_interval_read_begin(range->notifier);
| ^~~~~~~~~~~~~~~~~~~~~~~
| mmu_interval_check_retry
Quoting Stanislav:
: Documentation/mm/hmm.rst explicitly states:
:
: "Address space mirroring's main objective is to allow duplication of a
: range of CPU page table into a device page table; HMM helps keep both
: synchronized. A device driver that wants to mirror a process address
: space must start with the registration of a mmu_interval_notifier"
:
: I think making CONFIG_HMM_MIRROR dependent on the CONFIG_MMU_NOTIFIER
: is the right thing to do.
So select MMU_NOTIFIER if hmm.c is to be compiled.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Stanislav Kinsburskii <skinsburskii@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_probe_hits_wsum() could overflow in weird setups. Users could set
the weight unreasonably high. They could also set the aggregation
interval unreasonably high compared to the sampling interval. Such user
setup is unlikely. Even if such setup is used, damon_has_probe_weights()
always returns false, so the overflow cannot happen. The function may be
completed in future, though. Even if the overflow happens, the
consequence is degraded monitoring results for the unreasonable setup. It
is just a trivial user experience issue.
It is still better to be prevented unless the cost is expensive. Avoid
the overflow by adding the parameter validation in the core layer
parameters validation function.
Link: https://lore.kernel.org/20260710134651.18084-11-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
When any damon_probe->weight is set, DAMON will do only probe monitoring.
probe_hits is 'unsigned char'. It could overflow when the aggregation
interval is larger than the sampling interval times 256.
damon_as_probe_weights() always return false, so such overflow cannot
happen. Even if it happens, it only degrades the monitoring results.
That said, the overflow is not intentional. It is better to be prevented
as long as the cost is not expensive. Disallow the overflow by adding a
validation logic on the core layer parameters validation function.
Link: https://lore.kernel.org/20260710134651.18084-10-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
When probe weights are set, users may want DAMON monitoring results to be
optimized for the weights. For that, regions adjustment should work for
the weighted sum of probe hits. Extend damon_merge_regions_of() to detect
if the weights are set, and work with probe hits in the case.
The weights setup detection function is incomplete. It always returns
false. It is intentional, so that more changes to completely support
weights can be made in an incremental but safe way. Until the function is
completed, all changes depend on it is no-op, so DAMON works in the
current mode.
Link: https://lore.kernel.org/20260710134651.18084-9-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon: introduce data attributes only monitoring".
TL;DR: Introduce a way to get DAMON's best effort accuracy monitoring of
user-demanding non-access data attributes.
Background
==========
DAMON was initially designed for only access monitoring. It turned out
users want to get the information together with more data attributes. For
example, some users want to know how much of a hot memory region belongs
to huge pages or specific cgroups. Page level properties based monitoring
was introduced with commit 626ffabe67 ("mm/damon: clarify trying vs
applying on damos_stat kernel-doc comment") to fill the gap. Because it
works only at snapshot level and snapshot capturing in the mode can induce
high overhead, commit 45c49d9fd6 ("mm/damon/core: introduce struct
damon_probe") introduced data attributes monitoring.
Data attributes monitoring treats the attributes as only additional and
subordinate information. Data access monitoring is always turned on, and
regions are adjusted for best accuracy of the access information. In some
cases, users may be primarily interested in the attributes more than the
access. They might even not care about the access information at all.
Because DAMON treats data accesses as the only primary information, such
users cannot get high quality attributes information.
Design and Implementation
=========================
Introduce another way for treating data attributes as the primary
information. Add 'weight' property to each data attribute probe. When
any of the weights are set, the mode is enabled. Data access monitoring
is completely turned off in the mode. For region adjustment, the weighted
sum of probe hit counters is used instead of the nr_accesses.
Using the weights, users can specify to what attributes they are
interested in to what degree. DAMON will adjust the regions and provide
the best-effort quality monitoring that is optimized for the user demands.
Extend damon_operations for efficient use of probe hits. Update regions
merge and kdamond main logic to support the new mode. Add a new struct
field and a sysfs file for API callers and ABI users, respectively.
Test
====
On ~7 GiB memory idle system, run a simple AI-assisted program. The
program allocates and faults 2 GiB anonymous pages. Then, it does nothing
but wait until the user terminates it. Hence, the system ~2 GiB of
anonymous pages with no active accesses.
Monitor the distribution of the anonymous pages using DAMON attributes
monitoring mode, using DAMON user-space tool, damo [1].
$ sudo ./damo start --probe_filter allow anon
$ sudo ./damo report access --dont_merge_regions
heatmap: 00000000000000000000000000000000000000000000000399999995111111146666666666666666
# min/max temperatures: -2,470,000,000, -1,620,000,000, column size: 99.800 MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
# <start> <size> <freq> <age> <probe hits>
0 4.000 KiB 79.840 MiB 0 hz 24.700 s 2
1 79.844 MiB 718.562 MiB 0 hz 24.700 s 8
2 798.406 MiB 793.148 MiB 0 hz 24.700 s 7
3 1.554 GiB 797.828 MiB 0 hz 24.700 s 7
4 2.333 GiB 794.668 MiB 0 hz 24.600 s 8
5 3.109 GiB 791.117 MiB 0 hz 24.500 s 0
6 3.882 GiB 785.312 MiB 0 hz 24 s 2
7 4.649 GiB 787.867 MiB 0 hz 16.200 s 6
8 5.418 GiB 784.477 MiB 0 hz 23.300 s 6
9 6.184 GiB 783.820 MiB 0 hz 18.200 s 9
10 6.950 GiB 797.730 MiB 0 hz 18.900 s 7
11 7.729 GiB 69.625 MiB 0 hz 18.900 s 0
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
Note that the line after the line starting with "intervals:" is not
provided by the current version of 'damo'. I manually added the legends
line for easier understanding of these results.
Each of the 12 lines after the legend line shows the DAMON-found regions.
Each line shows 1) index of the region, 2) start address of the region, 3)
size of the region, 4) access frequency of the region, 5) age (how long
the access frequency on the region was kept) of the region, and finally 6)
the probe hit count.
Because data access is the primary information that adjusts region for,
and there is only nearly zero access on the system, regions are naively
adjusted with the same size. Still <probe hits> show different
distribution of the anonymous pages, but it is obviously very rough
information.
Switch to the attributes only mode and show how it changes the picture:
$ sudo ./damo tune --probe_filter allow anon --probe_weight 100
$ sudo ./damo report access --dont_merge_regions
heatmap: 88888888888888888889888999999889999999000004888888888888889999988888898888888888
# min/max temperatures: -4,430,000,000, 0, column size: 99.800 MiB
intervals: sample 5 ms aggr 100 ms (max access hz 200)
# <start> <size> <freq> <age> <probe hits>
0 4.000 KiB 60.445 MiB 0 hz 700 ms 0
1 60.449 MiB 1.363 MiB 0 hz 600 ms 18
2 61.812 MiB 144.000 KiB 0 hz 0 ns 1
3 61.953 MiB 1.922 MiB 0 hz 2.400 s 19
4 63.875 MiB 12.133 MiB 0 hz 200 ms 0
[...]
500 5.132 GiB 8.000 KiB 0 hz 2 m 15.800 s 20
501 5.132 GiB 8.000 KiB 0 hz 2 m 16.200 s 0
502 5.132 GiB 16.000 KiB 0 hz 2 m 16.900 s 20
503 5.132 GiB 24.000 KiB 0 hz 2 m 14.200 s 0
504 5.132 GiB 8.000 KiB 0 hz 2 m 14.900 s 20
[...]
923 7.534 GiB 126.637 MiB 0 hz 0 ns 6
924 7.658 GiB 252.000 KiB 0 hz 54.800 s 20
925 7.658 GiB 142.242 MiB 0 hz 300 ms 0
memory bw estimate: 0 B per second
total size: 7.797 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
As expected, regions are adjusted to provide the best accurate picture for
the anonymous pages distribution (<probe hits>). The region 0 (60.445 MiB
memory from the address 4.000 KiB) has nearly zero anonymous pages. The
region 1 (1.363 MiB memory from the address 60.449 MiB) is nearly full
with anonymous pages. Region 500 (8 KiB memory from the address 5.132
GiB) is certainly two anonymous pages.
Future Work
===========
Attributes only monitoring disables access monitoring. We will enable
that in future, by extending the supported attributes to include data
accesses. This patch series, and the future work are parts of the ongoing
project [2] for extending DAMON. The project aims to extend DAMON with
primitives other than page table accessed bits such as AMD IBS, Intel
PEBS, and Arm SPE, to provide more powerful and detailed information like
per-CPUs/threads/reads/writes monitoring.
Patches Sequence
================
Patch 1 introduces damon_probe->weight for specifying the weights of each
attribute. Patches 2-6 extends apply_probe() damon_ops callback to
efficiently support the new mode. Patch 7 fixes wrong use of abs() in the
regions merge code. Patch 8 extends regions merge function to work with
probe hits in the mode. Patch 8 also introduces the function for
detecting the mode enablement but always returns false, for safe and
incremental changes. Patches 9 and 10 adds user parameters validation to
prevent theoretical overflow of probe hits and the weighted sum. Patches
11-14 incrementally update kdamond_fn() to support the mode. Patch 15
completes the mode detection function implementation, so that the new mode
really works. Patch 16 introduces a new sysfs file for ABI users.
Finally, patches 17-19 respectively updates design, usage and ABI
documents for the new feature and interfaces.
[1] https://github.com/damonitor/damo
[2] https://lore.kernel.org/20260525225208.1179-1-sj@kernel.org/
This patch (of 19):
Add a new field, weight to damon_probe struct. The field is used to
specify the degree of the API caller's interest to the data attribute of
the probe.
Link: https://lore.kernel.org/20260710134651.18084-1-sj@kernel.org
Link: https://lore.kernel.org/20260710134651.18084-2-sj@kernel.org
Link: https://github.com/damonitor/damo [1]
Link: https://lore.kernel.org/20260525225208.1179-1-sj@kernel.org/ [2]
Signed-off-by: SJ Park <sj@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Since commit f002882ca3 ("mm: merge folio_is_secretmem() and
folio_fast_pin_allowed() into gup_fast_folio_allowed()"),
gup_fast_folio_allowed() falls back to the slow path for any order-0 folio
with a NULL mapping when CONFIG_SECRETMEM=y. This causes a performance
regression for drivers that allocate pages with alloc_page() and insert
them into VMAs via vm_insert_page(). These pages legitimately have a NULL
folio->mapping, but they cannot be secretmem pages.
Secretmem pages are always added to the secretmem inode's page cache via
filemap_add_folio(), which sets folio->mapping to the inode's i_mapping.
A folio with a NULL mapping can never be a secretmem folio. The
NULL-mapping check was intended to handle truncated file-backed pages (a
reject_file_backed concern), not secretmem detection.
When only check_secretmem is true (and reject_file_backed is false), a
NULL mapping is sufficient to prove the folio is not secretmem, so the
fast path can proceed.
Link: https://lore.kernel.org/20260708005745.164928-1-jhubbard@nvidia.com
Fixes: f002882ca3 ("mm: merge folio_is_secretmem() and folio_fast_pin_allowed() into gup_fast_folio_allowed()")
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Tested-by: Sourab Gupta <sougupta@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The KSM NUMA merge test allocates identical pages on different NUMA nodes
and verifies KSM behavior with merge_across_nodes enabled and disabled.
On systems with memoryless NUMA nodes, for example:
#numactl -H
available: 2 nodes (0,4)
.....
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 14825 MB
node 0 free: 1382 MB
node 4 cpus:
node 4 size: 0 MB
node 4 free: 0 MB
the test may attempt to allocate memory on a node without memory, causing
numa_alloc_onnode() to fail and resulting in a spurious test failure.
The test currently checks numa_num_configured_nodes() to determine whether
sufficient NUMA nodes are available. However, configured nodes do not
necessarily have memory.
Reuse the existing get_first_mem_node() and get_next_mem_node() helpers to
locate NUMA nodes that actually contain memory, and skip the test when
fewer than two such nodes are available.
Before patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
mbind: Invalid argument
ok 1 KSM NUMA merging
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
mbind: Invalid argument
not ok 1 KSM NUMA merging
Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
[FAIL]
not ok 2 ksm_tests -N -m 0 # exit=1
After patch:
---------------------------
running ./ksm_tests -N -m 1
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 1 ksm_tests -N -m 1
---------------------------
running ./ksm_tests -N -m 0
---------------------------
At least 2 NUMA nodes with memory must be available
ok 1
SKIP KSM NUMA merging
Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0
[PASS]
ok 2 ksm_tests -N -m 0
Link: https://lore.kernel.org/78a3b0e3fb94004c0710872c5bab6f7381b7d63c.1783446924.git.sayalip@linux.ibm.com
Fixes: e3820ab252 ("selftest/vm: fix ksm selftest to run with different NUMA topologies")
Co-developed-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Sayali Patil <sayalip@linux.ibm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Liam Howlett <liam@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
