Neha reports that mapped shmem aren't considered for NUMA balancing,
noting convergence problems and bandwidth bottlenecking for cachelib based
workloads on tiered memory systems.
Looking at the code and going through the git history, this doesn't
actually seem intentional:
Commit fc3147245d ("mm: numa: Limit NUMA scanning to migrate-on-fault
VMAs") added a vma_policy_mof() gate to task_numa_work() so VMAs whose
policy lacks MPOL_F_MOF are skipped from NUMA balancing scans. The
motivation was a real usecase: Oracle was pinning shared segments with
mbind(MPOL_BIND) so trapping faults was both expensive and pointless.
The handling of NULL from vm_ops->get_policy, however, treated "user
explicitly opted out" the same as "user never specified anything." For
VMAs whose shared policy is absent - the common case for shmem - the scan
was disabled too.
This issue is old. It probably hurts less in conventional NUMA. But it's
very noticeable on tiered systems, where entire tmpfs workingsets can get
stuck on lower-bandwidth memory.
Fix this by having vma_policy_mof() use __get_vma_policy() directly, and
thereby handle the fallback to task policy (-> preferred_node_policy() has
MPOL_F_MOF per default). Every other consumer of vm_ops->get_policy
already handles it this way, the scan-eligibility check was the outlier.
This preserves Mel's intended fix: don't scan stuff the user explicitly
pinned. But allow default policy vmas to participate in balancing.
Link: https://lore.kernel.org/20260629163337.1264881-1-hannes@cmpxchg.org
Fixes: fc3147245d ("mm: numa: Limit NUMA scanning to migrate-on-fault VMAs")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: Neha Gholkar <nehagholkar@gmail.com>
Tested-by: Neha Gholkar <nehagholkar@gmail.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Balbir Singh <balbirs@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit 49a4e7186b ("selftests/mm: thuge-gen: add setup of HugeTLB
pages") changed thuge-gen test to use common functions for reading hugetlb
attributes from sysfs, but it missed that the original read_free()
function special cased PAGE_SIZE tests.
For PAGE_SIZE tests, failure to read sysfs was ignored and read_free()
returned 0.
This allowed test_shmget() to essentially skip the check of how many huge
pages was consumed when it ran with PAGE_SIZE.
Commit 3199b0c09efa ("selftests/mm: fix read_file() return value check")
fixed checks for read_file() return value and this exposed the issue in
test_shmget() that checks the number of free hugetlb pages even for
PAGE_SIZE test, tries to access
/sys/kernel/mm/hugepages/hugepages-<PAGE_SIZE>/free_hugepages
and obviously fails there.
Gate the checks for free huge pages on size != getpagesize() and
initialize before and after variables to values matching PAGE_SIZE test.
Link: https://lore.kernel.org/20260812-selftests-thuge-gen-fix-v2-1-9adaa693e73b@kernel.org
Fixes: 49a4e7186b ("selftests/mm: thuge-gen: add setup of HugeTLB pages")
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Sarthak Sharma <sarthak.sharma@arm.com>
Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
If ->pmd_entry() sets walk->action = ACTION_AGAIN, the pmd_none() check is
retried. The PMD entry may be cleared at the point of retry.
In this case, if walk->ops->install_pte is not specified, the code
continues to the next PMD entry in the range without resetting
walk->action to ACTION_SUBTREE.
This leaves walk->action erroneously set to ACTION_AGAIN, which is
incorrect.
This was incorrect but not problematic up until commit 3b89863c3f
("mm/pagewalk: fix race between concurrent split and refault") which
updated walk_pud_range() to check for walk->action == ACTION_AGAIN upon
walk_pmd_range()'s return, causing the PUD walk to be retried.
In this case this results in duplicate walk callbacks being invoked,
which is erroneous and will break any caller that is not idempotent
with respect to this (and waste time for those which are). The result
is an out-of-bounds write, triggered by a local fuzzer:
[ 2.272695] ==================================================================
[ 2.273471] BUG: KASAN: slab-out-of-bounds in __mincore_unmapped_range+0x14f/0x190
[ 2.274302] Write of size 1 at addr ffff888008d9b000 by task poc/106
[ 2.274966]
[ 2.275154] CPU: 0 UID: 1000 PID: 106 Comm: poc Not tainted 7.2.0-rc6-00429-ga7c7074b58d2 #55 PREEMPT(lazy)
[ 2.275159] Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 2.275164] Call Trace:
[ 2.275170] <TASK>
[ 2.275172] dump_stack_lvl+0x53/0x70
[ 2.275200] print_report+0xd0/0x630
[ 2.275210] ? __pfx__raw_spin_lock_irqsave+0x10/0x10
[ 2.275219] ? irqentry_exit+0xd2/0x670
[ 2.275224] ? irqentry_exit+0xd2/0x670
[ 2.275226] ? __virt_addr_valid+0xef/0x1a0
[ 2.275239] ? __mincore_unmapped_range+0x14f/0x190
[ 2.275242] kasan_report+0xce/0x100
[ 2.275245] ? __mincore_unmapped_range+0x14f/0x190
[ 2.275248] __mincore_unmapped_range+0x14f/0x190
[ 2.275252] mincore_unmapped_range+0x45/0x70
[ 2.275254] walk_pgd_range+0xafc/0xfc0
[ 2.275261] ? __pfx_walk_pgd_range+0x10/0x10
[ 2.275264] ? __update_load_avg_se+0x3d1/0x670
[ 2.275275] __walk_page_range+0xc0/0x310
[ 2.275278] ? __pfx_find_vma+0x10/0x10
[ 2.275281] ? finish_task_switch.isra.0+0x16d/0x4f0
[ 2.275290] walk_page_range_mm_unsafe+0x26f/0x3a0
[ 2.275293] ? __pfx_mtree_load+0x10/0x10
[ 2.275298] ? __pfx_walk_page_range_mm_unsafe+0x10/0x10
[ 2.275302] ? __free_frozen_pages+0x54d/0x7e0
[ 2.275308] __do_sys_mincore+0x132/0x380
[ 2.275311] do_syscall_64+0xf9/0x540
[ 2.275316] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 2.275322] RIP: 0033:0x422ccd
[ 2.275326] Code: b3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
[ 2.275329] RSP: 002b:00007fffffffec18 EFLAGS: 00000287 ORIG_RAX: 000000000000001b
[ 2.275337] RAX: ffffffffffffffda RBX: 0000000000000066 RCX: 0000000000422ccd
[ 2.275339] RDX: 00000000004d0940 RSI: 0000000001000000 RDI: 00007ffff4000000
[ 2.275340] RBP: 00000000004d0940 R08: 0000000000000100 R09: 0000000000000100
[ 2.275342] R10: 0000000000000100 R11: 0000000000000287 R12: 20c49ba5e353f7cf
[ 2.275343] R13: 00000000004990d3 R14: 0000000000000000 R15: 0000000000000001
[ 2.275346] </TASK>
[ 2.275347]
[ 2.296904] The buggy address belongs to the object at ffff888008d9b000
[ 2.296904] which belongs to the cache sigqueue of size 80
[ 2.298151] The buggy address is located 0 bytes inside of
[ 2.298151] allocated 80-byte region [ffff888008d9b000, ffff888008d9b050)
[ 2.299408]
[ 2.299601] The buggy address belongs to the physical page:
[ 2.300191] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x8d9b
[ 2.301001] flags: 0x100000000000000(node=0|zone=1)
[ 2.301535] page_type: f5(slab)
[ 2.301884] raw: 0100000000000000 ffff888107e46780 dead000000000122 0000000000000000
[ 2.302687] raw: 0000000000000000 0000000800240024 00000000f5000000 0000000000000000
[ 2.303489] page dumped because: kasan: bad access detected
[ 2.304092]
[ 2.304276] Memory state around the buggy address:
[ 2.304801] ffff888008d9af00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2.305567] ffff888008d9af80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2.306340] >ffff888008d9b000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 2.307115] ^
[ 2.307474] ffff888008d9b080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 2.308237] ffff888008d9b100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 2.308997] ==================================================================
A specific example of this breaking things is mincore which walks an
internal cursor data structure a byte at a time on assumption that page
table entry callbacks are called only once for each entry.
Fix the problem by resetting walk->action to ACTION_SUBTREE prior to the
none check.
The pattern also exists in walk_pud_range() so fix it there too.
This issue was found through AI-based fuzzing.
Link: https://lore.kernel.org/20260811161949.3879321-2-imv4bel@gmail.com
Fixes: 3b89863c3f ("mm/pagewalk: fix race between concurrent split and refault")
Assisted-by: Claude:claude-opus-5
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Max Boone <mboone@akamai.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
A slot with a folio in the swap cache is freed when the folio leaves the
cache, not when its count drops. swap_put_entries_cluster() follows that
rule. swap_free_hibernation_slot() does not, it calls
__swap_cluster_free_entries() whether or not a folio sits on the slot.
Cluster readahead can put one there. It walks a raw page_cluster sized
window of offsets around the faulting entry, and a hibernation slot passes
__swap_cache_add_check() because it is not a folio and its count is not
zero. Freeing the slot then clears the entry under that folio.
The folio is now unreachable from the swap table, and the offset goes back
to the allocator. The folio is still on the LRU though, so reclaim can
pick it up later. It then takes the old offset out of folio->swap and
overwrites the table entry there, which by then may belong to someone
else.
This bug can trigger silent memory corruption, process crashes, or data
instability across completely unrelated userspace applications - typically
occurring when uswsusp is preparing the hibernation image.
I found this while working on giving hibernation slots their own marker in
the swap table, which I had discussed with Kairui.
(https://lore.kernel.org/linux-mm/abp7aDgYLrxF3Me8@KASONG-MC4/) As far as
I know there are no reports, so there is no Reported-by/Closes to add.
Check for a cached folio before freeing. The slot is then left in the
ordinary state where only the swap cache holds it, and it is freed when
the folio leaves the cache, either through the reclaim below or through
normal reclaim later.
Link: https://lore.kernel.org/20260811132209.2862708-2-youngjun.park@lge.com
Fixes: 0d6af9bcf3 ("mm, swap: use the swap table to track the swap count")
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
Acked-by: Kairui Song <kasong@tencent.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Barry Song <baohua@kernel.org>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit 0e2759afca ("page_counter: track failcnt only for legacy
cgroups") made failcnt accounting conditional on track_failcnt. It
enabled the flag for memcg->memory, but not for memcg->memsw or
memcg->tcpmem.
Consequently, memory.memsw.failcnt remains zero when the memory+swap limit
is hit. memory.kmem.tcp.limit_in_bytes still sets memcg->tcpmem.max, but
TCP charge failures are not reflected in memory.kmem.tcp.failcnt.
Enable failcnt accounting for both v1 counters.
To reproduce memory.memsw.failcnt:
CG=/sys/fs/cgroup/memory/memsw-test
LIMIT=33554432
mkdir "$CG"
echo "$LIMIT" > "$CG/memory.limit_in_bytes"
echo "$LIMIT" > "$CG/memory.memsw.limit_in_bytes"
Start a child process in the cgroup and make it allocate and touch 96 MiB
of memory, causing a memcg OOM.
cat "$CG/memory.memsw.failcnt"
Without the patch, memory.memsw.failcnt is 0. With the patch,
memory.memsw.failcnt is greater than 0.
To reproduce memory.kmem.tcp.failcnt:
CG=/sys/fs/cgroup/memory/tcpmem-test
LIMIT=65536
mkdir "$CG"
echo "$LIMIT" > "$CG/memory.kmem.tcp.limit_in_bytes"
Start a child process in the cgroup, create a TCP socket, and reserve
1 MiB of socket memory with SO_RESERVE_MEM. The reservation fails with
ENOMEM.
cat "$CG/memory.kmem.tcp.failcnt"
Without the patch, memory.kmem.tcp.failcnt is 0. With the patch,
memory.kmem.tcp.failcnt is greater than 0.
Link: https://lore.kernel.org/20260811030843.109104-1-guopeng.zhang@linux.dev
Closes: https://sashiko.dev/#/patchset/20260810074247.52747-1-guopeng.zhang@linux.dev?part=1
Fixes: 0e2759afca ("page_counter: track failcnt only for legacy cgroups")
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Tao Cui <cuitao@kylinos.cn>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
memcg_reparent_objcgs() has an inherent assumption that a folio's objcg is
the objcg of the folio's node. Folio migration across nodes breaks that
assumption: the new folio simply inherits the old folio's objcg while
living on a different node.
Once the assumption is broken, the reparenting of the folio's objcg and
the reparenting of the folio's LRU list are no longer atomic.
memcg_reparent_objcgs() handles one node per iteration and drops all the
locks in between, so the objcg gets reparented in the iteration for the
objcg's node while the LRU list gets spliced in the iteration for the
folio's node. Any LRU operation on that folio in between resolves its
lruvec through the objcg, and thus takes the lru_lock of the wrong memcg,
not the lru_lock of the list the folio is actually on.
Fix this by selecting the objcg by folio_nid() at charge time, and by
re-deriving it for the destination node in mem_cgroup_migrate() and
mem_cgroup_replace_folio().
Link: https://lore.kernel.org/20260807142406.443516-1-shakeel.butt@linux.dev
Fixes: f1cf8d2f36 ("mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Reported-by: Karl Erik Hofseth <karl.e.hofseth@opoint.com>
Closes: https://lore.kernel.org/all/anMmd1ADrDVwMO6v@work/
Co-developed-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Muchun Song <muchun.song@linux.dev>
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The memory-failure tests call cleanup() only after all result checks. A
failed ASSERT_* invokes fixture teardown and aborts the test, so it skips
cleanup() and leaves the injected page hardware-poisoned.
Invoke cleanup() from FIXTURE_TEARDOWN() instead. Guard it with
self->injection_attempted so tests that exit before injection do not try
to unpoison a page when no injection was attempted. Injection can poison
a page before returning an error or delivering SIGBUS, so teardown must
clean up after every injection attempt. This runs the existing HWPoison
and HardwareCorrupted checks on both normal and assertion-failure paths.
Link: https://lore.kernel.org/20260729091127.1001179-1-usama.anjum@arm.com
Fixes: ff4ef2fbd1 ("selftests/mm: add memory failure anonymous page test")
Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
shmem_evict_inode() ends with WARN_ON(inode->i_blocks) as a final
consistency check of shmem's block accounting. When it fires, the
inode-local counters die with the inode; what may linger is a small
residue in accounting kept outside the inode, such as per-mount or
per-user charges. No data is lost, and no corruption follows.
On kernels running with panic_on_warn=1, this accounting inconsistency
escalates to a full machine panic, which is disproportionate to the
impact.
Downgrade the WARN_ON() to a pr_warn() that reports the inode together
with its accounting counters (i_blocks, alloced, swapped, nrpages),
keeping the inconsistency visible in the logs.
The accounting bugs this check has caught over the years -- the swapout
race described in commit 0f3c42f522 ("tmpfs: change final i_blocks BUG
to WARNING") and the error recovery race fixed in commit 267a4c76bb
("tmpfs: fix shmem_evict_inode() warnings on i_blocks") -- are real and
should still be fixed; this change only removes the disproportionate
escalation.
One way to hit this race: soft_offline_in_use_page()'s fast path drops a
clean, unmapped shmem folio via mapping_evict_folio(), where the
xas_store() and the nrpages decrement are not atomic against a concurrent
shmem_evict_inode(); the final shmem_recalc_inode() can then read the
pre-decrement nrpages, compute freed = 0, and leave one page charged.
Same class as the races in 0f3c42f522 and 267a4c76bb, this time in the
under-count direction; reproduced on 7.2-rc4 with
madvise(MADV_SOFT_OFFLINE) racing MAP_FIXED replacement of a
shared-anonymous VMA.
[yujiacheng3@huawei.com: drop redundant casts in shmem_evict_inode() pr_warn]
Link: https://lore.kernel.org/20260729121201.776566-1-yujiacheng3@huawei.com
Link: https://lore.kernel.org/20260728091014.3876715-1-yujiacheng3@huawei.com
Fixes: 0f3c42f522 ("tmpfs: change final i_blocks BUG to WARNING")
Signed-off-by: Jiacheng Yu <yujiacheng3@huawei.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Yongqiang Liu <liuyongqiang13@huawei.com>
Cc: Christian Brauner <brauner@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
In zs_page_migrate(), locks are acquired in the following order:
1. write_lock(&pool->lock)
2. spin_lock(&class->lock)
3. zspage_write_trylock(zspage)
However, upon successful page migration, they were being released in
forward acquisition (FIFO) order:
1. write_unlock(&pool->lock)
2. spin_unlock(&class->lock)
3. zspage_write_unlock(zspage)
Fix the unlocking order to release locks in strict reverse (LIFO)
order of acquisition:
3. zspage_write_unlock(zspage)
2. spin_unlock(&class->lock)
1. write_unlock(&pool->lock)
Releasing locks in reverse order of acquisition adheres to standard
kernel locking hygiene, prevents potential lock ordering and lockdep
inconsistencies.
Link: https://lore.kernel.org/20260728055333.421080-1-richardycc@google.com
Signed-off-by: Richard Chang <richardycc@google.com>
Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Tested-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Martin Liu <liumartin@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
RWP tracks accesses by installing PAGE_NONE (protnone) PTEs, so its code
paths are gated on userfaultfd_rwp(). Without
CONFIG_ARCH_HAS_PTE_PROTNONE there is no PAGE_NONE -- <linux/pgtable.h>
defines it to a BUILD_BUG() stub, relying on callers folding such paths to
dead code via IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE).
userfaultfd_rwp() was not a compile-time constant, so the compiler could
not fold those paths. With an older compiler (gcc 8.5.0, sparc64) the
PAGE_NONE reference in move_pages_huge_pmd() survived to codegen:
mm/huge_memory.c:2874: _dst_pmd = pmd_modify(_dst_pmd, PAGE_NONE);
compiler_types.h:702: error: call to '__compiletime_assert_501'
declared with attribute error: BUILD_BUG failed
RWP cannot exist without protnone, so return a compile-time false when
CONFIG_ARCH_HAS_PTE_PROTNONE is unset; every RWP path then folds away.
Link: https://lore.kernel.org/amcitKvUvFYr8W38@thinkstation
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202607250853.VaJWGLeA-lkp@intel.com/
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
migrate_pages_batch() unmaps each folio before moving it, and every
unmap runs the mmu_notifier invalidate callbacks. On KVM hosts
try_to_migrate() ends up in kvm_mmu_notifier_invalidate_range_start() ->
tdp_mmu_zap_leafs(), which is expensive, so unmapping a large batch keeps
the CPU busy for a long time.
The loop already calls cond_resched(), but on PREEMPTION kernels that is
a no-op, and involuntary preemption is not a Tasks-RCU quiescent state.
A long batch therefore never reports a quiescent state, and the
migrating task (e.g. kcompactd) becomes a Tasks-RCU holdout, stalling the
Tasks-RCU grace period for minutes, which is common at Meta fleet:
INFO: rcu_tasks detected stalls on tasks:
0000000055349ecc: .. nvcsw: 1157401/1157401 holdout: 1 idle_cpu: -1/56 task:kcompactd0 state:R running task
Call Trace:
tdp_mmu_zap_leafs
tdp_mmu_next_root
gfn_to_pfn_cache_invalidate_start
kvm_mmu_notifier_invalidate_range_start
__mmu_notifier_invalidate_range_start
try_to_migrate_one
try_to_migrate
migrate_pages_batch
migrate_pages
compact_zone
compact_node
kcompactd
kthread
Use cond_resched_tasks_rcu_qs() so a quiescent state is reported even
when cond_resched() does nothing.
This has also been discussed at [1]
Link: https://lore.kernel.org/20260727-kcompact-v1-1-bdfefddd6874@debian.org
Link: https://lore.kernel.org/all/amdWVTs0WKOxguxP@gmail.com/ [1]
Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Gregory Price <gourry@gourry.net>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
As we deployed defrag_mode into Meta production, pressure spikes and
excessive swapping were observed on some workloads. Tracing confirmed
that this is unmovable/reclaimable requests spinning in the allocator and
direct reclaim, causing excessive amounts of swap.
The initial plan for defrag_mode was to rely on kswapd/kcompactd to
produce blocks, and if those are overwhelmed under high pressure, let the
allocator fall back (__rmqueue_steal()) after its retry loops. However,
that retrying results in more reclaim on some of these workloads than we'd
hoped, sometimes excessively so, spurred on by the !costly order
conditions in should_reclaim_retry().
The storms are dependent on the request type. Reclaim will inevitably
make room in existing movable blocks, since that's where the LRU pages
live. So if movable requests retry on reclaim, they make progress.
When non-movable requests spin in reclaim that isn't productive. They
cannot use the individually freed pages, and the process is unlikely to
accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They
spin and overreclaim excessively, which tanks performance and triggers
userspace guards like swap exhaustion or pressure based OOM.
To fix this, send non-movable requests, regardless of order, into
pageblock reclaim/compaction. This way, they help move things along to
meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and
excess OOM rates are no longer observed in production.
The longer-term plan is still to have all requests, including the movable
ones, help make blocks to spread the cost of defragmenting more evenly and
fairly; combined with proper watermarking to reduce allocation latencies
in the common case. However, doing this naively unearths scaling and
concurrency limitations in compaction that need to be addressed first.
Promoting just non-movables for now is the minimally viable bug fix for
the above issue.
[brendan.jackman@linux.dev: fix try_to_compact_pages() kerneldoc]
Link: https://lore.kernel.org/DK7NM9RPUJOD.11PNJJ5N2OBED@linux.dev
Link: https://lore.kernel.org/20260722150006.3848560-5-hannes@cmpxchg.org
Fixes: e3aa7df331 ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: "Brendan Jackman" <brendan.jackman@linux.dev>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The compaction capturing code assumes the allocation request order and
compaction target order are the same. That won't be true once defrag_mode
promotes sub-block allocations to pageblock-order compaction: compaction
targets the larger order, while capture should remain at the original
allocation order.
Move the capture_control to the page allocator and give it its own copies
of what the page freeing path matches against - zone, migratetype and the
allocation order - rather than reaching into compaction's live
compact_control. __alloc_pages_direct_compact() fills in migratetype and
order, and installs and hides current->capture_control around the whole
compaction call; try_to_compact_pages() aims capc->zone at each zone while
it is being compacted. compact_zone_order() no longer deals with capture
at all.
Pass the capture_control through try_to_compact_pages() /
compact_zone_order() in place of the bare struct page **.
No functional change.
Link: https://lore.kernel.org/20260722150006.3848560-4-hannes@cmpxchg.org
Fixes: e3aa7df331 ("mm: page_alloc: defrag_mode")
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Co-developed-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
While trying to fix a reclaim storm in defrag_mode, I noticed that
non-movable direct compaction is extremely inefficient.
When searching for space to evacuate, compaction only allows blocks of the
same type as the incoming request. This is to prevent migratetype
pollution, where a small non-movable request frees space in a movable
block and provokes the allocator to fall back and pollute it.
This protection is reasonable on one hand, but the downside is that it
makes non-movable direct compaction nearly useless: if we get the type
annotations right, by definition there aren't any movable pages inside the
non-movable blocks it is allowed to scan.
With defrag_mode, the goal is the production of whole blocks, which are
essentially type neutral: __rmqueue_claim() will convert them wholesale on
alloc. This makes type mixing and pollution a non-issue.
Fix the pollution gates to take the requested order into account, and
allow whole-block requests to scan blocks of other types.
The only exception is CMA blocks. That type is sticky and these blocks
cannot be claimed to other types. Continue to be strict with them, and
allow only explicit ALLOC_CMA requests and kcompactd to evacuate them.
Link: https://lore.kernel.org/20260722150006.3848560-3-hannes@cmpxchg.org
Fixes: e3aa7df331 ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Gregory Price <gourry@gourry.net>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: fix reclaim storms in defrag_mode", v2.
As we deployed vm.defrag_mode=1 in Meta production, some workloads
regressed with recurring pressure spikes and swap storms (which in turn
triggered userspace OOM rules on pressure and swap utilization levels).
Tracing pinned this to non-movable requests spinning and reclaiming
unproductively when kswapd/kcompactd are overwhelmed. Direct reclaim
predominantly frees up pages in movable blocks, but those requests cannot
use that space under defrag_mode rules; and it is unlikely to free up
whole blocks incidentally for __rmqueue_claim() to work.
This series fixes it by making non-movable requests participate in
pageblock production in the allocator slowpath - meaning, they will invoke
direct reclaim and direct compaction with pageblock_order.
That requires some small-ish adjustments up front in the allocator and the
compaction code: three prep patches and the fix last.
The series has been in production against one of the affected workloads
for several weeks and restores the OOM kill rate to !defrag_mode baseline.
This patch (of 4):
A subsequent patch will have some order-0 allocations participate in
compaction under defrag_mode, to stave off extfrag events.
Since this is a sprawling expansion of entry points, and compaction can
enter filesystem paths, add lockdep annotations that catches __GFP_FS
passing errors.
Direct reclaim has had this annotation for a while, and since reclaim and
compaction are usually used in conjunction, this is unlikely to unearth
old bugs. It's more about future proofing and peace of mind.
Link: https://lore.kernel.org/20260722150006.3848560-1-hannes@cmpxchg.org
Link: https://lore.kernel.org/20260722150006.3848560-2-hannes@cmpxchg.org
Fixes: e3aa7df331 ("mm: page_alloc: defrag_mode")
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Brendan Jackman <brendan.jackman@linux.dev>
Cc: Gregory Price <gourry@gourry.net>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
DAMOS_MIGRATE_{HOT,COLD} passes a target NUMA node to migrate_pages().
But alloc_migration_target() only treats mtc->nid as a preferred node
unless __GFP_THISNODE is set. Hence target allocation can fall back to
another node, and migrate_pages() can report success without placing the
folio on the requested target node.
Consider a two-node tiered system where node 0 is a fast tier and node 1
is a CPU-less slow tier such as CXL memory, and the user wants to promote
hot regions from node 1 to node 0 with a command like:
sudo damo start --ops vaddr --target_pid ${workload_pid} \
--damos_action migrate_hot 0 \
--damos_access_rate 70% max
Without the __GFP_THISNODE flag, when the memory allocator finds that node
0 is nearly full, it can fall back to node 1 without waking up kswapd.
Then the pages allocated for migrate_pages() are still on node 1, and the
regions that are expected to be promoted to node 0 are only moved to
different physical pages on node 1.
Meanwhile, both the mm_migrate_pages tracepoint and DAMOS's own sz_applied
statistics (reported via the damos_stat_after_apply_interval tracepoint)
show the migrations as successful, which makes the failure practically
invisible and hard to investigate.
Running a demotion-purpose DAMOS scheme alongside the promotion scheme
does not fully avoid this either. If demotion cannot keep up with the
promotion rate, allocation can still fall back to node 1 during promotion,
and the same misleading statistics show up.
Make DAMON's migration target allocation strict by setting __GFP_THISNODE,
so that a failed allocation on the target node is reported as a failure
instead of silently landing on a different node. This is consistent with
alloc_misplaced_dst_folio(), alloc_demote_folio(), and with
do_move_pages_to_node(), which all use __GFP_THISNODE for migrations to an
explicit destination node.
Link: https://lore.kernel.org/20260721135607.251869-1-sj@kernel.org
Signed-off-by: Jiahui Zhang <jiahuitry@outlook.com>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Honggyu Kim <honggyu.kim@sk.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The kernel-doc comment above enum damos_filter_type states that only the
anon and memcg type filters are handled by damon_operations (and therefore
accounted as 'tried'), and that DAMON_OPS_VADDR and DAMON_OPS_FVADDR do
not support those two filter types.
Neither is accurate anymore. damos_filter_for_ops() routes every filter
type except ADDR and TARGET to the operations layer, and the VADDR and
FVADDR operations (the latter being a copy of the former) handle all of
those types through damos_folio_filter_match() / damos_va_filter_out().
Update the comment to match the code.
Link: https://lore.kernel.org/20260721140011.269802-1-sj@kernel.org
Signed-off-by: Song Hu <husong@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/early_ioremap: clarify and clean up
early_ioremap_reset()".
__late_set_fixmap() and __late_clear_fixmap() are only used after
early_ioremap_reset() has been called, but the comment above them does not
say anything about that. So arm64, riscv and powerpc, whose
__set_fixmap() works before and after paging_init(), describe the same
situation in three different ways:
calls reset defines the macros
arm64 yes yes
riscv no yes
powerpc no no
Patch 1 documents when early_ioremap_reset() needs to be called and that
only architectures calling it need to define the macros.
Patches 2 and 3 remove the unneeded riscv macros, which are unreachable,
and the arm64 reset call and macros, which change nothing.
No functional change.
This patch (of 3):
__late_set_fixmap() and __late_clear_fixmap() are only used after
early_ioremap_reset() has been called.
arm64, riscv and powerpc all have a __set_fixmap() that works before and
after paging_init(), so they do not need to call early_ioremap_reset() or
define the macros, but they describe the same situation in three different
ways:
calls reset defines the macros
arm64 yes yes
riscv no yes
powerpc no no
The existing comment is vague and allows all three. Replace it with
comments that make it clear when the reset and the macros are needed.
No functional change.
Link: https://lore.kernel.org/20260708170647.362562-1-ekffu200098@gmail.com
Link: https://lore.kernel.org/20260708170647.362562-2-ekffu200098@gmail.com
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
The pagemap_ioctl selftest allocates memory via calloc() in several places
but does not check the return values. If calloc() fails, the subsequent
code will dereference a NULL pointer and crash.
Additionally, in sanity_tests(), the calloc() failure check incorrectly
uses MAP_FAILED (the mmap() error constant) instead of NULL. Since
calloc() returns NULL on failure, the check never triggers and a failed
allocation goes undetected.
Add NULL checks after each calloc() call, and fix the wrong error constant
in sanity_tests(). Use ksft_exit_fail_msg() consistent with the existing
error handling pattern in the file.
Link: https://lore.kernel.org/20260721063611.342-1-yanlonglong@kylinos.cn
Signed-off-by: longlong yan <yanlonglong@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: SJ Park <sj@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit a67fe41e21 ("mm: rmap: support batched unmapping for file large
folios") extended batched unmapping for file folios. That also required
making pte_install_uffd_wp_if_needed() support batching, but that was left
out for the time being. Correctness was maintained by stopping batching
if the VMA the folio belongs to is marked uffd-wp.
Now that cond_install_uffd_wp_ptes() supports batching, call it with the
full batch length and allow folio_unmap_pte_batch() to batch file folios
belonging to uffd-wp VMAs.
For file folios, if the uffd-wp bit is set, unmapping converts present
PTEs into uffd-wp markers. We must ensure that the same PTE range is not
reprocessed by the try_to_unmap_one() loop.
The page_vma_mapped_walk API ensures this: check_pte() only returns true
if any PFN in [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the PTE.
There is no PFN underlying a uffd-wp marker PTE, so check_pte() returns
false and the walk skips ahead until it reaches a present entry again.
Link: https://lore.kernel.org/20260720065508.2695106-4-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "Batch unmap of uffd-wp file folios", v2.
Currently, batched unmapping is supported if:
1) folio is a file folio, not belonging to uffd-wp VMA
2) folio is anonymous and not swapbacked (lazyfree), not belonging to
uffd-wp VMA
So the cases which are not supported are
1) folio belonging to uffd-wp VMA
2) folio is anonymous and swapbacked
It is easy to see that this adds a lot of cognitive load while reading
try_to_unmap_one - we need to remember throughout whether nr_pages == 1 or
> 1.
The uffd-wp handling in try_to_unmap_one is regarding preserving the
uffd-wp state for file folios via pte_install_uffd_wp_if_needed (for anon
folio, we handle that while constructing the swap pte).
Stop special casing on uffd-wp VMAs by simply adding batching support to
pte_install_uffd_wp_if_needed.
This patch (of 3):
pte_install_uffd_wp_if_needed() has grown too large for mm_inline.h. Move
it to memory.c.
This helper is only used inside mm/, so declare it in mm/internal.h
instead of a public header.
While at it, convert the comment to kerneldoc and rename the local
arguments from pte/pteval to ptep/pte so the pointer and PTE value are
easier to distinguish.
Link: https://lore.kernel.org/20260720065508.2695106-1-dev.jain@arm.com
Link: https://lore.kernel.org/20260720065508.2695106-2-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Since commit 304daa8132 ("maps4: add /proc/kpageflags interface"),
/proc/kpageflags directly operates on page->flags to determine page
status. Later, commit 1779754959 ("proc: export more page flags in
/proc/kpageflags") started using page helper functions when exposing new
flags, leading to a mix of both approaches.
For tail pages, the original code did not return corresponding status.
commit 0a71649cb7 ("/proc/kpageflags: return KPF_SLAB for slab tail
pages") and commit 832fc1de01 ("/proc/kpageflags: return KPF_BUDDY for
"tail" buddy pages") made tail slab/buddy pages also return corresponding
status. Then commit dee3d0bef2 ("proc: rewrite stable_page_flags()")
made all tail pages return the same status as their head page, except for
hwpoison and mapped flags. It also cached the folio's flags and operate
on the flags directly to avoid concurrency issues if using folio_test_*()
helpers.
Since commit 476d87d6a0 ("fs: stable_page_flags(): use
snapshot_page()"), we can now safely switch to folio_test_*() helpers
instead of directly operating on flags, which is more readable and
consistent with the rest of the kernel. Only convert cfolio-specific
flags (i.e., anon, ksm, swapcache) to folio_test_*() helpers, which
reduces redundant code. Keep others unchanged due to they aren't
folio-specific flags or coverting them doesn't cleanup.
No functional change is intended.
Link: https://lore.kernel.org/20260720033021.4091944-3-tujinjiang@huawei.com
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Luiz Capitulino <luizcap@redhat.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Nanyong Sun <sunnanyong@huawei.com>
Cc: Svetly Todorov <svetly.todorov@memverge.com>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_apply_min_nr_regions() repeatedly split each region until its size
becomes small enough to meet the user-defined low limit of the number of
regions. The loop assumes the split operation (damon_split_region_at())
will always succeed and create the new region. But the operation could
silently fail for memory allocation failures, for example.
If such failure happens and the region was the last region, the linked
list-based next region fetching returns invalid pointer. As a result,
invalid memory dereference and corruption could happen. Even if the
corner case is handled, it imposes stress to the allocator by trying split
regions for other targets. Fix the issue by breaking all the loops for
any region split failure.
This means there could be a min_nr_regions violation. It will only rarely
happen since the allocation is arguably too small to fail. Even if it
happens, it is only temporal. damon_apply_min_nr_regions() will be called
again after the aggregation interval.
The user impact of the issue should be minor, since the allocation is
arguably too small to fail. But, it could still theoretically happen, and
the consequence is very bad.
This issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260719155442.88794-1-sj@kernel.org
Link: https://lore.kernel.org/20260717011834.120715-1-sj@kernel.org [1]
Fixes: b1029f29eb ("mm/damon/core: split regions for min_nr_regions")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 7.1.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Since commit 42f994b714 ("mm/damon/core: implement scheme-specific apply
interval"), DAMOS scheme can be applied at any time. At that time,
nr_accesses may not be fully aggregated. But the quota prioritization
score is calculated using the not fully aggregated count. As a result,
the performance of DAMOS could be degraded. Fix by using
damon_nr_accesses_mvsum() instead.
The user impact of the issue is suboptimum DAMOS performance under certain
setups. Nonetheless, the bug was there from the beginning of the setup
availability. In other words, the suboptimum performance is the baseline
of the setup and hence it didn't cause regression. Also the extent of the
suboptimality was not big enough to be found from users and testers.
Still, this is a clear bug that is better to be fixed, and can be easily
fixed.
Link: https://lore.kernel.org/20260719161136.90191-1-sj@kernel.org
Fixes: 42f994b714 ("mm/damon/core: implement scheme-specific apply interval")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: <stable@vger.kernel.org> # 6.7.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
hugetlbfs_fill_super() allocates a hugepage subpool when size or min_size
mount options are specified. hugepage_new_subpool() may also reserve huge
pages for min_size.
If root dentry creation fails after the subpool is created, the failure
path frees the subpool with kfree(). This bypasses hugepage_put_subpool()
and can leave min_size reservations charged.
Use hugepage_put_subpool() on the failure path, matching the normal
put_super path.
Link: https://lore.kernel.org/20260720021900.1376309-1-chenyichong@uniontech.com
Fixes: 7ca02d0ae5 ("hugetlbfs: accept subpool min_size mount option and setup accordingly")
Signed-off-by: Yichong Chen <chenyichong@uniontech.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
In damon_migrate_folio_list(), we're redeclaring folio inside the first
while loop, but it just shadows the outer one. Since the second loop uses
the outer folio anyway, the inner declaration is pointless.
Remove it to consistently reuse the same variable throughout the
function and improve readability.
Link: https://lore.kernel.org/20260718002125.637104-1-lienze@kylinos.cn
Signed-off-by: Enze Li <lienze@kylinos.cn>
Reviewed-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damon_test_commit_target_regions_for() traverses expected values array
after damon_commit_target_regions() call. It assumes
damon_commit_target_regions() made expected number of regions. It might
not. Because the traversal is made based on the region count, it could do
out of bounds access to the expectation value array.
The consequent user impact (out-of-bound access) is quite bad. The
realistic user impact would be limited, though. It would affect only test
run setups.
Fix it by testing if the number of regions was also changed as expected
and exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-8-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: 603f67eb91 ("mm/damon/tests/core-kunit: add damon_commit_target_regions() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damos_test_commit_quota_goals_for() traverses damos quota goals after
damos_commit_quota_goals() call. It assumes damos_commit_quota_goals()
made expected numbers of goals. It might not. Because the traversal is
made based on destination struct length, it could do out of bounds access
for source expectation value array.
The consequent user impact (out-of-bound access ) is quite bad. The
realistic user impact would be limited though. It would affect only test
run setups.
Fix it by testing if the number of goals was also changed as expected and
exit early for the failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-7-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: d9adfb8a28 ("mm/damon/tests/core-kunit: add damos_commit_quota_goals() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
damos_test_commit_dests_for() traverse damos action destinations after
damos_commit_dests(). It assumes damos_commit_dests() made expected
numbers of destinations for source and destination structures. It might
not. Because the traversal is made based on destination struct length, it
could do out of bounds access for source value expectation.
The consequent user impact (out-of-bound access ) is quite bad. The
realistic user impact would be limited, though. It would affect only test
run setups.
Fix it by exiting early for the number of regions test failure.
The issue was discovered [1] by Sashiko.
Link: https://lore.kernel.org/20260718001442.87129-6-sj@kernel.org
Link: https://lore.kernel.org/20260713144757.39740-1-sj@kernel.org [1]
Fixes: eec573b8dd ("mm/damon/tests/core-kunit: add damos_commit_dests() test")
Signed-off-by: SJ Park <sj@kernel.org>
Cc: Brendan Higgins <brendan.higgins@linux.dev>
Cc: <stable@vger.kernel.org> # 6.19.x
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
