From 26444eb71465c9934d9d418ef69c43f61185329b Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:31 +0100 Subject: [PATCH 01/25] mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which this series has to deal with carefully. This mitigation resolves the issue by acquiring the mmap read lock on init_mm on vmap page table free if a ptdump is in progress. However the fix in this series would cause a deadlock if we were to simply apply it for arm64 without also reverting the change. This is because vmap may acquire the read lock before ptdump attempts to acquire the write lock, which then gets queued, and rwsem starvation rules mean that the (unacknowledged) nested mmap read lock in the arm64 code would also block, meaning the original read lock is never released and thus deadlock. This series works around this by #ifndef CONFIG_ARM64'ing the mmap read lock in vmap logic, then partially reverting commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), keeping the enablement of huge vmap support, and removing the ifdeffery with the partial revert patch. There are related issues that are also addressed in this series: * x86 page attribute logic, specifically Change Page Attributes (CPA), implements a feature whereby huge ranges can be collapsed into huge leaf entries. This can similarly cause a UAF when done in parallel with a ptdump walk, so similarly acquire the init_mm mmap lock to avoid this. * The CPA logic allows concurrent page table manipulation and CPA collapse, meaning the former risks accessing a page table the latter frees. Fix this by acquiring mmap write lock on init_mm across the whole CPA collapse operation and read lock on the page table manipulation. * x86 and arm64 permit walks of non-kernel mm's (both allowing efi mm walks, and in x86's case arbitrary mm's), so we ensure kernel mappings remain stable by locking the init_mm as well as the mm being walked. The ordering of patches is established for both strict dependencies (the arm64 partial revert in particular has to be done after the vmap changes) and logical ones (the non-kernel mm fix only makes sense once the vmap/CPA fixes are in place). This patch (of 3): Currently there is a nasty race between ptdump and vmap when attempting to map a huge P4D, PUD or PMD entry: * ptdump walks kernel page table ranges it doesn't own. * When vmap maps ranges it tries to promotes existing ones to huge page tables in vmap_try_huge_[p4d,pud,pmd]() at P4D, PUD and PMD level, freeing the lower page table in [p4d,pud,pmd]_free_[pud,pmd,pte]_page() when it succeeds. Both of these things can happen at the same time and as a result ptdump can access a freed page table, resulting in a use-after-free and memory corruption. This is possible because while ptdump_walk_pgd() holds both the mem hotplug lock and the mmap write lock before invoking walk_page_range_debug(), vmap takes no relevant locks at all. Fix this by holding the mmap read lock in vmap_try_huge_*() when freeing page tables. The read lock is sufficient: ptdump is the only walker that must be excluded and it holds the mmap write lock. Other holders of the read lock may run concurrently, but each exclusively owns the range it operates on and cannot reach the page tables freed here. We also hold the lock while assigning the huge page table entry, which means page table walkers observe only the huge or non-huge page table entry. We use a trylock to prevent ptdump from blocking vmap making forward progress. This is fine because it's an optimisation in any case, and thus the vmap can safely proceed regardless. All other kernel page table walkers that touch vmalloc ranges either exclusively own the memory walked or acquire the mmap lock, so this correctly excludes those walkers. One wrinkle here is commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which addresses the issue for arm64 only by explicitly acquiring the mmap read lock on kernel page table freeing should a concurrent ptdump be in progress. This is problematic as vmap may acquire the mmap read lock prior to ptdump attempting to acquire an mmap write lock, leading to a deadlock when the mmap read lock is slept upon on page table freeing due to rwsem anti-starvation. We work around this by predicating the mmap lock being taken on !CONFIG_ARM64 for the time being. With this patch applied, a follow up will partially revert commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") and at that stage remove the arm64 ifdeffery. We also update walk_page_range_debug() to assert the mmap write lock unconditionally and update the comment here to reflect this change. The issue has existed as long as ptdump was available and vmap freed page tables when promoting to a huge leaf entry, that is, since commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") for huge ioremap, and commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings") for huge vmalloc. Since the former is the earlier of the two we choose that for our Fixes tag. We also define a guard class for mmap_read_trylock() so we can use cleanup.h to make the scope handling cleaner in the implementation. This patch is based on work by David Carlier (linked), with gratitude! Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-0-8cc77dcc0018@kernel.org Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-1-8cc77dcc0018@kernel.org Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: syzbot+fd95a72470f5a44e464c@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a287988.39669fcc.33b062.00a0.GAE@google.com/T/ Link: https://lore.kernel.org/linux-mm/20260706203128.162335-1-devnexen@gmail.com/ Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Dev Jain Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/mmap_lock.h | 1 + mm/pagewalk.c | 22 ++++++++++-------- mm/vmalloc.c | 49 ++++++++++++++++++++++++++++++++------- 3 files changed, 53 insertions(+), 19 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 04b8f61ece5d..6b5c2390cc30 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -621,6 +621,7 @@ static inline void mmap_read_unlock(struct mm_struct *mm) DEFINE_GUARD(mmap_read_lock, struct mm_struct *, mmap_read_lock(_T), mmap_read_unlock(_T)) +DEFINE_GUARD_COND(mmap_read_lock, _try, mmap_read_trylock(_T)) static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) { diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 3ae2586ff45b..bbcfd68d0907 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -678,6 +678,8 @@ int walk_kernel_page_table_range_lockless(unsigned long start, unsigned long end * will also not lock the PTEs for the pte_entry() callback. * * This is for debugging purposes ONLY. + * + * The mmap write lock must be held. */ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, @@ -691,6 +693,16 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, .no_vma = true }; + /* + * When walking userland page tables, an mmap write lock must be held to + * account for munmap() downgrading to an mmap read lock when tearing + * down page tables. + * + * When walking kernel page tables, an mmap write lock must also be held + * to account for page table freeing on vmap huge page mapping. + */ + mmap_assert_write_locked(mm); + /* For convenience, we allow traversal of kernel mappings. */ if (mm == &init_mm) return walk_kernel_page_table_range(start, end, ops, @@ -700,16 +712,6 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, if (!check_ops_safe(ops)) return -EINVAL; - /* - * The mmap lock protects the page walker from changes to the page - * tables during the walk. However a read lock is insufficient to - * protect those areas which don't have a VMA as munmap() detaches - * the VMAs before downgrading to a read lock and actually tearing - * down PTEs/page tables. In which case, the mmap write lock should - * be held. - */ - mmap_assert_write_locked(mm); - return walk_pgd_range(start, end, &walk); } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1afca3568b9b..d5c4d2bb770b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -43,6 +43,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -158,10 +159,24 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, PMD_SIZE)) return 0; - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; + if (!pmd_present(*pmd)) + return pmd_set_huge(pmd, phys_addr, prot); - return pmd_set_huge(pmd, phys_addr, prot); + /* + * Acquire the mmap read lock to exclude ptdump, which walks + * kernel page tables it does not own under the mmap write lock. + * + * Concurrent read lock holders are safe: each exclusively owns + * the range it operates on and cannot reach this page table. + */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!pmd_free_pte_page(pmd, addr)) + return 0; + return pmd_set_huge(pmd, phys_addr, prot); + } } static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, @@ -210,10 +225,18 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, PUD_SIZE)) return 0; - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; + if (!pud_present(*pud)) + return pud_set_huge(pud, phys_addr, prot); - return pud_set_huge(pud, phys_addr, prot); + /* See comment in vmap_try_huge_pmd(). */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!pud_free_pmd_page(pud, addr)) + return 0; + return pud_set_huge(pud, phys_addr, prot); + } } static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, @@ -262,10 +285,18 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, P4D_SIZE)) return 0; - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; + if (!p4d_present(*p4d)) + return p4d_set_huge(p4d, phys_addr, prot); - return p4d_set_huge(p4d, phys_addr, prot); + /* See comment in vmap_try_huge_pmd(). */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!p4d_free_pud_page(p4d, addr)) + return 0; + return p4d_set_huge(p4d, phys_addr, prot); + } } static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, From 27c32e5538344b13c1505a08861e04620c125d47 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:34 +0100 Subject: [PATCH 02/25] mm/ptdump: always stabilise against page table freeing using init_mm Previous commits have established the invariant that kernel page table freeing is performed while an mmap read lock on init_mm is held, which fixes races between ptdump and kernel page table freeing over init_mm. However, x86 and arm64 can perform a ptdump over an mm other than init_mm via ptdump_walk_pgd() and since kernel memory ranges are shared across non-kernel mm's, this means that the race still exists for these cases. Fix this by acquiring a nested mmap write lock for init_mm in ptdump_walk_pgd(). This is safe as we take this after mmap write locking the mm, and nothing acquires the init_mm lock first before locking an arbitrary mm, so no deadlock is possible. Also update walk_page_range_debug() to assert that init_mm is write locked, add a comment explaining why and remove some redundant code, and eliminate the unnecessary and confusing invocation of walk_kernel_page_table_range(). We can safely remove the non-NULL check for walk.mm, as the mmap lock asserts would NULL pointer deref if it was (and of course no callers do this). The first point at which ptdump can race kernel page table freeing is commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table"), so we target this in the Fixes tag. Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-4-8cc77dcc0018@kernel.org Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: David Carlier Cc: Dev Jain Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Cc: Signed-off-by: Andrew Morton --- mm/pagewalk.c | 14 +++++++++----- mm/ptdump.c | 7 +++++++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index bbcfd68d0907..5d87c632a255 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -702,12 +702,16 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, * to account for page table freeing on vmap huge page mapping. */ mmap_assert_write_locked(mm); + /* + * x86, arm64 ptdump allow walks of efi mm's and x86 ptdump allows walks + * of arbitrary mm's. + * + * However, they both must also hold the init_mm lock to account for + * concurrent kernel page table freeing. + */ + mmap_assert_write_locked(&init_mm); - /* For convenience, we allow traversal of kernel mappings. */ - if (mm == &init_mm) - return walk_kernel_page_table_range(start, end, ops, - pgd, private); - if (start >= end || !walk.mm) + if (start >= end) return -EINVAL; if (!check_ops_safe(ops)) return -EINVAL; diff --git a/mm/ptdump.c b/mm/ptdump.c index 973020000096..5851096e6f65 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -178,11 +178,18 @@ void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd) get_online_mems(); mmap_write_lock(mm); + /* To stabilise kernel page tables we must hold the init_mm lock too. */ + if (mm != &init_mm) + mmap_write_lock_nested(&init_mm, SINGLE_DEPTH_NESTING); + while (range->start != range->end) { walk_page_range_debug(mm, range->start, range->end, &ptdump_ops, pgd, st); range++; } + + if (mm != &init_mm) + mmap_write_unlock(&init_mm); mmap_write_unlock(mm); put_online_mems(); From 9d3277b2c07ccc9508d648098b3bbb46c61b7f3c Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:35 +0100 Subject: [PATCH 03/25] arm64: remove redundant concurrent ptdump UAF mitigation This partially reverts commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), retaining vmalloc-huge support but eliminating the now redundant mitigation against a race between huge vmap page table freeing and ptdump, as this issue has now been fixed at core. We also simultaneously remove the arm64 if-deffery when acquiring the mmap read lock upon vmap huge page table promotion as it is no longer required. Note that this patch relies on the preceding vmalloc patch, and should not be backported alone. Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-5-8cc77dcc0018@kernel.org Fixes: fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Dev Jain Acked-by: Mike Rapoport (Microsoft) Acked-by: Kiryl Shutsemau (Meta) Acked-by: Will Deacon Reviewed-by: David Hildenbrand (Arm) Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: David Carlier Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton --- arch/arm64/include/asm/ptdump.h | 2 -- arch/arm64/mm/mmu.c | 43 +++------------------------------ arch/arm64/mm/ptdump.c | 11 ++------- mm/vmalloc.c | 15 +++--------- 4 files changed, 9 insertions(+), 62 deletions(-) diff --git a/arch/arm64/include/asm/ptdump.h b/arch/arm64/include/asm/ptdump.h index 5b374a6ab34a..50a195eda8ed 100644 --- a/arch/arm64/include/asm/ptdump.h +++ b/arch/arm64/include/asm/ptdump.h @@ -7,8 +7,6 @@ #include -DECLARE_STATIC_KEY_FALSE(arm64_ptdump_lock_key); - #ifdef CONFIG_PTDUMP #include diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 18a8b0d3714e..d4de88770ecf 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -49,8 +49,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -DEFINE_STATIC_KEY_FALSE(arm64_ptdump_lock_key); - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); @@ -1864,8 +1862,7 @@ int pmd_clear_huge(pmd_t *pmdp) return 1; } -static int __pmd_free_pte_page(pmd_t *pmdp, unsigned long addr, - bool acquire_mmap_lock) +int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { pte_t *table; pmd_t pmd; @@ -1877,25 +1874,13 @@ static int __pmd_free_pte_page(pmd_t *pmdp, unsigned long addr, return 1; } - /* See comment in pud_free_pmd_page for static key logic */ table = pte_offset_kernel(pmdp, addr); pmd_clear(pmdp); __flush_tlb_kernel_pgtable(addr); - if (static_branch_unlikely(&arm64_ptdump_lock_key) && acquire_mmap_lock) { - mmap_read_lock(&init_mm); - mmap_read_unlock(&init_mm); - } - pte_free_kernel(NULL, table); return 1; } -int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) -{ - /* If ptdump is walking the pagetables, acquire init_mm.mmap_lock */ - return __pmd_free_pte_page(pmdp, addr, /* acquire_mmap_lock = */ true); -} - int pud_free_pmd_page(pud_t *pudp, unsigned long addr) { pmd_t *table; @@ -1911,36 +1896,16 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) } table = pmd_offset(pudp, addr); - - /* - * Our objective is to prevent ptdump from reading a PMD table which has - * been freed. In this race, if pud_free_pmd_page observes the key on - * (which got flipped by ptdump) then the mmap lock sequence here will, - * as a result of the mmap write lock/unlock sequence in ptdump, give - * us the correct synchronization. If not, this means that ptdump has - * yet not started walking the pagetables - the sequence of barriers - * issued by __flush_tlb_kernel_pgtable() guarantees that ptdump will - * observe an empty PUD. - */ - pud_clear(pudp); - __flush_tlb_kernel_pgtable(addr); - if (static_branch_unlikely(&arm64_ptdump_lock_key)) { - mmap_read_lock(&init_mm); - mmap_read_unlock(&init_mm); - } - pmdp = table; next = addr; end = addr + PUD_SIZE; do { if (pmd_present(pmdp_get(pmdp))) - /* - * PMD has been isolated, so ptdump won't see it. No - * need to acquire init_mm.mmap_lock. - */ - __pmd_free_pte_page(pmdp, next, /* acquire_mmap_lock = */ false); + pmd_free_pte_page(pmdp, next); } while (pmdp++, next += PMD_SIZE, next != end); + pud_clear(pudp); + __flush_tlb_kernel_pgtable(addr); pmd_free(NULL, table); return 1; } diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 1c20144700d7..5a76c59b5ada 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -283,13 +283,6 @@ void note_page_flush(struct ptdump_state *pt_st) note_page(pt_st, 0, -1, pte_val(pte_zero)); } -static void arm64_ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm) -{ - static_branch_inc(&arm64_ptdump_lock_key); - ptdump_walk_pgd(st, mm, NULL); - static_branch_dec(&arm64_ptdump_lock_key); -} - void ptdump_walk(struct seq_file *s, struct ptdump_info *info) { unsigned long end = ~0UL; @@ -318,7 +311,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) } }; - arm64_ptdump_walk_pgd(&st.ptdump, info->mm); + ptdump_walk_pgd(&st.ptdump, info->mm, NULL); } static void __init ptdump_initialize(void) @@ -360,7 +353,7 @@ bool ptdump_check_wx(void) } }; - arm64_ptdump_walk_pgd(&st.ptdump, &init_mm); + ptdump_walk_pgd(&st.ptdump, &init_mm, NULL); if (st.wx_pages || st.uxn_pages) { pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN pages found\n", diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d5c4d2bb770b..f4fa227a8d7f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -169,10 +169,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, * Concurrent read lock holders are safe: each exclusively owns * the range it operates on and cannot reach this page table. */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!pmd_free_pte_page(pmd, addr)) return 0; return pmd_set_huge(pmd, phys_addr, prot); @@ -229,10 +226,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, return pud_set_huge(pud, phys_addr, prot); /* See comment in vmap_try_huge_pmd(). */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!pud_free_pmd_page(pud, addr)) return 0; return pud_set_huge(pud, phys_addr, prot); @@ -289,10 +283,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, return p4d_set_huge(p4d, phys_addr, prot); /* See comment in vmap_try_huge_pmd(). */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!p4d_free_pud_page(p4d, addr)) return 0; return p4d_set_huge(p4d, phys_addr, prot); From c2689266e5f70ecc960e64d40516518c038ae34e Mon Sep 17 00:00:00 2001 From: Ramin Moussavi Date: Mon, 27 Jul 2026 23:58:23 +0200 Subject: [PATCH 04/25] microblaze: restore the page alignment of swapper_pg_dir microblaze handles TLB misses in software, and the handler builds the address of the L1 entry by ORing the index into the page directory base instead of adding it (hw_exception_handler.S): bsrli r5, r3, PGDIR_SHIFT - 2 andi r5, r5, PAGE_SIZE - 4 /* Assume pgdir aligned on 4K boundary, no need for "andi r4,r4,0xfffff003" */ or r4, r4, r5 lwi r4, r4, 0 /* Get L1 entry */ The index is masked to the low 12 bits, so the OR only works if those bits of the base are zero -- which is exactly the assumption the comment states and the reason the masking of the base can be skipped. swapper_pg_dir had no alignment directive of its own. It was aligned because it followed empty_zero_page in head.S, and that one carried the .align 12: .section .data .global empty_zero_page .align 12 empty_zero_page: .space PAGE_SIZE .global swapper_pg_dir swapper_pg_dir: .space PAGE_SIZE Commit 6215d9f4470f ("arch, mm: consolidate empty_zero_page") removed empty_zero_page from head.S, and with it the .align 12 that -- despite sitting next to empty_zero_page -- was what page aligned swapper_pg_dir. Since then swapper_pg_dir lands wherever .data happens to put it, its low bits are no longer zero, and every kernel TLB miss ORs the index into a base with a nonzero offset. The resulting L1 lookups read the wrong words, no valid translation is ever installed, and the kernel spins in exceptions long before it can print anything. On qemu-system-microblazeel (petalogix-s3adsp1800) the console stays completely silent at 100% CPU; there is no oops and no guest error reported by qemu, which makes this awkward to diagnose. Give swapper_pg_dir the alignment it requires, rather than relying on a neighbour to provide it. microblaze has no noMMU variant left in mainline -- CONFIG_MMU is def_bool y and mmu_defconfig is the only defconfig -- so this is not a corner case: every mainline microblaze kernel since v7.1-rc1 fails to boot, including the v7.1 release. v7.0: swapper_pg_dir = 0xc05fd000 (aligned) v7.1-rc1: swapper_pg_dir = 0xc0603140 (offset 320) v7.1-rc1 + this fix: swapper_pg_dir = 0xc0604000 (aligned) next-20260726: swapper_pg_dir = 0xc0615140 (offset 320) next-20260726 + this fix: swapper_pg_dir = 0xc0616000 (aligned) Verified on qemu-system-microblazeel (petalogix-s3adsp1800) with mmu_defconfig and microblazeel gcc 12.5.0: v7.1-rc1 and next-20260726 both print nothing at all without the fix, and both boot to userspace with it. Link: https://lore.kernel.org/20260727215823.1422701-1-ramin.moussavi@yacoub.de Fixes: 6215d9f4470f ("arch, mm: consolidate empty_zero_page") Signed-off-by: Ramin Moussavi Cc: Michal Simek Cc: Mike Rapoport Cc: Signed-off-by: Andrew Morton --- arch/microblaze/kernel/head.S | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/microblaze/kernel/head.S b/arch/microblaze/kernel/head.S index 808019c3b7ac..9bd3e513c89b 100644 --- a/arch/microblaze/kernel/head.S +++ b/arch/microblaze/kernel/head.S @@ -39,6 +39,8 @@ #include .section .data +/* The MMU requires a page aligned page directory. */ +.align 12 .global swapper_pg_dir swapper_pg_dir: .space PAGE_SIZE From 86da3f7e1e609e1e8bfbab198af68467c5a015a5 Mon Sep 17 00:00:00 2001 From: Hugh Dickins Date: Mon, 27 Jul 2026 22:24:14 -0700 Subject: [PATCH 05/25] mm/filemap: __filemap_add_folio() restore index before retrying In __filemap_add_folio()'s split-a-conflict loop, xas_set_order() is applied repeatedly: each application modifies xas.xa_index, rounding it down according to the split_order attempted at that stage: and if all goes as intended, it eventually (or immediately) converges on an xas_try_split() to the required folio_order, with xas.xa_index now the same as index: then xas_store() puts the new folio into the xarray there. But if a new node was needed, and GFP_NOWAIT allocation did not get one, the lock is dropped, xas_nomem() used to allocate, and sequence retried. If (that part of) the xarray is unchanged when the lock is reacquired, no problem. But what if the conflict was meanwhile resolved by another thread (perhaps even doing the same thing, inserting a folio at that same index)? Isn't there a danger of now putting our folio into the xarray at an intermediate rounded-down index? With !folio_contains() bug to follow, when CONFIG_DEBUG_VM=y is checking for that. Fix this with an xas_set_order() to restore the original xas.xa_index at the bottom of the loop, so the retry does a full re-evaluation after reacquiring the lock, and cannot reach xas_store() with the wrong index. Production was suffering from rare SIGILLs and SIGSEGVs, executable text found a page away from where it belonged, !folio_contains() bug hit when debug enabled: symptoms not seen since this patch went in. Link: https://lore.kernel.org/562fbfa6-dd6d-0b6a-2461-ed2ff1173bc8@google.com Fixes: 200a89c159a7 ("mm/filemap: use xas_try_split() in __filemap_add_folio()") Signed-off-by: Hugh Dickins Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Cc: Chris J Arges Cc: David Hildenbrand Cc: Jan Kara Cc: Kairui Song Cc: Signed-off-by: Andrew Morton --- mm/filemap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 58eb9d240643..d721986d5f46 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -931,6 +931,12 @@ noinline int __filemap_add_folio(struct address_space *mapping, if (!xas_nomem(&xas, gfp)) break; + + /* + * Lock has been dropped: start again with the original index + * and order (but now with the memory reserved by xas_nomem()). + */ + xas_set_order(&xas, index, forder); } if (xas_error(&xas)) From 50124648db87fb63d9548b6d09deb6d2c6359dfa Mon Sep 17 00:00:00 2001 From: Brendan Jackman Date: Wed, 29 Jul 2026 12:01:48 +0000 Subject: [PATCH 06/25] MAINTAINERS: update address for Brendan Jackman Switch my entry in MAINTAINERS and .mailmap to my @linux.dev email address Link: https://lore.kernel.org/20260729-email-change-v1-1-666ae7c2b7fc@google.com Signed-off-by: Brendan Jackman Signed-off-by: Brendan Jackman Acked-by: Mike Rapoport (Microsoft) Acked-by: Lorenzo Stoakes (ARM) Acked-by: Zi Yan Cc: David Hildenbrand Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- .mailmap | 1 + MAINTAINERS | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index ca6dc2575802..7324e1af59c2 100644 --- a/.mailmap +++ b/.mailmap @@ -170,6 +170,7 @@ Boris Brezillon Boris Brezillon Boris Brezillon Brendan Higgins +Brendan Jackman Brian Avery Brian Cain Brian Cain diff --git a/MAINTAINERS b/MAINTAINERS index 48b2baa1541f..d28e9f59e921 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17160,7 +17160,7 @@ M: Andrew Morton M: Vlastimil Babka R: Suren Baghdasaryan R: Michal Hocko -R: Brendan Jackman +R: Brendan Jackman R: Johannes Weiner R: Zi Yan L: linux-mm@kvack.org From 33192a26cddea7a7e4ca66e5c3eebd36fa8be2bb Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 30 Jul 2026 11:55:47 +0100 Subject: [PATCH 07/25] mm/huge_memory: fix huge_zero_pfn race Patch series "mm/huge_memory: fix huge_zero_pfn race", v2. There is a subtle race in the reference-counted huge_zero_folio implementation. The fast path atomic logic fails to account for the fact that the shrinker (which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a racing get_huge_zero_folio() installed a valid value there. This results in huge_zero_folio being correctly set but huge_zero_pfn being set incorrectly and thus is_huge_zero_pfn() and consequently is_huge_zero_pmd() will misidentify the huge zero folio as being an ordinary THP folio. This can result in the huge zero folio being split and otherwise treated incorrectly. The solution to this is very subtle as there is an atomic fast path, and thus ordering in weakly ordered architectures has to be treated very carefully. The first commit fixes the issue by introducing a spinlock around huge_zero_[pfn, folio, refcount] write, with careful consideration paid to load/store ordering in the fast path. It is placed first and kept as small as possible so that it can be backported on its own. The second commit is a pure cleanup which reworks the CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent logic from the dynamically allocated one. This patch (of 2): If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by huge_zero_refcount and returned by mm_get_huge_zero_folio(). When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero. A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault. This is because shrink_huge_zero_folio_scan() might, if very unlucky, be preempted between setting huge_zero_refcount to zero and writing an invalid value. During this time get_huge_zero_folio() could write to huge_zero_pfn before shrink_huge_zero_folio_scan() resumes. In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio: CPU 0 CPU 1 =======================================|================================= shrink_huge_zero_folio_scan() | atomic_cmpxchg() sets refcount to 0 | xchg() sets huge_zero_folio to NULL | get_huge_zero_folio() | | atomic_inc_not_zero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid huge_zero_folio v | Write valid huge_zero_pfn Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite! This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split. Note that the issue is with huge_zero_pfn not huge_zero_folio, as get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set huge_zero_folio. Fix the issue by introducing a spinlock, huge_zero_lock, to prevent concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount. There needs to be significant care taken here to ensure correctness: The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is outside of the critical section, and means huge zero allocation is gated on zero huge_zero_refcount. The fast path doesn't use huge_zero_lock, so the critical section is irrelevant to it. So invariants are required - huge_zero_refcount MUST: * Only be set in the huge_zero_lock critical section to ensure serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount writes. * Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values so installation of the huge zero folio on read page fault ensures concurrent is_huge_zero_*() calls correctly identify the huge zero folio. * Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL respectively, and atomically. Establish these by: * Only setting huge_zero_refcount to zero or an absolute value in the huge_zero_lock critical section in get_huge_zero_folio() and shrink_huge_zero_folio_scan(), and always updating atomically there and elsewhere. * Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio() after huge_zero_[pfn, folio] are set. This is paired with atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a non-zero value if huge_zero_[pfn, folio] are set. * Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to ensure that it is set zero only when equal to 1 and set atomically. * atomic_cmpxchg() being fully ordered ensures this is done prior to huge_zero_[folio, pfn] being set to NULL and ~0UL respectively. Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg() in shrink_huge_zero_folio_scan() is now performed under the lock, and replace with an equally locked atomic_inc() to set the reference count should the caller be raced on huge zero folio installation. folio_put() naturally implies a full memory barrier so its ordering is maintained correctly. The huge zero folio also cannot be released except when the shrinker does so as it is non-LRU and non-rmappable. Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan()) can actually set huge_zero_refcount to zero, which is the count of mm's which have at least one huge zero folio installed plus one shrinker pin. Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE(). Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-1-c5d8a41b317f@kernel.org Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: Hengbin Zhang Closes: https://lore.kernel.org/linux-mm/20260727154001.4102341-1-uqbarz@gmail.com/ Suggested-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Hannes Reinecke Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 49 +++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 58cabe6af33d..f43852ef9944 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include "internal.h" @@ -78,6 +79,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, static bool split_underused_thp = true; static atomic_t huge_zero_refcount; +static DEFINE_SPINLOCK(huge_zero_lock); struct folio *huge_zero_folio __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; unsigned long huge_anon_orders_always __read_mostly; @@ -224,7 +226,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, static bool get_huge_zero_folio(void) { struct folio *zero_folio; -retry: + + /* Paired with atomic_set_release(). */ if (likely(atomic_inc_not_zero(&huge_zero_refcount))) return true; @@ -237,17 +240,22 @@ static bool get_huge_zero_folio(void) } /* Ensure zero folio won't have large_rmappable flag set. */ folio_clear_large_rmappable(zero_folio); - preempt_disable(); - if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) { - preempt_enable(); - folio_put(zero_folio); - goto retry; - } - WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); - /* We take additional reference here. It will be put back by shrinker */ - atomic_set(&huge_zero_refcount, 2); - preempt_enable(); + /* Paired with critical section in shrink_huge_zero_folio_scan(). */ + spin_lock(&huge_zero_lock); + if (huge_zero_folio) { + /* Somebody else already installed it. */ + atomic_inc(&huge_zero_refcount); + spin_unlock(&huge_zero_lock); + folio_put(zero_folio); + return true; + } + WRITE_ONCE(huge_zero_folio, zero_folio); + WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); + /* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */ + atomic_set_release(&huge_zero_refcount, 2); + spin_unlock(&huge_zero_lock); + count_vm_event(THP_ZERO_PAGE_ALLOC); return true; } @@ -297,15 +305,22 @@ static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, struct shrink_control *sc) { - if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { - struct folio *zero_folio = xchg(&huge_zero_folio, NULL); - BUG_ON(zero_folio == NULL); + struct folio *zero_folio; + + /* Paired with critical section in get_huge_zero_folio(). */ + scoped_guard(spinlock, &huge_zero_lock) { + /* Paired with atomic_inc_not_zero() in get_huge_zero_folio(). */ + if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) != 1) + return 0; + + zero_folio = huge_zero_folio; + VM_WARN_ON_ONCE(!zero_folio); + WRITE_ONCE(huge_zero_folio, NULL); WRITE_ONCE(huge_zero_pfn, ~0UL); - folio_put(zero_folio); - return HPAGE_PMD_NR; } - return 0; + folio_put(zero_folio); + return HPAGE_PMD_NR; } static struct shrinker *huge_zero_folio_shrinker; From 98bd3af0bb6bfdb0fb39cbfa05456acb374b691e Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 30 Jul 2026 11:55:48 +0100 Subject: [PATCH 08/25] mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic Rather than mixing the refcounted and non-refcounted CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so it is clear what happens when this configuration option is set and what happens when it is not. Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only introduce the refcount, lock and shrinker if !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown, abstract the huge zero folio allocation from refcounting. Also change a BUG_ON() to WARN_ON_ONCE() while we're at it. No functional change intended. Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand (Arm) Cc: Dev Jain Cc: Hannes Reinecke Cc: Hengbin Zhang Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 160 ++++++++++++++++++++++++++++------------------- 1 file changed, 94 insertions(+), 66 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f43852ef9944..f84fe7913d88 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -78,10 +78,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc); static bool split_underused_thp = true; +#define HUGE_ZERO_UNSET_PFN (~0UL) +struct folio *huge_zero_folio __read_mostly; +unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN; +#ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO static atomic_t huge_zero_refcount; static DEFINE_SPINLOCK(huge_zero_lock); -struct folio *huge_zero_folio __read_mostly; -unsigned long huge_zero_pfn __read_mostly = ~0UL; +static struct shrinker *huge_zero_folio_shrinker; +#endif + unsigned long huge_anon_orders_always __read_mostly; unsigned long huge_anon_orders_madvise __read_mostly; unsigned long huge_anon_orders_inherit __read_mostly; @@ -223,6 +228,47 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, return orders; } +static struct folio *alloc_huge_zero_folio(void) +{ + struct folio *zero_folio; + + zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) & + ~__GFP_MOVABLE, + HPAGE_PMD_ORDER); + if (!zero_folio) { + count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED); + return NULL; + } + folio_clear_large_rmappable(zero_folio); /* Explicitly not rmappable. */ + return zero_folio; +} + +#ifdef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO +static int __init huge_zero_init(void) +{ + huge_zero_folio = alloc_huge_zero_folio(); + if (!huge_zero_folio) { + pr_warn("Allocating persistent huge zero folio failed\n"); + } else { + huge_zero_pfn = folio_pfn(huge_zero_folio); + count_vm_event(THP_ZERO_PAGE_ALLOC); + } + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + return huge_zero_folio; +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ +} +#else static bool get_huge_zero_folio(void) { struct folio *zero_folio; @@ -231,15 +277,9 @@ static bool get_huge_zero_folio(void) if (likely(atomic_inc_not_zero(&huge_zero_refcount))) return true; - zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) & - ~__GFP_MOVABLE, - HPAGE_PMD_ORDER); - if (!zero_folio) { - count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED); + zero_folio = alloc_huge_zero_folio(); + if (unlikely(!zero_folio)) return false; - } - /* Ensure zero folio won't have large_rmappable flag set. */ - folio_clear_large_rmappable(zero_folio); /* Paired with critical section in shrink_huge_zero_folio_scan(). */ spin_lock(&huge_zero_lock); @@ -266,33 +306,7 @@ static void put_huge_zero_folio(void) * Counter should never go to zero here. Only shrinker can put * last reference. */ - BUG_ON(atomic_dec_and_test(&huge_zero_refcount)); -} - -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return huge_zero_folio; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - return READ_ONCE(huge_zero_folio); - - if (!get_huge_zero_folio()) - return NULL; - - if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); - - return READ_ONCE(huge_zero_folio); -} - -void mm_put_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); + WARN_ON_ONCE(atomic_dec_and_test(&huge_zero_refcount)); } static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, @@ -316,14 +330,53 @@ static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, zero_folio = huge_zero_folio; VM_WARN_ON_ONCE(!zero_folio); WRITE_ONCE(huge_zero_folio, NULL); - WRITE_ONCE(huge_zero_pfn, ~0UL); + WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN); } folio_put(zero_folio); return HPAGE_PMD_NR; } -static struct shrinker *huge_zero_folio_shrinker; +static int __init huge_zero_init(void) +{ + huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); + if (!huge_zero_folio_shrinker) { + shrinker_free(deferred_split_shrinker); + list_lru_destroy(&deferred_split_lru); + return -ENOMEM; + } + + huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; + huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; + shrinker_register(huge_zero_folio_shrinker); + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ + shrinker_free(huge_zero_folio_shrinker); +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + return READ_ONCE(huge_zero_folio); + + if (!get_huge_zero_folio()) + return NULL; + + if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); + + return READ_ONCE(huge_zero_folio); +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); +} +#endif /* CONFIG_PERSISTENT_HUGE_ZERO_FOLIO */ #ifdef CONFIG_SYSFS static ssize_t enabled_show(struct kobject *kobj, @@ -987,39 +1040,14 @@ static int __init thp_shrinker_init(void) deferred_split_shrinker->scan_objects = deferred_split_scan; shrinker_register(deferred_split_shrinker); - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) { - /* - * Bump the reference of the huge_zero_folio and do not - * initialize the shrinker. - * - * huge_zero_folio will always be NULL on failure. We assume - * that get_huge_zero_folio() will most likely not fail as - * thp_shrinker_init() is invoked early on during boot. - */ - if (!get_huge_zero_folio()) - pr_warn("Allocating persistent huge zero folio failed\n"); - return 0; - } - - huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); - if (!huge_zero_folio_shrinker) { - shrinker_free(deferred_split_shrinker); - list_lru_destroy(&deferred_split_lru); - return -ENOMEM; - } - - huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; - huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; - shrinker_register(huge_zero_folio_shrinker); - - return 0; + return huge_zero_init(); } static void __init thp_shrinker_exit(void) { - shrinker_free(huge_zero_folio_shrinker); shrinker_free(deferred_split_shrinker); list_lru_destroy(&deferred_split_lru); + huge_zero_shrinker_exit(); } static int __init hugepage_init(void) From 4194140a51201e76c02c666ba83c07a81cefb6cc Mon Sep 17 00:00:00 2001 From: Danila Tikhonov Date: Sat, 1 Aug 2026 18:33:33 +0300 Subject: [PATCH 09/25] mailmap: map old addresses to Danila Tikhonov Map my old jiaxyga.com and ProtonMail addresses to the current mainlining.org address. Link: https://lore.kernel.org/20260801153333.296218-1-danila@mainlining.org Signed-off-by: Danila Tikhonov Cc: Jakub Kacinski Signed-off-by: Andrew Morton --- .mailmap | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.mailmap b/.mailmap index 7324e1af59c2..086f81a5bc1c 100644 --- a/.mailmap +++ b/.mailmap @@ -233,6 +233,8 @@ Daniel Lezcano Daniel Lezcano Daniel Thompson Daniele Alessandrelli +Danila Tikhonov +Danila Tikhonov Danilo Krummrich David Brownell David Collins From a16fd3ad9d89b05475864da97327870464611736 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 3 Aug 2026 06:40:32 -0700 Subject: [PATCH 10/25] samples/damon/mtier: error out for zero quota goal target values Patch series "mm/damon: avoid division by zero from damos_quota_score()". DAMON_SAMPLE_MTIER and DAMON_LRU_SORT allow the user to trigger division by zero in damos_quota_score(). Avoid it by adding parameters validation checks. This patch (of 2): damos_quota_score() can trigger division by zero if the target_value is zero. DAMON_SAMPLE_MTIER lets users set the target_value via node0_mem_{used,free}_bp parameters. It doesn't guard zero value case, though. As a result, users can trigger division by zero. Fix the issue by returning an error when the user tries to start DAMON with zero node0_mem_{used,free}_bp parameter values. DAMON_SAMPLE_MTIER is just a sample module, but the consequence is quite bad. Also the zero node0_mem_free_bp parameter might look like a reasonable setup to some users. Hence, the issue might really happen in the real world. One reliable way to reproduce the issue is like below: # cd /sys/module/damon_sample_mtier/parameters # echo 4096 > node0_start_addr # echo 8192 > node0_end_addr # echo 8192 > node1_start_addr # echo 81920 > node1_end_addr # echo 0 > node0_mem_free_bp # echo Y > enabled # dmesg -w [...] [18792.235916] Oops: divide error: 0000 [#1] SMP NOPTI [...] [18792.242787] RIP: 0010:damos_quota_score+0x6f/0x480 [...] This issue was discovered [1] by Sashiko. Link: https://lore.kernel.org/20260803134034.15217-1-sj@kernel.org Link: https://lore.kernel.org/20260803134034.15217-2-sj@kernel.org Link: https://lore.kernel.org/20260801202657.117135-1-sj@kernel.org [1] Fixes: c5e67d40a102 ("samples/damon/mtier: add parameters for node0 memory usage") Signed-off-by: SJ Park Cc: # 6.17.x Signed-off-by: Andrew Morton --- samples/damon/mtier.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/samples/damon/mtier.c b/samples/damon/mtier.c index 3785b0c7ffb1..dae6929f827c 100644 --- a/samples/damon/mtier.c +++ b/samples/damon/mtier.c @@ -156,6 +156,9 @@ static struct damon_ctx *damon_sample_mtier_build_ctx(bool promote) if (!scheme) goto free_out; damon_set_schemes(ctx, &scheme, 1); + /* zero target value causes division by zero in damos_quota_store() */ + if (!node0_mem_used_bp || !node0_mem_free_bp) + goto free_out; quota_goal = damos_new_quota_goal( promote ? DAMOS_QUOTA_NODE_MEM_USED_BP : DAMOS_QUOTA_NODE_MEM_FREE_BP, From 06befa61c427e74319781e6f35a364cfc32dbae8 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 3 Aug 2026 06:40:33 -0700 Subject: [PATCH 11/25] mm/damon/lru_sort: error out for >10000 active_mem_bp damos_quota_score() can trigger division by zero if the target value is zero. DAMON_LRU_SORT lets users set the target value for the hot memory scheme via active_mem_bp parameter. It avoids setting it as the target value if the parameter value is zero. However, it also sets the cold memory scheme with a target value that is calculated as '10000 - active_mem_bp + 2'. Hence, if a user sets active_mem_bp 10002, the cold memory scheme's quota goal target value can be zero. As a result, division by zero can be triggered. Fix by returning an error when the user tries to start DAMON with >10000 active_mem_bp parameter value. It makes no sense to set active_mem_bp with 10002. It also requires module parameters write permission to reproduce the issue. That said, the consequence is quite bad. One reliable way to reproduce the issue is like below: # cd /sys/module/damon_lru_sort/parameters # echo 1000 > wmarks_high # echo 995 > wmarks_mid # echo 0 > wmarks_low # echo 10002 > active_mem_bp # echo Y > enabled # dmesg -w [...] [ 597.421247] Oops: divide error: 0000 [#1] SMP NOPTI [ 597.428848] RIP: 0010:damos_quota_score+0x6f/0x480 This issue was discovered [1] by Sashiko. Link: https://lore.kernel.org/20260803134034.15217-3-sj@kernel.org Link: https://lore.kernel.org/20260801213028.5127-1-sj@kernel.org [1] Fixes: 40d98d31cd70 ("mm/damon/lru_sort: support active:inactive memory ratio based auto-tuning") Signed-off-by: SJ Park Cc: # 7.0.x Signed-off-by: Andrew Morton --- mm/damon/lru_sort.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c index 32f41491b726..120b3fe3fc9d 100644 --- a/mm/damon/lru_sort.c +++ b/mm/damon/lru_sort.c @@ -233,6 +233,8 @@ static int damon_lru_sort_add_quota_goals(struct damos *hot_scheme, if (!active_mem_bp) return 0; + if (10000 < active_mem_bp) + return -EINVAL; goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, active_mem_bp); if (!goal) return -ENOMEM; From b9b6bad94c62cbccb9e0ad34635c49fc5f9c52cb Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 3 Aug 2026 06:46:44 -0700 Subject: [PATCH 12/25] mm/damon/reclaim: skip damon_call() if ctx has not started Patch series "mm/damon/{reclaim,lru_sort}: fix commit_inputs infinite hang". Writing 'Y' to commit_inputs parameters of DAMON_RECLAIM and DAMON_LRU_SORT before the modules were ever turned on causes infinite hang. Fix those. The issue was discovered [1] by Sashiko. This patch (of 2): DAMON_RECLAIM calls damon_call() for commit_inputs parameter user input if the DAMON context is initialized. The context could be initialized, but not yet successfully started. In the case, damon_call() could indefinitely hang. Read the comment on damon_call() for more detail. Fix the problem by memorizing if the DAMON context has ever successfully started, and skip damon_call() if it has not. This issue can easily be reproduced by writing Y to commit_inputs on a system that DAMON_RECLAIM was not turned on before. Link: https://lore.kernel.org/20260803134646.16640-1-sj@kernel.org Link: https://lore.kernel.org/20260803134646.16640-2-sj@kernel.org Link: https://lore.kernel.org/20260802173021.762-1-sj@kernel.org [1] Fixes: de3c60e1c831 ("mm/damon: add synchronous commit for commit_inputs") Signed-off-by: SJ Park Cc: Liew Rui Yan Cc: # 7.2.x Signed-off-by: Andrew Morton --- mm/damon/reclaim.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/damon/reclaim.c b/mm/damon/reclaim.c index 11b70d0a9a6f..a1a65b1270c8 100644 --- a/mm/damon/reclaim.c +++ b/mm/damon/reclaim.c @@ -276,6 +276,8 @@ static int damon_reclaim_commit_inputs_fn(void *arg) return damon_reclaim_apply_parameters(); } +static bool damon_reclaim_damon_has_started; + static int damon_reclaim_commit_inputs_store(const char *val, const struct kernel_param *kp) { @@ -296,11 +298,8 @@ static int damon_reclaim_commit_inputs_store(const char *val, if (!commit_inputs_request) return 0; - /* - * Skip damon_call() if ctx is not initialized to avoid - * NULL pointer dereference. - */ - if (!ctx) + /* Skip damon_call() if ctx has not successfully started. */ + if (!damon_reclaim_damon_has_started) return -EINVAL; err = damon_call(ctx, &control); @@ -347,6 +346,8 @@ static int damon_reclaim_turn(bool on) err = damon_start(&ctx, 1, true); if (err) return err; + if (!damon_reclaim_damon_has_started) + damon_reclaim_damon_has_started = true; return damon_call(ctx, &call_control); } From 0f1868310347f99f1b80d5c6a613ddc747288355 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Mon, 3 Aug 2026 06:46:45 -0700 Subject: [PATCH 13/25] mm/damon/lru_sort: skip damon_call() if ctx has not started DAMON_LRU_SORT calls damon_call() for commit_inputs parameter user input if the DAMON context is initialized. The context could be initialized, but not yet successfully started. In the case, damon_call() could indefinitely hang. Read the comment on damon_call() for more detail. Fix the problem by memorizing if the DAMON context has ever successfully started, and skip damon_call() if it has not. This issue can easily be reproduced by writing Y to the commit_inputs parameter file on a system that DAMON_LRU_SORT has not turned on before. Link: https://lore.kernel.org/20260803134646.16640-3-sj@kernel.org Fixes: de3c60e1c831 ("mm/damon: add synchronous commit for commit_inputs") Signed-off-by: SJ Park Cc: Liew Rui Yan Cc: # 7.2.x Signed-off-by: Andrew Morton --- mm/damon/lru_sort.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/damon/lru_sort.c b/mm/damon/lru_sort.c index 120b3fe3fc9d..729d94b8d77e 100644 --- a/mm/damon/lru_sort.c +++ b/mm/damon/lru_sort.c @@ -351,6 +351,8 @@ static int damon_lru_sort_commit_inputs_fn(void *arg) return damon_lru_sort_apply_parameters(); } +static bool damon_lru_sort_damon_has_started; + static int damon_lru_sort_commit_inputs_store(const char *val, const struct kernel_param *kp) { @@ -371,11 +373,8 @@ static int damon_lru_sort_commit_inputs_store(const char *val, if (!commit_inputs_request) return 0; - /* - * Skip damon_call() if ctx is not initialized to avoid - * NULL pointer dereference. - */ - if (!ctx) + /* Skip damon_call() if ctx has not successfully started. */ + if (!damon_lru_sort_damon_has_started) return -EINVAL; err = damon_call(ctx, &control); @@ -426,6 +425,8 @@ static int damon_lru_sort_turn(bool on) err = damon_start(&ctx, 1, true); if (err) return err; + if (!damon_lru_sort_damon_has_started) + damon_lru_sort_damon_has_started = true; return damon_call(ctx, &call_control); } From 8db4bab826ccc9ec10fa41736a48031cd338d392 Mon Sep 17 00:00:00 2001 From: Zhiling Zou Date: Thu, 23 Jul 2026 00:48:13 +0800 Subject: [PATCH 14/25] mm/page_table_check: skip special zero mappings page_table_check_set() and page_table_check_clear() account mappings based on PageAnon(). Shared zero-page PTEs and huge zero PMDs are special mappings, but page_table_check can still account them as file-backed pages. An unprivileged process can populate enough zero mappings to overflow file_map_count and hit the existing BUG_ON(). The PTE path can do this with the shared zero page, and the PMD path can do the same with huge zero mappings. Skip special zero mappings in the user page-table accounting paths. Keep the PTE-side pte_special() check, and identify huge zero PMDs from the mapped folio instead of pmd_special(). That covers architectures where pmd_special() is a no-op without adding huge_zero_pfn checks to the generic counter helpers. Link: https://lore.kernel.org/cover.1784717203.git.zhilinz@nebusec.ai Link: https://lore.kernel.org/e94478e4fb7912fb7e8ebebed5ce85d00dc9a69d.1784717203.git.zhilinz@nebusec.ai Fixes: df4e817b7108 ("mm: page table check") Signed-off-by: Zhiling Zou Signed-off-by: Ren Wei Reported-by: Vega Cc: Pasha Tatashin Assisted-by: Codex:gpt-5.4 Cc: Signed-off-by: Andrew Morton --- mm/page_table_check.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 53a8997ec043..2403f5a11410 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -151,18 +151,29 @@ void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, if (&init_mm == mm) return; - if (pte_user_accessible_page(mm, addr, pte)) + if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT); } EXPORT_SYMBOL(__page_table_check_pte_clear); +static inline bool page_table_check_huge_zero_pmd(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + if (!pfn_valid(pfn)) + return false; + + return is_huge_zero_folio(page_folio(pfn_to_page(pfn))); +} + void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, pmd_t pmd) { if (&init_mm == mm) return; - if (pmd_user_accessible_page(mm, addr, pmd)) + if (pmd_user_accessible_page(mm, addr, pmd) && + !page_table_check_huge_zero_pmd(pmd)) page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT); } EXPORT_SYMBOL(__page_table_check_pmd_clear); @@ -208,7 +219,7 @@ void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, for (i = 0; i < nr; i++) __page_table_check_pte_clear(mm, addr + PAGE_SIZE * i, ptep_get(ptep + i)); - if (pte_user_accessible_page(mm, addr, pte)) + if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) page_table_check_set(pte_pfn(pte), nr, pte_write(pte)); } EXPORT_SYMBOL(__page_table_check_ptes_set); @@ -238,7 +249,8 @@ void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr, for (i = 0; i < nr; i++) __page_table_check_pmd_clear(mm, addr + PMD_SIZE * i, *(pmdp + i)); - if (pmd_user_accessible_page(mm, addr, pmd)) + if (pmd_user_accessible_page(mm, addr, pmd) && + !page_table_check_huge_zero_pmd(pmd)) page_table_check_set(pmd_pfn(pmd), stride * nr, pmd_write(pmd)); } EXPORT_SYMBOL(__page_table_check_pmds_set); From aca1f2d5de17e138bc6c4859126b77e516b82541 Mon Sep 17 00:00:00 2001 From: Matt Fleming Date: Sat, 25 Jul 2026 11:14:19 +0100 Subject: [PATCH 15/25] mm/huge_memory: initialise workingset state before folio split xas_try_split() adds __GFP_ACCOUNT for page-cache xa_nodes, but __folio_split() leaves the xa_state's xa_lru unset. That lets a live, memcg-charged xa_node exist without being linked into the mapping's shadow_nodes list_lru; when reclaim later walks the list_lru it trips VM_WARN_ON(!css_is_dying()). Use mapping_set_update() to install both the workingset update callback and the shadow_nodes list_lru on the xa_state. Link: https://lore.kernel.org/20260725101419.3938406-1-matt@readmodwrite.com Fixes: 58729c04cf10 ("mm/huge_memory: add buddy allocator like (non-uniform) folio_split()") Signed-off-by: Matt Fleming Reported-by: syzbot+c5b060ce82921a2fd500@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=c5b060ce82921a2fd500 Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dave Chinner Cc: Dev Jain Cc: Kairui Song Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Muchun Song Cc: Nico Pache Cc: Roman Gushchin Cc: Ryan Roberts Cc: Shakeel Butt Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f84fe7913d88..5d94fa4c74fd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4076,7 +4076,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, gfp_t gfp; mapping = folio->mapping; - min_order = mapping_min_folio_order(folio->mapping); + min_order = mapping_min_folio_order(mapping); if (new_order < min_order) { ret = -EINVAL; goto out; @@ -4090,6 +4090,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } + mapping_set_update(&xas, mapping); + if (split_type == SPLIT_TYPE_UNIFORM) { xas_set_order(&xas, folio->index, new_order); xas_split_alloc(&xas, folio, old_order, gfp); From 5deb65c34e682e7c5f5df417a70e223e8fcc5f5a Mon Sep 17 00:00:00 2001 From: liyouhong Date: Sun, 26 Jul 2026 09:48:15 +0800 Subject: [PATCH 16/25] mm/damon/ops-common: putback folios on invalid migrate nid damon_pa_migrate() and damos_va_migrate() isolate folios into a local list and then call damon_migrate_pages(). When target_nid is invalid (including the scheme default NUMA_NO_NODE / -1), damon_migrate_pages() returns early without putting the folios back to the LRU. Callers then discard the list head while those folios remain isolated with an extra reference taken by folio_isolate_lru(). The pages stay off the LRU for as long as the mapping exists (anon active+inactive counts drop while RSS does not), and the leftover references can pin the pages after the mapping is gone. Put the folios back on the invalid-nid path so ignored migration requests still return them to the LRU. Link: https://lore.kernel.org/20260726014815.1280757-1-dayou5941@163.com Fixes: 7e6c3130690a ("mm/damon/ops-common: ignore migration request to invalid nodes") Assisted-by: Cursor:grok-4.5 Reviewed-by: SJ Park Signed-off-by: liyouhong Cc: Signed-off-by: Andrew Morton --- mm/damon/ops-common.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 6bdd1cfd3863..9c178c175832 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -392,8 +392,15 @@ unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid) return nr_migrated; if (target_nid < 0 || target_nid >= MAX_NUMNODES || - !node_state(target_nid, N_MEMORY)) + !node_state(target_nid, N_MEMORY)) { + while (!list_empty(folio_list)) { + struct folio *folio = lru_to_folio(folio_list); + + list_del(&folio->lru); + folio_putback_lru(folio); + } return nr_migrated; + } noreclaim_flag = memalloc_noreclaim_save(); From 1ec0e6b6f7321feb769f50d2f094a0aa6c2eda63 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 28 Jul 2026 07:04:03 -0700 Subject: [PATCH 17/25] mm/damon: adjust isolated pages stat for DAMOS_MIGRATE_{HOT,COLD} Callers of migrate_pages() should adjust NR_MIGRATED_{ANON,FILE} for isolations and putback of the folios. That for migration succeeded folios is done by migrate_pages(), in migrate_folio_done(). That for MR_DEMOTION reason is an exception though. DAMOS_MIGRATE_{HOT,COLD} call migrate_pages() but mistakenly not doing the stat adjustment. As a result, use of DAMOS_MIGRATE_{HOT,COLD} could corrupt the stat. It could confuse too_many_isolated(), make compaction and reclaim to behave in unexpected ways. The stat corruption can be reproduced and confirmed using DAMON user-space tool [1] on NUMA systems, like below. $ numactl --hardware available: 2 nodes (0-1) [...] $ sudo ./damo start --damos_action migrate_hot 1 $ sudo cat /proc/sys/vm/stat_refresh $ sudo dmesg [...] [ 80.215554] vmstat_refresh: nr_isolated_anon -5578 [ 80.216842] vmstat_refresh: nr_isolated_file -34400 This issue was discovered [2] by Sashiko. Link: https://lore.kernel.org/20260728140404.94476-1-sj@kernel.org Link: https://github.com/damonitor/damo [1] Link: https://lore.kernel.org/20260726164356.87940-1-sj@kernel.org [2] Fixes: b51820ebea65 ("mm/damon/paddr: introduce DAMOS_MIGRATE_COLD action for demotion") Signed-off-by: SJ Park Cc: Honggyu Kim Cc: Hyeongtak Ji Cc: # 6.11.x Signed-off-by: Andrew Morton --- mm/damon/ops-common.c | 4 ++++ mm/damon/paddr.c | 2 ++ mm/damon/vaddr.c | 3 ++- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 9c178c175832..0bcad6b1e5b9 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -375,6 +375,8 @@ static unsigned int damon_migrate_folio_list(struct list_head *folio_list, while (!list_empty(folio_list)) { folio = lru_to_folio(folio_list); list_del(&folio->lru); + node_stat_sub_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); folio_putback_lru(folio); } @@ -397,6 +399,8 @@ unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid) struct folio *folio = lru_to_folio(folio_list); list_del(&folio->lru); + node_stat_sub_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); folio_putback_lru(folio); } return nr_migrated; diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 5c2da45f988c..f7613ce279a2 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -350,6 +350,8 @@ static unsigned long damon_pa_migrate(struct damon_region *r, if (!folio_isolate_lru(folio)) goto put_folio; + node_stat_add_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); list_add(&folio->lru, &folio_list); put_folio: addr += folio_size(folio); diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index e73ec1ce016e..2c1c1952c008 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -649,7 +649,8 @@ static void damos_va_migrate_dests_add(struct folio *folio, isolate: if (!folio_isolate_lru(folio)) return; - + node_stat_add_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); list_add(&folio->lru, &migration_lists[i]); } From bf4ade7dbd76d4ec8697840e4ebb15ed77c5ec26 Mon Sep 17 00:00:00 2001 From: Shakeel Butt Date: Fri, 7 Aug 2026 07:24:05 -0700 Subject: [PATCH 18/25] memcg: keep folio's objcg same as its node memcg_reparent_objcgs() has an inherent assumption that a folio's objcg is the objcg of the folio's node. Folio migration across nodes breaks that assumption: the new folio simply inherits the old folio's objcg while living on a different node. Once the assumption is broken, the reparenting of the folio's objcg and the reparenting of the folio's LRU list are no longer atomic. memcg_reparent_objcgs() handles one node per iteration and drops all the locks in between, so the objcg gets reparented in the iteration for the objcg's node while the LRU list gets spliced in the iteration for the folio's node. Any LRU operation on that folio in between resolves its lruvec through the objcg, and thus takes the lru_lock of the wrong memcg, not the lru_lock of the list the folio is actually on. Fix this by selecting the objcg by folio_nid() at charge time, and by re-deriving it for the destination node in mem_cgroup_migrate() and mem_cgroup_replace_folio(). Link: https://lore.kernel.org/20260807142406.443516-1-shakeel.butt@linux.dev Fixes: f1cf8d2f36dc ("mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios") Signed-off-by: Johannes Weiner Signed-off-by: Shakeel Butt Reported-by: Karl Erik Hofseth Closes: https://lore.kernel.org/all/anMmd1ADrDVwMO6v@work/ Co-developed-by: Johannes Weiner Acked-by: Muchun Song Acked-by: Qi Zheng Cc: Michal Hocko Cc: Roman Gushchin Cc: Signed-off-by: Andrew Morton --- mm/memcontrol.c | 100 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 83 insertions(+), 17 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6dc4888a90f3..0fea772d3d78 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2907,10 +2907,9 @@ struct mem_cgroup *mem_cgroup_from_virt(void *p) return folio_memcg_check(virt_to_folio(p)); } -static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg, + int nid) { - int nid = numa_node_id(); - for (; memcg; memcg = parent_mem_cgroup(memcg)) { struct obj_cgroup *objcg = rcu_dereference(memcg->nodeinfo[nid]->objcg); @@ -2921,12 +2920,13 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return NULL; } -static inline struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +static inline struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg, + int nid) { struct obj_cgroup *objcg; rcu_read_lock(); - objcg = __get_obj_cgroup_from_memcg(memcg); + objcg = __get_obj_cgroup_from_memcg(memcg, nid); rcu_read_unlock(); return objcg; @@ -2970,7 +2970,7 @@ static struct obj_cgroup *current_objcg_update(void) rcu_read_lock(); memcg = mem_cgroup_from_task(current); - objcg = __get_obj_cgroup_from_memcg(memcg); + objcg = __get_obj_cgroup_from_memcg(memcg, numa_node_id()); rcu_read_unlock(); /* @@ -5120,7 +5120,7 @@ static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, int ret = 0; struct obj_cgroup *objcg; - objcg = get_obj_cgroup_from_memcg(memcg); + objcg = get_obj_cgroup_from_memcg(memcg, folio_nid(folio)); /* Do not account at the root objcg level. */ if (!obj_cgroup_is_root(objcg)) ret = try_charge_memcg(memcg, gfp, folio_nr_pages(folio)); @@ -5319,6 +5319,46 @@ void __mem_cgroup_uncharge_folios(struct folio_batch *folios) uncharge_batch(&ug); } +/* + * An LRU folio must hold the objcg belonging to its own node. + * + * memcg_reparent_objcgs() reparents a dying cgroup one node at a time: the + * folios on that node's LRU lists move to the parent and that node's objcg is + * redirected to the parent, atomically under the node's lru_lock. + * folio_lruvec_lock() relies on this to provide a stable folio<->lruvec + * binding. If a folio holds another node's objcg, its list membership and its + * lruvec resolution change in separate lock sections, and an LRU operation in + * between can re-add the folio to, and strand it on, the LRU list of a dead + * memcg. + * + * So when migration transfers the memcg state to a folio on another node, + * re-derive the objcg for the destination node. If the memcg is dying and the + * destination node has already been reparented, the lookup walks up to the + * nearest live ancestor - which is also where that node's LRU lists went. + * + * Returns the objcg to commit to @new, with a reference for the caller. + */ +static struct obj_cgroup *get_migration_objcg(struct folio *old, + struct folio *new) +{ + struct obj_cgroup *old_objcg, *new_objcg; + int new_nid = folio_nid(new); + + old_objcg = get_obj_cgroup_from_folio(old); + + if (folio_nid(old) == new_nid) + return old_objcg; + + rcu_read_lock(); + new_objcg = __get_obj_cgroup_from_memcg(obj_cgroup_memcg(old_objcg), + new_nid); + rcu_read_unlock(); + + obj_cgroup_put(old_objcg); + + return new_objcg; +} + /** * mem_cgroup_replace_folio - Charge a folio's replacement. * @old: Currently circulating folio. @@ -5347,21 +5387,28 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new) if (folio_memcg_charged(new)) return; - objcg = folio_objcg(old); - VM_WARN_ON_ONCE_FOLIO(!objcg, old); - if (!objcg) + VM_WARN_ON_ONCE_FOLIO(!folio_objcg(old), old); + if (!folio_objcg(old)) return; + objcg = get_migration_objcg(old, new); + rcu_read_lock(); memcg = obj_cgroup_memcg(objcg); - /* Force-charge the new page. The old one will be freed soon */ + + /* + * Force-charge the new page. The old one will be freed soon. + * + * The rootness of the committed objcg decides whether the final + * uncharge of @new goes through the page counters (see + * uncharge_folio()); charge them only if the uncharge will. + */ if (!obj_cgroup_is_root(objcg)) { page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); } - obj_cgroup_get(objcg); commit_charge(new, objcg); memcg1_commit_charge(new, memcg); rcu_read_unlock(); @@ -5373,14 +5420,15 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new) * @new: Replacement folio. * * Transfer the memcg data from the old folio to the new folio for migration. - * The old folio's data info will be cleared. Note that the memory counters - * will remain unchanged throughout the process. + * The old folio's data info will be cleared. The memory counters remain + * unchanged, unless the charge moves out of a fully reparented ancestry + * and has to be settled (see below). * * Both folios must be locked, @new->mapping must be set up. */ void mem_cgroup_migrate(struct folio *old, struct folio *new) { - struct obj_cgroup *objcg; + struct obj_cgroup *objcg, *new_objcg; VM_BUG_ON_FOLIO(!folio_test_locked(old), old); VM_BUG_ON_FOLIO(!folio_test_locked(new), new); @@ -5401,12 +5449,30 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) if (!objcg) return; - /* Transfer the charge and the objcg ref */ - commit_charge(new, objcg); + new_objcg = get_migration_objcg(old, new); + + /* + * @old was charged through a non-root objcg, so its charge is in the + * page counters. If the re-derivation walked up to the root objcg - + * @old's entire ancestry is dying and already reparented - the final + * uncharge of @new will skip the page counters (see uncharge_folio()). + * Settle them now: this is @old's eventual uncharge, moved up to the + * point where its charge record ends. + */ + if (obj_cgroup_is_root(new_objcg) && !obj_cgroup_is_root(objcg)) { + rcu_read_lock(); + memcg_uncharge(obj_cgroup_memcg(objcg), folio_nr_pages(old)); + rcu_read_unlock(); + } + + commit_charge(new, new_objcg); /* Warning should never happen, so don't worry about refcount non-0 */ WARN_ON_ONCE(folio_unqueue_deferred_split(old)); old->memcg_data = 0; + + /* @new holds its own reference now, drop @old's */ + obj_cgroup_put(objcg); } DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); From 3366ddf30edabf285b43685ed67c0a78282b3b65 Mon Sep 17 00:00:00 2001 From: Phillip Lougher Date: Thu, 6 Aug 2026 19:19:15 +0100 Subject: [PATCH 19/25] MAINTAINERS: remove git URL for Squashfs The squashfs-next.git URL hasn't been updated for many years, and it now doesn't exist. So remove it from the MAINTAINERS entry. Link: https://lore.kernel.org/20260806181916.617881-1-phillip@squashfs.org.uk Signed-off-by: Phillip Lougher Cc: Derek Barbosa Signed-off-by: Andrew Morton --- MAINTAINERS | 1 - 1 file changed, 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index d28e9f59e921..7afd6b353eaf 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -25553,7 +25553,6 @@ M: Phillip Lougher L: squashfs-devel@lists.sourceforge.net (subscribers-only) S: Maintained W: http://squashfs.org.uk -T: git git://git.kernel.org/pub/scm/linux/kernel/git/pkl/squashfs-next.git F: Documentation/filesystems/squashfs.rst F: fs/squashfs/ From bd24ce159b0e3212b2080f3a72ba4d89a57f7482 Mon Sep 17 00:00:00 2001 From: JP Kobryn Date: Thu, 6 Aug 2026 18:02:26 -0700 Subject: [PATCH 20/25] MAINTAINERS, mailmap: update email address for JP Kobryn Switch to my linux.dev address and add previous one to mailmap. Link: https://lore.kernel.org/20260807010226.8995-1-jp.kobryn@linux.dev Signed-off-by: JP Kobryn Acked-by: Shakeel Butt Cc: Roman Gushchin Signed-off-by: Andrew Morton --- .mailmap | 1 + MAINTAINERS | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index 086f81a5bc1c..103f92d4798c 100644 --- a/.mailmap +++ b/.mailmap @@ -459,6 +459,7 @@ Jorge Ramirez-Ortiz Josh Poimboeuf Jouni Malinen +JP Kobryn Juha Yrjola Juha Yrjola Juha Yrjola diff --git a/MAINTAINERS b/MAINTAINERS index 7afd6b353eaf..651d91bcb1a6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4982,7 +4982,7 @@ F: tools/lib/bpf/ BPF [MEMORY MANAGEMENT EXTENSIONS] M: Roman Gushchin -M: JP Kobryn +M: JP Kobryn M: Shakeel Butt L: bpf@vger.kernel.org L: linux-mm@kvack.org From 2100f04bc635ba0b328e6e27e5fb1742e421f620 Mon Sep 17 00:00:00 2001 From: Guodong Xu Date: Fri, 7 Aug 2026 18:12:34 -0400 Subject: [PATCH 21/25] mailmap: add entries for Guodong Xu Map my old Linaro and RISCstar email addresses to my current personal address. Neither former address receives mail anymore. Link: https://lore.kernel.org/20260807-b4-mailmap-guodong-xu-v2-1-f7c71bc6bd9f@gmail.com Signed-off-by: Guodong Xu Signed-off-by: Andrew Morton --- .mailmap | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.mailmap b/.mailmap index 103f92d4798c..61cdb662e6e8 100644 --- a/.mailmap +++ b/.mailmap @@ -325,6 +325,8 @@ Gokul Sriram Palanisamy Govindaraj Saminathan Guo Ren Guo Ren +Guodong Xu +Guodong Xu Guru Das Srinagesh Guru Das Srinagesh Guru Das Srinagesh From 25f52e81216884a7444bf07a606691feb09a94e3 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 10 Aug 2026 02:57:36 -0700 Subject: [PATCH 22/25] mm/vmscan: report RCU-tasks quiescent states in shrink_lruvec() I am seeing some rcu_tasks stalls in the Meta fleet during reclaim. INFO: rcu_tasks detected stalls on tasks: 0000000088620d09: .. nvcsw: 6735/6735 holdout: 1 idle_cpu: -1/8 task:GlobalCPUThread state:R running task pid:2552016 tgid:2524552 Call Trace: shrink_lruvec mem_cgroup_iter shrink_node do_try_to_free_pages try_to_free_pages __alloc_frozen_pages_noprof alloc_pages_noprof pte_alloc_one __pte_alloc handle_mm_fault Nothing promises direct reclaim returns in bounded time, and the scan loop in shrink_lruvec() only calls cond_resched(), which is a no-op on PREEMPTION kernels. Involuntary preemption is not a Tasks-RCU quiescent state, so the reclaiming task never reports one and becomes a holdout. Upgrade it to cond_resched_tasks_rcu_qs(), which reports a quiescent state even when cond_resched() does nothing. PS: This has been discussed in [1] Link: https://lore.kernel.org/20260810-rcu_task_shrink_lruvec-v1-1-4d9f7d5251cb@debian.org Link: https://lore.kernel.org/all/amdWVTs0WKOxguxP@gmail.com/ [1] Signed-off-by: Breno Leitao Reviewed-by: Paul E. McKenney Acked-by: Johannes Weiner Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: Barry Song Cc: David Hildenbrand Cc: Kairui Song Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Wei Xu Cc: Yuanchu Xie Cc: Signed-off-by: Andrew Morton --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 56708d1d2dfd..aa5c8096582d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5927,7 +5927,7 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) } } - cond_resched(); + cond_resched_tasks_rcu_qs(); if (nr_reclaimed < nr_to_reclaim || proportional_reclaim) continue; From 92192e9c5ee07efc657d3654bc264081fb0aa01b Mon Sep 17 00:00:00 2001 From: Guopeng Zhang Date: Tue, 11 Aug 2026 11:08:43 +0800 Subject: [PATCH 23/25] mm: memcg-v1: fix memsw and TCP failcnt accounting Commit 0e2759afcaf9 ("page_counter: track failcnt only for legacy cgroups") made failcnt accounting conditional on track_failcnt. It enabled the flag for memcg->memory, but not for memcg->memsw or memcg->tcpmem. Consequently, memory.memsw.failcnt remains zero when the memory+swap limit is hit. memory.kmem.tcp.limit_in_bytes still sets memcg->tcpmem.max, but TCP charge failures are not reflected in memory.kmem.tcp.failcnt. Enable failcnt accounting for both v1 counters. To reproduce memory.memsw.failcnt: CG=/sys/fs/cgroup/memory/memsw-test LIMIT=33554432 mkdir "$CG" echo "$LIMIT" > "$CG/memory.limit_in_bytes" echo "$LIMIT" > "$CG/memory.memsw.limit_in_bytes" Start a child process in the cgroup and make it allocate and touch 96 MiB of memory, causing a memcg OOM. cat "$CG/memory.memsw.failcnt" Without the patch, memory.memsw.failcnt is 0. With the patch, memory.memsw.failcnt is greater than 0. To reproduce memory.kmem.tcp.failcnt: CG=/sys/fs/cgroup/memory/tcpmem-test LIMIT=65536 mkdir "$CG" echo "$LIMIT" > "$CG/memory.kmem.tcp.limit_in_bytes" Start a child process in the cgroup, create a TCP socket, and reserve 1 MiB of socket memory with SO_RESERVE_MEM. The reservation fails with ENOMEM. cat "$CG/memory.kmem.tcp.failcnt" Without the patch, memory.kmem.tcp.failcnt is 0. With the patch, memory.kmem.tcp.failcnt is greater than 0. Link: https://lore.kernel.org/20260811030843.109104-1-guopeng.zhang@linux.dev Closes: https://sashiko.dev/#/patchset/20260810074247.52747-1-guopeng.zhang@linux.dev?part=1 Fixes: 0e2759afcaf9 ("page_counter: track failcnt only for legacy cgroups") Signed-off-by: Guopeng Zhang Acked-by: Johannes Weiner Acked-by: Michal Hocko Reviewed-by: Tao Cui Acked-by: Shakeel Butt Cc: Muchun Song Cc: Roman Gushchin Cc: Signed-off-by: Andrew Morton --- mm/memcontrol.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0fea772d3d78..065e48f7aa74 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4180,9 +4180,11 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) page_counter_init(&memcg->swap, &parent->swap, false); #ifdef CONFIG_MEMCG_V1 memcg->memory.track_failcnt = !memcg_on_dfl; + memcg->memsw.track_failcnt = !memcg_on_dfl; WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); page_counter_init(&memcg->kmem, &parent->kmem, false); page_counter_init(&memcg->tcpmem, &parent->tcpmem, false); + memcg->tcpmem.track_failcnt = !memcg_on_dfl; #endif } else { init_memcg_stats(); From 10d9012e83efedde8718ceaa5053f836e0c8596c Mon Sep 17 00:00:00 2001 From: Youngjun Park Date: Tue, 11 Aug 2026 22:22:06 +0900 Subject: [PATCH 24/25] mm, swap: don't free a hibernation slot that is in the swap cache A slot with a folio in the swap cache is freed when the folio leaves the cache, not when its count drops. swap_put_entries_cluster() follows that rule. swap_free_hibernation_slot() does not, it calls __swap_cluster_free_entries() whether or not a folio sits on the slot. Cluster readahead can put one there. It walks a raw page_cluster sized window of offsets around the faulting entry, and a hibernation slot passes __swap_cache_add_check() because it is not a folio and its count is not zero. Freeing the slot then clears the entry under that folio. The folio is now unreachable from the swap table, and the offset goes back to the allocator. The folio is still on the LRU though, so reclaim can pick it up later. It then takes the old offset out of folio->swap and overwrites the table entry there, which by then may belong to someone else. This bug can trigger silent memory corruption, process crashes, or data instability across completely unrelated userspace applications - typically occurring when uswsusp is preparing the hibernation image. I found this while working on giving hibernation slots their own marker in the swap table, which I had discussed with Kairui. (https://lore.kernel.org/linux-mm/abp7aDgYLrxF3Me8@KASONG-MC4/) As far as I know there are no reports, so there is no Reported-by/Closes to add. Check for a cached folio before freeing. The slot is then left in the ordinary state where only the swap cache holds it, and it is freed when the folio leaves the cache, either through the reclaim below or through normal reclaim later. Link: https://lore.kernel.org/20260811132209.2862708-2-youngjun.park@lge.com Fixes: 0d6af9bcf383 ("mm, swap: use the swap table to track the swap count") Signed-off-by: Youngjun Park Acked-by: Kairui Song Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Kemeng Shi Cc: Nhat Pham Cc: Signed-off-by: Andrew Morton --- mm/swapfile.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 78b49b0658ad..5e0ef8cfb250 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2196,7 +2196,14 @@ void swap_free_hibernation_slot(swp_entry_t entry) ci = swap_cluster_lock(si, offset); __swap_cluster_put_entry(ci, offset % SWAPFILE_CLUSTER); - __swap_cluster_free_entries(si, ci, offset % SWAPFILE_CLUSTER, 1); + /* + * A slot with a folio in the swap cache is freed when the folio + * leaves the cache, the same rule swap_put_entries_cluster() follows. + * Readahead can put a folio here, and freeing the slot now would + * leave that folio with no entry behind it. + */ + if (!swp_tb_is_folio(__swap_table_get(ci, offset % SWAPFILE_CLUSTER))) + __swap_cluster_free_entries(si, ci, offset % SWAPFILE_CLUSTER, 1); swap_cluster_unlock(ci); /* In theory readahead might add it to the swap cache by accident */ From aedf2efd18977e0cef7eb963166e2b1fcc0aa321 Mon Sep 17 00:00:00 2001 From: Hyunwoo Kim Date: Wed, 12 Aug 2026 01:18:57 +0900 Subject: [PATCH 25/25] mm/pagewalk: fix stale walk->action escaping walk_pmd_range() If ->pmd_entry() sets walk->action = ACTION_AGAIN, the pmd_none() check is retried. The PMD entry may be cleared at the point of retry. In this case, if walk->ops->install_pte is not specified, the code continues to the next PMD entry in the range without resetting walk->action to ACTION_SUBTREE. This leaves walk->action erroneously set to ACTION_AGAIN, which is incorrect. This was incorrect but not problematic up until commit 3b89863c3fa4 ("mm/pagewalk: fix race between concurrent split and refault") which updated walk_pud_range() to check for walk->action == ACTION_AGAIN upon walk_pmd_range()'s return, causing the PUD walk to be retried. In this case this results in duplicate walk callbacks being invoked, which is erroneous and will break any caller that is not idempotent with respect to this (and waste time for those which are). The result is an out-of-bounds write, triggered by a local fuzzer: [ 2.272695] ================================================================== [ 2.273471] BUG: KASAN: slab-out-of-bounds in __mincore_unmapped_range+0x14f/0x190 [ 2.274302] Write of size 1 at addr ffff888008d9b000 by task poc/106 [ 2.274966] [ 2.275154] CPU: 0 UID: 1000 PID: 106 Comm: poc Not tainted 7.2.0-rc6-00429-ga7c7074b58d2 #55 PREEMPT(lazy) [ 2.275159] Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 2.275164] Call Trace: [ 2.275170] [ 2.275172] dump_stack_lvl+0x53/0x70 [ 2.275200] print_report+0xd0/0x630 [ 2.275210] ? __pfx__raw_spin_lock_irqsave+0x10/0x10 [ 2.275219] ? irqentry_exit+0xd2/0x670 [ 2.275224] ? irqentry_exit+0xd2/0x670 [ 2.275226] ? __virt_addr_valid+0xef/0x1a0 [ 2.275239] ? __mincore_unmapped_range+0x14f/0x190 [ 2.275242] kasan_report+0xce/0x100 [ 2.275245] ? __mincore_unmapped_range+0x14f/0x190 [ 2.275248] __mincore_unmapped_range+0x14f/0x190 [ 2.275252] mincore_unmapped_range+0x45/0x70 [ 2.275254] walk_pgd_range+0xafc/0xfc0 [ 2.275261] ? __pfx_walk_pgd_range+0x10/0x10 [ 2.275264] ? __update_load_avg_se+0x3d1/0x670 [ 2.275275] __walk_page_range+0xc0/0x310 [ 2.275278] ? __pfx_find_vma+0x10/0x10 [ 2.275281] ? finish_task_switch.isra.0+0x16d/0x4f0 [ 2.275290] walk_page_range_mm_unsafe+0x26f/0x3a0 [ 2.275293] ? __pfx_mtree_load+0x10/0x10 [ 2.275298] ? __pfx_walk_page_range_mm_unsafe+0x10/0x10 [ 2.275302] ? __free_frozen_pages+0x54d/0x7e0 [ 2.275308] __do_sys_mincore+0x132/0x380 [ 2.275311] do_syscall_64+0xf9/0x540 [ 2.275316] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 2.275322] RIP: 0033:0x422ccd [ 2.275326] Code: b3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 [ 2.275329] RSP: 002b:00007fffffffec18 EFLAGS: 00000287 ORIG_RAX: 000000000000001b [ 2.275337] RAX: ffffffffffffffda RBX: 0000000000000066 RCX: 0000000000422ccd [ 2.275339] RDX: 00000000004d0940 RSI: 0000000001000000 RDI: 00007ffff4000000 [ 2.275340] RBP: 00000000004d0940 R08: 0000000000000100 R09: 0000000000000100 [ 2.275342] R10: 0000000000000100 R11: 0000000000000287 R12: 20c49ba5e353f7cf [ 2.275343] R13: 00000000004990d3 R14: 0000000000000000 R15: 0000000000000001 [ 2.275346] [ 2.275347] [ 2.296904] The buggy address belongs to the object at ffff888008d9b000 [ 2.296904] which belongs to the cache sigqueue of size 80 [ 2.298151] The buggy address is located 0 bytes inside of [ 2.298151] allocated 80-byte region [ffff888008d9b000, ffff888008d9b050) [ 2.299408] [ 2.299601] The buggy address belongs to the physical page: [ 2.300191] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x8d9b [ 2.301001] flags: 0x100000000000000(node=0|zone=1) [ 2.301535] page_type: f5(slab) [ 2.301884] raw: 0100000000000000 ffff888107e46780 dead000000000122 0000000000000000 [ 2.302687] raw: 0000000000000000 0000000800240024 00000000f5000000 0000000000000000 [ 2.303489] page dumped because: kasan: bad access detected [ 2.304092] [ 2.304276] Memory state around the buggy address: [ 2.304801] ffff888008d9af00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.305567] ffff888008d9af80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.306340] >ffff888008d9b000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 2.307115] ^ [ 2.307474] ffff888008d9b080: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 2.308237] ffff888008d9b100: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 2.308997] ================================================================== A specific example of this breaking things is mincore which walks an internal cursor data structure a byte at a time on assumption that page table entry callbacks are called only once for each entry. Fix the problem by resetting walk->action to ACTION_SUBTREE prior to the none check. The pattern also exists in walk_pud_range() so fix it there too. This issue was found through AI-based fuzzing. Link: https://lore.kernel.org/20260811161949.3879321-2-imv4bel@gmail.com Fixes: 3b89863c3fa4 ("mm/pagewalk: fix race between concurrent split and refault") Assisted-by: Claude:claude-opus-5 Signed-off-by: Hyunwoo Kim Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Max Boone Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton --- mm/pagewalk.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 5d87c632a255..d3bfece31933 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -126,6 +126,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pmd = pmd_offset(pud, addr); do { again: + walk->action = ACTION_SUBTREE; next = pmd_addr_end(addr, end); if (pmd_none(*pmd)) { if (has_install) @@ -138,8 +139,6 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, continue; } - walk->action = ACTION_SUBTREE; - /* * This implies that each ->pmd_entry() handler * needs to know about pmd_trans_huge() pmds @@ -196,6 +195,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pud = pud_offset(p4d, addr); do { again: + walk->action = ACTION_SUBTREE; next = pud_addr_end(addr, end); if (pud_none(*pud)) { if (has_install) @@ -208,8 +208,6 @@ static int walk_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, continue; } - walk->action = ACTION_SUBTREE; - if (ops->pud_entry) err = ops->pud_entry(pud, addr, next, walk); if (err)