Dev Jain ebcfc63d6b mm: abstract THP allocation ...

Patch series "Do not shatter hugezeropage on wp-fault", v7.

It was observed at [1] and [2] that the current kernel behaviour of
shattering a hugezeropage is inconsistent and suboptimal.  For a VMA with
a THP allowable order, when we write-fault on it, the kernel installs a
PMD-mapped THP.  On the other hand, if we first get a read fault, we get a
PMD pointing to the hugezeropage; subsequent write will trigger a
write-protection fault, shattering the hugezeropage into one writable
page, and all the other PTEs write-protected.  The conclusion being, as
compared to the case of a single write-fault, applications have to suffer
512 extra page faults if they were to use the VMA as such, plus we get the
overhead of khugepaged trying to replace that area with a THP anyway.

Instead, replace the hugezeropage with a THP on wp-fault.

[1]: https://lore.kernel.org/all/3743d7e1-0b79-4eaf-82d5-d1ca29fe347d@arm.com/
[2]: https://lore.kernel.org/all/1cfae0c0-96a2-4308-9c62-f7a640520242@arm.com/


This patch (of 2):

In preparation for the second patch, abstract away the THP allocation
logic present in the create_huge_pmd() path, which corresponds to the
faulting case when no page is present.

There should be no functional change as a result of applying this patch,
except that, as David notes at [1], a PMD-aligned address should be passed
to update_mmu_cache_pmd().

[1]: https://lore.kernel.org/all/ddd3fcd2-48b3-4170-bcaa-2fe66e093f43@redhat.com/

Link: https://lkml.kernel.org/r/20241008061746.285961-1-dev.jain@arm.com
Link: https://lkml.kernel.org/r/20241008061746.285961-2-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Lance Yang <ioworker0@gmail.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

2024-11-06 20:11:11 -08:00

..

damon

mm/damon: fix sparse warning for zero initializer

2024-11-05 16:56:21 -08:00

kasan

mm/vmalloc: combine all TLB flush operations of KASAN shadow virtual address into one operation

2024-11-05 16:56:21 -08:00

kfence

mm: kfence: fix elapsed time for allocated/freed track

2024-09-26 14:01:44 -07:00

kmsan

kmsan: do not pass NULL pointers as 0

2024-07-03 19:30:26 -07:00

backing-dev.c

writeback: support retrieving per group debug writeback stats of bdi

2024-05-05 17:53:51 -07:00

balloon_compaction.c

mm: remove MIGRATE_SYNC_NO_COPY mode

2024-07-03 19:30:00 -07:00

bootmem_info.c

bootmem: use kmemleak_free_part_phys in put_page_bootmem

2023-10-25 16:47:13 -07:00

cma_debug.c

…

cma_sysfs.c

mm/cma: add sysfs file 'release_pages_success'

2024-02-22 10:24:57 -08:00

cma.c

mm/cma: fix useless return in void function

2024-11-05 16:56:30 -08:00

cma.h

mm/cma: add sysfs file 'release_pages_success'

2024-02-22 10:24:57 -08:00

compaction.c

mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()

2024-09-03 21:15:47 -07:00

debug_page_alloc.c

mm: page_alloc: consolidate free page accounting

2024-04-25 20:56:04 -07:00

debug_page_ref.c

…

debug_vm_pgtable.c

mm/debug_vm_pgtable: Use pxdp_get() for accessing page table entries

2024-09-17 01:07:01 -07:00

debug.c

mm: support only one page_type per page

2024-09-03 21:15:43 -07:00

dmapool_test.c

mm/dmapool: add MODULE_DESCRIPTION()

2024-07-03 19:29:58 -07:00

dmapool.c

mm/mempool/dmapool: remove CONFIG_DEBUG_SLAB ifdefs

2023-12-05 11:17:58 +01:00

early_ioremap.c

mm/early_ioremap.c: improve the execution efficiency of early_ioremap_setup()

2023-06-09 16:25:56 -07:00

execmem.c

mm/execmem, arch: convert remaining overrides of module_alloc to execmem

2024-05-14 00:31:43 -07:00

fadvise.c

introduce fd_file(), convert all accessors to it.

2024-08-12 22:00:43 -04:00

fail_page_alloc.c

fault-inject: improve build for CONFIG_FAULT_INJECTION=n

2024-09-01 20:43:33 -07:00

failslab.c

fault-inject: improve build for CONFIG_FAULT_INJECTION=n

2024-09-01 20:43:33 -07:00

filemap.c

mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock()

2024-11-05 16:56:26 -08:00

folio-compat.c

mm: remove putback_lru_page()

2024-09-09 16:38:59 -07:00

gup_test.c

Merge mm-hotfixes-stable into mm-stable to pick up depended-upon changes.

2023-06-23 16:58:19 -07:00

gup_test.h

…

gup.c

gup: convert FOLL_TOUCH case in follow_page_pte() to folio

2024-11-06 20:11:08 -08:00

highmem.c

mm/highmem: make nr_free_highpages() return "unsigned long"

2024-07-03 19:30:06 -07:00

hmm.c

mm: provide mm_struct and address to huge_ptep_get()

2024-07-12 15:52:15 -07:00

huge_memory.c

mm: abstract THP allocation

2024-11-06 20:11:11 -08:00

hugetlb_cgroup.c

mm: memcg: don't call propagate_protected_usage() needlessly

2024-09-01 20:25:50 -07:00

hugetlb_vmemmap.c

mm/hugetlb_vmemmap: don't synchronize_rcu() without HVO

2024-09-01 20:25:45 -07:00

hugetlb_vmemmap.h

mm: hugetlb_vmemmap: fix reference to nonexistent file

2023-10-25 16:47:14 -07:00

hugetlb.c

mm/hugetlb: fix memfd_pin_folios resv_huge_pages leak

2024-09-26 14:01:43 -07:00

hwpoison-inject.c

mm/hwpoison: add MODULE_DESCRIPTION()

2024-07-03 19:29:58 -07:00

init-mm.c

mm: Deprecate pasid field

2023-12-12 10:11:32 +01:00

internal.h

mm: remove PageKsm()

2024-11-06 20:11:08 -08:00

interval_tree.c

…

io-mapping.c

…

ioremap.c

mm: ioremap: remove unneeded ioremap_allowed and iounmap_allowed

2023-08-18 10:12:36 -07:00

Kconfig

resource: remove dependency on SPARSEMEM from GET_FREE_REGION

2024-10-28 21:40:39 -07:00

Kconfig.debug

slub: Introduce CONFIG_SLUB_RCU_DEBUG

2024-08-27 14:12:51 +02:00

khugepaged.c

mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock()

2024-11-05 16:56:27 -08:00

kmemleak.c

mm/kmemleak: use IS_ERR_PCPU() for pointer in the percpu address space

2024-09-03 21:15:38 -07:00

ksm.c

mm: remove PageKsm()

2024-11-06 20:11:08 -08:00

list_lru.c

mm: list_lru: fix UAF for memory cgroup

2024-08-07 18:33:56 -07:00

maccess.c

mm: Fix copy_from_user_nofault().

2023-04-12 17:36:23 -07:00

madvise.c

mm/madvise: unrestrict process_madvise() for current process

2024-11-05 16:56:25 -08:00

Makefile

mm: introduce numa_emulation

2024-09-03 21:15:31 -07:00

mapping_dirty_helpers.c

mm: fix clean_record_shared_mapping_range kernel-doc

2023-08-24 16:20:30 -07:00

memblock.c

Merge tag 'memblock-v6.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock

2024-09-25 11:35:19 -07:00

memcontrol-v1.c

mm/thp: fix deferred split unqueue naming and locking

2024-11-05 16:49:54 -08:00

memcontrol-v1.h

memcg: cleanup with !CONFIG_MEMCG_V1

2024-09-17 01:07:00 -07:00

memcontrol.c

mm/memcontrol: add per-memcg pgpgin/pswpin counter

2024-11-05 16:56:21 -08:00

memfd.c

mm/hugetlb: simplify refs in memfd_alloc_folio

2024-09-26 14:01:44 -07:00

memory_hotplug.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

memory-failure.c

mm: migrate: add isolate_folio_to_list()

2024-09-03 21:15:59 -07:00

memory-tiers.c

memory tiers: use default_dram_perf_ref_source in log message

2024-09-26 14:01:44 -07:00

memory.c

mm/memory.c: remove stray newline at top of file

2024-11-06 20:11:11 -08:00

mempolicy.c

mm/mempolicy: fix comments for better documentation

2024-11-05 16:56:25 -08:00

mempool.c

mm: fix xyz_noprof functions calling profiled functions

2024-06-05 19:19:26 -07:00

memremap.c

mm: convert put_devmap_managed_page_refs() to put_devmap_managed_folio_refs()

2024-05-05 17:53:49 -07:00

memtest.c

memtest: use {READ,WRITE}_ONCE in memory scanning

2024-03-13 12:12:21 -07:00

migrate_device.c

mm: remap unused subpages to shared zeropage when splitting isolated thp

2024-09-09 16:39:03 -07:00

migrate.c

mm: migrate LRU_REFS_MASK bits in folio_migrate_flags

2024-11-05 16:56:29 -08:00

mincore.c

mm: provide mm_struct and address to huge_ptep_get()

2024-07-12 15:52:15 -07:00

mlock.c

Merge tag 'random-6.11-rc1-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/crng/random

2024-07-24 10:29:50 -07:00

mm_init.c

mm: drop CONFIG_HAVE_ARCH_NODEDATA_EXTENSION

2024-09-03 21:15:28 -07:00

mm_slot.h

…

mmap_lock.c

mm: mmap_lock: replace get_memcg_path_buf() with on-stack buffer

2024-07-03 19:30:26 -07:00

mmap.c

mm/mmap: teach generic_get_unmapped_area{_topdown} to handle hugetlb mappings

2024-11-06 20:11:09 -08:00

mmu_gather.c

mm/mmu_gather: improve cond_resched() handling with large folios and expensive page freeing

2024-02-22 15:27:17 -08:00

mmu_notifier.c

mm: move internal core VMA manipulation functions to own file

2024-09-01 20:25:54 -07:00

mmzone.c

mm: improve code consistency with zonelist_* helper functions

2024-09-01 20:25:55 -07:00

mprotect.c

mm: refactor map_deny_write_exec()

2024-11-05 16:49:55 -08:00

mremap.c

mm: mremap: move_ptes() use pte_offset_map_rw_nolock()

2024-11-05 16:56:28 -08:00

mseal.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

msync.c

…

nommu.c

mm: refactor arch_calc_vm_flag_bits() and arm64 MTE handling

2024-11-05 16:49:55 -08:00

numa_emulation.c

mm: introduce numa_emulation

2024-09-03 21:15:31 -07:00

numa_memblks.c

mm: numa_clear_kernel_node_hotplug: Add NUMA_NO_NODE check for node id

2024-10-28 21:40:40 -07:00

numa.c

mm: make range-to-target_node lookup facility a part of numa_memblks

2024-09-03 21:15:32 -07:00

oom_kill.c

mm: move mm flags to mm_types.h

2024-11-05 16:56:26 -08:00

page_alloc.c

mm/thp: fix deferred split unqueue naming and locking

2024-11-05 16:49:54 -08:00

page_counter.c

mm, memcg: cg2 memory{.swap,}.peak write handlers

2024-09-01 20:25:53 -07:00

page_ext.c

mm: don't account memmap per-node

2024-08-15 22:16:14 -07:00

page_idle.c

mm: page_idle: convert page idle to use a folio

2023-01-18 17:12:52 -08:00

page_io.c

mm: swap: make some count_mthp_stat() call-sites be THP-agnostic.

2024-11-06 20:11:09 -08:00

page_isolation.c

mm: remove migration for HugePage in isolate_single_pageblock()

2024-09-03 21:15:40 -07:00

page_owner.c

mm/page-owner: use gfp_nested_mask() instead of open coded masking

2024-05-19 14:40:44 -07:00

page_poison.c

mm/page_poison: replace kmap_atomic() with kmap_local_page()

2023-12-10 16:51:50 -08:00

page_reporting.c

mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER

2024-01-08 15:27:15 -08:00

page_reporting.h

…

page_table_check.c

mm/page_table_check: fix crash on ZONE_DEVICE

2024-06-15 10:43:04 -07:00

page_vma_mapped.c

mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock()

2024-11-05 16:56:28 -08:00

page-writeback.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

pagewalk.c

mm/pagewalk: fix usage of pmd_leaf()/pud_leaf() without present check

2024-10-28 21:40:38 -07:00

percpu-internal.h

mm: remove CONFIG_MEMCG_KMEM

2024-07-10 12:14:54 -07:00

percpu-km.c

…

percpu-stats.c

…

percpu-vm.c

percpu: clean up all mappings when pcpu_map_pages() fails

2024-04-25 20:55:49 -07:00

percpu.c

percpu: fix data race with pcpu_nr_empty_pop_pages

2024-11-06 20:11:11 -08:00

pgalloc-track.h

…

pgtable-generic.c

mm: pgtable: remove pte_offset_map_nolock()

2024-11-05 16:56:29 -08:00

process_vm_access.c

mm: refactor mm_access() to not return NULL

2024-11-05 16:56:23 -08:00

ptdump.c

mm: ptdump: add check_wx_pages debugfs attribute

2024-02-22 10:24:47 -08:00

readahead.c

Merge tag 'pull-stable-struct_fd' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs

2024-09-23 09:35:36 -07:00

rmap.c

mm: multi-gen LRU: use {ptep,pmdp}_clear_young_notify()

2024-11-03 10:47:03 -08:00

rodata_test.c

…

secretmem.c

secretmem: disable memfd_secret() if arch cannot set direct map

2024-10-09 12:47:19 -07:00

shmem_quota.c

shmem_quota: build the object file conditionally to the config option

2024-09-01 20:25:45 -07:00

shmem.c

mm: shmem: fix khugepaged activation policy for shmem

2024-11-05 16:56:20 -08:00

show_mem.c

mm/show_mem.c: report alloc tags in human readable units

2024-09-17 01:07:00 -07:00

shrinker_debug.c

mm: shrinker: use min() to improve shrinker_debugfs_scan_write()

2024-09-03 21:15:40 -07:00

shrinker.c

mm: shrinker: avoid memleak in alloc_shrinker_info

2024-10-31 20:27:04 -07:00

shuffle.c

…

shuffle.h

mm, treewide: rename MAX_ORDER to MAX_PAGE_ORDER

2024-01-08 15:27:15 -08:00

slab_common.c

mm: krealloc: Fix MTE false alarm in __do_krealloc

2024-10-29 10:40:53 +01:00

slab.h

mm, slab: suppress warnings in test_leak_destroy kunit test

2024-10-02 16:28:46 +02:00

slub.c

mm, slab: suppress warnings in test_leak_destroy kunit test

2024-10-02 16:28:46 +02:00

sparse-vmemmap.c

LoongArch: Set initial pte entry with PAGE_GLOBAL for kernel space

2024-10-21 22:11:19 +08:00

sparse.c

Merge tag 'x86-urgent-2024-09-01' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2024-09-01 14:43:08 -07:00

swap_cgroup.c

mm: attempt to batch free swap entries for zap_pte_range()

2024-09-03 21:15:33 -07:00

swap_slots.c

mm: swap: update get_swap_pages() to take folio order

2024-04-25 20:56:37 -07:00

swap_state.c

mm: add nr argument in mem_cgroup_swapin_uncharge_swap() helper to support large folios

2024-09-17 01:07:01 -07:00

swap.c

mm/thp: fix deferred split unqueue naming and locking

2024-11-05 16:49:54 -08:00

swap.h

mm: fix swap_read_folio_zeromap() for large folios with partial zeromap

2024-09-17 01:07:01 -07:00

swapfile.c

mm, swap: avoid over reclaim of full clusters

2024-10-30 20:14:11 -07:00

truncate.c

mm/truncate: reset xa_has_values flag on each iteration

2024-11-06 20:11:09 -08:00

usercopy.c

mm: Fix copy_from_user_nofault().

2023-04-12 17:36:23 -07:00

userfaultfd.c

mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock()

2024-11-05 16:56:28 -08:00

util.c

mm: only enforce minimum stack gap size if it's sensible

2024-09-01 20:26:02 -07:00

vma_internal.h

mm: remove duplicated include in vma_internal.h

2024-09-01 20:26:02 -07:00

vma.c

mm: remove misleading 'unlikely' hint in vms_gather_munmap_vmas()

2024-11-06 20:11:09 -08:00

vma.h

mm: refactor map_deny_write_exec()

2024-11-05 16:49:55 -08:00

vmalloc.c

mm/vmalloc: combine all TLB flush operations of KASAN shadow virtual address into one operation

2024-11-05 16:56:21 -08:00

vmpressure.c

eventfd: simplify eventfd_signal()

2023-11-28 14:08:38 +01:00

vmscan.c

mm: swap: make some count_mthp_stat() call-sites be THP-agnostic.

2024-11-06 20:11:09 -08:00

vmstat.c

mm: split underused THPs

2024-09-09 16:39:04 -07:00

workingset.c

cachestat: do not flush stats in recency check

2024-07-03 22:40:37 -07:00

z3fold.c

mm/z3fold: add __percpu annotation to *unbuddied pointer in struct z3fold_pool

2024-09-01 20:25:56 -07:00

zbud.c

mm: zpool: return pool size in pages

2024-04-25 20:55:48 -07:00

zpool.c

mm: zpool: return pool size in pages

2024-04-25 20:55:48 -07:00

zsmalloc.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

zswap.c

mm: zswap: delete comments for "value" member of 'struct zswap_entry'.

2024-10-09 12:47:19 -07:00