mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-31 08:15:07 -04:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'mm-stable-2026-08-18-18-39' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Browse Source 
Pull MM updates from Andrew Morton:

 - "mm: drop "sub" prefix from various places" (Dev Jain)

   page->folio conversion and a naming cleanup

 - "mm/kasan: remove redundant initialization for kasan_flag_write_only"
   (Igor Putko)

   KASAN cleanup work

 - "mm/filemap: reduce unnecessary xarray lookups" (Chi Zhiling)

   Small speedup in the pagecaache read code

 - "mm/percpu: Fix possible NOFS/NOIO reclaim recursion" (Kaitao Cheng)

   Improve the vmalloc code - mainly the avoidance of GFP_KERNEL
   allocations when the caller asked for GFP_NOFS or GFP_NOIO

 - "mm/kmemleak: avoid soft lockup when scanning task stacks" (Breno
   Leitao)

   Avoid a soft lockup watchdog trigger from the kmemleak scanning code
   in extreme situations

 - "mm/page_owner: misc cleanups" (Ye Liu)

   Cleanups to the page_owner code. For some reason lots of people have
   been working on the page_owner code this cycle.

 - "mm: convert to walk_page_range_vma() to eliminate find_vma()"
   (Kefeng Wang)

   Simplify and accelerate the page walking library function

 - "mm/migrate: preparatory cleanups for batch copy and offload"
   (Shivank Garg)

   Cleanups in the migration code

 - "mm/page_owner: add per-fd filter infrastructure for print_mode and
   NUMA filtering" (Zhen Ni)

   Per-fd filtering to page_owner in order to reduce the sometimes vast
   amount of output it can produce

 - "mm: Refactor bootmem gigantic hugepage allocation" (Muchun Song)

   Fixes and preparatory cleanups around bootmem HugeTLB handling,
   sparse initialization ordering, and related vmemmap setup

 - "mm/zsmalloc: reduce lock contention in zs_free()" (Wenchao Hao)

   Reduce lock contention in zs_free(), which dominates the unmap path
   under memory pressure on Android (LMK kills) and on x86 servers
   running zswap-heavy workloads.

   Up to 1.83x improvement in microbenchmarking.

 - "move alloc_tag.c file under mm/" (Suren Baghdasaryan)

 - "samples/damon: handle damon_{start,stop}() failures" (SJ Park)

   Fix improper handling of damon_start(), damon_stop(), and
   damon_call() failures across DAMON sample modules to prevent
   potential memory leaks, operation disruptions and use-after-free
   bugs

 - "mm/damon/sysfs: kobject_del() directories that users can
   create/remove" (SJ Park)

   Fix delayed sysfs directory removal under DEBUG_KOBJECT_RELEASE
   causeing creation failures due to duplicate directory names by adding
   missing kobject_del() calls before creating new directories

 - "mm: cleanup clear_not_present_full_ptes()" (David Hildenbrand)

   Clean up the core pte handling code

 - "selftests/damon: misc fixes for test bugs" (Kunwu Chan)

   Fix several bugs in the DAMON selftests

 - "selftests/damon: fix memcg_path staging handling" (Cheng Nie)

   Fix a bug in _damon_sysfs.py for damos_filter memcg_path setup, and
   add a test case for it in sysfs.py.

 - "selftests/damon: test kdamond refresh_ms" (Ruslan Valiyev)

   Selftest coverage for DAMON's refresh_ms sysfs feature by updating
   the test control module and verifying that scheme stats update
   automatically without manual intervention

 - "mm/damon: five misc fixups" (Akinobu Mita)

   Miscellaneous DAMON fixups.

 - "mm/damon/core: detect internal variation above max_nr_regions/2"
   (Jiayuan Chen)

   Fix DAMON's region splitting behavior when region counts exceed half
   the maximum budget by dynamically scaling down the split fraction as
   the limit approaches, preventing large regions from staying un-split,
   and add corresponding KUnit test coverage

 - "mm: preparatory patches for PMD level swap entries" (Usama Arif)

   Refactor and clean up PMD softleaf helpers, call sites, and
   architecture flags to lay the groundwork for a follow-up series that
   introduces PMD page table swap entries

 - "mm/damon: update, optimize, and clean up doc, tests, and code" (SJ
   Park)

   Update DAMON design and ABI documentation, expands unit and selftest
   coverage, optimize damon_commit_target_regions(), and clean up
   recently added sysfs interface code for better readability

 - "mm/vmpressure: reduce CPU, memory and code overhead on cgroup v2"
   (Usama Arif)

   Optimize vmpressure() by skipping unnecessary work on cgroup v2 for
   userspace event notifications and refactor v1-only eventfd handling
   into mm/memcontrol-v1.c to reduce memory overhead and code complexity

 - "selftests/mm: refactor pkey helpers and fix mmap error handling"
   (Hongfu Li)

   Refactor pkeys shared tracing and assertion helpers into a common
   file, unify protection key selftests to use consistent diagnostic
   logging and assertions, and enforce standardized MAP_FAILED return
   checks for mmap() calls across the tests

 - "mm/damon: optimize out nr_accesses_bp" (SJ Park)

   Replace the error-prone, continuously updated nr_accesses_bp field in
   damon_region with an on-demand moving sum function, reducing
   structure memory overhead and avoiding state corruption bugs

 - "Open HugeTLB allocation routine for more generic use" (Ackerley Tng)

   Decouple HugeTLB folio allocation from VMA dependencies by
   introducing hugetlb_alloc_folio(), enabling subsystems like
   guest_memfd to allocate HugeTLB folios without standard VMA
   reservations or pseudo-VMAs

 - "mm/damon: provide pseudo moving sum probe_hits" (SJ Park)

   Integrate DAMON's probe_hits attribute counter into the pseudo moving
   sum infrastructure, enabling real-time, online monitoring without
   waiting for full aggregation intervals

 - "mm: Some cleanups for page allocator APIs" (Brendan Jackman)

   Simplify and refactor the page allocator entry points and flags by
   unifying allocation paths, adding internal alloc_flags arguments, and
   eliminating redundant __ prefixed alloc_pages variants.

 - "Fix incorrect access of hugetlb pte entries" (Dev Jain)

   Enforce the consistent use of huge_ptep_get() instead of ptep_get()
   for HugeTLB entries and fixes an unaligned address issue in arm64's
   huge_ptep_get() implementation

 - "mm/damon: validate all parameters in the core" (SJ Park)

   Consolidate parameter validation into the DAMON core specifically
   within damon_start() and damon_commit_ctx() to centralize error
   checking, eliminate caller-side redundant checks and to improve
   maintenance efficiency

 - "tools/mm/page_owner_sort: fix filtering and cleanup issues" (Yichong
   Chen)

   Rename is_need() to filter_record() for clearer return semantics, fix
   per-record allocation memory leaks and bound output copies in
   search_pattern() to address an existing buffer issue

 - "memcg: bail out reclaim when memcg is dying" (Jiayuan Chen)

   Mitigate a system-wide stall which occurs when a cgroup is removed
   while one of its memory control files is doing synchronous reclaim

 - "mm/memory-failure: add panic option for unrecoverable pages" (Breno
   Leitao)

   Introduce an opt-in vm.panic_on_unrecoverable_memory_failure sysctl
   that immediately panics the kernel on unrecoverable memory errors in
   kernel-owned pages to preserve error context and prevent delayed,
   silent data corruption

 - "mm/damon: refactor damon_{start,stop,commit}() for simple error
   handling" (SJ Park)

   Refactor the DAMON core API functions to guarantee that all contexts
   are fully stopped when damon_start(), damon_stop(), or damon_commit()
   fail, eliminating the need for complex and error-prone caller-side
   cleanup code

 - "Keep tail page private zero at free and folio split" (Zi Yan)

   Add checks to ensure tail_page->private is zero when freeing compound
   or high-order pages and when promoting tail pages during large folio
   splits. By validating these fields at free and split time, it allows
   the removal of redundant private field clearing inside
   prep_compound_tail()

 - "mm: drop redundant lru_add_drain in anon folio reuse paths" (Barry
   Song)

   Eliminate redundant lru_add_drain() calls in
   wp_can_reuse_anon_folio() and do_swap_page() to reduce LRU lock
   contention and system overhead

   By validating folio refcounts against the LRU cache before draining
   and removing unnecessary drains in the swap path, it achieves up to a
   30.5% reduction in drain calls during heavy swap workloads

 - "mm: clean up folio LRU and swap declarations" (Jianyue Wu)

   Reorganize folio LRU and swap code by relocating page-cluster state
   to mm/swap_state.c, renaming mm/swap.c to mm/folio.c, and moving
   MM-internal reclaim declarations into mm/internal.h.

 - "userfaultfd: working set tracking for VM guest memory" (Kiryl
   Shutsemau)

   Add userfaultfd support for tracking the working set of VM guest
   memory, so a VMM can identify hot pages and reclaim cold ones to
   tiered or remote storage

 - "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 2)" (David
   Hildenbrand)

   Remove the remaining pieces of CONFIG_HAVE_BOOTMEM_INFO_NODE,
   performing some smaller cleanups around freeing of reserved vmemmap
   pages on the way.

 - "mm/damon: update probe hits for runtime parameter commits" (SJ Park)

   Ensure that DAMON's probe_hits attribute counter is properly updated
   when monitoring intervals are changed at runtime, matching the
   behavior of nr_accesses. To achieve this, it refactors and renames
   existing helper functions for shared use, applies the updates to
   probe_hits, and handles edge cases in damon_probe_hits_mvsum() to
   maintain measurement accuracy.

 - "KSM: performance optimizations for rmap_walk_ksm" (xu xin)

   Resolve a severe KSM reverse-mapping performance bottleneck where
   thousands of split VMAs sharing a single anon_vma cause extended lock
   contention.

   By adding an interval-filtering check during the rmap walk, it
   reduces worst-case anon_vma lock hold times from over 500ms down to
   under 2ms, preventing application freezes and latency spikes under
   memory pressure.

 - "mm: split a couple of headers from internal.h" (Mike Rapoport)

   Split declarations related to mm_init, memblock, vmalloc and sparse
   into new headers

 - "KSM: use linear_page_index in collect_procs_ksm()" (xu xin)

   Apply the interval tree optimization from rmap_walk_ksm() to
   collect_procs_ksm() to avoid iterating over non-matching VMAs during
   KSM memory error handling.

   It hoists loop-invariant address initialization and restricts the
   anon_vma_interval_tree_foreach walk to a targeted page offset range,
   reducing redundant checks and improving lookup efficiency.

 - "selftests/mm: avoid false failures in hugetlb and KSM tests" (Sayali
   Patil)

   Fix issues in the hugetlb and KSM MM selftest categories that can
   report failures when the prerequisites for the tests are not
   satisfied

 - "mm/damon: introduce data attributes only monitoring" (SJ Park)

   Introduce attribute-weighted region management in DAMON, allowing
   users to prioritize specific data attributes (such as page sizes or
   cgroups) over or instead of access monitoring.

   By assigning weights to attribute probes, DAMON can completely
   disable access tracking and adjust monitoring regions based on
   weighted probe-hit counters to optimize monitoring quality for
   attribute-focused workloads.

 - "mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings"
   (Stanislav Kinsburskii)

   Extend hmm_range_fault() to support userfaultfd-backed regions by
   allowing the mmap lock to be dropped during fault handling via a new
   hmm_range_fault_locked() helper.

   By accepting a locked pointer and signaling retry status when lock
   release occurs, it enables page fault resolution in userfaultfd
   regions while preserving backward compatibility for existing callers.

 - "mm: make VMA page offset handling more consistent" (Lorenzo Stoakes)

   Clean up and standardize how vma->vm_pgoff is accessed and
   manipulated across file-backed and anonymous mappings in the kernel

   It introduces dedicated helper functions such as vma_start_pgoff(),
   vma_end_pgoff(), vma_set_pgoff() and linear_page_delta() while
   renaming rmap interval tree helpers to better reflect their
   functionality.

   These changes establish a cleaner foundation for future work that
   will unify virtual page offset indexing for all anonymous and CoW'd
   folios.

 - "mm: handle device-private PMDs in walk callbacks" (Usama Arif)

   Address kernel panics and state corruption caused by MM walk
   callbacks reaching non-present device-private PMD swap entries
   created during HMM migrations

   It ensures that functions which acquire pmd_trans_huge_lock()
   properly recognize device-private PMDs instead of assuming a present
   THP or a standard migration entry.

 - "mm/rmap: Refactor try_to_unmap_one" (Dev Jain)

   Refactor try_to_unmap_one by modularizing Hugetlb,
   anonymous-lazyfree, and anonymous-swapbacked logic into dedicated
   functions, laying the structural groundwork for batched anonymous
   large folio unmapping.

 - "Docs/ABI/damon: sysfs ABI document fixes and additions" (Song Hu)

   Fix typos and fills in missing entries in the DAMON sysfs ABI
   document

 - "dax/kmem: atomic whole-device hotplug via sysfs" (Gregory Price)

   Introduce an atomic sysfs state attribute and supporting DAX/MM
   infrastructure to prevent userland races when offlining and removing
   entire memory regions

   By adding an unplugged state alongside standard online modes, it
   enables whole-device atomic hotplug control while preserving backward
   compatibility.

 - "mm: convert more vm_flags_t users to vma_flags_t" (Lorenzo Stoakes)

   Continue transitioning the kernel from the deprecated vm_flags_t type
   to vma_flags_t across core memory management infrastructure.

   It replaces legacy type usage in core functions such as do_mmap(),
   unmapped area allocation, mm->def_vma_flags, and VMA operations like
   mlock, mprotect, and mremap.

 - "Two small patches to clean up mm/mm_slot.h" (xu xin)

   Refactor mm_slot.h by introducing mm_slot_remove() to unify duplicate
   slot deletion sequences in khugepaged and KSM. It also adds code
   documentation explaining why mm_slot_lookup and mm_slot_insert must
   remain as preprocessor macros rather than static inline functions.

 - "mm/damon/core: hide core-private struct fields" (SJ Park)

   Clean up DAMON core structures by consistently marking internal-only
   fields with private: comment tags to prevent improper direct access
   from outer layers.

   It enforces encapsulation across core structures including
   damon_region, damon_target, and damon_ctx and updates DAMON_SYSFS to
   interact through approved access APIs instead of exposing raw struct
   members.

 - "mm/damon: unurgent fixes for infinite loop, NULL de-ref and races"
   (SJ Park)

   Address potential infinite loops, NULL dereferences, and race
   conditions identified in DAMON

   It fixes an infinite loop triggered by extreme user configurations, a
   NULL pointer dereference within unit tests and minor monitoring
   accuracy degradation caused by subtle runtime races.

 - "mm/page_alloc: fixes for free_pages_nolock() on RT/UP" (Brendan
   Jackman)

   Fix an NMI safety flaw in __free_frozen_pages() where freeing pages
   on non-SMP or PREEMPT_RT kernels can bypass can_spin_trylock() checks
   via non-PCP or isolated migration paths.

   It also resolves potential kernel crashes and privilege escalation
   risks triggered when BPF tracing runs in NMI context alongside memory
   hotplug or large allocation frees.

 - "mm/page_alloc: couple of followups for recent cleanups" (Brendan
   Jackman)

   Clean up and update page allocator nomenclature, documentation, and
   debug assertions.

   It aligns internal FPI_ flags with the public "nolock" naming
   convention, removes outdated internal implementation details from
   high-level page allocator comments, and eliminates obsolete
   VM_BUG_ON() assertions in allocation paths.

 - "mm/mseal: further cleanups" (Lorenzo Stoakes)

   Refactor and simplify the mseal implementation by clarifying API
   boundaries and removing unnecessary code complexity.

   It replaces generic do_mseal() usage outside the syscall with a
   dedicated mseal_mmap_page_zero() helper for MMAP_PAGE_ZERO,
   eliminates mm_struct parameters to enforce that sealing applies only
   to current->mm, and streamlines overall logic and comments with no
   functional changes intended.

 - "mm/vmscan: fix swappiness=max and clean up per-node proactive
   reclaim" (Ridong Chen)

   Resolve reclaim behavior bugs and clean up function parameters across
   memory reclaim paths

   It fixes swappiness=max in both standard reclaim and MGLRU so
   unswappable anonymous memory no longer falls back to evicting page
   cache, ensures reclaim_store() returns accurate error codes instead
   of collapsing all failures into -EAGAIN, and removes the obsolete
   gfp_mask parameter from __node_reclaim().

 - "mm: mincore: misc cleanups" (Kefeng Wang)

   Clean up and simplifies the mincore code. Most importantly, it
   removes the historical special behavior that always reports VM_PFNMAP
   pages as non-resident.

 - "mm/huge_memory: drop dead split helper variants" (Kiryl Shutsemau)

   Two trivial cleanups in the folio split API

 - "mm/damon: fix uninitialized DAMOS field and kunit exec expectation
   bugs" (SJ Park)

   Resolve minor operational and testing bugs in DAMON identified by
   Sashiko. It initializes the damos->last_applied field to prevent
   occasional efficiency degradation and fixes invalid memory accesses
   in DAMON KUnit tests during test failure handling.

 - "cleanup for stable_page_flags()" (Jinjiang Tu)

   Clean up and refactor stable_page_flags() used by /proc/kpageflags
   without altering functionality.

   It uses BIT_ULL() to prevent shift-overflow warnings on 64-bit flag
   bits, converts folio-specific flag checks to standard folio_test_*()
   helpers, and removes redundant CONFIG_PAGE_IDLE_FLAG handling.

 - "Batch unmap of uffd-wp file folios" (Dev Jain)

   Extend batched folio unmapping support to file folios within
   userfaultfd write-protect (uffd-wp) VMAs by adding batching
   capabilities to pte_install_uffd_wp_if_needed().

   This removes special-case restrictions on uffd-wp VMAs in
   try_to_unmap_one(), significantly simplifying the function's control
   flow and complexity.

 - "mm/early_ioremap: clarify and clean up early_ioremap_reset()"
   (Sang-Heon Jeon)

   Clarify and clean up the architecture-specific usage of
   __late_set_fixmap() and __late_clear_fixmap() after
   early_ioremap_reset()

   It adds explicit documentation regarding when early_ioremap_reset()
   must be called and removes redundant macro definitions and reset
   calls in the RISC-V and ARM64 architectures.

 - "mm: fix reclaim storms in defrag_mode" (Johannes Weiner)

   Address severe performance regressions, swap storms, and spurious
   OOMs caused by vm.defrag_mode=1 under high memory pressure in Meta
   production

   It updates the page allocator slowpath so non-movable allocation
   requests actively trigger direct reclaim and direct compaction at
   pageblock_order scale, allowing them to claim whole pageblocks rather
   than spinning unproductively.

 - "zram: lockmap tweaks" (Sebastian Siewior)

   Optimize and fix lockdep tracking for zram devices by consolidating
   per-entry lockmaps and isolate lock classes across multiple instances

   This reduces memory overhead by replacing per-entry lockdep_map
   instances with a single map per struct zram, and assigns a dynamic
   lock_class_key to each instance to prevent false deadlock reports
   when different zram devices are backed by distinct filesystems.

* tag 'mm-stable-2026-08-18-18-39' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (501 commits)
  selftests/mm: thuge-gen: fix test_shmget() for PAGE_SIZE check
  selftests/mm: unpoison pages in memory-failure teardown
  mm/shmem: downgrade final i_blocks check in shmem_evict_inode() to pr_warn()
  mm/khugepaged: replace mutex_lock/mutex_unlock usage with guard macro
  mm/zsmalloc: fix release order of locks in zs_page_migrate()
  Documentation: zram: remove sections numbering
  ksm: stop iterating VMAs when ksm_test_exit returns true
  mm: fold userfaultfd_rwp() to false without CONFIG_ARCH_HAS_PTE_PROTNONE
  mm/migrate: report RCU-tasks quiescent states in migrate_pages_batch()
  zram: use a custom key for each zram object
  zram: move lockmap to be per-zram instead per table
  selftests/mm: fix gup_longterm EINVAL error message
  mm: page_alloc: fix non-movable reclaim storm in defrag_mode
  mm: page_alloc: move capture_control to the page allocator
  mm: compaction: support non-movable compaction for pageblock requests
  mm: page_alloc: __GFP_FS lockdep annotation for direct compaction
  hugetlb: evaluate subpool free state while locked
  mm/damon: remove trailing semicolons after function definitions
  mm/damon/ops-common: prevent migration fallback to non-target nodes
  mm/damon: update outdated comment about DAMOS filter handling
  ...
This commit is contained in:
Linus Torvalds
2026-08-20 18:17:08 -07:00
parent 7f063b2f17 08745c6235
commit ed3b875bea
339 changed files with 12508 additions and 6365 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
CREDITS
View File

@@ -3104,6 +3104,10 @@ N: Jens Osterkamp
E: jens@de.ibm.com
D: Maintainer of Spidernet network driver for Cell
N: Kent Overstreet
E: kent.overstreet@linux.dev
D: Co-authored and contributed to Memory Allocation Profiling
N: Gadi Oxman
E: gadio@netvision.net.il
D: Original author and maintainer of IDE/ATAPI floppy/tape drivers

6
Documentation/ABI/testing/sysfs-block-zram
View File

@@ -152,15 +152,15 @@ Description:
compression algorithm parameters.
What: /sys/block/zram<id>/compressed_writeback
Date: Decemeber 2025
Date: December 2025
Contact: Richard Chang <richardycc@google.com>
Description:
The compressed_writeback device atrribute toggles compressed
The compressed_writeback device attribute toggles compressed
writeback feature.
What: /sys/block/zram<id>/writeback_batch_size
Date: November 2025
Contact: Sergey Senozhatsky <senozhatsky@chromium.org>
Description:
The writeback_batch_size device atrribute sets the maximum
The writeback_batch_size device attribute sets the maximum
number of in-flight writeback operations.

24
Documentation/ABI/testing/sysfs-bus-dax
View File

@@ -151,3 +151,27 @@ Description:
memmap_on_memory parameter for memory_hotplug. This is
typically set on the kernel command line -
memory_hotplug.memmap_on_memory set to 'true' or 'force'."
What: /sys/bus/dax/devices/daxX.Y/state
Contact: nvdimm@lists.linux.dev
Description:
(RW) Controls the state of the memory region.
Applies to all memory blocks associated with the device.
Only applies to dax_kmem devices.
Reading returns the current state; the writable states mirror
the per-block /sys/devices/system/memory/memoryX/state ABI::
"unplugged": memory blocks are not present
"online": memory is online, zone chosen by the kernel
"online_kernel": memory is online in ZONE_NORMAL
"online_movable": memory is online in ZONE_MOVABLE
"offline" (memory blocks are present but offline) may also be
reported - this happens when the device is bound while the
auto_online_blocks policy is "offline". It cannot be written,
as it's not useful and creates device destruction races.
A device can only be onlined from the "unplugged" state, so a
device must be returned to "unplugged" before it can be onlined
into a different state.

75
Documentation/ABI/testing/sysfs-kernel-mm-damon
View File

@@ -36,7 +36,10 @@ Description: Writing 'on' or 'off' to this file makes the kdamond starts or
kdamond. Writing 'clear_schemes_tried_regions' to the file
removes contents of the 'tried_regions' directory. Writing
'update_schemes_effective_quotas' to the file updates
'.../quotas/effective_bytes' files of this kdamond.
'.../quotas/effective_bytes' files of this kdamond. Writing
'update_tuned_intervals' to the file updates 'sample_us' and
'aggr_us' files under 'monitoring_attrs/intervals' with the
auto-tuned sampling and aggregation intervals.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/pid
Date: Mar 2022
@@ -112,7 +115,7 @@ Description: Writing a value to this file sets the update interval of the
DAMON context in microseconds as the value. Reading this file
returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/access_bp
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/access_bp
Date: Feb 2025
Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the monitoring intervals
@@ -120,7 +123,7 @@ Description: Writing a value to this file sets the monitoring intervals
the given time interval (aggrs in same directory), in bp
(1/10,000). Reading this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/aggrs
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/aggrs
Date: Feb 2025
Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the time interval to achieve
@@ -128,14 +131,14 @@ Description: Writing a value to this file sets the time interval to achieve
access events ratio (access_bp in same directory) within.
Reading this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/min_sample_us
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/min_sample_us
Date: Feb 2025
Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the minimum value of
auto-tuned sampling interval in microseconds. Reading this
file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intrvals_goal/max_sample_us
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/intervals_goal/max_sample_us
Date: Feb 2025
Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the maximum value of
@@ -144,7 +147,7 @@ Description: Writing a value to this file sets the maximum value of
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/nr_regions/min
WDate: Mar 2022
Date: Mar 2022
Contact: SJ Park <sj@kernel.org>
Description: Writing a value to this file sets the minimum number of
monitoring regions of the DAMON context as the value. Reading
@@ -157,6 +160,52 @@ Description: Writing a value to this file sets the maximum number of
monitoring regions of the DAMON context as the value. Reading
this file returns the value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/nr_probes
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for each DAMON probe named '0' to 'N-1' under the
probes/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/weight
Date: Jul 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
per-probe attribute weight.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/nr_filters
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for each DAMON probe filter named '0' to 'N-1'
under the filters/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/type
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the type of
the memory of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/path
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: If 'memcg' is written to the 'type' file, writing to and
reading from this file sets and gets the path to the memory
cgroup of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/matching
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether the filter is for
the memory of the 'type', or all except the 'type'.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/probes/<P>/filters/<F>/allow
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether to allow or reject
hitting the probe for the memory that satisfies the 'type' and
the 'matching' of the directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/targets/nr_targets
Date: Mar 2022
Contact: SJ Park <sj@kernel.org>
@@ -241,7 +290,7 @@ Description: Writing to and reading from this file sets and gets the maximum
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/min
Date: Mar 2022
Contact: SJ Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the manimum
Description: Writing to and reading from this file sets and gets the minimum
'nr_accesses' of the scheme's target regions.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/nr_accesses/max
@@ -601,3 +650,15 @@ Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the size of the memory in the region
that passed DAMON operations layer-handled filters of the
scheme in bytes.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/probes/
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Directory for DAMON data-attribute monitoring probes of the
region.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/<R>/probes/<P>/hits
Date: May 2026
Contact: SJ Park <sj@kernel.org>
Description: Reading this file returns the number of data-attribute
monitoring probe-hit positive samples of the region.

40
Documentation/admin-guide/blockdev/zram.rst
View File

@@ -61,8 +61,8 @@ and, in general case, something like::
should suffice.
1) Load Module
==============
Load Module
===========
::
@@ -73,8 +73,8 @@ This creates 4 devices: /dev/zram{0,1,2,3}
num_devices parameter is optional and tells zram how many devices should be
pre-created. Default: 1.
2) Select compression algorithm
===============================
Select compression algorithm
============================
Using comp_algorithm device attribute one can see available and
currently selected (shown in square brackets) compression algorithms,
@@ -93,8 +93,8 @@ Examples::
For the time being, the `comp_algorithm` content shows only compression
algorithms that are supported by zram.
3) Set compression algorithm parameters: Optional
=================================================
Set compression algorithm parameters: Optional
==============================================
Compression algorithms may support specific parameters which can be
tweaked for particular dataset. ZRAM has an `algorithm_params` device
@@ -124,8 +124,8 @@ better the compression ratio, it even can take negatives values for some
algorithms), for other algorithms `level` is acceleration level (the higher
the value the lower the compression ratio).
4) Set Disksize
===============
Set Disksize
============
Set disk size by writing the value to sysfs node 'disksize'.
The value can be either in bytes or you can use mem suffixes.
@@ -144,8 +144,8 @@ There is little point creating a zram of greater than twice the size of memory
since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the
size of the disk when not in use so a huge zram is wasteful.
5) Set memory limit: Optional
=============================
Set memory limit: Optional
==========================
Set memory limit by writing the value to sysfs node 'mem_limit'.
The value can be either in bytes or you can use mem suffixes.
@@ -163,8 +163,8 @@ Examples::
# To disable memory limit
echo 0 > /sys/block/zram0/mem_limit
6) Activate
===========
Activate
========
::
@@ -174,8 +174,8 @@ Examples::
mkfs.ext4 /dev/zram1
mount /dev/zram1 /tmp
7) Add/remove zram devices
==========================
Add/remove zram devices
=======================
zram provides a control interface, which enables dynamic (on-demand) device
addition and removal.
@@ -194,8 +194,8 @@ execute::
echo X > /sys/class/zram-control/hot_remove
8) Stats
========
Stats
=====
Per-device statistics are exported as various nodes under /sys/block/zram<id>/
@@ -296,16 +296,16 @@ a single line of text and contains the following stats separated by whitespace:
Unit: 4K bytes
============== =============================================================
9) Deactivate
==============
Deactivate
==========
::
swapoff /dev/zram0
umount /dev/zram1
10) Reset
=========
Reset
=====
Write any positive value to 'reset' sysfs node::

2
Documentation/admin-guide/cgroup-v1/cpusets.rst
View File

@@ -284,7 +284,7 @@ take action.
==>
Unless this feature is enabled by writing "1" to the special file
/dev/cpuset/memory_pressure_enabled, the hook in the rebalance
code of __alloc_pages() for this metric reduces to simply noticing
code of the page allocator for this metric reduces to simply noticing
that the cpuset_memory_pressure_enabled flag is zero. So only
systems that enable this feature will compute the metric.

19
Documentation/admin-guide/mm/damon/usage.rst
View File

@@ -73,9 +73,10 @@ comma (",").
│ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us
│ │ │ │ │ │ nr_regions/min,max
│ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes
│ │ │ │ │ │ │ 0/filters/nr_filters
│ │ │ │ │ │ │ │ 0/type,matching,allow,path
│ │ │ │ │ │ │ │ ...
│ │ │ │ │ │ │ 0/weight
│ │ │ │ │ │ │ │ filters/nr_filters
│ │ │ │ │ │ │ │ │ 0/type,matching,allow,path
│ │ │ │ │ │ │ │ │ ...
│ │ │ │ │ │ │ ...
│ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
│ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target
@@ -246,7 +247,7 @@ writing to and reading from the files.
Under ``nr_regions`` directory, two files for the lower-bound and upper-bound
of DAMON's monitoring regions (``min`` and ``max``, respectively), which
controls the monitoring overhead, exist. You can set and get the values by
writing to and rading from the files.
writing to and reading from the files.
For more details about the intervals and monitoring regions range, please refer
to the Design document (:doc:`/mm/damon/design`).
@@ -264,7 +265,7 @@ Please refer to the :ref:`design document of the feature
<damon_design_monitoring_intervals_autotuning>` for the internal of the tuning
mechanism. Reading and writing the four files under ``intervals_goal``
directory shows and updates the tuning parameters that described in the
:ref:design doc <damon_design_monitoring_intervals_autotuning>` with the same
:ref:`design doc <damon_design_monitoring_intervals_autotuning>` with the same
names. The tuning starts with the user-set ``sample_us`` and ``aggr_us``. The
tuning-applied current values of the two intervals can be read from the
``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to
@@ -286,6 +287,10 @@ In each probe directory, one directory, ``filters`` exists. The directory
contains files for installing filters for the probe, that is used to determine
the data attribute for the probe.
Each probe directory also contains ``weight`` file. Reading from and writing
to the file gets and sets the :ref:`attributes-only monitoring
<damon_design_attrs_only_monitoring>` weight for the attribute of the probe.
In the beginning, ``filters`` directory has only one file, ``nr_filters``.
Writing a number (``N``) to the file creates the number of child directories
named ``0`` to ``N-1``. Each directory represents each filter and works in a
@@ -377,7 +382,7 @@ schemes/<N>/
In each scheme directory, nine directories (``access_pattern``, ``quotas``,
``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``,
``stats``, and ``tried_regions``) and three files (``action``, ``target_nid``
and ``apply_interval``) exist.
and ``apply_interval_us``) exist.
The ``action`` file is for setting and getting the scheme's :ref:`action
<damon_design_damos_action>`. The keywords that can be written to and read
@@ -743,7 +748,7 @@ counter). Finally the tenth field (``X``) shows the ``age`` of the region
(refer to :ref:`design <damon_design_age_tracking>` for more details of the
counter).
If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would
If the event was ``damon:damos_before_apply``, the ``perf script`` output would
be somewhat like below::
kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136

13
Documentation/admin-guide/mm/pagemap.rst
View File

@@ -19,8 +19,11 @@ There are four components to pagemap:
* Bit 55 pte is soft-dirty (see
Documentation/admin-guide/mm/soft-dirty.rst)
* Bit 56 page exclusively mapped (since 4.2)
* Bit 57 pte is uffd-wp write-protected (since 5.13) (see
Documentation/admin-guide/mm/userfaultfd.rst)
* Bit 57 pte is tracked by userfaultfd (since 5.13) — in a
``VM_UFFD_WP`` VMA this indicates a write-protected PTE; in a
``VM_UFFD_RWP`` VMA it indicates an RWP-protected PTE. WP and
RWP are mutually exclusive per VMA, so the meaning is
unambiguous. See Documentation/admin-guide/mm/userfaultfd.rst.
* Bit 58 pte is a guard region (since 6.15) (see madvise (2) man page)
* Bits 59-60 zero
* Bit 61 page is file-page or shared-anon (since 3.5)
@@ -244,7 +247,8 @@ in this IOCTL:
Following flags about pages are currently supported:
- ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled
- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected
- ``PAGE_IS_WRITTEN`` - Page in a ``UFFDIO_REGISTER_MODE_WP`` VMA has been
written to since it was write-protected. Only reported inside such VMAs.
- ``PAGE_IS_FILE`` - Page is file backed
- ``PAGE_IS_PRESENT`` - Page is present in the memory
- ``PAGE_IS_SWAPPED`` - Page is in swapped
@@ -252,6 +256,9 @@ Following flags about pages are currently supported:
- ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed
- ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty
- ``PAGE_IS_GUARD`` - Page is a part of a guard region
- ``PAGE_IS_ACCESSED`` - Page in a ``UFFDIO_REGISTER_MODE_RWP`` VMA has been
accessed since RWP was applied. Only reported inside such VMAs. See
Documentation/admin-guide/mm/userfaultfd.rst for the RWP workflow.
The ``struct pm_scan_arg`` is used as the argument of the IOCTL.

2
Documentation/admin-guide/mm/transhuge.rst
View File

@@ -761,7 +761,7 @@ compact_fail
but failed.
It is possible to establish how long the stalls were using the function
tracer to record how long was spent in __alloc_pages() and
tracer to record how long was spent in the page allocator and
using the mm_page_alloc tracepoint to identify which allocations were
for huge pages.

269
Documentation/admin-guide/mm/userfaultfd.rst
View File

@@ -131,6 +131,16 @@ userfaults on the range registered. Not all ioctls will necessarily be
supported for all memory types (e.g. anonymous memory vs. shmem vs.
hugetlbfs), or all types of intercepted faults.
.. note::
Re-registering an already-registered range must not drop any of the
modes that install per-PTE markers — currently
``UFFDIO_REGISTER_MODE_WP`` and ``UFFDIO_REGISTER_MODE_RWP``. Doing
so would strand markers with no flag to describe them, so the call
is rejected with ``-EBUSY``; userspace must issue
``UFFDIO_UNREGISTER`` first. This differs from older kernels, which
silently replaced the mode bits on re-registration.
Userland can use the ``uffdio_register.ioctls`` to manage the virtual
address space in the background (to add or potentially also remove
memory from the ``userfaultfd`` registered range). This means a userfault
@@ -265,16 +275,16 @@ tracking and it can be different in a few ways:
- Dirty information will not get lost if the pte was zapped due to
various reasons (e.g. during split of a shmem transparent huge page).
- Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
set; dirty when uffd-wp bit cleared), it has different semantics on
some of the memory operations. For example: ``MADV_DONTNEED`` on
- Due to a reverted meaning of soft-dirty (page clean when the uffd bit
is set; dirty when the uffd bit is cleared), it has different semantics
on some of the memory operations. For example: ``MADV_DONTNEED`` on
anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
dirtying of memory by dropping uffd-wp bit during the procedure.
dirtying of memory by dropping the uffd bit during the procedure.
The user app can collect the "written/dirty" status by looking up the
uffd-wp bit for the pages being interested in /proc/pagemap.
uffd bit for the pages being interested in /proc/pagemap.
The page will not be under track of uffd-wp async mode until the page is
The page will not be under track of userfaultfd-wp async mode until the page is
explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault
that was tracked by async mode userfaultfd-wp is invalid.
@@ -297,6 +307,253 @@ transparent to the guest, we want that same address range to act as if it was
still poisoned, even though it's on a new physical host which ostensibly
doesn't have a memory error in the exact same spot.
Read-Write Protection
---------------------
``UFFDIO_REGISTER_MODE_RWP`` enables read-write protection tracking on a
memory range. It is similar to (but faster than) ``mprotect(PROT_NONE)``
combined with a signal handler; unlike ``mprotect(PROT_NONE)``, RWP only
traps accesses to *present* PTEs, so accesses to unpopulated addresses in a
protected range fall through to the normal missing-page path. It uses the
PROT_NONE hinting mechanism (same as NUMA balancing) to make pages
inaccessible while keeping them resident in memory. Works on anonymous,
shmem, and hugetlbfs memory.
RWP is designed for VM memory managers that need to track the working set
of guest memory for cold page eviction to tiered or remote storage.
**Setup:**
1. Open a userfaultfd and enable ``UFFD_FEATURE_RWP`` via ``UFFDIO_API``.
Optionally request ``UFFD_FEATURE_RWP_ASYNC`` as well — it requires
``UFFD_FEATURE_RWP`` to be set in the same ``UFFDIO_API`` call.
2. Register the guest memory range with ``UFFDIO_REGISTER_MODE_RWP``
(and ``UFFDIO_REGISTER_MODE_MISSING`` if evicted pages will need to be
fetched back from storage).
**Feature availability:**
RWP is built on top of two kernel primitives: a spare PTE bit owned by
userfaultfd (``CONFIG_HAVE_ARCH_USERFAULTFD_WP``) and architecture support
for present-but-inaccessible PTEs (``CONFIG_ARCH_HAS_PTE_PROTNONE``). When both
are available on a 64-bit kernel, the build selects
``CONFIG_USERFAULTFD_RWP=y`` and the ``VM_UFFD_RWP`` VMA flag becomes
available.
``UFFD_FEATURE_RWP`` and ``UFFD_FEATURE_RWP_ASYNC`` are unavailable when
the running kernel or architecture does not support them — for example
32-bit kernels (where ``VM_UFFD_RWP`` is unavailable), kernels built
without ``CONFIG_USERFAULTFD_RWP``, and architectures whose ptes cannot
carry the uffd bit at runtime (e.g. riscv without the ``SVRSW60T59B``
extension). Requesting an unsupported feature in
``uffdio_api.features`` makes ``UFFDIO_API`` fail with ``EINVAL`` and
leaves the userfaultfd context uninitialized; the structure is returned
zeroed, so the error path cannot be used to discover what the kernel
supports. The recommended probe sequence is therefore to open a
throwaway userfaultfd, call ``UFFDIO_API`` once with ``features = 0``,
inspect the returned bitmask, close that fd, then open the real one
and call ``UFFDIO_API`` again with only the supported features set.
**Protecting and Unprotecting:**
Use ``UFFDIO_RWPROTECT`` to protect or unprotect a range, mirroring the
``UFFDIO_WRITEPROTECT`` interface::
struct uffdio_rwprotect rwp = {
.range = { .start = addr, .len = len },
.mode = UFFDIO_RWPROTECT_MODE_RWP, /* protect */
};
ioctl(uffd, UFFDIO_RWPROTECT, &rwp);
Setting ``UFFDIO_RWPROTECT_MODE_RWP`` sets PROT_NONE on present PTEs in the
range. Pages stay resident and their physical frames are preserved — only
access permissions are removed.
Clearing ``UFFDIO_RWPROTECT_MODE_RWP`` restores normal VMA permissions and
wakes any faulting threads (unless ``UFFDIO_RWPROTECT_MODE_DONTWAKE`` is set).
**Scope of protection:**
RWP protection is a property of *present* PTEs. ``UFFDIO_RWPROTECT`` only
affects entries that are already populated. Unpopulated addresses within
the range remain unpopulated; when first accessed they fault through the
normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
``finish_fault()``) and the resulting PTE is not RWP-protected. To observe
the population itself, co-register the range with
``UFFDIO_REGISTER_MODE_MISSING``.
Protection is preserved across page reclaim: a page swapped out while
RWP-protected carries the marker on its swap entry, and swap-in restores
the PROT_NONE state so the first access after swap-in still faults. The
same applies to pages temporarily replaced by migration entries.
Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
memory, hole-punch on shmem, truncation of a file mapping — also drop the
RWP marker: the next access re-populates the range without protection.
Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
persistent RWP marker today. The user needs to re-arm the range with
``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
**Fault Handling:**
When a protected page is accessed:
- **Sync mode** (default): The faulting thread blocks and a
``UFFD_PAGEFAULT_FLAG_RWP`` message is delivered to the userfaultfd
handler. The handler resolves the fault with ``UFFDIO_RWPROTECT``
(clearing ``MODE_RWP``), which restores the PTE permissions and wakes
the faulting thread.
- **Async mode** (``UFFD_FEATURE_RWP_ASYNC``): The kernel automatically
restores PTE permissions and the thread continues without blocking. No
message is delivered to the handler.
**Runtime Mode Switching:**
``UFFDIO_SET_MODE`` toggles ``UFFD_FEATURE_RWP_ASYNC`` at runtime, allowing
the VMM to switch between lightweight async detection and safe sync
eviction without re-registering. The toggle takes ``mmap_write_lock()``
and calls ``vma_start_write()`` on each UFFD-armed VMA, draining
in-flight per-VMA-locked faults before the new mode takes effect.
**Working-set detection with PAGEMAP_SCAN:**
RWP-protected PTEs carry the uffd PTE bit; an access (and, in async mode, its
auto-resolution) clears it. ``PAGEMAP_SCAN`` reports ``PAGE_IS_ACCESSED`` once
the bit is clear on a ``VM_UFFD_RWP`` VMA, so a *non-inverted* scan reports the
pages that were touched during the interval -- the hot set::
struct pm_scan_arg arg = {
.size = sizeof(arg),
.start = guest_mem_start,
.end = guest_mem_end,
.vec = (uint64_t)regions,
.vec_len = regions_len,
.category_mask = PAGE_IS_ACCESSED,
.return_mask = PAGE_IS_ACCESSED,
};
long n = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
The returned ``page_region`` array lists the hot ranges. ``PAGE_IS_ACCESSED``
is set on an accessed page whether it is still present or has since been
swapped out, so the hot scan needs no ``PAGE_IS_PRESENT`` filter -- unpopulated
holes carry neither bit and are excluded on their own.
Track the hot set and reclaim everything else from the backing file (see the
workflow below). Do **not** invert the scan to enumerate "cold" pages
directly: an inverted scan reports only the ``VM_UFFD_RWP`` PTEs that are still
protected, i.e. the resident portion of *this* VMA. For a file mapping the
working set spans the whole file -- pages that live in the page cache but are
not mapped into this VMA (a pre-populated tmpfs file, or memory populated
through another mapping) are ``pte_none`` here, never appear in the scan, and
would never be considered for eviction even though they occupy memory. Driving
eviction from "file offsets minus the hot set" avoids that blind spot; a cold
PTE scan cannot. To additionally record the *first* access to a cached but
unmapped page (e.g. pre-populated content) as hot, co-register the range with
``UFFDIO_REGISTER_MODE_MINOR``: such accesses then fault as minor faults
instead of mapping the page silently.
**Cleanup:**
When the userfaultfd is closed or the range is unregistered, all PROT_NONE
PTEs are automatically restored to their normal VMA permissions. This
prevents pages from becoming permanently inaccessible.
**VMM Working Set Tracking Workflow:**
A typical VMM lifecycle for cold page eviction to tiered storage. Two
mappings of the same shmem (or hugetlbfs) file are used: ``guest_mem`` is
the RWP-registered mapping that vCPUs access through, and ``io_mem`` is a
private mapping for VMM-side I/O. Reading ``io_mem`` does not go through
the RWP-protected PTEs of ``guest_mem``, so the VMM's own ``pwrite()``
never traps on its own ::
/* One-time setup */
fd = memfd_create("guest", MFD_CLOEXEC);
ftruncate(fd, guest_size);
guest_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0); /* vCPU view, RWP-registered */
io_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0); /* VMM I/O view, unprotected */
uffd = userfaultfd(O_CLOEXEC | O_NONBLOCK);
struct uffdio_api api = {
.api = UFFD_API,
.features = UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
};
ioctl(uffd, UFFDIO_API, &api);
if (!(api.features & UFFD_FEATURE_RWP))
/* RWP unavailable on this kernel/arch -- fall back. */
ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
.range = { guest_mem, guest_size },
.mode = UFFDIO_REGISTER_MODE_RWP |
UFFDIO_REGISTER_MODE_MISSING,
});
/* Tracking loop */
while (vm_running) {
/* 1. Detection phase (async -- no vCPU stalls) */
ioctl(uffd, UFFDIO_RWPROTECT, &(struct uffdio_rwprotect){
.range = full_range,
.mode = UFFDIO_RWPROTECT_MODE_RWP });
sleep(tracking_interval);
/*
* 2. Switch to sync BEFORE scanning. In async mode a vCPU
* access races eviction: it would auto-resolve and mark the
* page hot just as the VMM writes it out and punches it,
* losing the update. Sync mode makes such accesses block and
* be delivered, freezing the hot snapshot for the rest of the
* iteration.
*/
ioctl(uffd, UFFDIO_SET_MODE,
&(struct uffdio_set_mode){
.disable = UFFD_FEATURE_RWP_ASYNC });
/* 3. Read the hot set: pages touched this interval. */
ioctl(pagemap_fd, PAGEMAP_SCAN, &(struct pm_scan_arg){
.category_mask = PAGE_IS_ACCESSED,
.return_mask = PAGE_IS_ACCESSED,
...
});
/*
* 4. Reclaim the file offsets that are NOT in the hot set.
* Driving this from the file's offset space (rather than from a
* cold PTE scan) also reclaims pages that are cached but not
* mapped into guest_mem, e.g. pre-populated content.
*/
for each non-hot offset range:
/* Read from io_mem -- bypasses RWP, no fault. */
pwrite(storage_fd, (char *)io_mem + off, len, off);
/* Drop the page from the shared file. */
fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
off, len);
/*
* Wake any vCPU blocked on the RWP fault for this range:
* fallocate() does not iterate ctx->fault_pending_wqh.
*/
ioctl(uffd, UFFDIO_WAKE, &(struct uffdio_range){
.start = (uintptr_t)guest_mem + off, .len = len });
/* 5. Resume async tracking */
ioctl(uffd, UFFDIO_SET_MODE,
&(struct uffdio_set_mode){
.enable = UFFD_FEATURE_RWP_ASYNC });
}
During step 4, a vCPU that accesses a ``guest_mem`` offset being evicted
blocks with a ``UFFD_PAGEFAULT_FLAG_RWP`` fault while the eviction is in
progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
fires, the vCPU retries the access, faults as ``MISSING``, and the
handler resolves it with ``UFFDIO_COPY`` from storage.
This workflow targets shmem and hugetlbfs (both support a private
``io_mem`` mapping over the same fd). Anonymous-memory backings need a
different inner-loop strategy because the VMM has no way to read the
page without going through the RWP-protected mapping.
QEMU/KVM
========

83
Documentation/admin-guide/sysctl/vm.rst
View File

@@ -19,9 +19,6 @@ The files in this directory can be used to tune the operation
of the virtual memory (VM) subsystem of the Linux kernel and
the writeout of dirty data to disk.
Default values and initialization routines for most of these
files can be found in mm/swap.c.
Currently, these files are in /proc/sys/vm:
- admin_reserve_kbytes
@@ -67,6 +64,7 @@ Currently, these files are in /proc/sys/vm:
- page-cluster
- page_lock_unfairness
- panic_on_oom
- panic_on_unrecoverable_memory_failure
- percpu_pagelist_high_fraction
- stat_interval
- stat_refresh
@@ -925,6 +923,85 @@ panic_on_oom=2+kdump gives you very strong tool to investigate
why oom happens. You can get snapshot.
panic_on_unrecoverable_memory_failure
======================================
When a hardware memory error (e.g. multi-bit ECC) hits a kernel page
that cannot be recovered by the memory failure handler, the default
behaviour is to ignore the error and continue operation. This is
dangerous because the corrupted data remains accessible to the kernel,
risking silent data corruption or a delayed crash when the poisoned
memory is next accessed.
When enabled, this sysctl triggers a panic on memory failure events
hitting kernel-owned pages that the handler cannot recover:
``PageReserved`` (firmware reservations, kernel image, vDSO, zero
page, and similar memblock-reserved regions), ``PageSlab``,
``PageTable``, and ``PageLargeKmalloc``. These are owned by the
kernel and the memory failure handler cannot reliably evict their
contents.
Other unrecoverable kernel-owned populations (vmalloc allocations,
kernel stack pages, ...) are not currently covered because the
handler has no page-type signal that distinguishes them from a
userspace folio temporarily off the LRU during migration or
compaction. Such pages still go through the standard
MF_MSG_GET_HWPOISON path: ``PG_hwpoison`` is set on them and a
delayed crash on the next access remains possible. Coverage may
grow as the handler gains stronger kernel-ownership signals.
Recoverable failure paths are also intentionally left out: in-flight
buddy allocations and other transient races with the page allocator
can reach the same diagnostic, and panicking on them would risk
killing the box for a page destined for userspace where the standard
SIGBUS recovery path applies. Pages whose state could not be
classified at all are not covered either, since an unknown state is
not a sound basis for a panic decision.
For many environments it is preferable to panic immediately with a clean
crash dump that captures the original error context, rather than to
continue and face a random crash later whose cause is difficult to
diagnose.
Use cases
---------
This option is most useful in environments where unattributed crashes
are expensive to debug or where data integrity must take precedence
over availability:
* Large fleets, where multi-bit ECC errors on kernel pages are observed
regularly and post-mortem analysis of an unrelated downstream crash
(often seconds to minutes after the original error) consumes
significant engineering effort.
* Systems configured with kdump, where panicking at the moment of the
hardware error produces a vmcore that still contains the faulting
address, the affected page state, and the originating MCE/GHES
record — context that is typically lost by the time a delayed crash
occurs.
* High-availability clusters that rely on fast, deterministic node
failure for failover, and prefer an immediate panic over silent data
corruption propagating to replicas or persistent storage.
* Kernel and platform developers reproducing hwpoison issues with
tools such as ``mce-inject`` or error-injection debugfs interfaces,
where panicking on the unrecoverable path makes regressions
immediately visible instead of surfacing as later, unrelated
failures.
= =====================================================================
0 Try to continue operation (default).
1 Panic immediately. If the ``panic`` sysctl is also non-zero then the
machine will be rebooted.
= =====================================================================
Example::
echo 1 > /proc/sys/vm/panic_on_unrecoverable_memory_failure
percpu_pagelist_high_fraction
=============================

3
Documentation/core-api/mm-api.rst
View File

@@ -73,6 +73,7 @@ Readahead
Writeback
---------
.. kernel-doc:: include/linux/writeback.h
.. kernel-doc:: mm/page-writeback.c
:export:
@@ -117,7 +118,7 @@ More Memory Management Functions
.. #kernel-doc:: mm/hmm.c (build warnings)
.. kernel-doc:: mm/memremap.c
.. kernel-doc:: mm/hugetlb.c
.. kernel-doc:: mm/swap.c
.. kernel-doc:: mm/folio.c
.. kernel-doc:: mm/memcontrol.c
.. #kernel-doc:: mm/memory-tiers.c (build warnings)
.. kernel-doc:: mm/shmem.c

3
Documentation/driver-api/basics.rst
View File

@@ -90,6 +90,9 @@ Reference counting
.. kernel-doc:: lib/refcount.c
:export:
.. kernel-doc:: include/linux/percpu-refcount.h
.. kernel-doc:: lib/percpu-refcount.c
Atomics
-------

1
Documentation/filesystems/proc.rst
View File

@@ -608,6 +608,7 @@ encoded manner. The codes are the following:
um userfaultfd missing tracking
uw userfaultfd wr-protect tracking
ui userfaultfd minor fault
ur userfaultfd read-write-protect tracking
ss shadow/guarded control stack page
sl sealed
lf lock on fault pages

43
Documentation/mm/damon/design.rst
View File

@@ -86,7 +86,7 @@ To know how user-space can do the configuration via :ref:`DAMON sysfs interface
documentation.
.. _damon_design_vaddr_target_regions_construction:
.. _damon_design_vaddr_target_regions_construction:
VMA-based Target Address Range Construction
-------------------------------------------
@@ -316,6 +316,31 @@ Another way to do this for higher accuracy is using :ref:`DAMOS filter
information in page level. But, because it is operated in page level, the
overhead is proportional to the size of the memory.
.. _damon_design_attrs_only_monitoring:
Data Attributes-only Monitoring
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Data access is the primary monitoring information for DAMON. Hence it
:ref:`adjusts <damon_design_data_attrs_monitoring>` regions using the access
:ref:`counter <damon_design_region_based_sampling>` (``nr_accesses``). In some
use cases, however, users may want some of :ref:`attributes
<damon_design_data_attrs_monitoring>` to be the primary information.
Data attributes-only monitoring mode supports this use case. For the mode,
each attribute probe has their priority weight value. Users can describe by
what combination of the attributes the primary information is decided, by
setting the priority weight value. If the total sum of the weights is not
zero, the mode is enabled. The regions adjustment mechanism uses the weighted
sum of the :ref:`probe hit counts <damon_design_data_attrs_monitoring>` instead
of ``nr_accesses`` in the case. When the mode is enabled, access monitoring is
automatically turned off. The access counter (``nr_accesses``) will always be
zero and not updated. Hence the mode is called Data Attributes "only"
monitoring.
Refer to the :ref:`admin guide <damon_usage_sysfs_probes>` to know how users
can use the mode.
Dynamic Target Space Updates Handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -686,9 +711,11 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to
(1/10,000).
- ``inactive_mem_bp``: Inactive to active + inactive (LRU) memory size ratio in
bp (1/10,000).
- ``node_eligible_mem_bp``: Scheme target access pattern-eligible memory ratio
of a node in bp (1/10,000).
``nid`` is optionally required for only ``node_mem_used_bp``,
``node_mem_free_bp``, ``node_memcg_used_bp`` and ``node_memcg_free_bp`` to
``nid`` is optionally required for ``node_mem_used_bp``, ``node_mem_free_bp``,
``node_memcg_used_bp``, ``node_memcg_free_bp`` and ``node_eligible_mem_bp`` to
point the specific NUMA node.
``path`` is optionally required for only ``node_memcg_used_bp`` and
@@ -930,11 +957,11 @@ control parameters for the usage would also need to be optimized for the
purpose.
To support such cases, yet more DAMON API user kernel modules that provide more
simple and optimized user space interfaces are available. Currently, two
modules for proactive reclamation and LRU lists manipulation are provided. For
more detail, please read the usage documents for those
(:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim` and
:doc:`/admin-guide/mm/damon/lru_sort`).
simple and optimized user space interfaces are available. Currently, three
modules for access monitoring statistics, proactive reclamation, and LRU lists
manipulation are provided. For more detail, please read the usage documents for
those (:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim`
and :doc:`/admin-guide/mm/damon/lru_sort`).
.. _damon_design_special_purpose_modules_exclusivity:

81
Documentation/mm/hmm.rst
View File

@@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the
update action to the range (mark range read only, or fully unmap, etc.). The
device must complete the update before the driver callback returns.
When the device driver wants to populate a range of virtual addresses, it can
use::
When the device driver wants to populate a range of virtual addresses, the
normal interface is::
int hmm_range_fault(struct hmm_range *range);
int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
unsigned long timeout);
It will trigger a page fault on missing or read-only entries if write access is
requested (see below). Page faults use the generic mm page fault code path just
like a CPU page fault. The usage pattern is::
like a CPU page fault.
The caller must not hold ``mmap_read_lock`` before the call.
``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and
allows ``handle_mm_fault()`` to drop it during fault handling. This is required
for VMAs whose fault handlers may release the mmap lock, for example regions
managed by ``userfaultfd``.
If the mmap lock is dropped or the range is invalidated, the function refreshes
``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned
if mmap lock acquisition is interrupted or a fatal signal is pending during
retry handling.
The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The
timeout exists to preserve caller policy for repeated mmu-notifier invalidation
and is checked between retry attempts. HMM does not interrupt page fault
handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is
exhausted before a stable range is obtained.
The usage pattern is::
int driver_populate_range(...)
{
struct hmm_range range;
unsigned long timeout;
...
timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
range.notifier = &interval_sub;
range.start = ...;
range.end = ...;
range.hmm_pfns = ...;
if (!mmget_not_zero(interval_sub->notifier.mm))
if (!mmget_not_zero(interval_sub.mm))
return -EFAULT;
again:
range.notifier_seq = mmu_interval_read_begin(&interval_sub);
mmap_read_lock(mm);
ret = hmm_range_fault(&range);
if (ret) {
mmap_read_unlock(mm);
if (ret == -EBUSY)
goto again;
return ret;
}
mmap_read_unlock(mm);
ret = hmm_range_fault_unlocked_timeout(&range, timeout);
if (ret)
goto out_put;
take_lock(driver->update);
if (mmu_interval_read_retry(&ni, range.notifier_seq) {
if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
release_lock(driver->update);
goto again;
}
@@ -200,13 +215,31 @@ like a CPU page fault. The usage pattern is::
* under the update lock */
release_lock(driver->update);
return 0;
ret = 0;
out_put:
mmput(interval_sub.mm);
return ret;
}
The driver->update lock is the same lock that the driver takes inside its
invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
update.
update. The retry check must use the same notifier and sequence number stored
in ``range`` by ``hmm_range_fault_unlocked_timeout()``.
Holding the mmap lock across HMM faults
=======================================
Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver
really needs to hold the mmap lock across work outside HMM, it can use::
int hmm_range_fault(struct hmm_range *range);
The mmap lock must be held by the caller and will remain held on return. This
interface cannot support VMAs whose fault handlers need to drop the mmap lock.
New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they
have a specific requirement to keep the mmap lock held across the call.
Leverage default_flags and pfn_flags_mask
=========================================
@@ -221,8 +254,8 @@ permission, it sets::
range->default_flags = HMM_PFN_REQ_FAULT;
range->pfn_flags_mask = 0;
and calls hmm_range_fault() as described above. This will fill fault all pages
in the range with at least read permission.
and calls the HMM range fault helper as described above. This will fault
all pages in the range with at least read permission.
Now let's say the driver wants to do the same except for one page in the range for
which it wants to have write permission. Now driver set::
@@ -236,9 +269,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with
write permission i.e., if the CPU pte does not have write permission set then HMM
will call handle_mm_fault().
After hmm_range_fault completes the flag bits are set to the current state of
the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is
writable.
After the HMM range fault helper completes the flag bits are set to the
current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be
set if the page is writable.
Represent and manage device memory from core kernel point of view
@@ -316,7 +349,7 @@ between device driver specific code and shared common code:
system memory and device private memory.
One of the first steps migrate_vma_setup() does is to invalidate other
device's MMUs with the ``mmu_notifier_invalidate_range_start(()`` and
device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and
``mmu_notifier_invalidate_range_end()`` calls around the page table
walks to fill in the ``args->src`` array with PFNs to be migrated.
The ``invalidate_range_start()`` callback is passed a

78
Documentation/mm/page_owner.rst
View File

@@ -65,7 +65,15 @@ un-tracking state.
Usage
=====
1) Build user-space helper::
1) Build user-space helpers:
::
To filter page_owner output:
cd tools/mm
make page_owner_filter
To sort and analyze page_owner output:
cd tools/mm
make page_owner_sort
@@ -74,7 +82,11 @@ Usage
3) Do the job that you want to debug.
4) Analyze information from page owner::
4) (Optional) Filter page_owner output::
./page_owner_filter -m handle -n 0,1,2 > filtered_page_owner.txt
5) Analyze information from page owner::
cat /sys/kernel/debug/page_owner_stacks/show_stacks > stacks.txt
cat stacks.txt
@@ -263,3 +275,65 @@ STANDARD FORMAT SPECIFIERS
f free whether the page has been released or not
st stacktrace stack trace of the page allocation
ator allocator memory allocator for pages
Filtering page_owner output
============================
page_owner supports filtering output at the kernel level before reading,
which reduces the amount of data that needs to be processed in userspace.
The page_owner_filter tool provides a convenient interface for this filtering
capability. It supports two types of filters:
1. **print_mode filter**: Control what information is printed for each page
- ``stack``: Print full stack traces (default, compatible with existing usage)
- ``handle``: Print only stack handle numbers (much faster, smaller output)
- ``stack_handle``: Print both stack traces and handle numbers
The ``handle`` mode uses numeric identifiers instead of full stack traces.
The mapping from handles to actual stack traces can be obtained via the
show_stacks_handles interface.
2. **NUMA node filter**: Filter pages by NUMA node ID
- Supports single node: ``-n 0``
- Multiple nodes: ``-n 0,1,2``
- Ranges: ``-n 0-3``
- Mixed format: ``-n 0,2-3,5``
Usage examples::
# Filter by print mode
./page_owner_filter -m handle
./page_owner_filter -m stack_handle
# Filter by NUMA node
./page_owner_filter -n 0
./page_owner_filter -n 0-3
# Combined filters
./page_owner_filter -m stack -n 0,1,2
./page_owner_filter -m handle -n 0,2-3
# Save to file
./page_owner_filter -m handle -o filtered_output.txt
The handle mode is particularly useful for monitoring and performance-critical
scenarios as it dramatically reduces output size. Testing shows handle mode can
reduce output size by ~66% (84MB vs 244MB) and improve read performance by ~4.4x
compared to full stack output.
The NUMA node filter is useful for NUMA-aware memory allocation analysis and debugging.
Behind the scenes, page_owner_filter opens /sys/kernel/debug/page_owner and
writes filter commands before reading the filtered output. The filtering uses
per-file-descriptor state, allowing each open() to have independent filter settings.
Each file descriptor maintains its own filter state, so you can have multiple
independent filtering operations running concurrently. For example, in different
terminals you can run different filters simultaneously::
# Terminal 1: Filter node 0
./page_owner_filter -n 0 > node0_output.txt
# Terminal 2: Filter node 1 (runs concurrently)
./page_owner_filter -n 1 > node1_output.txt

3
Documentation/mm/physical_memory.rst
View File

@@ -262,9 +262,6 @@ General
PFN that needs to be initialized. Defined only when
``CONFIG_DEFERRED_STRUCT_PAGE_INIT`` is enabled
``deferred_split_queue``
Per-node queue of huge pages that their split was deferred. Defined only when ``CONFIG_TRANSPARENT_HUGEPAGE`` is enabled.
``__lruvec``
Per-node lruvec holding LRU lists and related parameters. Used only when
memory cgroups are disabled. It should not be accessed directly, use

2
Documentation/mm/process_addrs.rst
View File

@@ -724,7 +724,7 @@ the zap and the invocation of :c:func:`!free_pgtables`.
Since it is assumed that all such steps have been taken, page table entries are
cleared without page table locks (in the :c:func:`!pgd_clear`, :c:func:`!p4d_clear`,
:c:func:`!pud_clear`, and :c:func:`!pmd_clear` functions.
:c:func:`!pud_clear`, and :c:func:`!pmd_clear` functions).
.. note:: It is possible for leaf page tables to be torn down independent of
the page tables above it as is done by

14
Documentation/userspace-api/mseal.rst
View File

@@ -50,8 +50,8 @@ mseal syscall signature
* The start address (``addr``) is not allocated.
* The end address (``addr`` + ``len``) is not allocated.
* A gap (unallocated memory) between start and end address.
- **-EPERM**:
* sealing is supported only on 64-bit CPUs, 32-bit is not supported.
- **-ENOSYS**:
* The kernel does not implement ``mseal()``.
**Note about error return**:
- For above error cases, users can expect the given memory range is
@@ -62,7 +62,8 @@ mseal syscall signature
memory range could happen. However, those cases should be rare.
**Architecture support**:
mseal only works on 64-bit CPUs, not 32-bit CPUs.
mseal is built only for 64-bit kernels. 32-bit kernels return
``-ENOSYS``.
**Idempotent**:
users can call mseal multiple times. mseal on an already sealed memory
@@ -131,11 +132,11 @@ Use cases
- Chrome browser: protect some security sensitive data structures.
- System mappings:
The system mappings are created by the kernel and includes vdso, vvar,
The system mappings are created by the kernel and include vdso, vvar,
vvar_vclock, vectors (arm compat-mode), sigpage (arm compat-mode), uprobes.
Those system mappings are readonly only or execute only, memory sealing can
protect them from ever changing to writable or unmmap/remapped as different
protect them from ever changing to writable or unmapped/remapped as different
attributes. This is useful to mitigate memory corruption issues where a
corrupted pointer is passed to a memory management system.
@@ -143,9 +144,6 @@ Use cases
the CONFIG_MSEAL_SYSTEM_MAPPINGS seals all system mappings of this
architecture.
The following architectures currently support this feature: x86-64, arm64,
loongarch and s390.
WARNING: This feature breaks programs which rely on relocating
or unmapping system mappings. Known broken software at the time
of writing includes CHECKPOINT_RESTORE, UML, gVisor, rr. Therefore

34
MAINTAINERS
View File

@@ -16903,24 +16903,13 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git fixes
F: Documentation/core-api/boot-time-mm.rst
F: include/linux/kho/abi/memblock.h
F: include/linux/memblock.h
F: mm/bootmem_info.c
F: mm/memblock.c
F: mm/memtest.c
F: mm/mm_init.c
F: mm/mm_init.h
F: mm/rodata_test.c
F: tools/testing/memblock/
MEMORY ALLOCATION PROFILING
M: Suren Baghdasaryan <surenb@google.com>
M: Kent Overstreet <kent.overstreet@linux.dev>
R: Hao Ge <hao.ge@linux.dev>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/mm/allocation-profiling.rst
F: include/linux/alloc_tag.h
F: include/linux/pgalloc_tag.h
F: lib/alloc_tag.c
MEMORY CONTROLLER DRIVERS
M: Krzysztof Kozlowski <krzk@kernel.org>
L: linux-kernel@vger.kernel.org
@@ -16965,6 +16954,16 @@ T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new
F: mm/
F: tools/mm/
MEMORY MANAGEMENT - ALLOCATION PROFILING (ALLOC TAG)
M: Suren Baghdasaryan <surenb@google.com>
R: Hao Ge <hao.ge@linux.dev>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/mm/allocation-profiling.rst
F: include/linux/alloc_tag.h
F: include/linux/pgalloc_tag.h
F: mm/alloc_tag.c
MEMORY MANAGEMENT - BALLOON
M: Andrew Morton <akpm@linux-foundation.org>
M: David Hildenbrand <david@kernel.org>
@@ -17012,6 +17011,7 @@ F: kernel/fork.c
F: mm/Kconfig
F: mm/debug.c
F: mm/folio-compat.c
F: mm/folio.c
F: mm/highmem.c
F: mm/init-mm.c
F: mm/internal.h
@@ -17024,6 +17024,7 @@ F: mm/pgtable-generic.c
F: mm/ptdump.c
F: mm/sparse-vmemmap.c
F: mm/sparse.c
F: mm/sparse.h
F: mm/util.c
F: mm/vmpressure.c
F: mm/vmstat.c
@@ -17112,7 +17113,6 @@ F: Documentation/admin-guide/mm/multigen_lru.rst
F: Documentation/mm/multigen_lru.rst
F: include/linux/mm_inline.h
F: include/linux/mmzone.h
F: mm/swap.c
F: mm/vmscan.c
F: mm/workingset.c
@@ -17197,6 +17197,7 @@ F: mm/debug_page_alloc.c
F: mm/debug_page_ref.c
F: mm/fail_page_alloc.c
F: mm/page_alloc.c
F: mm/page_alloc.h
F: mm/page_ext.c
F: mm/page_frag_cache.c
F: mm/page_isolation.c
@@ -17234,6 +17235,7 @@ R: Lance Yang <lance.yang@linux.dev>
L: linux-mm@kvack.org
S: Maintained
F: include/linux/rmap.h
F: mm/interval_tree.c
F: mm/page_vma_mapped.c
F: mm/rmap.c
F: tools/testing/selftests/mm/rmap.c
@@ -17263,7 +17265,6 @@ F: include/linux/swap.h
F: include/linux/swapfile.h
F: include/linux/swapops.h
F: mm/page_io.c
F: mm/swap.c
F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
@@ -17340,7 +17341,6 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: include/trace/events/mmap.h
F: fs/proc/task_mmu.c
F: fs/proc/task_nommu.c
F: mm/interval_tree.c
F: mm/mincore.c
F: mm/mlock.c
F: mm/mmap.c
@@ -28938,6 +28938,7 @@ W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: include/linux/vmalloc.h
F: mm/vmalloc.c
F: mm/vmalloc.h
F: lib/test_vmalloc.c
VME SUBSYSTEM
@@ -29916,8 +29917,11 @@ M: Minchan Kim <minchan@kernel.org>
M: Sergey Senozhatsky <senozhatsky@chromium.org>
L: linux-kernel@vger.kernel.org
S: Maintained
F: Documentation/ABI/testing/sysfs-block-zram
F: Documentation/ABI/testing/sysfs-class-zram
F: Documentation/admin-guide/blockdev/zram.rst
F: drivers/block/zram/
F: tools/testing/selftests/zram/
ZS DECSTATION Z85C30 SERIAL DRIVER
M: "Maciej W. Rozycki" <macro@orcam.me.uk>

4
arch/arm/mm/fault-armv.c
View File

@@ -132,7 +132,7 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma,
pgoff_t pgoff;
int aliases = 0;
pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >> PAGE_SHIFT);
pgoff = linear_page_index(vma, addr);
/*
* If we have any shared mappings that are in the same mm
@@ -140,7 +140,7 @@ make_coherent(struct address_space *mapping, struct vm_area_struct *vma,
* cache coherency.
*/
flush_dcache_mmap_lock(mapping);
vma_interval_tree_foreach(mpnt, &mapping->i_mmap, pgoff, pgoff) {
mapping_rmap_tree_foreach(mpnt, mapping, pgoff, pgoff) {
/*
* If we are using split PTE locks, then we need to take the pte
* lock. Otherwise we are using shared mm->page_table_lock which

2
arch/arm/mm/flush.c
View File

@@ -251,7 +251,7 @@ static void __flush_dcache_aliases(struct address_space *mapping, struct folio *
pgoff_end = pgoff + folio_nr_pages(folio) - 1;
flush_dcache_mmap_lock(mapping);
vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff_end) {
mapping_rmap_tree_foreach(vma, mapping, pgoff, pgoff_end) {
unsigned long start, offset, pfn;
unsigned int nr;

3
arch/arm64/Kconfig
View File

@@ -17,7 +17,7 @@ config ARM64
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_MEMORY_HOTPLUG
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CC_PLATFORM
select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
@@ -78,6 +78,7 @@ config ARM64
select ARCH_SUPPORTS_CFI
select ARCH_SUPPORTS_ATOMIC_RMW
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
select ARCH_SUPPORTS_PER_VMA_LOCK

3
arch/arm64/include/asm/fixmap.h
View File

@@ -114,9 +114,6 @@ void __init early_fixmap_init(void);
#define __early_set_fixmap __set_fixmap
#define __late_set_fixmap __set_fixmap
#define __late_clear_fixmap(idx) __set_fixmap((idx), 0, FIXMAP_PAGE_CLEAR)
extern void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot);
#include <asm-generic/fixmap.h>

8
arch/arm64/include/asm/pgtable-prot.h
View File

@@ -28,11 +28,11 @@
#define PTE_PRESENT_VALID_KERNEL (PTE_VALID | PTE_MAYBE_NG)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
#define PTE_UFFD_WP (_AT(pteval_t, 1) << 58) /* uffd-wp tracking */
#define PTE_SWP_UFFD_WP (_AT(pteval_t, 1) << 3) /* only for swp ptes */
#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultfd tracking */
#define PTE_SWP_UFFD (_AT(pteval_t, 1) << 3) /* only for swp ptes */
#else
#define PTE_UFFD_WP (_AT(pteval_t, 0))
#define PTE_SWP_UFFD_WP (_AT(pteval_t, 0))
#define PTE_UFFD (_AT(pteval_t, 0))
#define PTE_SWP_UFFD (_AT(pteval_t, 0))
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)

51
arch/arm64/include/asm/pgtable.h
View File

@@ -345,19 +345,19 @@ static inline pmd_t pmd_mknoncont(pmd_t pmd)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline int pte_uffd_wp(pte_t pte)
static inline int pte_uffd(pte_t pte)
{
return !!(pte_val(pte) & PTE_UFFD_WP);
return !!(pte_val(pte) & PTE_UFFD);
}
static inline pte_t pte_mkuffd_wp(pte_t pte)
static inline pte_t pte_mkuffd(pte_t pte)
{
return pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD_WP)));
return pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD)));
}
static inline pte_t pte_clear_uffd_wp(pte_t pte)
static inline pte_t pte_clear_uffd(pte_t pte)
{
return clear_pte_bit(pte, __pgprot(PTE_UFFD_WP));
return clear_pte_bit(pte, __pgprot(PTE_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -541,26 +541,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
static inline pte_t pte_swp_mkuffd(pte_t pte)
{
return set_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));
return set_pte_bit(pte, __pgprot(PTE_SWP_UFFD));
}
static inline int pte_swp_uffd_wp(pte_t pte)
static inline int pte_swp_uffd(pte_t pte)
{
return !!(pte_val(pte) & PTE_SWP_UFFD_WP);
return !!(pte_val(pte) & PTE_SWP_UFFD);
}
static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
return clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));
return clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#ifdef CONFIG_NUMA_BALANCING
/*
* See the comment in include/linux/pgtable.h
*/
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
/*
@@ -579,7 +576,7 @@ static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
#endif
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_present(pmd) pte_present(pmd_pte(pmd))
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
@@ -597,13 +594,13 @@ static inline int pmd_protnone(pmd_t pmd)
#define pmd_mkvalid_k(pmd) pte_pmd(pte_mkvalid_k(pmd_pte(pmd)))
#define pmd_mkinvalid(pmd) pte_pmd(pte_mkinvalid(pmd_pte(pmd)))
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
#define pmd_uffd_wp(pmd) pte_uffd_wp(pmd_pte(pmd))
#define pmd_mkuffd_wp(pmd) pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)))
#define pmd_clear_uffd_wp(pmd) pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)))
#define pmd_swp_uffd_wp(pmd) pte_swp_uffd_wp(pmd_pte(pmd))
#define pmd_swp_mkuffd_wp(pmd) pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)))
#define pmd_swp_clear_uffd_wp(pmd) \
pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)))
#define pmd_uffd(pmd) pte_uffd(pmd_pte(pmd))
#define pmd_mkuffd(pmd) pte_pmd(pte_mkuffd(pmd_pte(pmd)))
#define pmd_clear_uffd(pmd) pte_pmd(pte_clear_uffd(pmd_pte(pmd)))
#define pmd_swp_uffd(pmd) pte_swp_uffd(pmd_pte(pmd))
#define pmd_swp_mkuffd(pmd) pte_pmd(pte_swp_mkuffd(pmd_pte(pmd)))
#define pmd_swp_clear_uffd(pmd) \
pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
@@ -1519,7 +1516,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
* Encode and decode a swap entry:
* bits 0-1: present (must be zero)
* bits 2: remember PG_anon_exclusive
* bit 3: remember uffd-wp state
* bit 3: remember uffd state
* bits 6-10: swap type
* bit 11: PTE_PRESENT_INVALID (must be zero)
* bits 12-61: swap offset
@@ -1538,10 +1535,10 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pmd(swp) __pmd((swp).val)
#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
/*
* Ensure that there are not more swap files than can be encoded in the kernel

2
arch/arm64/kernel/setup.c
View File

@@ -347,8 +347,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
request_standard_resources();
early_ioremap_reset();
if (acpi_disabled)
psci_dt_init();
else

7
arch/csky/abiv1/inc/abi/cacheflush.h
View File

@@ -42,7 +42,12 @@ static inline void flush_anon_page(struct vm_area_struct *vma,
* Use cache_wbinv_all() here and need to be improved in future.
*/
extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end);
#define flush_cache_vmap(start, end) cache_wbinv_all()
static inline void flush_cache_vmap(unsigned long start, unsigned long end)
{
cache_wbinv_all();
}
#define flush_cache_vmap_early(start, end) do { } while (0)
#define flush_cache_vunmap(start, end) cache_wbinv_all()

3
arch/loongarch/Kconfig
View File

@@ -12,7 +12,7 @@ config LOONGARCH
select ARCH_NEEDS_DEFER_KASAN
select ARCH_DISABLE_KASAN_INLINE
select ARCH_ENABLE_MEMORY_HOTPLUG
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_CPU_FINALIZE_INIT
select ARCH_HAS_CURRENT_STACK_POINTER
@@ -67,6 +67,7 @@ config LOONGARCH
select ARCH_SUPPORTS_LTO_CLANG
select ARCH_SUPPORTS_LTO_CLANG_THIN
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
select ARCH_HAS_PTE_PROTNONE if 64BIT
select ARCH_SUPPORTS_NUMA_BALANCING if NUMA
select ARCH_SUPPORTS_PER_VMA_LOCK
select ARCH_SUPPORTS_RT

4
arch/loongarch/include/asm/pgtable.h
View File

@@ -630,7 +630,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#ifdef CONFIG_NUMA_BALANCING
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline long pte_protnone(pte_t pte)
{
return (pte_val(pte) & _PAGE_PROTNONE);
@@ -640,7 +640,7 @@ static inline long pmd_protnone(pmd_t pmd)
{
return (pmd_val(pmd) & _PAGE_PROTNONE);
}
#endif /* CONFIG_NUMA_BALANCING */
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_leaf(pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0)
#define pud_leaf(pud) ((pud_val(pud) & _PAGE_HUGE) != 0)

4
arch/mips/kernel/vdso.c
View File

@@ -91,8 +91,8 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
/* Map delay slot emulation page */
base = do_mmap(NULL, STACK_TOP, PAGE_SIZE, PROT_READ | PROT_EXEC,
MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED, 0, 0, &unused,
NULL);
MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED,
EMPTY_VMA_FLAGS, 0, &unused, NULL);
if (IS_ERR_VALUE(base)) {
ret = base;
goto out;

2
arch/nios2/mm/cacheflush.c
View File

@@ -82,7 +82,7 @@ static void flush_aliases(struct address_space *mapping, struct folio *folio)
pgoff = folio->index;
flush_dcache_mmap_lock_irqsave(mapping, flags);
vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff + nr - 1) {
mapping_rmap_tree_foreach(vma, mapping, pgoff, pgoff + nr - 1) {
unsigned long start;
if (vma->vm_mm != mm)

2
arch/parisc/kernel/cache.c
View File

@@ -503,7 +503,7 @@ void flush_dcache_folio(struct folio *folio)
* on machines that support equivalent aliasing
*/
flush_dcache_mmap_lock_irqsave(mapping, flags);
vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff + nr - 1) {
mapping_rmap_tree_foreach(vma, mapping, pgoff, pgoff + nr - 1) {
unsigned long offset = pgoff - vma->vm_pgoff;
unsigned long pfn = folio_pfn(folio);

10
arch/powerpc/include/asm/book3s/64/pgtable.h
View File

@@ -490,13 +490,13 @@ static inline pte_t pte_clear_soft_dirty(pte_t pte)
}
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
#ifdef CONFIG_NUMA_BALANCING
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_raw(pte) & cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE | _PAGE_RWX)) ==
cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE);
}
#endif /* CONFIG_NUMA_BALANCING */
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline bool pte_hw_valid(pte_t pte)
{
@@ -1060,19 +1060,19 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)
#define pmd_mksoft_dirty(pmd) pte_pmd(pte_mksoft_dirty(pmd_pte(pmd)))
#define pmd_clear_soft_dirty(pmd) pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd)))
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
#define pmd_swp_mksoft_dirty(pmd) pte_pmd(pte_swp_mksoft_dirty(pmd_pte(pmd)))
#define pmd_swp_soft_dirty(pmd) pte_swp_soft_dirty(pmd_pte(pmd))
#define pmd_swp_clear_soft_dirty(pmd) pte_pmd(pte_swp_clear_soft_dirty(pmd_pte(pmd)))
#endif
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
#ifdef CONFIG_NUMA_BALANCING
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
#endif /* CONFIG_NUMA_BALANCING */
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_write(pmd) pte_write(pmd_pte(pmd))

7
arch/powerpc/mm/book3s64/radix_pgtable.c
View File

@@ -1314,7 +1314,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
* covering out both edges.
*/
unsigned long addr;
unsigned long addr_pfn = start_pfn;
unsigned long next;
pgd_t *pgd;
p4d_t *p4d;
@@ -1335,7 +1334,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
if (pmd_leaf(READ_ONCE(*pmd))) {
/* existing huge mapping. Skip the range */
addr_pfn += (PMD_SIZE >> PAGE_SHIFT);
next = pmd_addr_end(addr, end);
continue;
}
@@ -1348,11 +1346,11 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
* page whose VMEMMAP_RESERVE_NR pages were mapped and
* this request fall in those pages.
*/
addr_pfn += 1;
next = addr + PAGE_SIZE;
continue;
} else {
unsigned long nr_pages = pgmap_vmemmap_nr(pgmap);
unsigned long addr_pfn = page_to_pfn((struct page *)addr);
unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages);
pte_t *tail_page_pte;
@@ -1376,7 +1374,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
if (!pte)
return -ENOMEM;
addr_pfn += 2;
next = addr + 2 * PAGE_SIZE;
continue;
}
@@ -1392,7 +1389,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
return -ENOMEM;
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
addr_pfn += 1;
next = addr + PAGE_SIZE;
continue;
}
@@ -1402,7 +1398,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn,
return -ENOMEM;
vmemmap_verify(pte, node, addr, addr + PAGE_SIZE);
addr_pfn += 1;
next = addr + PAGE_SIZE;
continue;
}

6
arch/powerpc/mm/book3s64/slice.c
View File

@@ -659,7 +659,8 @@ unsigned long arch_get_unmapped_area(struct file *filp,
unsigned int psize;
if (radix_enabled())
return generic_get_unmapped_area(filp, addr, len, pgoff, flags, vm_flags);
return generic_get_unmapped_area(filp, addr, len, pgoff, flags,
legacy_to_vma_flags(vm_flags));
if (filp && is_file_hugepages(filp))
psize = file_to_psize(filp);
@@ -679,7 +680,8 @@ unsigned long arch_get_unmapped_area_topdown(struct file *filp,
unsigned int psize;
if (radix_enabled())
return generic_get_unmapped_area_topdown(filp, addr0, len, pgoff, flags, vm_flags);
return generic_get_unmapped_area_topdown(filp, addr0, len,
pgoff, flags, legacy_to_vma_flags(vm_flags));
if (filp && is_file_hugepages(filp))
psize = file_to_psize(filp);

13
arch/powerpc/mm/hugetlbpage.c
View File

@@ -104,17 +104,14 @@ void __init pseries_add_gpage(u64 addr, u64 page_size, unsigned long number_of_p
}
}
static int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate)
static __init void *pseries_alloc_bootmem_huge_page(struct hstate *hstate)
{
struct huge_bootmem_page *m;
void *m;
if (nr_gpages == 0)
return 0;
return NULL;
m = phys_to_virt(gpage_freearray[--nr_gpages]);
gpage_freearray[nr_gpages] = 0;
list_add(&m->list, &huge_boot_pages[0]);
m->hstate = hstate;
m->flags = 0;
return 1;
return m;
}
bool __init hugetlb_node_alloc_supported(void)
@@ -124,7 +121,7 @@ bool __init hugetlb_node_alloc_supported(void)
#endif
int __init alloc_bootmem_huge_page(struct hstate *h, int nid)
void *__init arch_alloc_bootmem_huge_page(struct hstate *h, int nid)
{
#ifdef CONFIG_PPC_BOOK3S_64

3
arch/powerpc/platforms/Kconfig.cputype
View File

@@ -87,6 +87,7 @@ config PPC_BOOK3S_64
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_SPLIT_PMD_PTLOCK
select ARCH_SUPPORTS_HUGETLBFS
select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select HAVE_MOVE_PMD
select HAVE_MOVE_PUD
@@ -112,7 +113,7 @@ config PPC_THP
depends on PPC_RADIX_MMU || (PPC_64S_HASH_MMU && PAGE_SIZE_64KB)
select HAVE_ARCH_TRANSPARENT_HUGEPAGE
select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
choice
prompt "CPU selection"

3
arch/riscv/Kconfig
View File

@@ -22,7 +22,7 @@ config RISCV
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_BINFMT_FLAT
select ARCH_HAS_CC_CAN_LINK
@@ -73,6 +73,7 @@ config RISCV
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT && MMU
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU
select ARCH_SUPPORTS_PER_VMA_LOCK if MMU
select ARCH_HAS_PTE_PROTNONE if MMU
select ARCH_SUPPORTS_RT
select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK
select ARCH_SUPPORTS_SCHED_MC if SMP

3
arch/riscv/include/asm/fixmap.h
View File

@@ -55,9 +55,6 @@ enum fixed_addresses {
#define __early_set_fixmap __set_fixmap
#define __late_set_fixmap __set_fixmap
#define __late_clear_fixmap(idx) __set_fixmap((idx), 0, FIXMAP_PAGE_CLEAR)
extern void __set_fixmap(enum fixed_addresses idx,
phys_addr_t phys, pgprot_t prot);

12
arch/riscv/include/asm/pgtable-bits.h
View File

@@ -40,20 +40,20 @@
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
/* ext_svrsw60t59b: Bit(60) for uffd-wp tracking */
#define _PAGE_UFFD_WP \
/* ext_svrsw60t59b: Bit(60) for userfaultfd tracking */
#define _PAGE_UFFD \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
(1UL << 60) : 0)
/*
* Bit 4 is not involved into swap entry computation, so we
* can borrow it for swap page uffd-wp tracking.
* can borrow it for swap page userfaultfd tracking.
*/
#define _PAGE_SWP_UFFD_WP \
#define _PAGE_SWP_UFFD \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
_PAGE_USER : 0)
#else
#define _PAGE_UFFD_WP 0
#define _PAGE_SWP_UFFD_WP 0
#define _PAGE_UFFD 0
#define _PAGE_SWP_UFFD 0
#endif
#define _PAGE_TABLE _PAGE_PRESENT

67
arch/riscv/include/asm/pgtable.h
View File

@@ -410,37 +410,37 @@ static inline pte_t pte_wrprotect(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
#define pgtable_supports_uffd_wp() \
#define pgtable_supports_uffd() \
riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)
static inline bool pte_uffd_wp(pte_t pte)
static inline bool pte_uffd(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_UFFD_WP);
return !!(pte_val(pte) & _PAGE_UFFD);
}
static inline pte_t pte_mkuffd_wp(pte_t pte)
static inline pte_t pte_mkuffd(pte_t pte)
{
return pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD_WP));
return pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD));
}
static inline pte_t pte_clear_uffd_wp(pte_t pte)
static inline pte_t pte_clear_uffd(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_UFFD_WP));
return __pte(pte_val(pte) & ~(_PAGE_UFFD));
}
static inline bool pte_swp_uffd_wp(pte_t pte)
static inline bool pte_swp_uffd(pte_t pte)
{
return !!(pte_val(pte) & _PAGE_SWP_UFFD_WP);
return !!(pte_val(pte) & _PAGE_SWP_UFFD);
}
static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
static inline pte_t pte_swp_mkuffd(pte_t pte)
{
return __pte(pte_val(pte) | _PAGE_SWP_UFFD_WP);
return __pte(pte_val(pte) | _PAGE_SWP_UFFD);
}
static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_SWP_UFFD_WP));
return __pte(pte_val(pte) & ~(_PAGE_SWP_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -534,10 +534,7 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
PAGE_SIZE)
#endif
#ifdef CONFIG_NUMA_BALANCING
/*
* See the comment in include/asm-generic/pgtable.h
*/
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE;
@@ -547,7 +544,7 @@ static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
#endif
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
/* Modify page protection bits */
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
@@ -909,34 +906,34 @@ static inline pud_t pud_mkspecial(pud_t pud)
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline bool pmd_uffd_wp(pmd_t pmd)
static inline bool pmd_uffd(pmd_t pmd)
{
return pte_uffd_wp(pmd_pte(pmd));
return pte_uffd(pmd_pte(pmd));
}
static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
static inline pmd_t pmd_mkuffd(pmd_t pmd)
{
return pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)));
return pte_pmd(pte_mkuffd(pmd_pte(pmd)));
}
static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
static inline pmd_t pmd_clear_uffd(pmd_t pmd)
{
return pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)));
return pte_pmd(pte_clear_uffd(pmd_pte(pmd)));
}
static inline bool pmd_swp_uffd_wp(pmd_t pmd)
static inline bool pmd_swp_uffd(pmd_t pmd)
{
return pte_swp_uffd_wp(pmd_pte(pmd));
return pte_swp_uffd(pmd_pte(pmd));
}
static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)
{
return pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)));
return pte_pmd(pte_swp_mkuffd(pmd_pte(pmd)));
}
static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)
{
return pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)));
return pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -956,7 +953,7 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
return pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd)));
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
static inline bool pmd_swp_soft_dirty(pmd_t pmd)
{
return pte_swp_soft_dirty(pmd_pte(pmd));
@@ -971,7 +968,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
{
return pte_pmd(pte_swp_clear_soft_dirty(pmd_pte(pmd)));
}
#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
@@ -1180,7 +1177,7 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
* bit 0: _PAGE_PRESENT (zero)
* bit 1 to 2: (zero)
* bit 3: _PAGE_SWP_SOFT_DIRTY
* bit 4: _PAGE_SWP_UFFD_WP
* bit 4: _PAGE_SWP_UFFD
* bit 5: _PAGE_PROT_NONE (zero)
* bit 6: exclusive marker
* bits 7 to 11: swap type
@@ -1218,10 +1215,10 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE);
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pmd(swp) __pmd((swp).val)
#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
/*
* In the RV64 Linux scheme, we give the user half of the virtual-address space

3
arch/s390/Kconfig
View File

@@ -82,7 +82,7 @@ config S390
select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE
select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
select ARCH_HAS_CC_CAN_LINK
select ARCH_HAS_CPU_FINALIZE_INIT
select ARCH_HAS_CURRENT_STACK_POINTER
@@ -148,6 +148,7 @@ config S390
select ARCH_SUPPORTS_HUGETLBFS
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && CC_IS_CLANG
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
select ARCH_SUPPORTS_PER_VMA_LOCK

12
arch/s390/include/asm/hugetlb.h
View File

@@ -77,20 +77,20 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,
__set_huge_pte_at(mm, addr, ptep, pte_wrprotect(pte));
}
#define __HAVE_ARCH_HUGE_PTE_MKUFFD_WP
static inline pte_t huge_pte_mkuffd_wp(pte_t pte)
#define __HAVE_ARCH_HUGE_PTE_MKUFFD
static inline pte_t huge_pte_mkuffd(pte_t pte)
{
return pte;
}
#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP
static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)
#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD
static inline pte_t huge_pte_clear_uffd(pte_t pte)
{
return pte;
}
#define __HAVE_ARCH_HUGE_PTE_UFFD_WP
static inline int huge_pte_uffd_wp(pte_t pte)
#define __HAVE_ARCH_HUGE_PTE_UFFD
static inline int huge_pte_uffd(pte_t pte)
{
return 0;
}

6
arch/s390/include/asm/pgtable.h
View File

@@ -842,7 +842,7 @@ static inline int pte_same(pte_t a, pte_t b)
return pte_val(a) == pte_val(b);
}
#ifdef CONFIG_NUMA_BALANCING
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return pte_present(pte) && !(pte_val(pte) & _PAGE_READ);
@@ -853,7 +853,7 @@ static inline int pmd_protnone(pmd_t pmd)
/* pmd_leaf(pmd) implies pmd_present(pmd) */
return pmd_leaf(pmd) && !(pmd_val(pmd) & _SEGMENT_ENTRY_READ);
}
#endif
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline bool pte_swp_exclusive(pte_t pte)
{
@@ -903,7 +903,7 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
return clear_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_SOFT_DIRTY));
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
#define pmd_swp_soft_dirty(pmd) pmd_soft_dirty(pmd)
#define pmd_swp_mksoft_dirty(pmd) pmd_mksoft_dirty(pmd)
#define pmd_swp_clear_soft_dirty(pmd) pmd_clear_soft_dirty(pmd)

4
arch/s390/mm/vmem.c
View File

@@ -39,7 +39,6 @@ static void __ref *vmem_alloc_pages(unsigned int order)
static void vmem_free_pages(unsigned long addr, int order, struct vmem_altmap *altmap)
{
unsigned int nr_pages = 1 << order;
struct page *page;
if (altmap) {
@@ -49,8 +48,7 @@ static void vmem_free_pages(unsigned long addr, int order, struct vmem_altmap *a
page = virt_to_page((void *)addr);
if (PageReserved(page)) {
/* allocated from memblock */
while (nr_pages--)
free_reserved_page(page++);
free_reserved_pages(page, order);
} else {
free_pages(addr, order);
}

4
arch/sparc/include/asm/pgtable_64.h
View File

@@ -945,10 +945,6 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
#define pte_clear(mm,addr,ptep) \
set_pte_at((mm), (addr), (ptep), __pte(0UL))
#define __HAVE_ARCH_PTE_CLEAR_NOT_PRESENT_FULL
#define pte_clear_not_present_full(mm,addr,ptep,fullmm) \
__set_pte_at((mm), (addr), (ptep), __pte(0UL), (fullmm))
#ifdef DCACHE_ALIASING_POSSIBLE
#define __HAVE_ARCH_MOVE_PTE
#define move_pte(pte, old_addr, new_addr) \

3
arch/x86/Kconfig
View File

@@ -70,7 +70,7 @@ config X86
select ARCH_ENABLE_HUGEPAGE_MIGRATION if X86_64 && HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if (PGTABLE_LEVELS > 2) && (X86_64 || X86_PAE)
select ARCH_ENABLE_THP_MIGRATION if X86_64 && TRANSPARENT_HUGEPAGE
select ARCH_HAS_PMD_SOFTLEAVES if X86_64 && TRANSPARENT_HUGEPAGE
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_CPU_ATTACK_VECTORS if CPU_MITIGATIONS
select ARCH_HAS_CACHE_LINE_SIZE
@@ -125,6 +125,7 @@ config X86
select ARCH_SUPPORTS_DEBUG_PAGEALLOC
select ARCH_SUPPORTS_HUGETLBFS
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if X86_64
select ARCH_HAS_PTE_PROTNONE if X86_64
select ARCH_SUPPORTS_NUMA_BALANCING if X86_64
select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096
select ARCH_SUPPORTS_CFI if X86_64

6
arch/x86/events/intel/ds.c
View File

@@ -832,7 +832,7 @@ static void *dsalloc_pages(size_t size, gfp_t flags, int cpu)
int node = cpu_to_node(cpu);
struct page *page;
page = __alloc_pages_node(node, flags | __GFP_ZERO, order);
page = alloc_pages_node(node, flags | __GFP_ZERO, order);
return page ? page_address(page) : NULL;
}
@@ -1088,9 +1088,9 @@ void init_arch_pebs_on_cpu(int cpu)
/*
* 4KB-aligned pointer of the output buffer
* (__alloc_pages_node() return page aligned address)
* (alloc_pages_node() returns page aligned address)
* Buffer Size = 4KB * 2^SIZE
* contiguous physical buffer (__alloc_pages_node() with order)
* contiguous physical buffer (alloc_pages_node() with order)
*/
arch_pebs_base = virt_to_phys(cpuc->pebs_vaddr) | PEBS_BUFFER_SHIFT;
wrmsrq_on_cpu(cpu, MSR_IA32_PEBS_BASE, arch_pebs_base);

58
arch/x86/include/asm/pgtable.h
View File

@@ -411,19 +411,19 @@ static inline pte_t pte_wrprotect(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline int pte_uffd_wp(pte_t pte)
static inline int pte_uffd(pte_t pte)
{
return pte_flags(pte) & _PAGE_UFFD_WP;
return pte_flags(pte) & _PAGE_UFFD;
}
static inline pte_t pte_mkuffd_wp(pte_t pte)
static inline pte_t pte_mkuffd(pte_t pte)
{
return pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD_WP));
return pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD));
}
static inline pte_t pte_clear_uffd_wp(pte_t pte)
static inline pte_t pte_clear_uffd(pte_t pte)
{
return pte_clear_flags(pte, _PAGE_UFFD_WP);
return pte_clear_flags(pte, _PAGE_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -526,19 +526,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline int pmd_uffd_wp(pmd_t pmd)
static inline int pmd_uffd(pmd_t pmd)
{
return pmd_flags(pmd) & _PAGE_UFFD_WP;
return pmd_flags(pmd) & _PAGE_UFFD;
}
static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
static inline pmd_t pmd_mkuffd(pmd_t pmd)
{
return pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD_WP));
return pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD));
}
static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
static inline pmd_t pmd_clear_uffd(pmd_t pmd)
{
return pmd_clear_flags(pmd, _PAGE_UFFD_WP);
return pmd_clear_flags(pmd, _PAGE_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -985,11 +985,7 @@ static inline int pmd_present(pmd_t pmd)
return pmd_flags(pmd) & (_PAGE_PRESENT | _PAGE_PROTNONE | _PAGE_PSE);
}
#ifdef CONFIG_NUMA_BALANCING
/*
* These work without NUMA balancing but the kernel does not care. See the
* comment in include/linux/pgtable.h
*/
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT))
@@ -1001,7 +997,7 @@ static inline int pmd_protnone(pmd_t pmd)
return (pmd_flags(pmd) & (_PAGE_PROTNONE | _PAGE_PRESENT))
== _PAGE_PROTNONE;
}
#endif /* CONFIG_NUMA_BALANCING */
#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline int pmd_none(pmd_t pmd)
{
@@ -1545,7 +1541,7 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
static inline pmd_t pmd_swp_mksoft_dirty(pmd_t pmd)
{
return pmd_set_flags(pmd, _PAGE_SWP_SOFT_DIRTY);
@@ -1564,34 +1560,34 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
static inline pte_t pte_swp_mkuffd(pte_t pte)
{
return pte_set_flags(pte, _PAGE_SWP_UFFD_WP);
return pte_set_flags(pte, _PAGE_SWP_UFFD);
}
static inline int pte_swp_uffd_wp(pte_t pte)
static inline int pte_swp_uffd(pte_t pte)
{
return pte_flags(pte) & _PAGE_SWP_UFFD_WP;
return pte_flags(pte) & _PAGE_SWP_UFFD;
}
static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
return pte_clear_flags(pte, _PAGE_SWP_UFFD_WP);
return pte_clear_flags(pte, _PAGE_SWP_UFFD);
}
static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)
{
return pmd_set_flags(pmd, _PAGE_SWP_UFFD_WP);
return pmd_set_flags(pmd, _PAGE_SWP_UFFD);
}
static inline int pmd_swp_uffd_wp(pmd_t pmd)
static inline int pmd_swp_uffd(pmd_t pmd)
{
return pmd_flags(pmd) & _PAGE_SWP_UFFD_WP;
return pmd_flags(pmd) & _PAGE_SWP_UFFD;
}
static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)
{
return pmd_clear_flags(pmd, _PAGE_SWP_UFFD_WP);
return pmd_clear_flags(pmd, _PAGE_SWP_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */

16
arch/x86/include/asm/pgtable_types.h
View File

@@ -31,7 +31,7 @@
#define _PAGE_BIT_SPECIAL _PAGE_BIT_SOFTW1
#define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1
#define _PAGE_BIT_UFFD_WP _PAGE_BIT_SOFTW2 /* userfaultfd wrprotected */
#define _PAGE_BIT_UFFD _PAGE_BIT_SOFTW2 /* userfaultfd tracking */
#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */
#define _PAGE_BIT_KERNEL_4K _PAGE_BIT_SOFTW3 /* page must not be converted to large */
@@ -39,7 +39,7 @@
#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5 /* Saved Dirty bit (leaf) */
#define _PAGE_BIT_NOPTISHADOW _PAGE_BIT_SOFTW5 /* No PTI shadow (root PGD) */
#else
/* Shared with _PAGE_BIT_UFFD_WP which is not supported on 32 bit */
/* Shared with _PAGE_BIT_UFFD which is not supported on 32 bit */
#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW2 /* Saved Dirty bit (leaf) */
#define _PAGE_BIT_NOPTISHADOW _PAGE_BIT_SOFTW2 /* No PTI shadow (root PGD) */
#endif
@@ -111,11 +111,11 @@
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
#define _PAGE_UFFD_WP (_AT(pteval_t, 1) << _PAGE_BIT_UFFD_WP)
#define _PAGE_SWP_UFFD_WP _PAGE_USER
#define _PAGE_UFFD (_AT(pteval_t, 1) << _PAGE_BIT_UFFD)
#define _PAGE_SWP_UFFD _PAGE_USER
#else
#define _PAGE_UFFD_WP (_AT(pteval_t, 0))
#define _PAGE_SWP_UFFD_WP (_AT(pteval_t, 0))
#define _PAGE_UFFD (_AT(pteval_t, 0))
#define _PAGE_SWP_UFFD (_AT(pteval_t, 0))
#endif
#if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE)
@@ -129,7 +129,7 @@
/*
* The hardware requires shadow stack to be Write=0,Dirty=1. However,
* there are valid cases where the kernel might create read-only PTEs that
* are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In
* are dirty (e.g., fork(), mprotect(), userfaultfd, soft-dirty tracking). In
* this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit,
* to avoid creating a wrong "shadow stack" PTEs. Such PTEs have
* (Write=0,SavedDirty=1,Dirty=0) set.
@@ -151,7 +151,7 @@
#define _COMMON_PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \
_PAGE_SPECIAL | _PAGE_ACCESSED | \
_PAGE_DIRTY_BITS | _PAGE_SOFT_DIRTY | \
_PAGE_CC | _PAGE_UFFD_WP)
_PAGE_CC | _PAGE_UFFD)
#define _PAGE_CHG_MASK (_COMMON_PAGE_CHG_MASK | _PAGE_PAT)
#define _HPAGE_CHG_MASK (_COMMON_PAGE_CHG_MASK | _PAGE_PSE | _PAGE_PAT_LARGE)

3
arch/x86/kernel/cpu/sgx/virt.c
View File

@@ -9,6 +9,7 @@
#include <linux/miscdevice.h>
#include <linux/mm.h>
#include <linux/mman.h>
#include <linux/pagemap.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
#include <linux/slab.h>
@@ -41,7 +42,7 @@ static int __sgx_vepc_fault(struct sgx_vepc *vepc,
WARN_ON(!mutex_is_locked(&vepc->lock));
/* Calculate index of EPC page in virtual EPC's page_array */
index = vma->vm_pgoff + PFN_DOWN(addr - vma->vm_start);
index = linear_page_index(vma, addr);
epc_page = xa_load(&vepc->page_array, index);
if (epc_page)

2
arch/x86/kvm/vmx/vmx.c
View File

@@ -3029,7 +3029,7 @@ struct vmcs *alloc_vmcs_cpu(bool shadow, int cpu, gfp_t flags)
struct page *pages;
struct vmcs *vmcs;
pages = __alloc_pages_node(node, flags, 0);
pages = alloc_pages_node(node, flags, 0);
if (!pages)
return NULL;
vmcs = page_address(pages);

134
arch/x86/mm/init_64.c
View File

@@ -33,7 +33,6 @@
#include <linux/nmi.h>
#include <linux/gfp.h>
#include <linux/kcore.h>
#include <linux/bootmem_info.h>
#include <asm/processor.h>
#include <asm/bios_ebda.h>
@@ -1000,32 +999,12 @@ int arch_add_memory(int nid, u64 start, u64 size,
return add_pages(nid, start_pfn, nr_pages, params);
}
static void free_reserved_pages(struct page *page, unsigned long nr_pages)
static void __meminit free_pagetable(struct page *page)
{
while (nr_pages--)
free_reserved_page(page++);
}
static void __meminit free_pagetable(struct page *page, int order)
{
/* bootmem page has reserved flag */
if (PageReserved(page)) {
unsigned long nr_pages = 1 << order;
#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE
enum bootmem_type type = bootmem_type(page);
if (type == MIX_SECTION_INFO) {
while (nr_pages--)
put_page_bootmem(page++);
} else {
free_reserved_pages(page, nr_pages);
}
#else
free_reserved_pages(page, nr_pages);
#endif
} else {
if (PageReserved(page))
free_reserved_page(page);
else
pagetable_free(page_ptdesc(page));
}
}
static void __meminit free_vmemmap_pages(struct page *page, unsigned int order,
@@ -1033,19 +1012,12 @@ static void __meminit free_vmemmap_pages(struct page *page, unsigned int order,
{
unsigned long nr_pages = 1u << order;
if (altmap) {
if (altmap)
vmem_altmap_free(altmap, nr_pages);
} else if (PageReserved(page)) {
if (IS_ENABLED(CONFIG_HAVE_BOOTMEM_INFO_NODE) &&
bootmem_type(page) == SECTION_INFO) {
while (nr_pages--)
put_page_bootmem(page++);
} else {
free_reserved_pages(page, nr_pages);
}
} else {
else if (PageReserved(page))
free_reserved_pages(page, order);
else
__free_pages(page, order);
}
}
static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
@@ -1060,7 +1032,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
}
/* free a pte table */
free_pagetable(pmd_page(*pmd), 0);
free_pagetable(pmd_page(*pmd));
spin_lock(&init_mm.page_table_lock);
pmd_clear(pmd);
spin_unlock(&init_mm.page_table_lock);
@@ -1078,7 +1050,7 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud)
}
/* free a pmd table */
free_pagetable(pud_page(*pud), 0);
free_pagetable(pud_page(*pud));
spin_lock(&init_mm.page_table_lock);
pud_clear(pud);
spin_unlock(&init_mm.page_table_lock);
@@ -1096,7 +1068,7 @@ static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
}
/* free a pud table */
free_pagetable(p4d_page(*p4d), 0);
free_pagetable(p4d_page(*p4d));
spin_lock(&init_mm.page_table_lock);
p4d_clear(p4d);
spin_unlock(&init_mm.page_table_lock);
@@ -1313,16 +1285,6 @@ void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
static struct kcore_list kcore_vsyscall;
static void __init register_page_bootmem_info(void)
{
#if defined(CONFIG_NUMA) || defined(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)
int i;
for_each_online_node(i)
register_page_bootmem_info_node(NODE_DATA(i));
#endif
}
/*
* Pre-allocates page-table pages for the vmalloc area in the kernel page-table.
* Only the level which needs to be synchronized between all page-tables is
@@ -1385,14 +1347,6 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
/*
* Must be done after boot memory is put on freelist, because here we
* might set fields in deferred struct pages that have not yet been
* initialized, and memblock_free_all() initializes all the reserved
* deferred pages for us.
*/
register_page_bootmem_info();
/* Register memory areas for /proc/kcore */
if (get_gate_vma(&init_mm))
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER);
@@ -1591,72 +1545,6 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
return err;
}
#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE
void register_page_bootmem_memmap(unsigned long section_nr,
struct page *start_page, unsigned long nr_pages)
{
unsigned long addr = (unsigned long)start_page;
unsigned long end = (unsigned long)(start_page + nr_pages);
unsigned long next;
pgd_t *pgd;
p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
unsigned int nr_pmd_pages;
struct page *page;
for (; addr < end; addr = next) {
pte_t *pte = NULL;
pgd = pgd_offset_k(addr);
if (pgd_none(*pgd)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
continue;
}
get_page_bootmem(section_nr, pgd_page(*pgd), MIX_SECTION_INFO);
p4d = p4d_offset(pgd, addr);
if (p4d_none(*p4d)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
continue;
}
get_page_bootmem(section_nr, p4d_page(*p4d), MIX_SECTION_INFO);
pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
continue;
}
get_page_bootmem(section_nr, pud_page(*pud), MIX_SECTION_INFO);
pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
continue;
}
if (!boot_cpu_has(X86_FEATURE_PSE) || !pmd_leaf(*pmd)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
get_page_bootmem(section_nr, pmd_page(*pmd),
MIX_SECTION_INFO);
pte = pte_offset_kernel(pmd, addr);
if (pte_none(*pte))
continue;
get_page_bootmem(section_nr, pte_page(*pte),
SECTION_INFO);
} else {
next = pmd_addr_end(addr, end);
nr_pmd_pages = (next - addr) >> PAGE_SHIFT;
page = pmd_page(*pmd);
while (nr_pmd_pages--)
get_page_bootmem(section_nr, page++,
SECTION_INFO);
}
}
}
#endif
void __meminit vmemmap_populate_print_last(void)
{
if (p_start) {

2
arch/x86/virt/hw.c
View File

@@ -196,7 +196,7 @@ static __init int __x86_vmx_init(void)
struct page *page;
struct vmcs *vmcs;
page = __alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
page = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
if (WARN_ON_ONCE(!page)) {
x86_vmx_exit();
return -ENOMEM;

23
drivers/accel/amdxdna/aie2_ctx.c
View File

@@ -1048,7 +1048,7 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
bool found;
int ret;
timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
again:
found = false;
down_write(&xdna->notifier_lock);
@@ -1089,24 +1089,9 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
return -EFAULT;
}
mapp->range.notifier_seq = mmu_interval_read_begin(&mapp->notifier);
mmap_read_lock(mm);
ret = hmm_range_fault(&mapp->range);
mmap_read_unlock(mm);
if (ret) {
if (time_after(jiffies, timeout)) {
ret = -ETIME;
goto put_mm;
}
if (ret == -EBUSY) {
amdxdna_umap_put(mapp);
mmput(mm);
goto again;
}
ret = hmm_range_fault_unlocked_timeout(&mapp->range, timeout);
if (ret)
goto put_mm;
}
down_write(&xdna->notifier_lock);
if (mmu_interval_read_retry(&mapp->notifier, mapp->range.notifier_seq)) {
@@ -1124,7 +1109,7 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
put_mm:
amdxdna_umap_put(mapp);
mmput(mm);
return ret;
return ret == -EBUSY ? -ETIME : ret;
}
int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq)

9
drivers/base/memory.c
View File

@@ -46,6 +46,15 @@ int mhp_online_type_from_str(const char *str)
}
return -EINVAL;
}
EXPORT_SYMBOL_GPL(mhp_online_type_from_str);
const char *mhp_online_type_to_str(int online_type)
{
if (online_type < 0 || online_type >= (int)ARRAY_SIZE(online_type_to_str))
return NULL;
return online_type_to_str[online_type];
}
EXPORT_SYMBOL_GPL(mhp_online_type_to_str);
#define to_memory_block(dev) container_of(dev, struct memory_block, dev)

26
drivers/block/zram/zram_drv.c
View File

@@ -57,15 +57,6 @@ static size_t huge_class_size;
static const struct block_device_operations zram_devops;
static void slot_free(struct zram *zram, u32 index);
#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map)
static void slot_lock_init(struct zram *zram, u32 index)
{
static struct lock_class_key __key;
lockdep_init_map(slot_dep_map(zram, index), "zram->table[index].lock",
&__key, 0);
}
/*
* entry locking rules:
@@ -84,8 +75,8 @@ static __must_check bool slot_trylock(struct zram *zram, u32 index)
unsigned long *lock = &zram->table[index].__lock;
if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) {
mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_);
lock_acquired(slot_dep_map(zram, index), _RET_IP_);
mutex_acquire(&zram->table_lock_map, 0, 1, _RET_IP_);
lock_acquired(&zram->table_lock_map, _RET_IP_);
return true;
}
@@ -96,16 +87,16 @@ static void slot_lock(struct zram *zram, u32 index)
{
unsigned long *lock = &zram->table[index].__lock;
mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_);
mutex_acquire(&zram->table_lock_map, 0, 0, _RET_IP_);
wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE);
lock_acquired(slot_dep_map(zram, index), _RET_IP_);
lock_acquired(&zram->table_lock_map, _RET_IP_);
}
static void slot_unlock(struct zram *zram, u32 index)
{
unsigned long *lock = &zram->table[index].__lock;
mutex_release(slot_dep_map(zram, index), _RET_IP_);
mutex_release(&zram->table_lock_map, _RET_IP_);
clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock);
}
@@ -1980,11 +1971,12 @@ static void zram_meta_free(struct zram *zram, u64 disksize)
zs_destroy_pool(zram->mem_pool);
vfree(zram->table);
zram->table = NULL;
lockdep_unregister_key(&zram->table_lock_key);
}
static bool zram_meta_alloc(struct zram *zram, u64 disksize)
{
size_t num_pages, index;
size_t num_pages;
num_pages = disksize >> PAGE_SHIFT;
zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table)));
@@ -2001,8 +1993,8 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)
if (!huge_class_size)
huge_class_size = zs_huge_class_size(zram->mem_pool);
for (index = 0; index < num_pages; index++)
slot_lock_init(zram, index);
lockdep_register_key(&zram->table_lock_key);
lockdep_init_map(&zram->table_lock_map, "zram->table[index].lock", &zram->table_lock_key, 0);
return true;
}

3
drivers/block/zram/zram_drv.h
View File

@@ -74,7 +74,6 @@ struct zram_table_entry {
#endif
} attr;
};
struct lockdep_map dep_map;
};
struct zram_stats {
@@ -107,6 +106,8 @@ struct zram_stats {
struct zram {
struct zram_table_entry *table;
struct lockdep_map table_lock_map;
struct lock_class_key table_lock_key;
struct zs_pool *mem_pool;
struct zcomp *comps[ZRAM_MAX_COMPS];
struct zcomp_params params[ZRAM_MAX_COMPS];

2
drivers/dax/bus.h
View File

@@ -16,6 +16,8 @@ struct dax_region;
#define IORESOURCE_DAX_STATIC BIT(0)
#define IORESOURCE_DAX_KMEM BIT(1)
#define DAX_KMEM_UNPLUGGED (-1) /* Do not create memory blocks */
struct dax_region *alloc_dax_region(struct device *parent, int region_id,
struct range *range, int target_node, unsigned int align,
unsigned long flags);

506
drivers/dax/kmem.c
View File

@@ -33,9 +33,7 @@ static int dax_kmem_range(struct dev_dax *dev_dax, int i, struct range *r)
struct dev_dax_range *dax_range = &dev_dax->ranges[i];
struct range *range = &dax_range->range;
/* memory-block align the hotplug range */
r->start = ALIGN(range->start, memory_block_size_bytes());
r->end = ALIGN_DOWN(range->end + 1, memory_block_size_bytes()) - 1;
*r = memory_block_aligned_range(range);
if (r->start >= r->end) {
r->start = range->start;
r->end = range->end;
@@ -47,6 +45,8 @@ static int dax_kmem_range(struct dev_dax *dev_dax, int i, struct range *r)
struct dax_kmem_data {
const char *res_name;
int mgid;
int state;
struct mutex lock; /* protects hotplug state transitions */
struct resource *res[];
};
@@ -65,16 +65,331 @@ static void kmem_put_memory_types(void)
mt_put_memory_types(&kmem_memory_types);
}
/* True for the online states a kmem dax device can hold. */
static bool dax_kmem_state_is_online(int state)
{
return state == MMOP_ONLINE ||
state == MMOP_ONLINE_KERNEL ||
state == MMOP_ONLINE_MOVABLE;
}
/**
* dax_kmem_do_hotplug - hotplug memory for dax kmem device
* @dev_dax: the dev_dax instance
* @data: the dax_kmem_data structure with resource tracking
* @online_type: the online policy to use for the memory blocks
*
* Hotplugs all ranges in the dev_dax region as system memory with the
* provided online policy (offline, online, online_movable, online_kernel).
*
* Returns the number of successfully mapped ranges, or negative error.
*/
static int dax_kmem_do_hotplug(struct dev_dax *dev_dax,
struct dax_kmem_data *data,
int online_type)
{
struct device *dev = &dev_dax->dev;
int i, rc, added = 0;
mhp_t mhp_flags;
if (dax_kmem_state_is_online(data->state))
return -EINVAL;
if (online_type < MMOP_OFFLINE || online_type > MMOP_ONLINE_MOVABLE)
return -EINVAL;
for (i = 0; i < dev_dax->nr_range; i++) {
struct range range;
rc = dax_kmem_range(dev_dax, i, &range);
if (rc)
continue;
/*
* init_resources() is best-effort: if a reservation conflict
* occurs it keeps the range but leaves res[i]=NULL. For hotplug
* on probe systems, this means kmem will partially online.
*
* We have to keep this behavior not to break those systems.
* For those systems - atomicity only applies to valid ranges.
*/
if (!data->res[i])
continue;
mhp_flags = MHP_NID_IS_MGID;
if (dev_dax->memmap_on_memory)
mhp_flags |= MHP_MEMMAP_ON_MEMORY;
/*
* Ensure that future kexec'd kernels will not treat
* this as RAM automatically.
*/
rc = __add_memory_driver_managed(data->mgid, range.start,
range_len(&range), kmem_name, mhp_flags,
online_type);
if (rc) {
dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
i, range.start, range.end);
/*
* Release the reservation for the range that failed to
* add so a later hotremove does not try to remove memory
* that was never added.
*/
if (data->res[i]) {
remove_resource(data->res[i]);
kfree(data->res[i]);
data->res[i] = NULL;
}
if (added)
continue;
return rc;
}
added++;
}
return added;
}
/**
* dax_kmem_init_resources - create memory regions for dax kmem
* @dev_dax: the dev_dax instance
* @data: the dax_kmem_data structure with resource tracking
*
* Initializes all the resources for the DAX
*
* Returns the number of successfully mapped ranges, or negative error.
*/
static int dax_kmem_init_resources(struct dev_dax *dev_dax,
struct dax_kmem_data *data)
{
struct device *dev = &dev_dax->dev;
int i, rc, mapped = 0;
for (i = 0; i < dev_dax->nr_range; i++) {
struct resource *res;
struct range range;
rc = dax_kmem_range(dev_dax, i, &range);
if (rc)
continue;
/* Skip ranges already added */
if (data->res[i])
continue;
/* Region is permanently reserved if hotremove fails. */
res = request_mem_region(range.start, range_len(&range),
data->res_name);
if (!res) {
dev_warn(dev, "mapping%d: %#llx-%#llx could not reserve region\n",
i, range.start, range.end);
/*
* Once some memory has been onlined we can't
* assume that it can be un-onlined safely.
*/
if (mapped)
continue;
return -EBUSY;
}
data->res[i] = res;
/*
* Set flags appropriate for System RAM. Leave ..._BUSY clear
* so that add_memory() can add a child resource. Do not
* inherit flags from the parent since it may set new flags
* unknown to us that will break add_memory() later.
*/
res->flags = IORESOURCE_SYSTEM_RAM;
mapped++;
}
return mapped;
}
#ifdef CONFIG_MEMORY_HOTREMOVE
/**
* dax_kmem_do_hotremove - hot-remove memory for dax kmem device
* @dev_dax: the dev_dax instance
* @data: the dax_kmem_data structure with resource tracking
*
* Offlines and removes every currently-added range in the dev_dax region
* atomically: either all ranges are offlined and removed, or none are and
* the device is returned to its prior state.
*
* Returns 0 on success, or a negative errno on failure.
*/
static int dax_kmem_do_hotremove(struct dev_dax *dev_dax,
struct dax_kmem_data *data)
{
struct device *dev = &dev_dax->dev;
struct range *ranges;
int i, nr_ranges = 0, rc;
ranges = kmalloc_objs(*ranges, dev_dax->nr_range);
if (!ranges)
return -ENOMEM;
/* Collect the ranges that were actually added during probe. */
for (i = 0; i < dev_dax->nr_range; i++) {
struct range range;
if (!data->res[i])
continue;
if (dax_kmem_range(dev_dax, i, &range))
continue;
ranges[nr_ranges++] = range;
}
/* Nothing added means nothing to remove. */
if (!nr_ranges) {
kfree(ranges);
return 0;
}
rc = offline_and_remove_memory_ranges(ranges, nr_ranges);
kfree(ranges);
if (rc) {
/* Recoverable: the ranges rolled back, nothing is leaked yet. */
dev_err(dev, "hotremove failed, device left online: %d\n", rc);
return rc;
}
/* All ranges removed; release the reserved resources. */
for (i = 0; i < dev_dax->nr_range; i++) {
if (!data->res[i])
continue;
remove_resource(data->res[i]);
kfree(data->res[i]);
data->res[i] = NULL;
}
return 0;
}
#else
static int dax_kmem_do_hotremove(struct dev_dax *dev_dax,
struct dax_kmem_data *data)
{
return -EBUSY;
}
#endif /* CONFIG_MEMORY_HOTREMOVE */
/**
* dax_kmem_cleanup_resources - remove the dax memory resources
* @dev_dax: the dev_dax instance
* @data: the dax_kmem_data structure with resource tracking
*
* Removes all resources in the dev_dax region.
*/
static void dax_kmem_cleanup_resources(struct dev_dax *dev_dax,
struct dax_kmem_data *data)
{
int i;
/*
* If the device unbind occurs before memory is hotremoved, we can never
* remove the memory (requires reboot). Attempting an offline operation
* here may cause deadlock and a failure to finish the unbind.
*
* Note: This leaks the resources.
*/
if (WARN(((data->state != DAX_KMEM_UNPLUGGED) &&
(data->state != MMOP_OFFLINE)),
"Hotplug memory regions stuck online until reboot"))
return;
for (i = 0; i < dev_dax->nr_range; i++) {
if (!data->res[i])
continue;
remove_resource(data->res[i]);
kfree(data->res[i]);
data->res[i] = NULL;
}
}
static int dax_kmem_parse_state(const char *buf)
{
int online_type;
/* "unplugged" is kmem-specific - the rest map to MMOP_ */
if (sysfs_streq(buf, "unplugged"))
return DAX_KMEM_UNPLUGGED;
online_type = mhp_online_type_from_str(buf);
/* Disallow "offline": it's not useful and creates race conditions */
if (online_type == MMOP_OFFLINE)
return -EINVAL;
return online_type;
}
static ssize_t state_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dax_kmem_data *data = dev_get_drvdata(dev);
const char *state_str;
if (data->state == DAX_KMEM_UNPLUGGED)
state_str = "unplugged";
else
state_str = mhp_online_type_to_str(data->state);
return sysfs_emit(buf, "%s\n", state_str ?: "unknown");
}
static ssize_t state_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t len)
{
struct dev_dax *dev_dax = to_dev_dax(dev);
struct dax_kmem_data *data = dev_get_drvdata(dev);
int online_type;
int rc;
online_type = dax_kmem_parse_state(buf);
if (online_type < DAX_KMEM_UNPLUGGED)
return online_type;
guard(mutex)(&data->lock);
/* Already in requested state */
if (data->state == online_type)
return len;
if (online_type == DAX_KMEM_UNPLUGGED) {
rc = dax_kmem_do_hotremove(dev_dax, data);
if (rc)
return rc;
data->state = DAX_KMEM_UNPLUGGED;
return len;
}
/* Onlining is only allowed from the unplugged state. */
if (data->state != DAX_KMEM_UNPLUGGED)
return -EBUSY;
/* Re-acquire resources if previously unplugged, otherwise no-op */
rc = dax_kmem_init_resources(dev_dax, data);
if (rc < 0)
return rc;
rc = dax_kmem_do_hotplug(dev_dax, data, online_type);
if (rc < 0) {
/* Total failure, drop the reservations we took. */
dax_kmem_cleanup_resources(dev_dax, data);
return rc;
}
data->state = online_type;
return len;
}
static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
{
struct device *dev = &dev_dax->dev;
unsigned long total_len = 0, orig_len = 0;
struct dax_kmem_data *data;
struct memory_dev_type *mtype;
int i, rc, mapped = 0;
mhp_t mhp_flags;
int i, rc;
int numa_node;
int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
int online_type = mhp_get_default_online_type();
/*
* Ensure good NUMA information for the persistent memory.
@@ -133,68 +448,26 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
if (rc < 0)
goto err_reg_mgid;
data->mgid = rc;
for (i = 0; i < dev_dax->nr_range; i++) {
struct resource *res;
struct range range;
rc = dax_kmem_range(dev_dax, i, &range);
if (rc)
continue;
/* Region is permanently reserved if hotremove fails. */
res = request_mem_region(range.start, range_len(&range), data->res_name);
if (!res) {
dev_warn(dev, "mapping%d: %#llx-%#llx could not reserve region\n",
i, range.start, range.end);
/*
* Once some memory has been onlined we can't
* assume that it can be un-onlined safely.
*/
if (mapped)
continue;
rc = -EBUSY;
goto err_request_mem;
}
data->res[i] = res;
/*
* Set flags appropriate for System RAM. Leave ..._BUSY clear
* so that add_memory() can add a child resource. Do not
* inherit flags from the parent since it may set new flags
* unknown to us that will break add_memory() below.
*/
res->flags = IORESOURCE_SYSTEM_RAM;
mhp_flags = MHP_NID_IS_MGID;
if (dev_dax->memmap_on_memory)
mhp_flags |= MHP_MEMMAP_ON_MEMORY;
/*
* Ensure that future kexec'd kernels will not treat
* this as RAM automatically.
*/
rc = add_memory_driver_managed(data->mgid, range.start,
range_len(&range), kmem_name, mhp_flags);
if (rc) {
dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
i, range.start, range.end);
remove_resource(res);
kfree(res);
data->res[i] = NULL;
if (mapped)
continue;
goto err_request_mem;
}
mapped++;
}
data->state = DAX_KMEM_UNPLUGGED;
mutex_init(&data->lock);
dev_set_drvdata(dev, data);
rc = dax_kmem_init_resources(dev_dax, data);
if (rc < 0)
goto err_resources;
rc = dax_kmem_do_hotplug(dev_dax, data, online_type);
if (rc < 0)
goto err_hotplug;
data->state = online_type;
return 0;
err_request_mem:
err_hotplug:
dax_kmem_cleanup_resources(dev_dax, data);
err_resources:
dev_set_drvdata(dev, NULL);
memory_group_unregister(data->mgid);
err_reg_mgid:
kfree(data->res_name);
@@ -206,61 +479,71 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
}
#ifdef CONFIG_MEMORY_HOTREMOVE
/*
* Remove the device's added ranges with remove_memory().
* Unlike the sysfs unplug path it never offlines and fails if the blocks are
* online (-EBUSY), so it is safe from unbind. Failures leak until reboot.
*
* Returns 0 only if every added range was removed.
*/
static int dax_kmem_remove_ranges(struct dev_dax *dev_dax,
struct dax_kmem_data *data)
{
struct device *dev = &dev_dax->dev;
int i, rc = 0;
for (i = 0; i < dev_dax->nr_range; i++) {
struct range range;
if (!data->res[i] || dax_kmem_range(dev_dax, i, &range))
continue;
if (remove_memory(range.start, range_len(&range))) {
dev_warn(dev, "mapping%d: %#llx-%#llx stuck online until reboot\n",
i, range.start, range.end);
rc = -EBUSY;
continue;
}
remove_resource(data->res[i]);
kfree(data->res[i]);
data->res[i] = NULL;
}
return rc;
}
static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
{
int i, success = 0;
int node = dev_dax->target_node;
struct device *dev = &dev_dax->dev;
struct dax_kmem_data *data = dev_get_drvdata(dev);
/*
* We have one shot for removing memory, if some memory blocks were not
* offline prior to calling this function remove_memory() will fail, and
* there is no way to hotremove this memory until reboot because device
* unbind will succeed even if we return failure.
* Remove every range that is still added. dax_kmem_remove_ranges()
* uses remove_memory(), which never offlines: an online block fails
* with -EBUSY rather than deadlocking an uninterruptible unbind.
*
* data->state only tracks daxX.Y/state writes, so it can be stale if
* blocks were toggled via memoryX/state. Do not trust it here and
* attempt simply remove_memory() - which reports the true state of
* each range anyway. Anything left online is leaked until reboot.
*/
for (i = 0; i < dev_dax->nr_range; i++) {
struct range range;
int rc;
rc = dax_kmem_range(dev_dax, i, &range);
if (rc)
continue;
/* range was never added during probe */
if (!data->res[i]) {
success++;
continue;
}
rc = remove_memory(range.start, range_len(&range));
if (rc == 0) {
remove_resource(data->res[i]);
kfree(data->res[i]);
data->res[i] = NULL;
success++;
continue;
}
if (dax_kmem_remove_ranges(dev_dax, data)) {
dev_err(dev, "Hotplug regions stuck online until reboot\n");
any_hotremove_failed = true;
dev_err(dev,
"mapping%d: %#llx-%#llx cannot be hotremoved until the next reboot\n",
i, range.start, range.end);
return;
}
if (success >= dev_dax->nr_range) {
memory_group_unregister(data->mgid);
kfree(data->res_name);
kfree(data);
dev_set_drvdata(dev, NULL);
/*
* Clear the memtype association on successful unplug.
* If not, we have memory blocks left which can be
* offlined/onlined later. We need to keep memory_dev_type
* for that. This implies this reference will be around
* till next reboot.
*/
clear_node_memory_type(node, NULL);
}
memory_group_unregister(data->mgid);
kfree(data->res_name);
kfree(data);
dev_set_drvdata(dev, NULL);
/*
* Clear the memtype association on successful unplug.
* If not, we have memory blocks left which can be
* offlined/onlined later. We need to keep memory_dev_type
* for that. This implies this reference will be around
* till next reboot.
*/
clear_node_memory_type(node, NULL);
}
#else
static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
@@ -276,10 +559,21 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
}
#endif /* CONFIG_MEMORY_HOTREMOVE */
static DEVICE_ATTR_RW(state);
static struct attribute *dev_dax_kmem_attrs[] = {
&dev_attr_state.attr,
NULL,
};
ATTRIBUTE_GROUPS(dev_dax_kmem);
static struct dax_device_driver device_dax_kmem_driver = {
.probe = dev_dax_kmem_probe,
.remove = dev_dax_kmem_remove,
.type = DAXDRV_KMEM_TYPE,
.drv = {
.dev_groups = dev_dax_kmem_groups,
},
};
static int __init dax_kmem_init(void)

2
drivers/gpu/drm/drm_gem.c
View File

@@ -1252,7 +1252,7 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size,
}
vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_writecombine(vma_get_page_prot(vma));
vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
}

2
drivers/gpu/drm/drm_gem_dma_helper.c
View File

@@ -540,7 +540,7 @@ int drm_gem_dma_mmap(struct drm_gem_dma_object *dma_obj, struct vm_area_struct *
vm_flags_mod(vma, VM_DONTDUMP | VM_DONTEXPAND, VM_PFNMAP);
if (dma_obj->map_noncoherent) {
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
ret = dma_mmap_pages(drm_dev_dma_dev(dma_obj->base.dev),
vma, vma->vm_end - vma->vm_start,

2
drivers/gpu/drm/drm_gem_shmem_helper.c
View File

@@ -768,7 +768,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
return ret;
vm_flags_set(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
if (shmem->map_wc)
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);

60
drivers/gpu/drm/drm_gpusvm.c
View File

@@ -773,8 +773,7 @@ enum drm_gpusvm_scan_result drm_gpusvm_scan_mm(struct drm_gpusvm_range *range,
.end = end,
.dev_private_owner = dev_private_owner,
};
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
enum drm_gpusvm_scan_result state = DRM_GPUSVM_SCAN_UNPOPULATED, new_state;
unsigned long *pfns;
unsigned long npages = npages_in_range(start, end);
@@ -788,22 +787,7 @@ enum drm_gpusvm_scan_result drm_gpusvm_scan_mm(struct drm_gpusvm_range *range,
hmm_range.hmm_pfns = pfns;
retry:
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
mmap_read_lock(range->gpusvm->mm);
while (true) {
err = hmm_range_fault(&hmm_range);
if (err == -EBUSY) {
if (time_after(jiffies, timeout))
break;
hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);
continue;
}
break;
}
mmap_read_unlock(range->gpusvm->mm);
err = hmm_range_fault_unlocked_timeout(&hmm_range, timeout);
if (err)
goto err_free;
@@ -1423,6 +1407,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
void *zdd;
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long remaining;
unsigned long i, j;
unsigned long npages = npages_in_range(pages_start, pages_end);
unsigned long num_dma_mapped;
@@ -1437,7 +1422,9 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
struct dma_iova_state *state = &svm_pages->state;
retry:
if (time_after(jiffies, timeout))
remaining = timeout - jiffies;
if (time_after_eq(jiffies, timeout))
return -EBUSY;
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
@@ -1454,21 +1441,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
}
hmm_range.hmm_pfns = pfns;
while (true) {
mmap_read_lock(mm);
err = hmm_range_fault(&hmm_range);
mmap_read_unlock(mm);
if (err == -EBUSY) {
if (time_after(jiffies, timeout))
break;
hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);
continue;
}
break;
}
err = hmm_range_fault_unlocked_timeout(&hmm_range, remaining);
mmput(mm);
if (err)
goto err_free;
@@ -1741,8 +1714,7 @@ int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
.end = drm_gpusvm_range_end(range),
.dev_private_owner = NULL,
};
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long *pfns;
unsigned long npages = npages_in_range(drm_gpusvm_range_start(range),
drm_gpusvm_range_end(range));
@@ -1759,24 +1731,12 @@ int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
}
hmm_range.hmm_pfns = pfns;
while (!time_after(jiffies, timeout)) {
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
if (time_after(jiffies, timeout)) {
err = -ETIME;
break;
}
mmap_read_lock(mm);
err = hmm_range_fault(&hmm_range);
mmap_read_unlock(mm);
if (err != -EBUSY)
break;
}
err = hmm_range_fault_unlocked_timeout(&hmm_range, timeout);
kvfree(pfns);
mmput(mm);
return err;
return err == -EBUSY ? -ETIME : err;
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);

5
drivers/gpu/drm/etnaviv/etnaviv_gem.c
View File

@@ -6,6 +6,7 @@
#include <drm/drm_prime.h>
#include <drm/drm_print.h>
#include <linux/dma-mapping.h>
#include <linux/pagemap.h>
#include <linux/shmem_fs.h>
#include <linux/spinlock.h>
#include <linux/vmalloc.h>
@@ -132,7 +133,7 @@ static int etnaviv_gem_mmap_obj(struct etnaviv_gem_object *etnaviv_obj,
vm_flags_set(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
vm_page_prot = vm_get_page_prot(vma->vm_flags);
vm_page_prot = vma_get_page_prot(vma);
if (etnaviv_obj->flags & ETNA_BO_WC) {
vma->vm_page_prot = pgprot_writecombine(vm_page_prot);
@@ -188,7 +189,7 @@ static vm_fault_t etnaviv_gem_fault(struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
pgoff = linear_page_delta(vma, vmf->address);
pfn = page_to_pfn(pages[pgoff]);

6
drivers/gpu/drm/exynos/exynos_drm_gem.c
View File

@@ -377,13 +377,13 @@ static int exynos_drm_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct
/* non-cachable as default. */
if (exynos_gem->flags & EXYNOS_BO_CACHABLE)
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
else if (exynos_gem->flags & EXYNOS_BO_WC)
vma->vm_page_prot =
pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
pgprot_writecombine(vma_get_page_prot(vma));
else
vma->vm_page_prot =
pgprot_noncached(vm_get_page_prot(vma->vm_flags));
pgprot_noncached(vma_get_page_prot(vma));
ret = exynos_drm_gem_mmap_buffer(exynos_gem, vma);
if (ret)

2
drivers/gpu/drm/gma500/gem.c
View File

@@ -288,7 +288,7 @@ static vm_fault_t psb_gem_fault(struct vm_fault *vmf)
/* Page relative to the VMA start - we must calculate this ourselves
because vmf->pgoff is the fake GEM offset */
page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
page_offset = linear_page_delta(vma, vmf->address);
/* CPU view of the page, don't go via the GART for CPU writes */
if (pobj->stolen)

12
drivers/gpu/drm/i915/gem/i915_gem_mman.c
View File

@@ -112,7 +112,7 @@ i915_gem_mmap_ioctl(struct drm_device *dev, void *data,
vma = find_vma(mm, addr);
if (vma && __vma_matches(vma, obj->base.filp, addr, args->size))
vma->vm_page_prot =
pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
pgprot_writecombine(vma_get_page_prot(vma));
else
addr = -ENOMEM;
mmap_write_unlock(mm);
@@ -1024,7 +1024,7 @@ i915_gem_object_mmap(struct drm_i915_gem_object *obj,
fput(anon);
if (obj->ops->mmap_ops) {
vma->vm_page_prot = pgprot_decrypted(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_decrypted(vma_get_page_prot(vma));
vma->vm_ops = obj->ops->mmap_ops;
vma->vm_private_data = obj->base.vma_node.driver_private;
return 0;
@@ -1035,7 +1035,7 @@ i915_gem_object_mmap(struct drm_i915_gem_object *obj,
switch (mmo->mmap_type) {
case I915_MMAP_TYPE_WC:
vma->vm_page_prot =
pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
pgprot_writecombine(vma_get_page_prot(vma));
vma->vm_ops = &vm_ops_cpu;
break;
@@ -1043,19 +1043,19 @@ i915_gem_object_mmap(struct drm_i915_gem_object *obj,
GEM_WARN_ON(1);
fallthrough;
case I915_MMAP_TYPE_WB:
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
vma->vm_ops = &vm_ops_cpu;
break;
case I915_MMAP_TYPE_UC:
vma->vm_page_prot =
pgprot_noncached(vm_get_page_prot(vma->vm_flags));
pgprot_noncached(vma_get_page_prot(vma));
vma->vm_ops = &vm_ops_cpu;
break;
case I915_MMAP_TYPE_GTT:
vma->vm_page_prot =
pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
pgprot_writecombine(vma_get_page_prot(vma));
vma->vm_ops = &vm_ops_gtt;
break;
}

5
drivers/gpu/drm/msm/msm_gem.c
View File

@@ -9,6 +9,7 @@
#include <linux/spinlock.h>
#include <linux/shmem_fs.h>
#include <linux/dma-buf.h>
#include <linux/pagemap.h>
#include <drm/drm_dumb_buffers.h>
#include <drm/drm_prime.h>
@@ -360,7 +361,7 @@ static vm_fault_t msm_gem_fault(struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
pgoff = linear_page_delta(vma, vmf->address);
pfn = page_to_pfn(pages[pgoff]);
@@ -1124,7 +1125,7 @@ static int msm_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struct
struct msm_gem_object *msm_obj = to_msm_bo(obj);
vm_flags_set(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
vma->vm_page_prot = msm_gem_pgprot(msm_obj, vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = msm_gem_pgprot(msm_obj, vma_get_page_prot(vma));
return 0;
}

2
drivers/gpu/drm/nouveau/nouveau_gem.c
View File

@@ -55,7 +55,7 @@ static vm_fault_t nouveau_ttm_fault(struct vm_fault *vmf)
goto error_unlock;
nouveau_bo_del_io_reserve_lru(bo);
prot = vm_get_page_prot(vma->vm_flags);
prot = vma_get_page_prot(vma);
ret = ttm_bo_vm_fault_reserved(vmf, prot, TTM_BO_VM_NUM_PREFAULT);
nouveau_bo_add_io_reserve_lru(bo);
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))

20
drivers/gpu/drm/nouveau/nouveau_svm.c
View File

@@ -678,20 +678,22 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
range.end = notifier->notifier.interval_tree.last + 1;
while (true) {
if (time_after(jiffies, timeout)) {
long remaining = timeout - jiffies;
/*
* The HMM timeout only bounds retries while HMM is walking and
* faulting the range. This fault is handled by a kernel worker,
* so fatal signals from the faulting process cannot stop an
* endless stream of invalidations here.
*/
if (time_after_eq(jiffies, timeout)) {
ret = -EBUSY;
goto out;
}
range.notifier_seq = mmu_interval_read_begin(range.notifier);
mmap_read_lock(mm);
ret = hmm_range_fault(&range);
mmap_read_unlock(mm);
if (ret) {
if (ret == -EBUSY)
continue;
ret = hmm_range_fault_unlocked_timeout(&range, remaining);
if (ret)
goto out;
}
mutex_lock(&svmm->mutex);
if (mmu_interval_read_retry(range.notifier,

2
drivers/gpu/drm/omapdrm/omap_fbdev.c
View File

@@ -84,7 +84,7 @@ static int omap_fbdev_pan_display(struct fb_var_screeninfo *var, struct fb_info
static int omap_fbdev_fb_mmap(struct fb_info *info, struct vm_area_struct *vma)
{
vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_writecombine(vma_get_page_prot(vma));
return fb_deferred_io_mmap(info, vma);
}

11
drivers/gpu/drm/omapdrm/omap_gem.c
View File

@@ -5,6 +5,7 @@
*/
#include <linux/dma-mapping.h>
#include <linux/pagemap.h>
#include <linux/seq_file.h>
#include <linux/shmem_fs.h>
#include <linux/spinlock.h>
@@ -359,7 +360,7 @@ static vm_fault_t omap_gem_fault_1d(struct drm_gem_object *obj,
pgoff_t pgoff;
/* We don't use vmf->pgoff since that has the fake offset: */
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
pgoff = linear_page_delta(vma, vmf->address);
if (omap_obj->pages) {
omap_gem_cpu_sync_page(obj, pgoff);
@@ -407,7 +408,7 @@ static vm_fault_t omap_gem_fault_2d(struct drm_gem_object *obj,
const int m = DIV_ROUND_UP(omap_obj->width << fmt, PAGE_SIZE);
/* We don't use vmf->pgoff since that has the fake offset: */
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
pgoff = linear_page_delta(vma, vmf->address);
/*
* Actual address we start mapping at is rounded down to previous slot
@@ -537,9 +538,9 @@ static int omap_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struc
vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_IO | VM_MIXEDMAP);
if (omap_obj->flags & OMAP_BO_WC) {
vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_writecombine(vma_get_page_prot(vma));
} else if (omap_obj->flags & OMAP_BO_UNCACHED) {
vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_noncached(vma_get_page_prot(vma));
} else {
/*
* We do have some private objects, at least for scanout buffers
@@ -557,7 +558,7 @@ static int omap_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struc
vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node);
vma_set_file(vma, obj->filp);
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
}
vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);

2
drivers/gpu/drm/panthor/panthor_gem.c
View File

@@ -776,7 +776,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v
}
vm_flags_set(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
if (should_map_wc(bo))
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);

2
drivers/gpu/drm/rockchip/rockchip_drm_gem.c
View File

@@ -255,7 +255,7 @@ static int rockchip_drm_gem_object_mmap(struct drm_gem_object *obj,
*/
vm_flags_mod(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP, VM_PFNMAP);
vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_writecombine(vma_get_page_prot(vma));
vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
if (rk_obj->pages)

5
drivers/gpu/drm/tegra/gem.c
View File

@@ -13,6 +13,7 @@
#include <linux/dma-buf.h>
#include <linux/iommu.h>
#include <linux/module.h>
#include <linux/pagemap.h>
#include <linux/vmalloc.h>
#include <drm/drm_drv.h>
@@ -564,7 +565,7 @@ static vm_fault_t tegra_bo_fault(struct vm_fault *vmf)
if (!bo->pages)
return VM_FAULT_SIGBUS;
offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
offset = linear_page_delta(vma, vmf->address);
page = bo->pages[offset];
return vmf_insert_page(vma, vmf->address, page);
@@ -601,7 +602,7 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma)
vma->vm_pgoff = vm_pgoff;
} else {
pgprot_t prot = vm_get_page_prot(vma->vm_flags);
pgprot_t prot = vma_get_page_prot(vma);
vm_flags_mod(vma, VM_MIXEDMAP, VM_PFNMAP);

7
drivers/gpu/drm/ttm/ttm_bo_vm.c
View File

@@ -32,6 +32,7 @@
#define pr_fmt(fmt) "[TTM] " fmt
#include <linux/export.h>
#include <linux/pagemap.h>
#include <drm/ttm/ttm_bo.h>
#include <drm/ttm/ttm_placement.h>
@@ -208,9 +209,9 @@ vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
if (unlikely(err != 0))
return VM_FAULT_SIGBUS;
page_offset = ((address - vma->vm_start) >> PAGE_SHIFT) +
vma->vm_pgoff - drm_vma_node_start(&bo->base.vma_node);
page_last = vma_pages(vma) + vma->vm_pgoff -
page_offset = linear_page_index(vma, address) -
drm_vma_node_start(&bo->base.vma_node);
page_last = vma_end_pgoff(vma) -
drm_vma_node_start(&bo->base.vma_node);
if (unlikely(page_offset >= PFN_UP(bo->base.size)))

2
drivers/gpu/drm/virtio/virtgpu_vram.c
View File

@@ -55,7 +55,7 @@ static int virtio_gpu_vram_mmap(struct drm_gem_object *obj,
vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node);
vm_flags_set(vma, VM_MIXEDMAP | VM_DONTEXPAND);
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
vma->vm_ops = &virtio_gpu_vram_vm_ops;

2
drivers/gpu/drm/vmwgfx/vmwgfx_page_dirty.c
View File

@@ -481,7 +481,7 @@ vm_fault_t vmw_bo_vm_fault(struct vm_fault *vmf)
if (vbo->dirty && vbo->dirty->method == VMW_BO_DIRTY_MKWRITE)
prot = vm_get_page_prot(vma->vm_flags & ~VM_SHARED);
else
prot = vm_get_page_prot(vma->vm_flags);
prot = vma_get_page_prot(vma);
ret = ttm_bo_vm_fault_reserved(vmf, prot, num_prefault);
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))

2
drivers/gpu/drm/xe/xe_device.c
View File

@@ -281,7 +281,7 @@ static vm_fault_t barrier_fault(struct vm_fault *vmf)
pgprot_t prot;
int idx;
prot = vm_get_page_prot(vma->vm_flags);
prot = vma_get_page_prot(vma);
if (drm_dev_enter(dev, &idx)) {
unsigned long pfn;

2
drivers/gpu/drm/xe/xe_mmio_gem.c
View File

@@ -149,7 +149,7 @@ static int xe_mmio_gem_mmap(struct drm_gem_object *base, struct vm_area_struct *
/* Set vm_pgoff (used as a fake buffer offset by DRM) to 0 */
vma->vm_pgoff = 0;
vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
vma->vm_page_prot = pgprot_noncached(vma_get_page_prot(vma));
vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP |
VM_DONTCOPY | VM_NORESERVE);

2
drivers/gpu/drm/xen/xen_drm_front_gem.c
View File

@@ -80,7 +80,7 @@ static int xen_drm_front_gem_object_mmap(struct drm_gem_object *gem_obj,
* which is mapped as Normal Inner Write-Back Outer Write-Back
* Inner-Shareable.
*/
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
/*
* vm_operations_struct.fault handler will be called if CPU access

54
drivers/hv/mshv_regions.c
View File

@@ -381,46 +381,6 @@ int mshv_region_get(struct mshv_mem_region *region)
return kref_get_unless_zero(&region->mreg_refcount);
}
/**
* mshv_region_hmm_fault_and_lock - Handle HMM faults and lock the memory region
* @region: Pointer to the memory region structure
* @range: Pointer to the HMM range structure
*
* This function performs the following steps:
* 1. Reads the notifier sequence for the HMM range.
* 2. Acquires a read lock on the memory map.
* 3. Handles HMM faults for the specified range.
* 4. Releases the read lock on the memory map.
* 5. If successful, locks the memory region mutex.
* 6. Verifies if the notifier sequence has changed during the operation.
* If it has, releases the mutex and returns -EBUSY to match with
* hmm_range_fault() return code for repeating.
*
* Return: 0 on success, a negative error code otherwise.
*/
static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
struct hmm_range *range)
{
int ret;
range->notifier_seq = mmu_interval_read_begin(range->notifier);
mmap_read_lock(region->mreg_mni.mm);
ret = hmm_range_fault(range);
mmap_read_unlock(region->mreg_mni.mm);
if (ret)
return ret;
mutex_lock(&region->mreg_mutex);
if (mmu_interval_read_retry(range->notifier, range->notifier_seq)) {
mutex_unlock(&region->mreg_mutex);
cond_resched();
return -EBUSY;
}
return 0;
}
/**
* mshv_region_range_fault - Handle memory range faults for a given region.
* @region: Pointer to the memory region structure.
@@ -452,13 +412,19 @@ static int mshv_region_range_fault(struct mshv_mem_region *region,
range.start = region->start_uaddr + page_offset * HV_HYP_PAGE_SIZE;
range.end = range.start + page_count * HV_HYP_PAGE_SIZE;
do {
ret = mshv_region_hmm_fault_and_lock(region, &range);
} while (ret == -EBUSY);
again:
ret = hmm_range_fault_unlocked_timeout(&range, 0);
if (ret)
goto out;
mutex_lock(&region->mreg_mutex);
if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
mutex_unlock(&region->mreg_mutex);
cond_resched();
goto again;
}
for (i = 0; i < page_count; i++)
region->mreg_pages[page_offset + i] = hmm_pfn_to_page(pfns[i]);

18
drivers/infiniband/core/umem_odp.c
View File

@@ -329,7 +329,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
struct mm_struct *owning_mm = umem_odp->umem.owning_mm;
int pfn_index, dma_index, ret = 0, start_idx;
unsigned int page_shift, hmm_order, pfn_start_idx;
unsigned long num_pfns, current_seq;
unsigned long num_pfns;
struct hmm_range range = {};
unsigned long timeout;
@@ -363,26 +363,18 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
}
range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]);
timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
retry:
current_seq = range.notifier_seq =
mmu_interval_read_begin(&umem_odp->notifier);
mmap_read_lock(owning_mm);
ret = hmm_range_fault(&range);
mmap_read_unlock(owning_mm);
if (unlikely(ret)) {
if (ret == -EBUSY && !time_after(jiffies, timeout))
goto retry;
ret = hmm_range_fault_unlocked_timeout(&range, timeout);
if (unlikely(ret))
goto out_put_mm;
}
start_idx = (range.start - ib_umem_start(umem_odp)) >> page_shift;
dma_index = start_idx;
mutex_lock(&umem_odp->umem_mutex);
if (mmu_interval_read_retry(&umem_odp->notifier, current_seq)) {
if (mmu_interval_read_retry(&umem_odp->notifier, range.notifier_seq)) {
mutex_unlock(&umem_odp->umem_mutex);
goto retry;
}

5
drivers/misc/sgi-xp/xpc_uv.c
View File

@@ -170,9 +170,8 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name,
mq->mmr_blade = uv_cpu_to_blade_id(cpu);
nid = cpu_to_node(cpu);
page = __alloc_pages_node(nid,
GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE,
pg_order);
page = alloc_pages_node(nid, GFP_KERNEL | __GFP_ZERO | __GFP_THISNODE,
pg_order);
if (page == NULL) {
dev_err(xpc_part, "xpc_create_gru_mq_uv() failed to alloc %d "
"bytes of memory on nid=%d for GRU mq\n", mq_size, nid);

2
drivers/net/ethernet/fungible/funeth/funeth_rx.c
View File

@@ -103,7 +103,7 @@ static int funeth_alloc_page(struct funeth_rxq *q, struct funeth_rxbuf *rb,
if (cache_get(q, rb))
return 0;
p = __alloc_pages_node(node, gfp | __GFP_NOWARN, 0);
p = alloc_pages_node(node, gfp | __GFP_NOWARN, 0);
if (unlikely(!p))
return -ENOMEM;

3
drivers/vfio/pci/nvgrace-gpu/main.c
View File

@@ -11,6 +11,7 @@
#include <linux/jiffies.h>
#include <linux/sched.h>
#include <linux/pci-p2pdma.h>
#include <linux/pagemap.h>
#include <linux/pm_runtime.h>
#include <linux/memory-failure.h>
@@ -385,7 +386,7 @@ static unsigned long addr_to_pgoff(struct vm_area_struct *vma,
u64 pgoff = vma->vm_pgoff &
((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
return ((addr - vma->vm_start) >> PAGE_SHIFT) + pgoff;
return linear_page_delta(vma, addr) + pgoff;
}
static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault *vmf,

3
drivers/vfio/pci/vfio_pci_core.c
View File

@@ -20,6 +20,7 @@
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/notifier.h>
#include <linux/pagemap.h>
#include <linux/pci.h>
#include <linux/pm_runtime.h>
#include <linux/slab.h>
@@ -1781,7 +1782,7 @@ static vm_fault_t vfio_pci_mmap_huge_fault(struct vm_fault *vmf,
struct vm_area_struct *vma = vmf->vma;
struct vfio_pci_core_device *vdev = vma->vm_private_data;
unsigned long addr = vmf->address & ~((PAGE_SIZE << order) - 1);
unsigned long pgoff = (addr - vma->vm_start) >> PAGE_SHIFT;
unsigned long pgoff = linear_page_delta(vma, addr);
unsigned long pfn = vma_to_pfn(vma) + pgoff;
vm_fault_t ret = VM_FAULT_FALLBACK;

2
drivers/video/fbdev/core/fb_io_fops.c
View File

@@ -169,7 +169,7 @@ int fb_io_mmap(struct fb_info *info, struct vm_area_struct *vma)
len = info->fix.mmio_len;
}
vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
vma->vm_page_prot = vma_get_page_prot(vma);
vma->vm_page_prot = pgprot_framebuffer(vma->vm_page_prot, vma->vm_start,
vma->vm_end, start);

2
fs/aio.c
View File

@@ -604,7 +604,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)
ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size,
PROT_READ | PROT_WRITE,
MAP_SHARED, 0, 0, &unused, NULL);
MAP_SHARED, EMPTY_VMA_FLAGS, 0, &unused, NULL);
mmap_write_unlock(mm);
if (IS_ERR((void *)ctx->mmap_base)) {
ctx->mmap_size = 0;

7
fs/binfmt_elf.c
View File

@@ -1353,11 +1353,8 @@ static int load_elf_binary(struct linux_binprm *bprm)
emulate the SVr4 behavior. Sigh. */
error = vm_mmap(NULL, 0, PAGE_SIZE, PROT_READ | PROT_EXEC,
MAP_FIXED | MAP_PRIVATE, 0);
retval = do_mseal(0, PAGE_SIZE, 0);
if (retval)
pr_warn_ratelimited("pid=%d, couldn't seal address 0, ret=%d.\n",
task_pid_nr(current), retval);
if (!error)
mseal_mmap_page_zero();
}
regs = current_pt_regs();

2
fs/dax.c
View File

@@ -1201,7 +1201,7 @@ static int dax_writeback_one(struct xa_state *xas, struct dax_device *dax_dev,
/* Walk all mappings of a given index of a file and writeprotect them */
i_mmap_lock_read(mapping);
vma_interval_tree_foreach(vma, &mapping->i_mmap, index, end) {
mapping_rmap_tree_foreach(vma, mapping, index, end) {
pfn_mkclean_range(pfn, count, index, vma);
cond_resched();
}

23
fs/hugetlbfs/inode.c
View File

@@ -184,7 +184,8 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
if (addr)
addr0 = ALIGN(addr, huge_page_size(h));
return mm_get_unmapped_area_vmflags(file, addr0, len, pgoff, flags, 0);
return mm_get_unmapped_area_vmaflags(file, addr0, len, pgoff, flags,
EMPTY_VMA_FLAGS);
}
/*
@@ -380,7 +381,6 @@ static void hugetlb_unmap_file_folio(struct hstate *h,
struct address_space *mapping,
struct folio *folio, pgoff_t index)
{
struct rb_root_cached *root = &mapping->i_mmap;
struct hugetlb_vma_lock *vma_lock;
unsigned long pfn = folio_pfn(folio);
struct vm_area_struct *vma;
@@ -394,7 +394,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h,
i_mmap_lock_write(mapping);
retry:
vma_lock = NULL;
vma_interval_tree_foreach(vma, root, start, end - 1) {
mapping_rmap_tree_foreach(vma, mapping, start, end - 1) {
v_start = vma_offset_start(vma, start);
v_end = vma_offset_end(vma, end);
@@ -460,8 +460,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h,
}
static void
hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,
zap_flags_t zap_flags)
hugetlb_vmdelete_list(struct address_space *mapping, pgoff_t start,
pgoff_t end, zap_flags_t zap_flags)
{
struct vm_area_struct *vma;
@@ -470,7 +470,8 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end,
* unmapped. Note, end is exclusive, whereas the interval tree takes
* an inclusive "last".
*/
vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) {
mapping_rmap_tree_foreach(vma, mapping, start,
end ? end - 1 : ULONG_MAX) {
unsigned long v_start;
unsigned long v_end;
@@ -615,8 +616,7 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset)
i_size_write(inode, offset);
i_mmap_lock_write(mapping);
if (mapping_mapped(mapping))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0,
ZAP_FLAG_DROP_MARKER);
hugetlb_vmdelete_list(mapping, pgoff, 0, ZAP_FLAG_DROP_MARKER);
i_mmap_unlock_write(mapping);
remove_inode_hugepages(inode, offset, LLONG_MAX);
}
@@ -676,7 +676,7 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len)
/* Unmap users of full pages in the hole. */
if (hole_end > hole_start) {
if (mapping_mapped(mapping))
hugetlb_vmdelete_list(&mapping->i_mmap,
hugetlb_vmdelete_list(mapping,
hole_start >> PAGE_SHIFT,
hole_end >> PAGE_SHIFT, 0);
}
@@ -1129,8 +1129,7 @@ static void hugetlbfs_put_super(struct super_block *sb)
if (sbi) {
sb->s_fs_info = NULL;
if (sbi->spool)
hugepage_put_subpool(sbi->spool);
hugepage_put_subpool(sbi->spool);
kfree(sbi);
}
@@ -1419,7 +1418,7 @@ hugetlbfs_fill_super(struct super_block *sb, struct fs_context *fc)
goto out_free;
return 0;
out_free:
kfree(sbinfo->spool);
hugepage_put_subpool(sbinfo->spool);
kfree(sbinfo);
return -ENOMEM;
}

56
fs/proc/page.c
View File

@@ -148,8 +148,6 @@ u64 stable_page_flags(const struct page *page)
const struct folio *folio;
struct page_snapshot ps;
unsigned long k;
unsigned long mapping;
bool is_anon;
u64 u = 0;
/*
@@ -157,24 +155,21 @@ u64 stable_page_flags(const struct page *page)
* it differentiates a memory hole from a page with no flags
*/
if (!page)
return 1 << KPF_NOPAGE;
return BIT_ULL(KPF_NOPAGE);
snapshot_page(&ps, page);
folio = &ps.folio_snapshot;
k = folio->flags.f;
mapping = (unsigned long)folio->mapping;
is_anon = mapping & FOLIO_MAPPING_ANON;
/*
* pseudo flags for the well known (anonymous) memory mapped pages
*/
if (folio_mapped(folio))
u |= 1 << KPF_MMAP;
if (is_anon) {
u |= 1 << KPF_ANON;
if ((mapping & FOLIO_MAPPING_FLAGS) == FOLIO_MAPPING_KSM)
u |= 1 << KPF_KSM;
u |= BIT_ULL(KPF_MMAP);
if (folio_test_anon(folio)) {
u |= BIT_ULL(KPF_ANON);
if (folio_test_ksm(folio))
u |= BIT_ULL(KPF_KSM);
}
/*
@@ -184,36 +179,32 @@ u64 stable_page_flags(const struct page *page)
if (ps.idx == 0)
u |= kpf_copy_bit(k, KPF_COMPOUND_HEAD, PG_head);
else
u |= 1 << KPF_COMPOUND_TAIL;
u |= BIT_ULL(KPF_COMPOUND_TAIL);
if (folio_test_hugetlb(folio))
u |= 1 << KPF_HUGE;
u |= BIT_ULL(KPF_HUGE);
else if (folio_test_large(folio) &&
folio_test_large_rmappable(folio)) {
/* Note: we indicate any THPs here, not just PMD-sized ones */
u |= 1 << KPF_THP;
u |= BIT_ULL(KPF_THP);
} else if (is_huge_zero_pfn(ps.pfn)) {
u |= 1 << KPF_ZERO_PAGE;
u |= 1 << KPF_THP;
u |= BIT_ULL(KPF_ZERO_PAGE);
u |= BIT_ULL(KPF_THP);
} else if (is_zero_pfn(ps.pfn)) {
u |= 1 << KPF_ZERO_PAGE;
u |= BIT_ULL(KPF_ZERO_PAGE);
}
if (ps.flags & PAGE_SNAPSHOT_PG_BUDDY)
u |= 1 << KPF_BUDDY;
u |= BIT_ULL(KPF_BUDDY);
if (ps.flags & PAGE_SNAPSHOT_PG_IDLE)
u |= BIT_ULL(KPF_IDLE);
if (folio_test_offline(folio))
u |= 1 << KPF_OFFLINE;
u |= BIT_ULL(KPF_OFFLINE);
if (folio_test_pgtable(folio))
u |= 1 << KPF_PGTABLE;
u |= BIT_ULL(KPF_PGTABLE);
if (folio_test_slab(folio))
u |= 1 << KPF_SLAB;
#if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT)
u |= kpf_copy_bit(k, KPF_IDLE, PG_idle);
#else
if (ps.flags & PAGE_SNAPSHOT_PG_IDLE)
u |= 1 << KPF_IDLE;
#endif
u |= BIT_ULL(KPF_SLAB);
u |= kpf_copy_bit(k, KPF_LOCKED, PG_locked);
u |= kpf_copy_bit(k, KPF_DIRTY, PG_dirty);
@@ -225,16 +216,15 @@ u64 stable_page_flags(const struct page *page)
u |= kpf_copy_bit(k, KPF_ACTIVE, PG_active);
u |= kpf_copy_bit(k, KPF_RECLAIM, PG_reclaim);
#define SWAPCACHE ((1 << PG_swapbacked) | (1 << PG_swapcache))
if ((k & SWAPCACHE) == SWAPCACHE)
u |= 1 << KPF_SWAPCACHE;
u |= kpf_copy_bit(k, KPF_SWAPBACKED, PG_swapbacked);
if (folio_test_swapcache(folio))
u |= BIT_ULL(KPF_SWAPCACHE);
u |= kpf_copy_bit(k, KPF_SWAPBACKED, PG_swapbacked);
u |= kpf_copy_bit(k, KPF_UNEVICTABLE, PG_unevictable);
u |= kpf_copy_bit(k, KPF_MLOCKED, PG_mlocked);
#ifdef CONFIG_MEMORY_FAILURE
if (u & (1 << KPF_HUGE))
if (u & BIT_ULL(KPF_HUGE))
u |= kpf_copy_bit(k, KPF_HWPOISON, PG_hwpoison);
else
u |= kpf_copy_bit(ps.page_snapshot.flags.f, KPF_HWPOISON, PG_hwpoison);

119
fs/proc/task_mmu.c
View File

@@ -1237,6 +1237,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
[ilog2(VM_UFFD_MINOR)] = "ui",
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
#ifdef CONFIG_USERFAULTFD_RWP
[ilog2(VM_UFFD_RWP)] = "ur",
#endif
#ifdef CONFIG_ARCH_HAS_USER_SHADOW_STACK
[ilog2(VM_SHADOW_STACK)] = "ss",
#endif
@@ -2035,14 +2038,14 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
page = vm_normal_page(vma, addr, pte);
if (pte_soft_dirty(pte))
flags |= PM_SOFT_DIRTY;
if (pte_uffd_wp(pte))
if (pte_uffd(pte))
flags |= PM_UFFD_WP;
} else {
softleaf_t entry;
if (pte_swp_soft_dirty(pte))
flags |= PM_SOFT_DIRTY;
if (pte_swp_uffd_wp(pte))
if (pte_swp_uffd(pte))
flags |= PM_UFFD_WP;
entry = softleaf_from_pte(pte);
if (pm->show_pfn) {
@@ -2108,7 +2111,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_PRESENT;
if (pmd_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_uffd_wp(pmd))
if (pmd_uffd(pmd))
flags |= PM_UFFD_WP;
if (pm->show_pfn)
frame = pmd_pfn(pmd) + idx;
@@ -2127,9 +2130,10 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_SWAP;
if (pmd_swp_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
if (pmd_swp_uffd_wp(pmd))
if (pmd_swp_uffd(pmd))
flags |= PM_UFFD_WP;
page = softleaf_to_page(entry);
if (softleaf_has_pfn(entry))
page = softleaf_to_page(entry);
}
if (page) {
@@ -2232,14 +2236,14 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,
!hugetlb_pmd_shared(ptep))
flags |= PM_MMAP_EXCLUSIVE;
if (huge_pte_uffd_wp(pte))
if (huge_pte_uffd(pte))
flags |= PM_UFFD_WP;
flags |= PM_PRESENT;
if (pm->show_pfn)
frame = pte_pfn(pte) +
((addr & ~hmask) >> PAGE_SHIFT);
} else if (pte_swp_uffd_wp_any(pte)) {
} else if (pte_swp_uffd_any(pte)) {
flags |= PM_UFFD_WP;
}
@@ -2280,7 +2284,7 @@ static const struct mm_walk_ops pagemap_ops = {
* Bits 5-54 swap offset if swapped
* Bit 55 pte is soft-dirty (see Documentation/admin-guide/mm/soft-dirty.rst)
* Bit 56 page exclusively mapped
* Bit 57 pte is uffd-wp write-protected
* Bit 57 pte is tracked by userfaultfd (uffd-wp or RWP)
* Bit 58 pte is a guard region
* Bits 59-60 zero
* Bit 61 page is file-page or shared-anon
@@ -2415,7 +2419,7 @@ static int pagemap_release(struct inode *inode, struct file *file)
PAGE_IS_FILE | PAGE_IS_PRESENT | \
PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \
PAGE_IS_HUGE | PAGE_IS_SOFT_DIRTY | \
PAGE_IS_GUARD)
PAGE_IS_GUARD | PAGE_IS_ACCESSED)
#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
struct pagemap_scan_private {
@@ -2434,15 +2438,17 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
if (pte_none(pte)) {
/*
* An unpopulated pte carries no uffd-wp marker, i.e. it is not
* write-protected, the same condition under which the present
* and swap cases below report PAGE_IS_WRITTEN. Report it here
* too so this generic path agrees with the PAGE_IS_WRITTEN fast
* path in pagemap_scan_pmd_entry(), which reports pte_none as
* written and, under PM_SCAN_WP_MATCHING, arms a marker. The
* fast path applies no VMA test, so neither does this.
* An unpopulated pte carries no uffd bit, i.e. it is not
* write-protected. The PAGE_IS_WRITTEN fast path in
* pagemap_scan_pmd_entry() is now gated on a VM_UFFD_WP VMA;
* gate the pte_none report here the same way so the two paths
* still agree. RWP has no such fast path and an unpopulated
* page is not part of the RWP working set, so it is reported as
* neither.
*/
return PAGE_IS_WRITTEN;
if (userfaultfd_wp(vma))
return PAGE_IS_WRITTEN;
return 0;
}
if (pte_present(pte)) {
@@ -2450,8 +2456,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
categories = PAGE_IS_PRESENT;
if (!pte_uffd_wp(pte))
categories |= PAGE_IS_WRITTEN;
if (!pte_uffd(pte)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
if (p->masks_of_interest & PAGE_IS_FILE) {
page = vm_normal_page(vma, addr, pte);
@@ -2468,8 +2478,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
categories = PAGE_IS_SWAPPED;
if (!pte_swp_uffd_wp_any(pte))
categories |= PAGE_IS_WRITTEN;
if (!pte_swp_uffd_any(pte)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
entry = softleaf_from_pte(pte);
if (softleaf_is_guard_marker(entry))
@@ -2493,13 +2507,13 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
pte_t old_pte;
old_pte = ptep_modify_prot_start(vma, addr, pte);
ptent = pte_mkuffd_wp(old_pte);
ptent = pte_mkuffd(old_pte);
ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
} else if (pte_none(ptent)) {
set_pte_at(vma->vm_mm, addr, pte,
make_pte_marker(PTE_MARKER_UFFD_WP));
} else {
ptent = pte_swp_mkuffd_wp(ptent);
ptent = pte_swp_mkuffd(ptent);
set_pte_at(vma->vm_mm, addr, pte, ptent);
}
}
@@ -2518,8 +2532,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
struct page *page;
categories |= PAGE_IS_PRESENT;
if (!pmd_uffd_wp(pmd))
categories |= PAGE_IS_WRITTEN;
if (!pmd_uffd(pmd)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
if (p->masks_of_interest & PAGE_IS_FILE) {
page = vm_normal_page_pmd(vma, addr, pmd);
@@ -2533,8 +2551,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
categories |= PAGE_IS_SOFT_DIRTY;
} else {
categories |= PAGE_IS_SWAPPED;
if (!pmd_swp_uffd_wp(pmd))
categories |= PAGE_IS_WRITTEN;
if (!pmd_swp_uffd(pmd)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
if (pmd_swp_soft_dirty(pmd))
categories |= PAGE_IS_SOFT_DIRTY;
@@ -2557,17 +2579,18 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
if (pmd_present(pmd)) {
old = pmdp_invalidate_ad(vma, addr, pmdp);
pmd = pmd_mkuffd_wp(old);
pmd = pmd_mkuffd(old);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
} else if (pmd_is_migration_entry(pmd)) {
pmd = pmd_swp_mkuffd_wp(pmd);
pmd = pmd_swp_mkuffd(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#ifdef CONFIG_HUGETLB_PAGE
static unsigned long pagemap_hugetlb_category(pte_t pte)
static unsigned long pagemap_hugetlb_category(struct vm_area_struct *vma,
pte_t pte)
{
unsigned long categories = PAGE_IS_HUGE;
@@ -2582,8 +2605,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
if (pte_present(pte)) {
categories |= PAGE_IS_PRESENT;
if (!huge_pte_uffd_wp(pte))
categories |= PAGE_IS_WRITTEN;
if (!huge_pte_uffd(pte)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
if (!PageAnon(pte_page(pte)))
categories |= PAGE_IS_FILE;
if (is_zero_pfn(pte_pfn(pte)))
@@ -2593,8 +2620,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
} else {
categories |= PAGE_IS_SWAPPED;
if (!pte_swp_uffd_wp_any(pte))
categories |= PAGE_IS_WRITTEN;
if (!pte_swp_uffd_any(pte)) {
if (userfaultfd_wp(vma))
categories |= PAGE_IS_WRITTEN;
if (userfaultfd_rwp(vma))
categories |= PAGE_IS_ACCESSED;
}
if (pte_swp_soft_dirty(pte))
categories |= PAGE_IS_SOFT_DIRTY;
}
@@ -2621,12 +2652,12 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
if (softleaf_is_migration(entry)) {
set_huge_pte_at(vma->vm_mm, addr, ptep,
pte_swp_mkuffd_wp(ptent), psize);
pte_swp_mkuffd(ptent), psize);
} else {
pte_t old_pte, new_pte;
old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
new_pte = huge_pte_mkuffd_wp(old_pte);
new_pte = huge_pte_mkuffd(old_pte);
huge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);
}
}
@@ -2859,8 +2890,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
pte_t ptent = ptep_get(pte);
if ((pte_present(ptent) && pte_uffd_wp(ptent)) ||
pte_swp_uffd_wp_any(ptent))
if ((pte_present(ptent) && pte_uffd(ptent)) ||
pte_swp_uffd_any(ptent))
continue;
make_uffd_wp_pte(vma, addr, pte, ptent);
if (!flush_end)
@@ -2870,15 +2901,16 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
goto flush_and_return;
}
if (!p->arg.category_anyof_mask && !p->arg.category_inverted &&
if (userfaultfd_wp(vma) && !p->arg.category_anyof_mask &&
!p->arg.category_inverted &&
p->arg.category_mask == PAGE_IS_WRITTEN &&
p->arg.return_mask == PAGE_IS_WRITTEN) {
for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
unsigned long next = addr + PAGE_SIZE;
pte_t ptent = ptep_get(pte);
if ((pte_present(ptent) && pte_uffd_wp(ptent)) ||
pte_swp_uffd_wp_any(ptent))
if ((pte_present(ptent) && pte_uffd(ptent)) ||
pte_swp_uffd_any(ptent))
continue;
ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN,
p, addr, &next);
@@ -2945,7 +2977,8 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
/* Go the short route when not write-protecting pages. */
pte = huge_ptep_get(walk->mm, start, ptep);
categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
categories = p->cur_vma_category |
pagemap_hugetlb_category(vma, pte);
if (!pagemap_scan_is_interesting_page(categories, p))
return 0;
@@ -2957,7 +2990,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
pte = huge_ptep_get(walk->mm, start, ptep);
categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
categories = p->cur_vma_category | pagemap_hugetlb_category(vma, pte);
if (!pagemap_scan_is_interesting_page(categories, p))
goto out_unlock;

4
include/asm-generic/fixmap.h
View File

@@ -21,7 +21,7 @@
#define __fix_to_virt(x) (FIXADDR_TOP - ((x) << PAGE_SHIFT))
#define __virt_to_fix(x) ((FIXADDR_TOP - ((x)&PAGE_MASK)) >> PAGE_SHIFT)
#ifndef __ASSEMBLY__
#ifndef __ASSEMBLER__
/*
* 'index to address' translation. If anyone tries to use the idx
* directly without translation, we catch the bug with a NULL-deference
@@ -97,5 +97,5 @@ static inline unsigned long virt_to_fix(const unsigned long vaddr)
#define set_fixmap_io(idx, phys) \
__set_fixmap(idx, phys, FIXMAP_PAGE_IO)
#endif /* __ASSEMBLY__ */
#endif /* __ASSEMBLER__ */
#endif /* __ASM_GENERIC_FIXMAP_H */

4
include/asm-generic/getorder.h
View File

@@ -2,7 +2,7 @@
#ifndef __ASM_GENERIC_GETORDER_H
#define __ASM_GENERIC_GETORDER_H
#ifndef __ASSEMBLY__
#ifndef __ASSEMBLER__
#include <linux/compiler.h>
#include <linux/log2.h>
@@ -47,6 +47,6 @@ static __always_inline __attribute_const__ int get_order(unsigned long size)
#endif
}
#endif /* __ASSEMBLY__ */
#endif /* __ASSEMBLER__ */
#endif /* __ASM_GENERIC_GETORDER_H */

18
include/asm-generic/hugetlb.h
View File

@@ -37,24 +37,24 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)
return pte_modify(pte, newprot);
}
#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD_WP
static inline pte_t huge_pte_mkuffd_wp(pte_t pte)
#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD
static inline pte_t huge_pte_mkuffd(pte_t pte)
{
return huge_pte_wrprotect(pte_mkuffd_wp(pte));
return huge_pte_wrprotect(pte_mkuffd(pte));
}
#endif
#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP
static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)
#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD
static inline pte_t huge_pte_clear_uffd(pte_t pte)
{
return pte_clear_uffd_wp(pte);
return pte_clear_uffd(pte);
}
#endif
#ifndef __HAVE_ARCH_HUGE_PTE_UFFD_WP
static inline int huge_pte_uffd_wp(pte_t pte)
#ifndef __HAVE_ARCH_HUGE_PTE_UFFD
static inline int huge_pte_uffd(pte_t pte)
{
return pte_uffd_wp(pte);
return pte_uffd(pte);
}
#endif

Some files were not shown because too many files have changed in this diff Show More