KVM/arm64 updates for 7.2
* New features:
- None. Zilch. Nada. Que dalle.
* Fixes and other improvements:
- Significant cleanup of the vgic-v5 PPI support which was merged in
7.1. This makes the code more maintainable, and squashes a couple
of bugs in the meantime.
- Set of fixes for the handling of the MMU in an NV context,
particularly VNCR-triggered faults. S1POE support is fixed
as well.
- Large set of pKVM fixes, mostly addressing recurring issues
around hypervisor tracking of donated pages in obscure cases
where the donation could fail and leave things in a bizarre
state.
- Fixes for the so-called "lazy vgic init", which resulted in
sleeping operations in non-preemptible sections. This turned
out to be far more invasive than initially expected...
- Reduce the overhead of L1/L2 context switch by not touching
the FP registers.
- Fix the way non-implemented page sizes are dealt with when
a guest insist on using them for S2 translation.
- The usual set of low-impact fixes and cleanups all over the map.
* kvm-arm64/nv-mmu-7.2:
: .
: Assorted collection of fixes for NV MMU bugs
:
: - Correctly plug AT S1E1A handling in the emulation backend
:
: - Make CPTR_EL2.E0POE depend on FEAT_S1POE
:
: - Drop the reference on the page if the VNCR translation
: races with an MMU notifier
:
: - Correctly synthesise an SEA if a page table walk fails due
: to a guest error
:
: - Fully invalidate the VNCR TLB and fixmap when translating
: for a new VNCR
:
: - Restart S1 walk when the S2 walk fails due to a race condition
:
: - Correctly return -EAGAIN when a S1 walk fails
:
: - Fix block mapping validity check in stage-1 walker for 64kB pages
:
: - Fix potential NULL dereference when performing an EL2 TLBI targeting
: the VNCR page
:
: - Hold kvm->mmu_lock while initialising the vncr_tlb pointer
: .
KVM: arm64: nv: Hold kvm->mmu_lock while initialising vcpu->arch.vncr_tlb
KVM: arm64: nv: Avoid dereferencing NULL VNCR pseudo-TLB
KVM: arm64: Fix block mapping validity check in stage-1 walker
KVM: arm64: nv: Restart stage-1 walk if stage-2 desc update fails
KVM: arm64: Restart instruction upon race in __kvm_at_s12()
KVM: arm64: nv: Inject SEA TTW when desc update can't write to GPA
KVM: arm64: nv: Fully update VNCR fixmap state in kvm_translate_vncr()
KVM: arm64: Don't leak PFN when kvm_translate_vncr() races MMU notifier
arm64: cpufeature: Expose ID_AA64ISAR2_EL1.ATS1A to KVM
KVM: arm64: Wire AT S1E1A in the system instruction handling table
KVM: arm64: Key CPTR_EL2.E0POE propagation on FEAT_S1POE
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/misc-7.2:
: .
: - Check for a valid vcpu pointer upon deactivating traps when handling
: a HYP panic in VHE mode
:
: - Make the __deactivate_fgt() macro use its arguments instead of the
: surrounding context
:
: - Don't bother with initialising TPIDR_EL2 in the hyp stubs, as this
: is already taken care of in more obvious places
:
: - Drop the unused kvm_arch pointer passed to __load_stage2()
:
: - Return -EOPNOTSUPP when a hypercall fails for some reason, instead of
: returning whatever was in the result structure
:
: - Make the ITS ABI selection helpers return void, which avoids wondering
: about the nature of the return code (always 0)
: .
KVM: arm64: vgic-its: Make ABI commit helpers return void
KVM: arm64: Set a Linux errno on SMCCC error in kvm_call_hyp_nvhe()
KVM: arm64: Remove @arch from __load_stage2()
KVM: arm64: Don't populate TPIDR_EL2 in finalise_el2()
KVM: arm64: Fix __deactivate_fgt macro parameter typo
KVM: arm64: Guard against NULL vcpu on VHE hyp panic path
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/vgic-v5-PPI-fixes:
: .
: Substantial cleanup of the vgic-v5 PPI support. From the original
: cover letter:
:
: "With the GICv5 PPi support merged in, it has become obvious that a few
: things could be improved, both from the correctness and maintainability
: angles."
: .
KVM: arm64: Fix arch timer interrupts for GICv3-on-GICv5 guests
irqchip/gic-v5: Immediately exec priority drop following activate
Documentation: KVM: Clarify that PMU_V3_IRQ IntID requirements for GICv5
Documentation: KVM: Fix typos in VGICv5 documentation
KVM: arm64: selftests: Improve error handling for GICv5 PPI selftest
KVM: arm64: selftests: Cleanup unused vars in GICv5 PPI selftest
KVM: arm64: selftests: Add missing GIC CDEN to no-vgic-v5 selftest
KVM: arm64: vgic-v5: Atomically assign bits to PPI DVI bitmap
KVM: arm64: vgic-v5: Add missing trap handing for NV triage
KVM: arm64: vgic-v5: Limit support to 64 PPIs
KVM: arm64: vgic: Rationalise per-CPU irq accessor
KVM: arm64: vgic-v5: Drop defensive checks from vgic_v5_ppi_queue_irq_unlock()
KVM: arm64: vgic: Consolidate vgic_allocate_private_irqs_locked()
KVM: arm64: vgic: Constify struct irq_ops usage
KVM: arm64: vgic-v5: Drop pointless ARM64_HAS_GICV5_CPUIF check
KVM: arm64: vgic-v5: Remove use of __assign_bit() with a constant
KVM: arm64: vgic-v5: Move PPI caps into kvm_vgic_global_state
KVM: arm64: vgic-v5: Add for_each_visible_v5_ppi() iterator
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/pkvm-fixes-7.2:
: .
: Assorted pKVM fixes for 7.2:
:
: - Ensure that the vcpu memcache is filled in a number of cases (donate,
: share, selftest)
:
: - Fix vmemmap page order handling by resetting it when initialising the
: memory pool
:
: - Don't leak page references on failed memory donation
:
: - Add sanity-check for refcounted pages when donating/sharing pages
:
: - Clear __hyp_running_vcpu on state flush
:
: - Check LR upper bound against a trusted value
:
: - Assorted fixes for the host-side tracking of the pages shared with
: EL2 as a result of some Sashiko testing from Fuad
:
: - Correctly forward HCR_EL2.VSE from host to guest, so that protected
: guests can see SErrors
: .
KVM: arm64: Roll back partial shares on kvm_share_hyp() failure
KVM: arm64: Avoid host/hyp share desync on unshare hypercall failure
KVM: arm64: Free hyp-share tracking node when share hypercall fails
KVM: arm64: Flush HCR_EL2.VSE to deliver SErrors to pKVM guests
KVM: arm64: Bound used_lrs when flushing the pKVM hyp vCPU
KVM: arm64: Clear __hyp_running_vcpu when flushing the pKVM hyp vCPU
KVM: arm64: Pre-check vcpu memcache for host->guest donate
KVM: arm64: Pre-check vcpu memcache for host->guest share
KVM: arm64: Seed pkvm_ownership_selftest vcpu memcache
KVM: arm64: Add fail-safe for refcounted pages in __pkvm_hyp_donate_host
KVM: arm64: Fix __pkvm_init_vm error path
KVM: arm64: Reset page order in pKVM hyp_pool
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/nv-granule-sizes:
: .
: Tidying up of the behaviour when the selected page size in not
: implemented, courtesy of Wei-Lin Chang. From the initial cover
: letter:
:
: "This small series fixes the granule size selection for software stage-1
: and stage-2 walks. Previously we treat the guest's TCR/VTCR.TGx as-is
: and use the encoded granule size for the walks. However this is
: incorrect if the granule sizes are not advertised in the guest's
: ID_AA64MMFR0_EL1.TGRAN*. The architecture specifies that when an
: unsupported size is programed in TGx, it must be treated as an
: implemented size. Fix this by choosing an available one while
: prioritizing PAGE_SIZE."
: .
KVM: arm64: Fallback to a supported value for unsupported guest TGx
KVM: arm64: nv: Use literal granule size in TLBI range calculation
KVM: arm64: Factor out TG0/1 decoding of VTCR and TCR
KVM: arm64: nv: Rename vtcr_to_walk_info() to setup_s2_walk()
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/nv-fp-elision:
: .
: Significantly reduce the overhead of the context switch between L1 and
: L2 guests by eliding the save/restore of the FP/SIMD/SVE registers, as
: this state is shared between the two guests, and therefore can be left
: live.
: .
KVM: arm64: nv: Don't save/restore FP register during a nested ERET or exception
KVM: arm64: nv: Track L2 to L1 exception emulation
Signed-off-by: Marc Zyngier <maz@kernel.org>
* kvm-arm64/no-lazy-vgic-init:
: .
: Fix an ugly situation where the vgic lazy init could happen in
: non-preemtible contexts such as vcpu reset, resulting in lockdep
: splats.
:
: This requires revamping the way in-kernel emulation of devices
: (timers, PMU) are presenting their interrupt to the vgic, and
: make sure there is no need to init the vgic on the back of that.
: .
KVM: arm64: vgic-v2: Don't init the vgic on in-kernel interrupt injection
KVM: arm64: vgic-v2: Force vgic init on injection outside the run loop
KVM: arm64: pmu: Kill the PMU interrupt level cache
KVM: arm64: timer: Kill the per-timer irq level cache
KVM: arm64: Simplify userspace notification of interrupt state
KVM: arm64: timer: Repaint kvm_timer_{should,irq_can}_fire() to kvm_timer_{pending,enabled}()
Signed-off-by: Marc Zyngier <maz@kernel.org>
Sashiko reports that there is a race between initialising vncr_tlb
and making use of it, as we don't hold the mmu_lock at this point.
Additionally, it identifies a memory leak, should userspace repeatedly
invokes the KVM_RUN ioctl after a failure of kvm_arch_vcpu_run_pid_change(),
as we assign vncr_tlb blindly on first run, irrespective of prior
allocations.
Slap the two bugs in one go by taking the kvm->mmu_lock on assigning
vncr_tlb, preventing the race for good, and by checking that vncr_tlb
is indeed NULL prior to allocation.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Link: https://lore.kernel.org/r/20260607180815.85FBC1F00893@smtp.kernel.org
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260608081108.2244133-1-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
VNCR TLB invalidation occurs from MMU notifiers or TLBI instructions,
and either can race against a vcpu not being onlined yet (no pseudo-TLB
allocated). Similarly, the TLB might be invalid, and the invalidation
should be skipped in this case.
Both kvm_invalidate_vncr_ipa() and kvm_invalidate_vncr_va() are
expected to perform the same checks, except that the latter doesn't
check for the allocation and blindly dereferences the pointer.
Solve this by introducing a new iterator built on top of the usual
kvm_for_each_vcpu() that checks for both of the above conditions,
and convert the two users to it.
Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Link: https://lore.kernel.org/r/aiUvSbrWndQeUPc8@v4bel
Fixes: 4ffa72ad8f ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2")
Cc: stable@vger.kernel.org
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260607175745.297793-1-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Pull misc fixes from Andrew Morton:
"11 hotfixes. 9 are for MM. 8 are cc:stable and the remaining 3 address
post-7.1 issues or aren't considered suitable for backporting.
Thre's a two-patch series "mm/damon/{reclaim,lru_sort}: handle ctx
allocation failures" from SeongJae Park which fixes a couple of DAMON
-ENOMEM bloopers. The rest are singletons - please see the individual
changelogs for details"
* tag 'mm-hotfixes-stable-2026-06-08-20-51' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
mm/mincore: handle non-swap entries before !CONFIG_SWAP guard
arm64: mm: call pagetable dtor when freeing hot-removed page tables
mm/list_lru: drain before clearing xarray entry on reparent
mm/huge_memory: use correct flags for device private PMD entry
mm/damon/lru_sort: handle ctx allocation failure
mm/damon/reclaim: handle ctx allocation failure
zram: fix use-after-free in zram_bvec_write_partial()
MAINTAINERS: update Baoquan He's email address
tools headers UAPI: sync linux/taskstats.h for procacct.c
mm/cma_sysfs: skip inactive CMA areas in sysfs
ipc/shm: serialize orphan cleanup with shm_nattch updates
For the 64K granule size, FEAT_LPA determines whether a level 1 mapping
is allowed. Using the result of has_52bit_pa() is too restrictive, as it
also checks the selected output addressi size in TCR.(I)PS. Fix it by
only checking FEAT_LPA.
Fixes: 5da3a3b27a ("KVM: arm64: Expand valid block mappings to FEAT_LPA/LPA2 support")
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260605185255.2431996-1-weilin.chang@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
If kvm_call_hyp_nvhe() fails with an SMCCC error code, we WARN().
However, the returned value isn't initialized and the caller might get
garbage or 0 which is likely to be interpreted as success.
Set a default -EOPNOTSUPP error value, ensuring all callers get the
message when hypercalls fail.
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
Acked-by: Will Deacon <will@kernel.org>
Reviewed-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260603110312.2909844-1-vdonnefort@google.com
[maz: changed error value to -EOPNOTSUPP as suggested by Will,
tidied up change log]
Signed-off-by: Marc Zyngier <maz@kernel.org>
kvm_share_hyp() shares a range one page at a time. If share_pfn_hyp()
fails partway through, the pages already shared by this call are left
shared, while the caller treats the whole range as failed and never
unshares them.
Unshare those pages before returning the error. If an unshare itself
fails the page is leaked: it stays shared with the hypervisor and is
no longer reusable for pKVM, but no isolation guarantee is broken, so
WARN and continue. Not expected in practice.
Fixes: a83e2191b7 ("KVM: arm64: pkvm: Refcount the pages shared with EL2")
Suggested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Link: https://patch.msgid.link/20260529121755.2923500-4-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
unshare_pfn_hyp() erases the tracking node from hyp_shared_pfns
and frees it before invoking __pkvm_host_unshare_hyp. If the
hypercall fails (e.g. EL2 refcount still held, or page-state
mismatch), the host loses its record while EL2 still holds the
share, breaking later share/unshare attempts on the same pfn.
Invoke the hypercall first; erase and free only on success.
Document at the kvm_unshare_hyp() call site that the WARN_ON() is
left non-fatal: a failed unshare leaks the page (it stays shared
with the hypervisor) but breaks no isolation guarantee.
Fixes: 52b28657eb ("KVM: arm64: pkvm: Unshare guest structs during teardown")
Reported-by: Sashiko (local):gemini-3.1-pro
Suggested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Link: https://patch.msgid.link/20260529121755.2923500-3-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
share_pfn_hyp() inserts a tracking node into hyp_shared_pfns and
then invokes __pkvm_host_share_hyp. If the hypercall rejects the
share (page-state mismatch at EL2), the node stays in the tree
with refcount 1: a phantom share that leaks the allocation and
that a later unshare will trust.
Erase the node and free it on hypercall failure.
Fixes: a83e2191b7 ("KVM: arm64: pkvm: Refcount the pages shared with EL2")
Reported-by: Sashiko (local):gemini-3.1-pro
Suggested-by: Vincent Donnefort <vdonnefort@google.com>
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Vincent Donnefort <vdonnefort@google.com>
Link: https://patch.msgid.link/20260529121755.2923500-2-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
With pKVM enabled, the host injects a virtual SError by setting
HCR_EL2.VSE on its vCPU copy, but flush_hyp_vcpu() only flows TWI/TWE
into the hyp vCPU that runs, so VSE never reaches it and a deferred
(masked) SError is never delivered. VSE is a host-owned injection
control, not a trap-configuration bit, so restricting the host's
trap-register values should not have dropped it.
Flow it on entry; sync_hyp_vcpu() already copies hcr_el2 back, so
delivery is reflected to the host. THis makes it consistent with
the existing forwarding of VSESR_EL2, which qualifies the Serror.
Fixes: b56680de9c ("KVM: arm64: Initialize trap register values in hyp in pKVM")
Reported-by: Sashiko (local):gemini-3.1-pro
Signed-off-by: Fuad Tabba <tabba@google.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260531154548.1505799-1-tabba@google.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
flush_hyp_vcpu() copies the host vGIC state into the hyp's private vCPU
on every run. The vGIC list register save and restore use used_lrs as
their loop bound and expect it to stay within the number of implemented
list registers. While this is generally the case, flush_hyp_vcpu()
copies vgic_v3 verbatim and does not enforce this, so a value provided
by the host is used at EL2 to index vgic_lr[] and access ICH_LR<n>_EL2
(host -> EL2).
Fix by clamping used_lrs to the number of implemented list registers
after the copy, as the trusted path already does in
vgic_flush_lr_state(). The number of implemented list registers is
constant after init, so it is replicated once from
kvm_vgic_global_state.nr_lr into hyp_gicv3_nr_lr rather than read on
every entry.
Cc: stable@vger.kernel.org
Fixes: be66e67f17 ("KVM: arm64: Use the pKVM hyp vCPU structure in handle___kvm_vcpu_run()")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260606175614.83273-3-imv4bel@gmail.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
flush_hyp_vcpu() copies the host vCPU context into the hyp's private
vCPU on every run. ctxt_to_vcpu() expects a guest context to have a
NULL __hyp_running_vcpu, which is only ever set on the host context, so
that it resolves the vCPU via container_of(). While this is generally
the case, flush_hyp_vcpu() copies the context verbatim and does not
enforce this, so a value provided by the host is dereferenced at EL2
(host -> EL2).
Fix by clearing __hyp_running_vcpu after the copy.
Cc: stable@vger.kernel.org
Fixes: be66e67f17 ("KVM: arm64: Use the pKVM hyp vCPU structure in handle___kvm_vcpu_run()")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Fuad Tabba <tabba@google.com>
Tested-by: Fuad Tabba <tabba@google.com>
Link: https://patch.msgid.link/20260606175614.83273-2-imv4bel@gmail.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
Pull Rust fixes from Miguel Ojeda:
"Toolchain and infrastructure:
- Fix 'rustc-option' (the Makefile one) when cross-compiling that
leads to build or boot failures in certain configs
- Work around a Rust compiler bug (already fixed for Rust 1.98.0)
thats lead to boot failures in certain configs due to missing
'uwtable' LLVM module flags
- Support a Rust compiler change (starting with Rust 1.98.0) in the
unstable target specification JSON files
- Forbid Rust + arm + KASAN configs, which do not build
'kernel' crate:
- Fix NOMMU build by adding a missing helper"
* tag 'rust-fixes-7.1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux:
rust: x86: support Rust >= 1.98.0 target spec
rust: arm64: set uwtable llvm module flag for CONFIG_UNWIND_TABLES
rust: helpers: add is_vmalloc_addr wrapper for NOMMU builds
rust: kasan/kbuild: fix rustc-option when cross-compiling
ARM: Do not select HAVE_RUST when KASAN is enabled
Pull kvm fixes from Paolo Bonzini:
"arm64:
- Correctly drop the ITS translation cache reference when it actually
gets invalidated
- Take the SRCU lock for SW page table walks
- Restore POR_EL0 access to host EL0, avoiding POR_EL0 becoming
inaccessible from EL0 after running a guest
- Reassign nested_mmus array behind mmu_lock, ensuring that vcpu init
and MMU notifiers are mutually exclusive
- Correctly handle FEAT_XNX at stage-2
s390:
- More fixes for the new page table management and nested
virtualization
x86:
- More fixes for GHCB issues:
- Read start/end indices of page size change requests exactly once
per vmexit
- Unmap and unpin the GHCB as needed on vCPU free"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (23 commits)
KVM: arm64: Correctly identify executable PTEs at stage-2
KVM: arm64: nv: Fix handling of XN[0] when !FEAT_XNX
KVM: arm64: Reassign nested_mmus array behind mmu_lock
KVM: arm64: Restore POR_EL0 access to host EL0
KVM: arm64: Take the SRCU lock for page table walks in fault injection and AT emulation
KVM: arm64: vgic-its: Drop the translation cache reference only for the erased entry
KVM: SEV: Unmap and unpin the GHCB as needed on vCPU free
KVM: SEV: Decouple the need to sync the GHCB SA from the need to free the SA
KVM: SEV: Move sev_free_vcpu() down below sev_es_unmap_ghcb()
KVM: Don't WARN if memory is dirtied without a vCPU when the VM is dying
KVM: SEV: Read start/end indices of PSC requests exactly once per #VMGEXIT
KVM: SEV: Add an anonymous "psc" struct to track current PSC metadata
KVM: SEV: Make it more obvious when KVM is writing back the current PSC index
KVM: s390: Remove ptep_zap_softleaf_entry()
KVM: s390: Fix possible reference leak in fault-in code
KVM: s390: Prevent memslots outside the ASCE range
KVM: s390: Lock pte when making page secure
KVM: s390: Fix fault-in code
KVM: s390: vsie: Fix rmap handling in _do_shadow_crste()
KVM: s390: Fix guest / virtual address confusion in _essa_clear_cbrl()
...
KVM/arm64 fixes for 7.1, take #5
- Correctly drop the ITS translation cache reference when it actually
gets invalidated
- Take the SRCU lock for SW page table walks
- Restore POR_EL0 access to host EL0, avoiding POR_EL0 becoming
inaccessible from EL0 after running a guest
- Reassign nested_mmus array behind mmu_lock, ensuring that vcpu init
and MMU notifiers are mutually exclusive
- Correctly handle FEAT_XNX at stage-2
KVM invalidates the I-cache before installing an executable PTE on
implementations without DIC. Unfortunately, support for FEAT_XNX
broke this check as KVM_PTE_LEAF_ATTR_HI_S2_XN was expanded to a
bitfield.
Fix it by reusing kvm_pgtable_stage2_pte_prot() and testing the abstract
permission bits instead.
Fixes: 2608563b46 ("KVM: arm64: Add support for FEAT_XNX stage-2 permissions")
Reported-by: Sashiko (gemini/gemini-3.1-pro-preview)
Signed-off-by: Oliver Upton <oupton@kernel.org>
Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260602165901.52800-3-oupton@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
kvm->arch.nested_mmus[] is walked under kvm->mmu_lock, including from the
MMU notifier path (kvm_unmap_gfn_range() -> kvm_nested_s2_unmap()), which
can run at any time. kvm_vcpu_init_nested() reallocates the array and frees
the old buffer while holding only kvm->arch.config_lock, so such a walker
can reference the freed array.
Allocate the new array outside of mmu_lock, as the allocation can sleep.
Under the lock, copy the existing entries, fix up the back pointers and
reassign the array. Free the old buffer after dropping the lock, as
kvfree() can sleep as well.
Fixes: 4f128f8e1a ("KVM: arm64: nv: Support multiple nested Stage-2 mmu structures")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/aiKIVVeIr1aAB1yp@v4bel
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger,kernel.org
walk_s1() and kvm_walk_nested_s2() expect to be called while holding
kvm->srcu to guard against memslot changes. While this is generally
the case, __kvm_at_s12() and __kvm_find_s1_desc_level() call into the
respective walkers without taking kvm->srcu.
Fix by acquiring kvm->srcu prior to the table walk in both instances.
Cc: stable@vger.kernel.org
Fixes: 50f77dc87f ("KVM: arm64: Populate level on S1PTW SEA injection")
Fixes: be04cebf3e ("KVM: arm64: nv: Add emulation of AT S12E{0,1}{R,W}")
Suggested-by: Oliver Upton <oupton@kernel.org>
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/aiAZfdeyanIvP8SD@v4bel
Signed-off-by: Marc Zyngier <maz@kernel.org>
vgic_its_invalidate_cache() walks the per-ITS translation cache with
xa_for_each() and drops the cache's reference on each entry with
vgic_put_irq(). It puts the iterated pointer, though, rather than the
value returned by xa_erase().
The function is called from contexts that do not exclude one another: the
ITS command handlers hold its_lock, the GITS_CTLR write path holds
cmd_lock, and the path that clears EnableLPIs in a redistributor's
GICR_CTLR holds neither. Two or more of them can drain the same cache
concurrently, and if each one observes the same entry, erases it and then
puts it, the single reference the cache holds on that entry is dropped
more than once. The entry can then be freed while an ITE still maps it.
xa_erase() is atomic and returns the previous entry, so put only the entry
that this context actually removed. The cache reference is then dropped
exactly once per entry even when the invalidations run concurrently, and
the behavior is unchanged when only one context runs.
Fixes: 8201d1028c ("KVM: arm64: vgic-its: Maintain a translation cache per ITS")
Signed-off-by: Hyunwoo Kim <imv4bel@gmail.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/ah2c5lu4JbUg7dj-@v4bel
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: stable@vger.kernel.org
Since 5e8eb9aeed ("arm64: mm: always call PTE/PMD ctor in
__create_pgd_mapping()") page-table allocation on ARM64 always calls
pagetable_{pte,pmd,pud,p4d}_ctor(). This sets the page_type to
PGTY_table, increments NR_PAGETABLE and possible allocates a PTL. However
the matching pagetable_dtor() calls were never added.
With DEBUG_VM enabled on kernel versions prior to v6.17 without
2dfcd1608f ("mm/page_alloc: let page freeing clear any set page type")
this leads to the following warning when freeing these pages due to
page->page_type sharing page->_mapcount:
BUG: Bad page state in process ... pfn:284fbb
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x284fbb
flags: 0x17fffc000000000(node=0|zone=2|lastcpupid=0x1ffff)
page_type: f2(table)
page dumped because: nonzero mapcount
Call trace:
bad_page+0x13c/0x160
__free_frozen_pages+0x6cc/0x860
___free_pages+0xf4/0x180
free_pages+0x54/0x80
free_hotplug_page_range.part.0+0x58/0x90
free_empty_tables+0x438/0x500
__remove_pgd_mapping.constprop.0+0x60/0xa8
arch_remove_memory+0x48/0x80
try_remove_memory+0x158/0x1d8
offline_and_remove_memory+0x138/0x180
It can also lead to leaking the ptl allocation if ALLOC_SPLIT_PTLOCKS is
defined and incorrect NR_PAGETABLE stats. Fix this by calling
pagetable_dtor() in free_hotplug_pgtable_page() prior to freeing the page
to undo the effects of calling pagetable_*_ctor().
Link: https://lore.kernel.org/20260521032730.2104017-1-apopple@nvidia.com
Fixes: 5e8eb9aeed ("arm64: mm: always call PTE/PMD ctor in __create_pgd_mapping()")
Signed-off-by: Alistair Popple <apopple@nvidia.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
kvm_walk_nested_s2() returns -EAGAIN as an indication that an underlying
descriptor update fails due to a race. The expectation is that the
caller restart translation, yet walk_s1() actually synthesizes an abort.
Propagate the -EAGAIN return out of walk_s1(), relying on callers to
restart the translation fetch.
Fixes: e4c7dfac2f ("KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW")
Signed-off-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602235450.103057-6-oupton@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
__kvm_at_s*() are expected to return -EAGAIN if the page table walk
raced with a concurrent update to a page table descriptor, which is
interpreted as a signal to restart the trapping instruction.
While this mostly works, __kvm_at_s12() silently eats the return from
__kvm_at_s1e01() and consumes an uninitialized PAR value. Propagate the
nonzero return instead.
Fixes: 92c6443222 ("KVM: arm64: Propagate PTW errors up to AT emulation")
Signed-off-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602235450.103057-5-oupton@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Similar to the handling of descriptor reads, inject an SEA during TTW
when the descriptor access fails for reasons other than a race, such as
a read-only memslot or a bad HVA.
Fixes: bff8aa213d ("KVM: arm64: Implement HW access flag management in stage-1 SW PTW")
Fixes: e4c7dfac2f ("KVM: arm64: nv: Implement HW access flag management in stage-2 SW PTW")
Signed-off-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602235450.103057-4-oupton@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
kvm_translate_vncr() first invalidates the pseudo-TLB entry and
corresponding fixmap in anticipation of installing a new translation.
While the fixmap invalidation does clear the mapping from host stage-1,
it does not clear the L1_VNCR_MAPPED flag. Depending on the state of the
VNCR TLB at vcpu_put(), this could potentially precipitate a BUG_ON() if
vt->cpu is reset.
Share a helper with kvm_vcpu_put_hw_mmu(), ensuring that KVM's view of
the VNCR fixmap is in sync with the state of the VNCR TLB. Give it a
slightly verbose name to make it obvious that it is meant to be used
local to a CPU, unlike other VNCR TLB maintenance.
Fixes: 069a05e535 ("KVM: arm64: nv: Handle VNCR_EL2-triggered faults")
Signed-off-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602235450.103057-3-oupton@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
KVM needs to know if the HW implements FEAT_ATS1A in order to correctly
sanitise HFGITR_EL2.ATS1E1A, which otherwise defaults to RES0 and
AT S1E1A traps are handled as UNDEF.
Solves this by exposing ID_AA64ISAR2_EL1.ATS1A to the rest of the kernel.
Fixes: ff987ffc0c ("KVM: arm64: nv: Add support for FEAT_ATS1A")
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602155430.2088142-4-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Despite having handling code for AT S1E1A, the instruction was
never plugged into the system instruction table, leading to an
exception being injected in the guest.
If the guest is Linux and using the __kvm_at() helper, the exception
is actually handled in the helper, and KVM continues more or less
silently by reentering the guest. Not exactly what you'd expect.
Fix this by plugging the emulation code where required.
Fixes: ff987ffc0c ("KVM: arm64: nv: Add support for FEAT_ATS1A")
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602155430.2088142-3-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
We propagate CPTR_EL2.E0POE from a L1 into the L0 configuration, but
we key this on the L1 guest supporting FEAT_S2POE. This is obviously
wrong, as this bit is solely concerned with Stage-1 translation.
Fix this by making the update depend on FEAT_S1POE.
Fixes: cd931bd609 ("KVM: arm64: nv: Add additional trap setup for CPTR_EL2")
Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Reviewed-by: Oliver Upton <oupton@kernel.org>
Link: https://patch.msgid.link/20260602155430.2088142-2-maz@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
Pull SoC fixes from Arnd Bergmann:
"Following the previous set of fixes, this addresses another
significant number of small issues found in firmware drivers (tee,
optee, qcomtee, qcom ice, exynos acpm) drivers through various tools.
This is about error handling, resource leaks, concurrency and a
use-after-free bug.
The fixes for the Qualcomm ICE driver also introduce interface changes
in the UFS and MMC drivers using it.
Outside of firmware drivers, there are a few fixes across the tree:
- Minor driver code mistakes in the Atmel EBI memory controller, the
i.MX soc ID driver and socfpga boot logic
- A defconfig change to avoid a boot time regression on multiple
qualcomm boards
- Device tree fixes for qualcomm, at91 and gemini, addressing mostly
minor configuration mistakes"
* tag 'soc-fixes-7.1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc: (28 commits)
firmware: samsung: acpm: Fix infinite loop on sequence number exhaustion
firmware: samsung: acpm: Fix missing LKMM barriers in sequence allocator
firmware: samsung: acpm: Fix false timeouts and Use-After-Free in polling
ARM: dts: gemini: Fix partition offsets
ARM: socfpga: Fix OF node refcount leak in SMP setup
soc: qcom: ice: Fix the error code when 'qcom,ice' property is not found
arm64: dts: qcom: eliza: Add power-domain and iface clk for ice node
arm64: dts: qcom: milos: Add power-domain and iface clk for ice node
tee: qcomtee: add missing va_end in early return qcomtee_object_user_init()
tee: fix params_from_user() error path in tee_ioctl_supp_recv
tee: shm: fix shm leak in register_shm_helper()
tee: fix tee_ioctl_object_invoke_arg padding
arm64: defconfig: Enable PCI M.2 power sequencing driver
scsi: ufs: ufs-qcom: Remove NULL check from devm_of_qcom_ice_get()
mmc: sdhci-msm: Remove NULL check from devm_of_qcom_ice_get()
soc: qcom: ice: Return proper error codes from devm_of_qcom_ice_get() instead of NULL
soc: qcom: ice: Return -ENODEV if the ICE platform device is not found
soc: qcom: ice: Fix race between qcom_ice_probe() and of_qcom_ice_get()
ARM: dts: microchip: sam9x7: fix GMAC clock configuration
firmware: samsung: acpm: Fix mailbox channel leak on probe error
...
Due to a rustc bug [1] the -Cforce-unwind-tables=y flag only emits the
uwtable annotation for functions, but not for the module. This means
that compiler-generated functions such as 'asan.module_ctor' do not
receive the uwtable annotation.
When CONFIG_UNWIND_PATCH_PAC_INTO_SCS is enabled, this leads to boot
failures because the dwarf information emitted for the kasan
constructors is wrong, which causes the SCS boot patching code to
patch the constructor in an illegal manner. Specifically, the paciasp
instruction is patched, but the autiasp instruction is not. This
mismatch leads to a crash when the constructor is called during boot.
==================================================================
BUG: KASAN: global-out-of-bounds in do_basic_setup+0x4c/0x90
Read of size 8 at addr ffffffe3cc7eb488 by task swapper/0/1
Specifically the faulting instruction is the (*fn)() to invoke the
constructor in do_ctors() of the init/main.c file.
Once the fix lands in rustc, this flag can be made conditional on the
rustc version. Note that passing the flag on a rustc with the fix
present has no effect.
[ The fix [1] has landed for Rust 1.98.0 (expected release on
2026-08-20).
Thus add a version check as discussed.
- Miguel ]
Fixes: d077242d68 ("rust: support for shadow call stack sanitizer")
Cc: stable@kernel.org
Link: https://github.com/rust-lang/rust/pull/156973 [1]
Reported-by: Bo Ye <bo.ye@mediatek.com>
Debugged-by: Isaac Manjarres <isaacmanjarres@google.com>
Debugged-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Isaac Manjarres <isaacmanjarres@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://patch.msgid.link/20260527-uwtable-module-flag-v1-1-caa41342be4b@google.com
[ Adjusted link and comment. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
KVM/arm64 fixes for 7.1, take #4
- Restore CONFIG_PKVM_DISABLE_STAGE2_ON_PANIC to its former glory by
making sure the config symbol is correctly spelled out in the code
- Don't reset the AArch32 view of the PMU counters to zero when the
guest is writing to them
- Fix an assorted collection of memory leaks in the newly added tracing
code
- Fix the capping of ZCR_EL2 which could be used in an unsanitised way
by an L2 guest
ZCR_EL2 can be updated by a VHE guest hypervisor either using ZCR_EL2
(which traps) or ZCR_EL1 (which does not trap). KVM handles both in
different way:
- on ZCR_EL2 trap, ZCR_EL2.LEN is immediately capped at the VM's own
VL limit. This has the potential to break existing SW that relies
on the full LEN field to be stateful.
- on ZCR_EL1 access, we do absolutely nothing.
On restoring the SVE context for an L2 guest, we directly restore the
guest hypervisor's view of ZCR_EL2 into the physical ZCR_EL2. If the
guest's view of the register was updated using the ZCR_EL2 accessor,
the value has already been sanitised (with the caveat mentioned above).
But if the guest used ZCR_EL1, the raw value is written into the HW,
and the L2 guest can now access VLs that it shouldn't.
Fix all the above by moving the VL capping to the restore points,
ensuring that:
- the HW is always programmed with a capped value, irrespective of
the accessor being used,
- the ZCR_EL2.LEN field is always completely stateful, irrespective
of the accessor being used.
Additionally, move ZCR_EL2 to be a sanitised register, ensuring that
only the LEN field is actually stateful. This requires some creative
construction of the RES0 mask, as the sysreg generation script does
not yet generate RAZ/WI fields.
Fixes: b3d29a8230 ("KVM: arm64: nv: Handle ZCR_EL2 traps")
Signed-off-by: Mark Brown <broonie@kernel.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260529-kvm-arm64-fix-zcr-len-nv-v2-1-86cad51992bd@kernel.org
[maz: rewrote commit message, tidy up access_zcr_el2()]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Qualcomm Arm64 defconfig fixes for v7.1
A number of targets now depends on the M.2 PCIe power sequencing driver,
enable this to keep these devices functional with a defconfig build.
* tag 'qcom-arm64-defconfig-fixes-for-7.1' of https://git.kernel.org/pub/scm/linux/kernel/git/qcom/linux:
arm64: defconfig: Enable PCI M.2 power sequencing driver
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Qualcomm Arm64 DeviceTree fixes for v7.1
Add missing power-domain and iface clocks for the ICE node of Eliza and
Milos to avoid the validation errors that resulted from late binding
changes. Also drop the reference clock for the USB QMP PHYs, for the
same reason.
Avoid touching the 20'th I2C bus on the Hamoa-based (X Elite) Dell
laptops, as this conflicts with the battery management firmware.
* tag 'qcom-arm64-fixes-for-7.1' of https://git.kernel.org/pub/scm/linux/kernel/git/qcom/linux:
arm64: dts: qcom: eliza: Add power-domain and iface clk for ice node
arm64: dts: qcom: milos: Add power-domain and iface clk for ice node
arm64: dts: qcom: x1-dell-thena: remove i2c20 (battery SMBus) and reserve its pins
arm64: dts: qcom: glymur: Drop RPMh CXO clocks from QMP PHYs
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
When KVM derives the translation granule for emulated stage-1 and
stage-2 walks, it decodes TCR/VTCR.TGx and treats the granule as-is.
This is wrong when the guest programs a granule size that is not
advertised in the guest's ID_AA64MMFR0_EL1.TGRAN* fields.
Architecturally, such a value must be treated as an implemented granule
size. Choose an available one while prioritizing PAGE_SIZE.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260414000334.3947257-5-weilin.chang@arm.com
[maz: minor tidying up]
Signed-off-by: Marc Zyngier <maz@kernel.org>
TLBI handling derives the invalidation range from guest VTCR_EL2.TG0 in
get_guest_mapping_ttl() and compute_tlb_inval_range(). Switch these to
use a helper that returns the decoded VTCR_EL2.TG0 granule size instead
of decoding it inline.
This keeps the granule size derivation in one place and prepares for
following changes that adjust the effective granule size.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260414000334.3947257-4-weilin.chang@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
The current code decodes TCR.TG0/TG1 and VTCR.TG0 inline at several
places. Extract this logic into helpers so the granule size can be
derived in one place. This enables us to alter the effective granule
size in the same place, which we will do in a later patch.
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
Link: https://patch.msgid.link/20260414000334.3947257-3-weilin.chang@arm.com
Signed-off-by: Marc Zyngier <maz@kernel.org>
Currently, it is not necessary for __finalise_el2() to configure
TPIDR_EL2:
* The hyp stub code does not consume the value of TPIDR_EL2.
* On the boot cpu, TPIDR_EL1 is used for the percpu offset until the
ARM64_HAS_VIRT_HOST_EXTN cpucap is detected and boot alternatives
are patched. Before boot alternatives are patched,
cpu_copy_el2regs() will copy TPIDR_EL1 into TPIDR_EL2. It is not
necessary for __finalise_el2() to initialise TPIDR_EL2 before this.
* Secondary CPUs are brought up after boot alternatives have been
patched, and __secondary_switched() will initialize TPIDR_EL2 in
'init_cpu_task', after finalise_el2() calls __finalise_el2()
* KVM hyp code which may consume TPIDR_EL2 is brought up after all
secondaries have been booted, once TPIDR_El2 has been configured on
all CPUs.
Remove the redundant initialisation from __finalise_el2().
Cc: Oliver Upton <oupton@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Reviewed-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260518153127.6078-1-will@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
