* for-next/mm:
arm64/efi: Avoid voluntary preemption with efi_mm installed
arm64: mm: Unmap kernel data/bss entirely from the linear map
arm64: mm: fix accidental linear mapping of no-map reserved memory
arm64: pgtable: convert pte_present() from macro to static inline
arm64: mm: Treat all devices as dma-coherent when CLIDR_EL1.LoC == 0
* for-next/misc:
arm64: Disable KCSAN instrumentation in delay.o
MAINTAINERS: arm64: Add Mark Rutland as an official Reviewer
arm64: smp: Fix IPI teardown for GICv5 flow
arm64: futex: Consolidate 'old == new' check in __lsui_cmpxchg32()
arm64: ftrace: allow DIRECT_CALLS without CALL_OPS
arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
* for-next/coco:
arm64: RSI: fix field-spanning write warning in attestation token init
virt: arm-cca-guest: Drop unused assignment of platform_device_id driver data
arm64/coco: Add pKVM as a CC platform
arm64/mm: Simplify SWIOTLB setup in arch_mm_preinit()
virt: arm-cca-guest: use migrate_disable() for attestation token requests
Gus reports a bad kernel memory access when using software PAN
(CONFIG_ARM64_SW_TTBR0_PAN=y) on a machine with support for EFI runtime
services:
Unable to handle kernel access to user memory outside uaccess routines
at virtual address 00000000f322ff30
Mem abort info:
ESR = 0x0000000096000004
FSC = 0x04: level 0 translation fault
Internal error: Oops: 0000000096000004 [#1] SMP
Workqueue: efi_rts_wq efi_call_rts
pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : efi_call_rts+0xd8/0x288
Call trace:
efi_call_rts+0xd8/0x288 (P)
process_one_work+0x178/0x4f8
worker_thread+0x194/0x328
This is because the fpsimd context management code called from
__efi_fpsimd_begin() can preempt voluntarily, returning later to the EFI
code with an incorrect value for TTBR0_EL1 thanks to the deferred mm
switching used by the software PAN implementation.
Since EFI runtime services cannot preempt voluntarily and because the
fpsimd switching code does not rely on the TTBR0_EL1 mappings, simply
reorder the fpsimd switch so that it occurs before we change the
page-table.
Cc: Ard Biesheuvel <ardb@kernel.org>
Reported-by: Gus Bourg <gus@bourg.net>
Tested-by: Gus Bourg <gus@bourg.net>
Fixes: a5baf582f4 ("arm64/efi: Call EFI runtime services without disabling preemption")
Link: https://lore.kernel.org/all/20260806000144.3388823-1-gus@bourg.net/
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
The following BTI exception was seen when loading a livepatch module:
Internal error: Oops - BTI: 0000000036000001 [#1] SMP
pstate: 634004c9 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=jc)
pc : kill_orphaned_pgrp+0x0/0x150
lr : do_exit+0x498/0xaf0 [livepatch_combined]
The problem is that the patch module's do_exit() is branching to a
static function in vmlinux using a module PLT veneer (indirect branch),
but the target function doesn't have a BTI landing pad.
Clang 21+ omits the landing pad for static functions which can only be
reached by a direct branch. That's normally fine for ordinary modules
which only branch to global exported functions, but Mark Brown points
out [1] that this isn't guaranteed if the module branches between
sections. Futhermore, livepatch modules use klp relocations to reference
arbitrary kernel symbols, so with CONFIG_RANDOMIZE_MODULE_REGION_FULL
the module is far enough from the kernel that every R_AARCH64_CALL26
needs a PLT.
Put Clang 21+ in the naughty corner alongside GCC, which suffers from
the same issue, by disabling CONFIG_ARM64_BTI_KERNEL until we have a
version of the toolchain with the problem resolved.
Cc: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/da06bbd3-d04b-4d0f-b331-f5b91bc373a5@sirena.org.uk [1]
Fixes: fd1e0fd71f ("arm64: Implement HAVE_LIVEPATCH")
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
[will: Stitched together commit message, diff and bug number]
Signed-off-by: Will Deacon <will@kernel.org>
KCSAN relies on udelay() for injecting delays. To avoid recursively
triggering a watchpoint, where KCSAN sets up watchpoint on an address
that is accessed by udelay() in the same thread, disable instrumentation
in arm64's delay implementation.
Paul found a manifestation of this as follows:
| BUG: KCSAN: data-race in __delay / set_need_resched_current
|
| read (marked) to 0xffff000005899b48 of 8 bytes by interrupt on cpu 8:
| __delay+0xb0/0x378
| __udelay+0x4c/0x60
| kcsan_setup_watchpoint+0x3b4/0x820
| __tsan_unaligned_write4+0x228/0x26c
| set_need_resched_current+0x138/0x1a8
| rcu_exp_handler+0x418/0x4a0
| __flush_smp_call_function_queue+0x36c/0x4a0
| generic_smp_call_function_single_interrupt+0x20/0x30
| ipi_handler+0xec/0x558
| handle_percpu_devid_irq+0x220/0x2a0
| generic_handle_domain_irq+0x84/0xb4
| gic_handle_irq+0x64/0x144
| call_on_irq_stack+0x30/0x48
| do_interrupt_handler+0x80/0xb8
| el1_interrupt+0x3c/0x60
| el1h_64_irq_handler+0x18/0x24
| el1h_64_irq+0x6c/0x70
| smp_call_function_single+0x18c/0x25c
| sync_rcu_exp_select_node_cpus+0x534/0x8bc
| rcu_exp_sel_wait_wake+0x358/0xef4
| wait_rcu_exp_gp+0x30/0x44
| kthread_worker_fn+0x1b4/0x5dc
| kthread+0x1d8/0x204
| ret_from_fork+0x10/0x20
|
| write to 0xffff000005899b4c of 4 bytes by interrupt on cpu 8:
| set_need_resched_current+0x138/0x1a8
| [...]
This matches what is already done in arch/x86/lib/Makefile.
Reported-by: "Paul E. McKenney" <paulmck@kernel.org>
Fixes: dd03762ab6 ("arm64: Enable KCSAN")
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
When a user unbinds an MSC and that MSC is the only MSC left for a
component then the corresponding mpam_component will be freed. If the user
then goes on to read the schemata file in the resctrl filesystem then the
mpam_component will be accessed from resctrl_arch_get_config() leading to a
use after free.
As the MPAM driver is not a module the unbind sysfs interface is the only
way to trigger the remove. Instead of dealing with the complexity of
allowing some unused MSC to unbind just remove the unbind sysfs interface.
Fixes: f04046f257 ("arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate")
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
If a user unbinds an MSC after mpam_disable() has been run in response
to an error interrupt then a dereference of a NULL pointer occurs as
mpam_disable() sets the drvdata to NULL. Add an early return to the driver
remove callback to avoid this.
Fixes: f04046f257 ("arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate")
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The linear aliases of the kernel text and rodata are also mapped
read-only in the linear map. Given that the contents of these regions
are mostly identical to the version in the loadable image, mapping them
read-only and leaving their contents visible is a reasonable hardening
measure.
Data and bss, however, are now also mapped read-only but the contents of
these regions are more likely to contain data that we'd rather not leak.
So let's unmap these entirely in the linear map when the kernel is
running normally.
When going into hibernation or waking up from it, these regions need to
be mapped, so map the region initially, and toggle the valid bit so
map/unmap the region as needed.
Doing so is required because pages covering the kernel image are marked
as PageReserved, and therefore disregarded for snapshotting by the
hibernate logic unless they are mapped.
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Kevin Brodsky <kevin.brodsky@arm.com>
Cc: Liz Prucka <lizprucka@google.com>
Cc: Seth Jenkins <sethjenkins@google.com>
Cc: Kees Cook <kees@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Jann Horn <jannh@google.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Commit 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
renamed the cpu_supports_bbml2_noabort() helper to cpu_supports_bbml3(),
as the Linux-defined "noabort" semantics have now been incorporated into
the architecture under the BBML3 feature.
Update the caller in the SMMUv3 SVA driver to use the new function.
There is a slightly oddity in that the SMMUv3 architecture already
defined BBML2 in such a way that aborts were prohibited, so we use the
BBML3 feature on the CPU to enable BBML2 in the SMMU.
Fixes: 94104e3cfa ("arm64: cpufeature: Rename BBML2_NOABORT as BBML3")
Signed-off-by: Will Deacon <will@kernel.org>
When reserved-memory regions with the "no-map" property are not
page-aligned, the kernel may accidentally map them into the linear
mapping, contradicting the no-map semantics.
The root cause is a mismatch between /proc/iomem's address boundaries
and the actual page table mapping boundaries:
1. /proc/iomem derives its ranges from memblock via
memblock_region_reserved_base_pfn/memblock_region_reserved_end_pfn,
which perform PFN rounding so the displayed boundaries are
page-aligned. This gives the impression that the no-map region
occupies whole pages.
2. However, memblock_mark_nomap() splits memblock.memory regions at
exact byte boundaries (memblock_isolate_range preserves raw DT
base/size with no alignment). When for_each_mem_range iterates the
non-NOMAP regions adjacent to a no-map region, it returns start/end
values that are NOT page-aligned — they are the precise byte
boundaries from the memblock split.
3. These sub-page-aligned values are passed to
__create_pgd_mapping_locked(), which does:
phys &= PAGE_MASK;
addr = virt & PAGE_MASK;
end = PAGE_ALIGN(virt + size);
The downward rounding of phys via PAGE_MASK extends the mapped
range backward into the adjacent no-map region, effectively
including no-map memory in the linear mapping.
For example, with 64K pages, reserved_region@A2000000 (base=0xA2000000,
size=0x8000, no-map) causes for_each_mem_range to return
start=0xA2008000 for the next mappable region. After phys &= PAGE_MASK,
the actual mapping starts at 0xA2000000 — the entire no-map region is
incorrectly mapped.
Fix this by rounding the mappable range inward to PAGE_SIZE boundaries
before passing it to __map_memblock: start is rounded UP and end is
rounded DOWN. This ensures the mapped area never overlaps with adjacent
no-map regions. The cost is at most one page of unmapped gap at each
boundary, which is preferable to violating no-map semantics.
Signed-off-by: liulhong617 <liulhong617@gmail.com>
Signed-off-by: Will Deacon <will@kernel.org>
Overriding the idle mechanism might be useful for debugging and performance
testing. Add a cmdline parameter for it, similar to the existing idle=
parameter already present for the x86 and ppc architectures.
It is also useful on platforms where the WFI instruction misbehaves,
such as Apple Silicon SoCs. Generally, a misbehaving instruction should
be treated as an erratum and patched using the alternatives framework.
However, in the Apple Silicon case we need more flexibility because it is
difficult to detect whether the erratum applies. For example, Linux VMs
inside macOS have the same MIDR and may even seem like they're running
in EL2 in the case of NV, but should continue using WFI (it's trapped and
handled correctly by the hypervisor there). Thus, we prefer to
let the m1n1 bootloader add the idle=nop parameter[1].
Link[1]: https://lore.kernel.org/all/99b69262-e54b-424e-baa2-96ef7013b87a@kernel.org/
Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Yureka Lilian <yureka@cyberchaos.dev>
Signed-off-by: Will Deacon <will@kernel.org>
Add ID_AA64MMFR2_EL1.BBM based BBML3 feature detection in
cpu_supports_bbml3() so that cpus with the feature would
not have to be added into MIDR based supports_bbml3_list.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
[will: Tidy up cpu_supports_bbml3()]
Signed-off-by: Will Deacon <will@kernel.org>
- As bbml2_noabort is functionally equivalent to bbml3,
rename cpu/system_supports_bbml2_noabort to
cpu/system_supports_bbml3.
The ARM64 capability name is also renamed accordingly.
- As BBML2_NOABORT or the equivalent BBML3 is the
kernel requirement for setting up linear map with
block/contpte mappings and not BBML2, replace all
bbml2 references with bbml3.
FEAT_BBML3, is introduced as part of 2025 Architecture Extensions.
https://developer.arm.com/documentation/109697/2026_03/2025-Architecture-Extensions
No functional changes are introduced with this patch.
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Add below cpus to the midr list, which supports
BBML2_NOABORT.
Cortex A520(AE)
Cortex A715
Cortex A720(AE)
Cortex A725
Neoverse N3
C1-Nano
C1-Pro
C1-Ultra
C1-Premium
C1-Ultra and C1-Premium both suffer from erratum 3683289,
where Break-Before-Make must be followed to avoid a livelock.
For both CPUs, the erratum is fixed from r1p1.
Hence we do not enable BBML2_NOABORT for CPU revisions <= r1p0.
The relevant SDENs are:
* C1-Ultra: https://developer.arm.com/documentation/111077/9-00/
* C1-Premium: https://developer.arm.com/documentation/111078/9-00/
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
pte_present() is used as the basis for both pmd_present() and pud_present().
It is currently implemented as a macro composed of pte_val() and
pte_present_invalid().
When pte_present() or its higher-level variants are used directly with
ptep_get() or pXdp_get(), for example:
pte_present(ptep_get(pte));
pmd_present(pmdp_get(pmd));
pud_present(pudp_get(pud));
the macro expansion causes the compiler to evaluate the argument twice,
resulting in redundant loads. For example, pte_present() expands to:
!pte_val(READ_ONCE(*pte) || pte_present_invalid(READ_ONCE(*pte))
A typical example is pud_free_pmd_page(), where the expansion of
pmd_present() generates:
...
/* pmd_present() (x20 = pmdp) */
1b88: f9400288 ldr x8, [x20] // read pmdp.
1b8c: f9000fa8 str x8, [x29, #0x18]
1b90: 3707fec8 tbnz w8, #0x0, 0x1b68 <pud_free_pmd_page+0xd0>
1b94: f9400288 ldr x8, [x20] // redundant read of pmdp.
1b98: 8a170109 and x9, x8, x23
1b9c: f9000fa8 str x8, [x29, #0x18]
1ba0: f120013f cmp x9, #0x800
1ba4: 54fffe20 b.eq 0x1b68 <pud_free_pmd_page+0xd0>
1ba8: 17fffff4 b 0x1b78 <pud_free_pmd_page+0xe0>
...
Convert pte_present() to static inline function so that prevent the
generation of redundant code and move pte_valid() and
pte_present_invalid() further up so the inline function can use them.
After this change, the generated code becomes:
...
/* pmd_present() (x20 = pmdp) */
1a30: f9400288 ldr x8, [x20]
1a34: 8a170109 and x9, x8, x23
1a38: f9000fa8 str x8, [x29, #0x18]
1a3c: f120013f cmp x9, #0x800
1a40: 54fffe80 b.eq 0x1a10 <pud_free_pmd_page+0xd0>
1a44: 3607fee8 tbz w8, #0x0, 0x1a20 <pud_free_pmd_page+0xe0>
1a48: 17fffff2 b 0x1a10 <pud_free_pmd_page+0xd0>
...
This eliminates the redundant load and also reduces code size at
call sites using this pattern. For example, pud_free_pmd_page() shrinks
from 7,500 bytes to 7,148 bytes, a reduction of approximately 4.7%.
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
Mark has been part of the arm64 furniture for as long as the code has
existed in mainline and is consistently one of the most active and
knowledgeable reviewers that we have.
Make it official.
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
T241-MPAM-6 causes all MBWU counter formats to count 64-byte
requests instead of bytes. Commit dc48eb1ff2 excluded the 63-bit
MSMON_MBWU_LWD format while scaling the shorter counters. Systems
selecting the preferred 63-bit counter consequently report bandwidth
values that are 64 times too small.
Apply the scale to both the sampled value and overflow correction for
the 63-bit format. Unsigned arithmetic retains modulo-u64 behavior
when the scaled counter range exceeds u64.
Fixes: dc48eb1ff2 ("arm_mpam: Add workaround for T241-MPAM-6")
Link: https://lore.kernel.org/lkml/20240816131432.993859-1-sdonthineni@nvidia.com/
Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
Reviewed-by: Fenghua Yu <fenghuay@nvidia.com>
Tested-by: Fenghua Yu <fenghuay@nvidia.com>
Reviewed-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
MPAM is able to emulate ABMC, i.e. mbm_event mode, by making memory
bandwidth monitors assignable. Rather than supporting the 'default'
mbm_assign_mode always use 'mbm_event' mode even if there are sufficient
memory bandwidth monitors. The per monitor event configuration is only
provided by resctrl when in 'mbm_event' mode and so only allowing
'mbm_event' mode will make it easier to support per-monitor event
configuration for MPAM. For the moment, the only event supported is
mbm_total_event with no bandwidth type configuration. The 'mbm_assign_mode'
file will still show 'default' when there is no support for memory
bandwidth monitoring.
The monitors need to be allocated from the driver, and mapped to whichever
control/monitor group resctrl wants to use them with.
Add a second array to hold the monitor values indexed by resctrl's cntr_id.
When CDP is in use, two monitors are needed so the available number of
counters halves. Platforms with one monitor will have zero monitors when
CDP is in use.
Co-developed-by: James Morse <james.morse@arm.com>
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Tested-by: Fenghua Yu <fenghuay@nvidia.com>
Tested-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Fenghua Yu <fenghuay@nvidia.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
resctrl has two types of bandwidth counters, NUMA-local and global. MPAM
can only count globally; either using MSC at the L3 cache or in the memory
controllers. When global and local equate to the same thing continue just
to call it global.
Pick the corresponding MPAM classes to back the MBM counters. As resctrl
requires all monitors to be at the L3 cache, we can only use the counters
at the memory controllers when they have the same topology as the L3 cache
and the traffic they see if the same. In particular, for the bandwidth
counters at the memory controllers to be exposed to resctrl it is required
there is a single L3 cache and a single NUMA node as otherwise cross NUMA
traffic will be counted at the wrong instance.
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Ben Horgan <ben.horgan@arm.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Tested-by: Zeng Heng <zengheng4@huawei.com>
Tested-by: Fenghua Yu <fenghuay@nvidia.com>
Tested-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Fenghua Yu <fenghuay@nvidia.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Signed-off-by: Will Deacon <will@kernel.org>
Sashiko reported that during CPU offlining, __cpu_disable() is
executed by the stopper thread via take_cpu_down() with local
interrupts disabled. __cpu_disable() calls ipi_teardown(), which
invokes ipi_lpi_disable(). For the GICv5 flow, this eventually calls
the sleepable disable_irq().
This can be reproduced easily with CONFIG_DEBUG_ATOMIC_SLEEP=y by
offlining a CPU:
BUG: sleeping function called from invalid context at kernel/irq/manage.c:702
in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 20, name: migration/1
preempt_count: 1, expected: 0
no locks held by migration/1/20.
irq event stamp: 186
hardirqs last enabled at (185): [<ffff800080b084c8>] _raw_spin_unlock_irq+0x38/0x68
hardirqs last disabled at (186): [<ffff8000801f8e08>] multi_cpu_stop+0xc8/0x190
softirqs last enabled at (80): [<ffff8000800c48b8>] handle_softirqs+0x410/0x468
softirqs last disabled at (75): [<ffff8000800102f4>] __do_softirq+0x1c/0x28
Fix this by using disable_irq_nosync() instead, which is safe in this
atomic context.
Fixes: ba1004f861 ("arm64: smp: Support non-SGIs for IPIs")
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
There is no functional requirement to keep the entries in arm64's
tools/cpucaps file sorted alphabetically and, in fact, they have fallen
out of order over time. Given that only the arm64 tree touches this file
and the perceived benefit of sorting was to help with conflicts, just
remove the stale comment and accept that we're collectively not very
good at the alphabet.
In the limited cases where a specific ordering is important, we enforce
it through build-time assertions (e.g. in can_use_gic_priorities()).
Cc: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Reported-by: Marc Zyngier <maz@kernel.org>
Acked-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
On systems where CLIDR_EL1.LoC == 0, no cache maintenance is required
when cleaning or invalidating to the Point of Coherency and therefore
all DMA agents can be treated as coherent.
Extend arch_setup_dma_ops() to take CLIDR_EL1.LoC into account when
setting the DMA ops for a device, emitting a warning message if the
firmware advertises a non-coherent device on a fully coherent system.
Cc: Steffen Eiden <seiden@linux.ibm.com>
Cc: Andreas Grapentin <gra@linux.ibm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Commit 7f16357378 ("arm64: proton-pack: Fix hard lockup due to print in
scheduler context") moved the "mitigation disabled" printks into
spectre_print_disabled_mitigations(). For spectre-v2 and spectre-v4 only
the pr_info_once() calls were removed, but for spectre-bhb the whole
branch went with the print:
- } else if (cpu_mitigations_off() || __nospectre_bhb) {
- pr_info_once("spectre-bhb mitigation disabled ...\n");
spectre_bhb_enable_mitigation() therefore no longer tests __nospectre_bhb
or cpu_mitigations_off() and the mitigation is enabled regardless of the
command line. The parameter is still parsed and its flag is still checked
by spectre_print_disabled_mitigations(), so the kernel prints "spectre-bhb
mitigation disabled by command-line option" while
/sys/devices/system/cpu/vulnerabilities/spectre_v2 reports "Mitigation:
CSV2, BHB" and the vectors are switched to EL1_VECTOR_BHB_LOOP.
The only remaining escape is the SPECTRE_VULNERABLE arm at the top of the
chain, which a CSV2 core never reaches, so from Cortex-A76 and Neoverse N1
onwards both nospectre_bhb and mitigations=off are ignored. Both are
documented in Documentation/admin-guide/kernel-parameters.txt.
The identical mistake was made on the neighbouring compile-time-option
branch immediately before this regression and fixed shortly afterwards;
this command-line branch was missed.
build_bhb_mitigation() in arch/arm64/net/bpf_jit_comp.c still tests both
flags, so nospectre_bhb currently keeps the exception-vector loop while
dropping the cBPF epilogue mitigation.
Restore the check, folded into a spectre_bhb_mitigations_off() helper
alongside its spectre_v2/v4 counterparts, and use it for the boot-time
print in spectre_print_disabled_mitigations() as well. The print itself
already lives there and does not need restoring.
Tested under QEMU with -cpu neoverse-n1 (CSV2, no ECBHB, no CLRBHB).
Before, spectre_v2 read "Mitigation: CSV2, BHB" with and without the
option; after, nospectre_bhb and mitigations=off both give "Mitigation:
CSV2, but not BHB" and a boot without either is unchanged.
Fixes: 7f16357378 ("arm64: proton-pack: Fix hard lockup due to print in scheduler context")
Assisted-by: Claude:claude-opus-5
Cc: stable@vger.kernel.org
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
Signed-off-by: Will Deacon <will@kernel.org>
The "visible" column was dropped so that last line's extra pieces
should also be dropped to prevent a docs build error:
Documentation/arch/arm64/cpu-feature-registers.rst:170: ERROR: Malformed table.
Right border not aligned or missing.
+------------------------------+---------+
| Name | bits |
+------------------------------+---------+
| RNDR | [63-60] |
+------------------------------+---------+
| TS | [55-52] |
+------------------------------+---------+
...
+------------------------------+---------+
| AES | [7-4] |
+------------------------------+---------+---------+ [docutils]
Fixes: bc7f7ddc64 ("arm64: Remove hidden bitfields from cpu-feature-registers.rst")
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Will Deacon <will@kernel.org>
The LSUI futex implementation relies on a cmpxchg() loop to implement
FUTEX_OP_XOR, as the architecture doesn't provide unprivileged *EOR
atomics. Since the unprivileged 'CAST' instructions used to implement
the cmpxchg() can only operate on 64-bit memory locations, the
__lsui_cmpxchg32() helper function performs a song and dance to marshall
the 32-bit futex value into the correct part of a 64-bit register and
fill the remaining bytes with the neighbouring data.
A consequence of this structure is that the 'CAST' failure/success
condition ends up being split into two separate 32-bit checks across
__lsui_cmpxchg32() and its caller. This is a little fiddly to read and
introduces some additional local variables which can be avoided if the
check is done in one place.
Tweak __lsui_cmpxchg32() so that it performs the full 64-bit check on
the value returned from the 'CAST' instruction and returns success to
its caller only in the case that the cmpxchg() operation has succeeded.
With that in place, simplify the outer loop in __lsui_futex_atomic_eor()
to pass 'oldval' by reference and return unless the cmpxchg() operation
returns -EAGAIN. __lsui_futex_cmpxchg() then swallows the -EAGAIN if the
futex word has changed.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Yeoreum Yun <yeoreum.yun@arm.com>
Reviewed-by: Yeoreum Yun <yeoreum.yun@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The challenge is passed in registers a1 through a8. However, copying to
®s.a1 makes FORTIFY treat the destination as the single a1 field,
resulting in a field-spanning write warning. [1]
Overlay the SMCCC register structure with an RSI-specific argument
layout and copy the challenge into an explicit 64-byte array. This keeps
the existing a1-a8 argument encoding while giving the copy a correctly
sized destination object.
[1]
memcpy: detected field-spanning write (size 64) of single field "®s.a1" at ./arch/arm64/include/asm/rsi_cmds.h:119 (size 8)
WARNING: ./arch/arm64/include/asm/rsi_cmds.h:119 at rsi_attestation_token_init+0xdc/0xf8 [arm_cca_guest], CPU#0: cat/3314
Fixes: b880a80011 ("arm64: rsi: Add RSI definitions")
Signed-off-by: Kohei Enju <enju.kohei@fujitsu.com>
Signed-off-by: Will Deacon <will@kernel.org>
We currently have a visibility column in the tables for the registers in
cpu-feature-registers.rst but this is always "y" for every register other
than ID_AA64PFR0_EL1. Given that the documentation of the full set of
bitfields is readily available in the architecture documentation it is
redundant for us to explicitly document things we don't advertise, and the
kernel documentation will inevitably lag the architecture.
Just remove the visibility column and hidden bitfields.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
In order to make it a bit easier to work with sort the list of registers in
cpu-feature-registers.rst lexically. There should be no content changes
resulting from this patch.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
We have been rather lax in updating the list of visible bitfields in the
ID registers in cpu-feature-registers.rst, it is currently missing several
of the registers and quite a few bitfields in existing registers. Bring it
into sync with current -next.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
cpu-feature-regsters.rst documents the set of userspace visible ID
registers. At present the section for each register is numbered, this has
lead to the registers being documented in a haphazard order as new ones
have been added to the end of the list to avoid renumbering. Remove the
numbers so we can avoid this problem in future.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Drop the CALL_OPS requirement from the HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
select. Configurations that keep CALL_OPS (!CFI clang builds, and GCC builds
without CC_OPTIMIZE_FOR_SIZE) are unchanged. CALL_OPS-less configurations
take the ftrace_caller ops-dispatch path for out-of-range direct calls,
trading the per-callsite fast path for working BPF trampolines; in-range
attachments still branch directly with no overhead. GCC -Os builds also gain
DIRECT_CALLS as a side effect. That is intended: s390 and loongarch already
ship DIRECT_CALLS without any per-callsite fast path.
Assisted-by: Claude:unspecified
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Clayton Craft <craftyguy@postmarketos.org>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
ftrace_modify_call() is guarded by CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS and
calls ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec)) directly, which only
exists when CALL_OPS is enabled.
Generic ftrace also needs ftrace_modify_call() when
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled, to retarget a callsite
between two non-FTRACE_ADDR destinations, as happens when a direct
trampoline is modified. The next patch allows DIRECT_CALLS without CALL_OPS,
so widen the guard to cover both configurations and switch the body to the
ftrace_rec_update_ops() wrapper, which already has a stub for the !CALL_OPS
case. ftrace_make_call() already uses the same wrapper today.
No functional change: with CALL_OPS enabled, ftrace_rec_update_ops() expands
to the exact call this replaces.
Assisted-by: Claude:unspecified
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Clayton Craft <craftyguy@postmarketos.org>
Reviewed-by: Puranjay Mohan <puranjay@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
observed by a peripheral before an older, non-overlapping Device-nGnR*
store to the same peripheral. This breaks the program-order guarantee
that software expects for Device-nGnR* accesses and can leave a
peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
- A PE executes a Device-nGnR* store followed by a younger
Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program
order, and there are no intervening Device-nGnR* loads.
- There is no DSB or full DMB between the store and the load.
- Specific microarchitectural and timing conditions occur.
Insert a DMB OSH immediately before each raw MMIO load on affected CPUs.
As a full barrier, DMB OSH orders the older Device store before the
younger Device load and prevents the erroneous observation.
Add the barrier directly to the __raw_read*() helpers, independently of
the existing device-load-acquire alternative. On affected CPUs this adds
one DMB OSH per raw MMIO load, including each load used by
memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.
Co-developed-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
Link: https://lore.kernel.org/all/akPQ8F3OgER621UP@willie-the-truck/
Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
The driver explicitly sets the .driver_data member of struct
platform_device_id to zero without relying on that value. Drop this
unused assignment.
While touching this array use a named initializer for .name.
Signed-off-by: Uwe Kleine-König (The Capable Hub) <u.kleine-koenig@baylibre.com>
Signed-off-by: Will Deacon <will@kernel.org>
arm64 implements CPPC FFH feedback-counter reads using AMU counters.
Because those counters must be sampled on the target CPU, reading the
delivered and reference counters separately widens the observation window
between them.
Implement the paired FFH feedback-counter read hook on arm64 and sample
both AMU counters together before decoding the requested CPC register
values.
Also factor the FFH bitfield extraction logic into a helper and reuse
it from the existing single-counter FFH read path.
Tested-by: Sumit Gupta <sumitg@nvidia.com>
Reviewed-by: Sumit Gupta <sumitg@nvidia.com>
Tested-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Reviewed-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-by: Jeremy Linton <jeremy.linton@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
cppc_get_perf_ctrs() reads the delivered and reference performance
counters one at a time.
Allow architectures to provide both FFH feedback counters in one
operation when that either narrows the sampling window or avoids extra
cross-CPU reads. Add a small FFH-specific hook for that case and fall
back to the existing per-register reads when unsupported.
Tested-by: Sumit Gupta <sumitg@nvidia.com>
Reviewed-by: Sumit Gupta <sumitg@nvidia.com>
Tested-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Reviewed-by: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
Signed-off-by: Pengjie Zhang <zhangpengjie2@huawei.com>
Acked-by: Rafael J. Wysocki (Intel) <rafael@kernel.org>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Reviewed-by: Jeremy Linton <jeremy.linton@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
pKVM does support memory encryption, expose that to the rest of
the kernel through cc_platform_has()
At the moment, all devices inside the guest are emulated which
requires its memory to be shared back to the host (decrypted), so
set force_dma_unencrypted() to always return true.
Although, typically pKVM guests rely on restricted-dma-pools to
bounce traffic, with this change, it is possible to solely rely on
the default SWIOTLB for that (assuming the appropriate size is set
from the command line)
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
At the moment, arch_mm_preinit() checks if the system has limited
addressing or is running under CCA to enable SWIOTLB, only after to
be forced to true anyway if it was false due to
CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC being unconditionally true for
arm64.
Simplify this logic, by making it clear that SWIOTLB is always used
but its size depends on the address layout of the system.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Tested-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
The RSI attestation token init and continue calls must be issued from
the same CPU. arm_cca_report_new() currently snapshots the CPU number
and uses smp_call_function_single() to issue those calls on that CPU.
With CONFIG_DEBUG_PREEMPT=y, the smp_processor_id() call used for the
snapshot triggers a debug splat [0] because it runs in preemptible
context. The snapshot does not pin the task to that CPU; it is only used
to choose the target CPU for smp_call_function_single(), which can fail
if that CPU is no longer available.
Use migrate_disable() and issue the token init and continue operations
directly, without the smp_call_function_single() callbacks. This keeps
the token request sequence on the same CPU while preserving a sleepable
context for the GFP_KERNEL allocations needed after the init call.
[0]
BUG: using smp_processor_id() in preemptible [00000000] code: cca-workload-at/264
caller is debug_smp_processor_id+0x20/0x30
CPU: 0 UID: 0 PID: 264 Comm: cca-workload-at Not tainted 7.1.0-rc1-00044-g55542ab273f2 #80 PREEMPT(lazy)
Hardware name: linux,dummy-virt (DT)
Call trace:
[...]
check_preemption_disabled+0xd8/0xf8
debug_smp_processor_id+0x20/0x30
arm_cca_report_new+0x48/0x278
tsm_report_read+0x154/0x1f8
tsm_report_outblob_read+0x20/0x38
configfs_bin_read_iter+0x118/0x208
vfs_read+0x220/0x318
[...]
Fixes: 7999edc484 ("virt: arm-cca-guest: TSM_REPORT support for realms")
Signed-off-by: Kohei Enju <enju.kohei@fujitsu.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Tested-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Reviewed-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Steven Price <steven.price@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
