Commit Graph

1234121 Commits

Author SHA1 Message Date
Will Deacon
ccaeeec529 Merge branch 'for-next/lpa2-prep' into for-next/core
* for-next/lpa2-prep:
  arm64: mm: get rid of kimage_vaddr global variable
  arm64: mm: Take potential load offset into account when KASLR is off
  arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
  arm64: Add ARM64_HAS_LPA2 CPU capability
  arm64/mm: Add FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2]
  arm64/mm: Update tlb invalidation routines for FEAT_LPA2
  arm64/mm: Add lpa2_is_enabled() kvm_lpa2_is_enabled() stubs
  arm64/mm: Modify range-based tlbi to decrement scale
2024-01-04 12:27:42 +00:00
Will Deacon
88619527b4 Merge branch 'for-next/kbuild' into for-next/core
* for-next/kbuild:
  efi/libstub: zboot: do not use $(shell ...) in cmd_copy_and_pad
  arm64: properly install vmlinuz.efi
  arm64: replace <asm-generic/export.h> with <linux/export.h>
  arm64: vdso32: rename 32-bit debug vdso to vdso32.so.dbg
2024-01-04 12:27:35 +00:00
Will Deacon
79eb42b269 Merge branch 'for-next/fpsimd' into for-next/core
* for-next/fpsimd:
  arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
  arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
  arm64: fpsimd: Drop unneeded 'busy' flag
2024-01-04 12:27:29 +00:00
Will Deacon
e90a8a210f Merge branch 'for-next/early-idreg-overrides' into for-next/core
* for-next/early-idreg-overrides:
  arm64/kernel: Move 'nokaslr' parsing out of early idreg code
  arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit
  arm64: idreg-override: Avoid sprintf() for simple string concatenation
  arm64: idreg-override: avoid strlen() to check for empty strings
  arm64: idreg-override: Avoid parameq() and parameqn()
  arm64: idreg-override: Prepare for place relative reloc patching
  arm64: idreg-override: Omit non-NULL checks for override pointer
2024-01-04 12:27:13 +00:00
Will Deacon
3f35db4e68 Merge branch 'for-next/cpufeature' into for-next/core
* for-next/cpufeature:
  arm64: Align boot cpucap handling with system cpucap handling
  arm64: Cleanup system cpucap handling
  arm64: Kconfig: drop KAISER reference from KPTI option description
  arm64: mm: Only map KPTI trampoline if it is going to be used
  arm64: Get rid of ARM64_HAS_NO_HW_PREFETCH
2024-01-04 12:26:56 +00:00
Masahiro Yamada
97ba4416d6 efi/libstub: zboot: do not use $(shell ...) in cmd_copy_and_pad
You do not need to use $(shell ...) in recipe lines, as they are already
executed in a shell. An alternative solution is $$(...), which is an
escaped sequence of the shell's command substituion, $(...).

For this case, there is a reason to avoid $(shell ...).

Kbuild detects command changes by using the if_changed macro, which
compares the previous command recorded in .*.cmd with the current
command from Makefile. If they differ, Kbuild re-runs the build rule.

To diff the commands, Make must expand $(shell ...) first. It means that
hexdump is executed every time, even when nothing needs rebuilding. If
Kbuild determines that vmlinux.bin needs rebuilding, hexdump will be
executed again to evaluate the 'cmd' macro, one more time to really
build vmlinux.bin, and finally yet again to record the expanded command
into .*.cmd.

Replace $(shell ...) with $$(...) to avoid multiple, unnecessay shell
evaluations. Since Make is agnostic about the shell code, $(...), the
if_changed macro compares the string "$(hexdump -s16 -n4 ...)" verbatim,
so hexdump is run only for building vmlinux.bin.

For the same reason, $(shell ...) in EFI_ZBOOT_OBJCOPY_FLAGS should be
eliminated.

While I was here, I replaced '&&' with ';' because a command for
if_changed is executed with 'set -e'.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231218080127.907460-1-masahiroy@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-19 10:02:40 +00:00
Josef Bacik
7b21ed7d11 arm64: properly install vmlinuz.efi
If you select CONFIG_EFI_ZBOOT, we will generate vmlinuz.efi, and then
when we go to install the kernel we'll install the vmlinux instead
because install.sh only recognizes Image.gz as wanting the compressed
install image.  With CONFIG_EFI_ZBOOT we don't get the proper kernel
installed, which means it doesn't boot, which makes for a very confused
and subsequently angry kernel developer.

Fix this by properly installing our compressed kernel if we've enabled
CONFIG_EFI_ZBOOT.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Cc: <stable@vger.kernel.org> # 6.1.x
Fixes: c37b830fef ("arm64: efi: enable generic EFI compressed boot")
Reviewed-by: Simon Glass <sjg@chromium.org>
Link: https://lore.kernel.org/r/6edb1402769c2c14c4fbef8f7eaedb3167558789.1702570674.git.josef@toxicpanda.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-17 12:31:53 +00:00
Mark Rutland
eb15d707c2 arm64: Align boot cpucap handling with system cpucap handling
Currently the detection+enablement of boot cpucaps is separate from the
patching of boot cpucap alternatives, which means there's a period where
cpus_have_cap($CAP) and alternative_has_cap($CAP) may be mismatched.

It would be preferable to manage the boot cpucaps in the same way as the
system cpucaps, both for clarity and to minimize the risk of accidental
usage of code relying upon an alternative which has not yet been
patched.

This patch aligns the handling of boot cpucaps with the handling of
system cpucaps:

* The existing setup_boot_cpu_capabilities() function is moved to be
  closer to the setup_system_capabilities() and setup_system_features()
  functions so that they're more clearly related and more likely to be
  updated together in future.

* The patching of boot cpucap alternatives is moved into
  setup_boot_cpu_capabilities(), immediately after boot cpucaps are
  detected and enabled.

* A new setup_boot_cpu_features() function is added to mirror
  setup_system_features(); this handles initialization of cpucap data
  structures and calls setup_boot_cpu_capabilities(). This makes
  init_cpu_features() a closer mirror to update_cpu_features(), and
  makes smp_prepare_boot_cpu() a closer mirror to smp_cpus_done().

Importantly, while these changes alter the structure of the code, they
retain the existing order of calls to:

  init_cpu_features(); // prefix initializing feature regs
  init_cpucap_indirect_list();
  detect_system_supports_pseudo_nmi();
  update_cpu_capabilities(SCOPE_BOOT_CPU | SCOPE_LOCAL_CPU);
  enable_cpu_capabilities(SCOPE_BOOT_CPU);
  apply_boot_alternatives();

... and hence there should be no functional change as a result of this
patch; this is purely a structural cleanup.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20231212170910.3745497-3-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-13 16:02:01 +00:00
Mark Rutland
63a2d92e14 arm64: Cleanup system cpucap handling
Recent changes to remove cpus_have_const_cap() introduced new users of
cpus_have_cap() in the period between detecting system cpucaps and
patching alternatives. It would be preferable to defer these until after
the relevant cpucaps have been patched so that these can use the usual
feature check helper functions, which is clearer and has less risk of
accidental usage of code relying upon an alternative which has not yet
been patched.

This patch reworks the system-wide cpucap detection and patching to
minimize this transient period:

* The detection, enablement, and patching of system cpucaps is moved
  into a new setup_system_capabilities() function so that these can be
  grouped together more clearly, with no other functions called in the
  period between detection and patching. This is called from
  setup_system_features() before the subsequent checks that depend on
  the cpucaps.

  The logging of TTBR0 PAN and cpucaps with a mask is also moved here to
  keep these as close as possible to update_cpu_capabilities().

  At the same time, comments are corrected and improved to make the
  intent clearer.

* As hyp_mode_check() only tests system register values (not hwcaps) and
  must be called prior to patching, the call to hyp_mode_check() is
  moved before the call to setup_system_features().

* In setup_system_features(), the use of system_uses_ttbr0_pan() is
  restored, now that this occurs after alternatives are patched. This is
  a partial revert of commit:

    53d62e995d ("arm64: Avoid cpus_have_const_cap() for ARM64_HAS_PAN")

* In sve_setup() and sme_setup(), the use of system_supports_sve() and
  system_supports_sme() respectively are restored, now that these occur
  after alternatives are patched. This is a partial revert of commit:

    a76521d160 ("arm64: Avoid cpus_have_const_cap() for ARM64_{SVE,SME,SME2,FA64}")

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20231212170910.3745497-2-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-13 16:02:01 +00:00
Ard Biesheuvel
2632e25217 arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
Now that kernel mode FPSIMD state is context switched along with other
task state, we can enable the existing logic that keeps track of which
task's FPSIMD state the CPU is holding in its registers. If it is the
context of the task that we are switching to, we can elide the reload of
the FPSIMD state from memory.

Note that we also need to check whether the FPSIMD state on this CPU is
the most recent: if a task gets migrated away and back again, the state
in memory may be more recent than the state in the CPU. So add another
CPU id field to task_struct to keep track of this. (We could reuse the
existing CPU id field used for user mode context, but that might result
in user state to be discarded unnecessarily, given that two distinct
CPUs could be holding the most recent user mode state and the most
recent kernel mode state)

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Mark Brown <broonie@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20231208113218.3001940-9-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 14:31:55 +00:00
Ard Biesheuvel
aefbab8e77 arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
Currently, the FPSIMD register file is not preserved and restored along
with the general registers on exception entry/exit or context switch.
For this reason, we disable preemption when enabling FPSIMD for kernel
mode use in task context, and suspend the processing of softirqs so that
there are no concurrent uses in the kernel. (Kernel mode FPSIMD may not
be used at all in other contexts).

Disabling preemption while doing CPU intensive work on inputs of
potentially unbounded size is bad for real-time performance, which is
why we try and ensure that SIMD crypto code does not operate on more
than ~4k at a time, which is an arbitrary limit and requires assembler
code to implement efficiently.

We can avoid the need for disabling preemption if we can ensure that any
in-kernel users of the NEON will not lose the FPSIMD register state
across a context switch. And given that disabling softirqs implicitly
disables preemption as well, we will also have to ensure that a softirq
that runs code using FPSIMD can safely interrupt an in-kernel user.

So introduce a thread_info flag TIF_KERNEL_FPSTATE, and modify the
context switch hook for FPSIMD to preserve and restore the kernel mode
FPSIMD to/from struct thread_struct when it is set. This avoids any
scheduling blackouts due to prolonged use of FPSIMD in kernel mode,
without the need for manual yielding.

In order to support softirq processing while FPSIMD is being used in
kernel task context, use the same flag to decide whether the kernel mode
FPSIMD state needs to be preserved and restored before allowing FPSIMD
to be used in softirq context.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Mark Brown <broonie@kernel.org>
Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20231208113218.3001940-8-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 14:31:54 +00:00
Ard Biesheuvel
9b19700e62 arm64: fpsimd: Drop unneeded 'busy' flag
Kernel mode NEON will preserve the user mode FPSIMD state by saving it
into the task struct before clobbering the registers. In order to avoid
the need for preserving kernel mode state too, we disallow nested use of
kernel mode NEON, i..e, use in softirq context while the interrupted
task context was using kernel mode NEON too.

Originally, this policy was implemented using a per-CPU flag which was
exposed via may_use_simd(), requiring the users of the kernel mode NEON
to deal with the possibility that it might return false, and having NEON
and non-NEON code paths. This policy was changed by commit
13150149aa ("arm64: fpsimd: run kernel mode NEON with softirqs
disabled"), and now, softirq processing is disabled entirely instead,
and so may_use_simd() can never fail when called from task or softirq
context.

This means we can drop the fpsimd_context_busy flag entirely, and
instead, ensure that we disable softirq processing in places where we
formerly relied on the flag for preventing races in the FPSIMD preserve
routines.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Mark Brown <broonie@kernel.org>
Tested-by: Geert Uytterhoeven <geert+renesas@glider.be>
Link: https://lore.kernel.org/r/20231208113218.3001940-7-ardb@google.com
[will: Folded in fix from CAMj1kXFhzbJRyWHELCivQW1yJaF=p07LLtbuyXYX3G1WtsdyQg@mail.gmail.com]
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 14:29:16 +00:00
Ard Biesheuvel
50f176175e arm64/kernel: Move 'nokaslr' parsing out of early idreg code
Parsing and ignoring 'nokaslr' can be done from anywhere, except from
the code that runs very early and is therefore built with limitations on
the kind of relocations it is permitted to use.

So move it to a source file that is part of the ordinary kernel build.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-63-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:53 +00:00
Ard Biesheuvel
ea48626f8f arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit
All ID register value overrides are =0 with the exception of the nokaslr
pseudo feature which uses =1. In order to remove the dependency on
kstrtou64(), which is part of the core kernel and no longer usable once
we move idreg-override into the early mini C runtime, let's just parse a
single hex digit (with optional leading 0x) and set the output value
accordingly.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-62-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:53 +00:00
Ard Biesheuvel
060260a6be arm64: idreg-override: Avoid sprintf() for simple string concatenation
Instead of using sprintf() with the "%s.%s=" format, where the first
string argument is always the same in the inner loop of match_options(),
use simple memcpy() for string concatenation, and move the first copy to
the outer loop. This removes the dependency on sprintf(), which will be
difficult to fulfil when we move this code into the early mini C
runtime.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-61-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:53 +00:00
Ard Biesheuvel
bcf1eed3f8 arm64: idreg-override: avoid strlen() to check for empty strings
strlen() is a costly way to decide whether a string is empty, as in that
case, the first character will be NUL so we can check for that directly.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-60-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:52 +00:00
Ard Biesheuvel
dc3f5aae06 arm64: idreg-override: Avoid parameq() and parameqn()
The only way parameq() and parameqn() deviate from the ordinary string
and memory routines is that they ignore the difference between dashes
and underscores.

Since we copy each command line argument into a buffer before passing it
to parameq() and parameqn() numerous times, let's just convert all
dashes to underscores just once, and update the alias array accordingly.

This also helps reduce the dependency on kernel APIs that are no longer
available once we move this code into the early mini C runtime.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-59-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:52 +00:00
Ard Biesheuvel
01fd29092a arm64: idreg-override: Prepare for place relative reloc patching
The ID reg override handling code uses a rather elaborate data structure
that relies on statically initialized absolute address values in pointer
fields. This means that this code cannot run until relocation fixups
have been applied, and this is unfortunate, because it means we cannot
discover overrides for KASLR or LVA/LPA without creating the kernel
mapping and performing the relocations first.

This can be solved by switching to place-relative relocations, which can
be applied by the linker at build time. This means some additional
arithmetic is required when dereferencing these pointers, as we can no
longer dereference the pointer members directly.

So let's implement this for idreg-override.c in a preliminary way, i.e.,
convert all the references in code to use a special accessor that
produces the correct absolute value at runtime.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-58-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:52 +00:00
Ard Biesheuvel
cbc59c9a4e arm64: idreg-override: Omit non-NULL checks for override pointer
Now that override pointers are always set, we can drop the various
non-NULL checks that we have in the code.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-57-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:13:52 +00:00
Ard Biesheuvel
376f5a3bd7 arm64: mm: get rid of kimage_vaddr global variable
We store the address of _text in kimage_vaddr, but since commit
09e3c22a86 ("arm64: Use a variable to store non-global mappings
decision"), we no longer reference this variable from modules so we no
longer need to export it.

In fact, we don't need it at all so let's just get rid of it.

Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Link: https://lore.kernel.org/r/20231129111555.3594833-46-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:06:28 +00:00
Ard Biesheuvel
a22fc8e102 arm64: mm: Take potential load offset into account when KASLR is off
We enable CONFIG_RELOCATABLE even when CONFIG_RANDOMIZE_BASE is
disabled, and this permits the loader (i.e., EFI) to place the kernel
anywhere in physical memory as long as the base address is 64k aligned.

This means that the 'KASLR' case described in the header that defines
the size of the statically allocated page tables could take effect even
when CONFIG_RANDMIZE_BASE=n. So check for CONFIG_RELOCATABLE instead.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Link: https://lore.kernel.org/r/20231129111555.3594833-45-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:06:27 +00:00
Ard Biesheuvel
3dfdc2750c arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
In subsequent patches, mark portions of the early C code will be marked
as __init.  Unfortunarely, __init implies __latent_entropy, and this
would result in the early C code being instrumented in an unsafe manner.

Disable the latent entropy plugin for the early C code.

Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lore.kernel.org/r/20231129111555.3594833-44-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-12 11:06:27 +00:00
Ard Biesheuvel
7540f70df9 arm64: Kconfig: drop KAISER reference from KPTI option description
KAISER is a reference to the KASLR hardening technique that already
existed before Meltdown happened, and by now, it is sufficiently obscure
that mentioning it does not actually clarify anything. So remove this
reference, and replace it with KPTI.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20231127120049.2258650-8-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-11 11:40:38 +00:00
Ard Biesheuvel
8885c7398f arm64: mm: Only map KPTI trampoline if it is going to be used
Avoid creating the fixmap entries for the KPTI trampoline if KPTI is not
in use.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20231127120049.2258650-7-ardb@google.com
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-11 11:40:38 +00:00
Masahiro Yamada
8fd7588fd4 arm64: replace <asm-generic/export.h> with <linux/export.h>
Commit ddb5cdbafa ("kbuild: generate KSYMTAB entries by modpost")
deprecated <asm-generic/export.h>, which is now a wrapper of
<linux/export.h>.

Replace #include <asm-generic/export.h> with #include <linux/export.h>.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Link: https://lore.kernel.org/r/20231126151045.1556686-1-masahiroy@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-05 14:25:30 +00:00
Marc Zyngier
103423ad7e arm64: Get rid of ARM64_HAS_NO_HW_PREFETCH
Back in 2016, it was argued that implementations lacking a HW
prefetcher could be helped by sprinkling a number of PRFM
instructions in strategic locations.

In 2023, the one platform that presumably needed this hack is no
longer in active use (let alone maintained), and an quick
experiment shows dropping this hack only leads to a 0.4% drop
on a full kernel compilation (tested on a MT30-GS0 48 CPU system).

Given that this is pretty much in the noise department and that
it may give odd ideas to other implementers, drop the hack for
good.

Suggested-by: Will Deacon <will@kernel.org>
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Link: https://lore.kernel.org/r/20231122133754.1240687-1-maz@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-05 12:02:52 +00:00
Masahiro Yamada
a099bec7a8 arm64: vdso32: rename 32-bit debug vdso to vdso32.so.dbg
'make vdso_install' renames arch/arm64/kernel/vdso32/vdso.so.dbg to
vdso32.so during installation, which allows 64-bit and 32-bit vdso
files to be installed in the same directory.

However, arm64 is the only architecture that requires this renaming.

To simplify the vdso_install logic, rename the in-tree vdso file so
its base name matches the installed file name.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Link: https://lore.kernel.org/r/20231117125620.1058300-1-masahiroy@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
2023-12-05 11:49:53 +00:00
Ryan Roberts
b1366d21da arm64: Add ARM64_HAS_LPA2 CPU capability
Expose FEAT_LPA2 as a capability so that we can take advantage of
alternatives patching in the hypervisor.

Although FEAT_LPA2 presence is advertised separately for stage1 and
stage2, the expectation is that in practice both stages will either
support or not support it. Therefore, we combine both into a single
capability, allowing us to simplify the implementation. KVM requires
support in both stages in order to use LPA2 since the same library is
used for hyp stage 1 and guest stage 2 pgtables.

Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20231127111737.1897081-6-ryan.roberts@arm.com
2023-11-27 15:03:50 +00:00
Anshuman Khandual
e477c8c483 arm64/mm: Add FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2]
PAGE_SIZE support is tested against possible minimum and maximum values for
its respective ID_AA64MMFR0.TGRAN field, depending on whether it is signed
or unsigned. But then FEAT_LPA2 implementation needs to be validated for 4K
and 16K page sizes via feature specific ID_AA64MMFR0.TGRAN values. Hence it
adds FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2] values per ARM ARM (0487G.A).

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20231127111737.1897081-5-ryan.roberts@arm.com
2023-11-27 15:03:50 +00:00
Ryan Roberts
c910f2b655 arm64/mm: Update tlb invalidation routines for FEAT_LPA2
FEAT_LPA2 impacts tlb invalidation in 2 ways; Firstly, the TTL field in
the non-range tlbi instructions can now validly take a 0 value as a
level hint for the 4KB granule (this is due to the extra level of
translation) - previously TTL=0b0100 meant no hint and was treated as
0b0000. Secondly, The BADDR field of the range-based tlbi instructions
is specified in 64KB units when LPA2 is in use (TCR.DS=1), whereas it is
in page units otherwise. Changes are required for tlbi to continue to
operate correctly when LPA2 is in use.

Solve the first problem by always adding the level hint if the level is
between [0, 3] (previously anything other than 0 was hinted, which
breaks in the new level -1 case from kvm). When running on non-LPA2 HW,
0 is still safe to hint as the HW will fall back to non-hinted. While we
are at it, we replace the notion of 0 being the non-hinted sentinel with
a macro, TLBI_TTL_UNKNOWN. This means callers won't need updating
if/when translation depth increases in future.

The second issue is more complex: When LPA2 is in use, use the non-range
tlbi instructions to forward align to a 64KB boundary first, then use
range-based tlbi from there on, until we have either invalidated all
pages or we have a single page remaining. If the latter, that is done
with non-range tlbi. We determine whether LPA2 is in use based on
lpa2_is_enabled() (for kernel calls) or kvm_lpa2_is_enabled() (for kvm
calls).

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20231127111737.1897081-4-ryan.roberts@arm.com
2023-11-27 15:03:50 +00:00
Ryan Roberts
936a4ec281 arm64/mm: Add lpa2_is_enabled() kvm_lpa2_is_enabled() stubs
Add stub functions which is initially always return false. These provide
the hooks that we need to update the range-based TLBI routines, whose
operands are encoded differently depending on whether lpa2 is enabled or
not.

The kernel and kvm will enable the use of lpa2 asynchronously in future,
and part of that enablement will involve fleshing out their respective
hook to advertise when it is using lpa2.

Since the kernel's decision to use lpa2 relies on more than just whether
the HW supports the feature, it can't just use the same static key as
kvm. This is another reason to use separate functions. lpa2_is_enabled()
is already implemented as part of Ard's kernel lpa2 series. Since kvm
will make its decision solely based on HW support, kvm_lpa2_is_enabled()
will be defined as system_supports_lpa2() once kvm starts using lpa2.

Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20231127111737.1897081-3-ryan.roberts@arm.com
2023-11-27 15:03:50 +00:00
Ryan Roberts
e2768b798a arm64/mm: Modify range-based tlbi to decrement scale
In preparation for adding support for LPA2 to the tlb invalidation
routines, modify the algorithm used by range-based tlbi to start at the
highest 'scale' and decrement instead of starting at the lowest 'scale'
and incrementing. This new approach makes it possible to maintain 64K
alignment as we work through the range, until the last op (at scale=0).
This is required when LPA2 is enabled. (This part will be added in a
subsequent commit).

This change is separated into its own patch because it will also impact
non-LPA2 systems, and I want to make it easy to bisect in case it leads
to performance regression (see below for benchmarks that suggest this
should not be a problem).

The original commit (d1d3aa98 "arm64: tlb: Use the TLBI RANGE feature in
arm64") stated this as the reason for _incrementing_ scale:

  However, in most scenarios, the pages = 1 when flush_tlb_range() is
  called. Start from scale = 3 or other proper value (such as scale
  =ilog2(pages)), will incur extra overhead. So increase 'scale' from 0
  to maximum.

But pages=1 is already special cased by the non-range invalidation path,
which will take care of it the first time through the loop (both in the
original commit and in my change), so I don't think switching to
decrement scale should have any extra performance impact after all.

Indeed benchmarking kernel compilation, a TLBI-heavy workload, suggests
that this new approach actually _improves_ performance slightly (using a
virtual machine on Apple M2):

Table shows time to execute kernel compilation workload with 8 jobs,
relative to baseline without this patch (more negative number is
bigger speedup). Repeated 9 times across 3 system reboots:

| counter   |       mean |     stdev |
|:----------|-----------:|----------:|
| real-time |      -0.6% |      0.0% |
| kern-time |      -1.6% |      0.5% |
| user-time |      -0.4% |      0.1% |

Reviewed-by: Oliver Upton <oliver.upton@linux.dev>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20231127111737.1897081-2-ryan.roberts@arm.com
2023-11-27 15:03:50 +00:00
Linus Torvalds
2cc14f52ae Linux 6.7-rc3 v6.7-rc3 2023-11-26 19:59:33 -08:00
Linus Torvalds
5b2b1173a9 Merge tag 'trace-v6.7-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull tracing fixes from Steven Rostedt::
 "Eventfs fixes:

   - With the usage of simple_recursive_remove() recommended by Al Viro,
     the code should not be calling "d_invalidate()" itself. Doing so is
     causing crashes. The code was calling d_invalidate() on the race of
     trying to look up a file while the parent was being deleted. This
     was detected, and the added dentry was having d_invalidate() called
     on it, but the deletion of the directory was also calling
     d_invalidate() on that same dentry.

   - A fix to not free the eventfs_inode (ei) until the last dput() was
     called on its ei->dentry made the ei->dentry exist even after it
     was marked for free by setting the ei->is_freed. But code elsewhere
     still was checking if ei->dentry was NULL if ei->is_freed is set
     and would trigger WARN_ON if that was the case. That's no longer
     true and there should not be any warnings when it is true.

   - Use GFP_NOFS for allocations done under eventfs_mutex. The
     eventfs_mutex can be taken on file system reclaim, make sure that
     allocations done under that mutex do not trigger file system
     reclaim.

   - Clean up code by moving the taking of inode_lock out of the helper
     functions and into where they are needed, and not use the parameter
     to know to take it or not. It must always be held but some callers
     of the helper function have it taken when they were called.

   - Warn if the inode_lock is not held in the helper functions.

   - Warn if eventfs_start_creating() is called without a parent. As
     eventfs is underneath tracefs, all files created will have a parent
     (the top one will have a tracefs parent).

  Tracing update:

   - Add Mathieu Desnoyers as an official reviewer of the tracing subsystem"

* tag 'trace-v6.7-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
  MAINTAINERS: TRACING: Add Mathieu Desnoyers as Reviewer
  eventfs: Make sure that parent->d_inode is locked in creating files/dirs
  eventfs: Do not allow NULL parent to eventfs_start_creating()
  eventfs: Move taking of inode_lock into dcache_dir_open_wrapper()
  eventfs: Use GFP_NOFS for allocation when eventfs_mutex is held
  eventfs: Do not invalidate dentry in create_file/dir_dentry()
  eventfs: Remove expectation that ei->is_freed means ei->dentry == NULL
2023-11-26 19:48:20 -08:00
Linus Torvalds
d2da77f431 Merge tag 'parisc-for-6.7-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux
Pull parisc architecture fixes from Helge Deller:
 "This patchset fixes and enforces correct section alignments for the
  ex_table, altinstructions, parisc_unwind, jump_table and bug_table
  which are created by inline assembly.

  Due to not being correctly aligned at link & load time they can
  trigger unnecessarily the kernel unaligned exception handler at
  runtime. While at it, I switched the bug table to use relative
  addresses which reduces the size of the table by half on 64-bit.

  We still had the ENOSYM and EREMOTERELEASE errno symbols as left-overs
  from HP-UX, which now trigger build-issues with glibc. We can simply
  remove them.

  Most of the patches are tagged for stable kernel series.

  Summary:

   - Drop HP-UX ENOSYM and EREMOTERELEASE return codes to avoid glibc
     build issues

   - Fix section alignments for ex_table, altinstructions, parisc unwind
     table, jump_table and bug_table

   - Reduce size of bug_table on 64-bit kernel by using relative
     pointers"

* tag 'parisc-for-6.7-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux:
  parisc: Reduce size of the bug_table on 64-bit kernel by half
  parisc: Drop the HP-UX ENOSYM and EREMOTERELEASE error codes
  parisc: Use natural CPU alignment for bug_table
  parisc: Ensure 32-bit alignment on parisc unwind section
  parisc: Mark lock_aligned variables 16-byte aligned on SMP
  parisc: Mark jump_table naturally aligned
  parisc: Mark altinstructions read-only and 32-bit aligned
  parisc: Mark ex_table entries 32-bit aligned in uaccess.h
  parisc: Mark ex_table entries 32-bit aligned in assembly.h
2023-11-26 09:59:39 -08:00
Linus Torvalds
4892711ace Merge tag 'x86-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 microcode fixes from Ingo Molnar:
 "Fix/enhance x86 microcode version reporting: fix the bootup log spam,
  and remove the driver version announcement to avoid version confusion
  when distros backport fixes"

* tag 'x86-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86/microcode: Rework early revisions reporting
  x86/microcode: Remove the driver announcement and version
2023-11-26 08:42:42 -08:00
Linus Torvalds
e81fe50520 Merge tag 'perf-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 perf event fix from Ingo Molnar:
 "Fix a bug in the Intel hybrid CPUs hardware-capabilities enumeration
  code resulting in non-working events on those platforms"

* tag 'perf-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  perf/x86/intel: Correct incorrect 'or' operation for PMU capabilities
2023-11-26 08:34:12 -08:00
Linus Torvalds
1d0dbc3d16 Merge tag 'locking-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking fix from Ingo Molnar:
 "Fix lockdep block chain corruption resulting in KASAN warnings"

* tag 'locking-urgent-2023-11-26' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  lockdep: Fix block chain corruption
2023-11-26 08:30:11 -08:00
Linus Torvalds
4515866db1 Merge tag '6.7-rc2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6
Pull smb client fixes from Steve French:

 - use after free fix in releasing multichannel interfaces

 - fixes for special file types (report char, block, FIFOs properly when
   created e.g. by NFS to Windows)

 - fixes for reporting various special file types and symlinks properly
   when using SMB1

* tag '6.7-rc2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6:
  smb: client: introduce cifs_sfu_make_node()
  smb: client: set correct file type from NFS reparse points
  smb: client: introduce ->parse_reparse_point()
  smb: client: implement ->query_reparse_point() for SMB1
  cifs: fix use after free for iface while disabling secondary channels
2023-11-26 08:22:27 -08:00
Linus Torvalds
090472ed9c Merge tag 'usb-6.7-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
Pull USB / PHY / Thunderbolt fixes from Greg KH:
 "Here are a number of reverts, fixes, and new device ids for 6.7-rc3
  for the USB, PHY, and Thunderbolt driver subsystems. Include in here
  are:

   - reverts of some PHY drivers that went into 6.7-rc1 that shouldn't
     have been merged yet, the author is reworking them based on review
     comments as they were using older apis that shouldn't be used
     anymore for newer drivers

   - small thunderbolt driver fixes for reported issues

   - USB driver fixes for a variety of small issues in dwc3, typec,
     xhci, and other smaller drivers.

   - new device ids for usb-serial and onboard_usb_hub drivers.

  All of these have been in linux-next with no reported issues"

* tag 'usb-6.7-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb: (33 commits)
  USB: serial: option: add Luat Air72*U series products
  USB: dwc3: qcom: fix ACPI platform device leak
  USB: dwc3: qcom: fix software node leak on probe errors
  USB: dwc3: qcom: fix resource leaks on probe deferral
  USB: dwc3: qcom: simplify wakeup interrupt setup
  USB: dwc3: qcom: fix wakeup after probe deferral
  dt-bindings: usb: qcom,dwc3: fix example wakeup interrupt types
  usb: misc: onboard-hub: add support for Microchip USB5744
  dt-bindings: usb: microchip,usb5744: Add second supply
  usb: misc: ljca: Fix enumeration error on Dell Latitude 9420
  USB: serial: option: add Fibocom L7xx modules
  USB: xhci-plat: fix legacy PHY double init
  usb: typec: tipd: Supply also I2C driver data
  usb: xhci-mtk: fix in-ep's start-split check failure
  usb: dwc3: set the dma max_seg_size
  usb: config: fix iteration issue in 'usb_get_bos_descriptor()'
  usb: dwc3: add missing of_node_put and platform_device_put
  USB: dwc2: write HCINT with INTMASK applied
  usb: misc: ljca: Drop _ADR support to get ljca children devices
  usb: cdnsp: Fix deadlock issue during using NCM gadget
  ...
2023-11-25 18:22:42 -08:00
Linus Torvalds
b46ae77f67 Merge tag 'xfs-6.7-fixes-3' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
Pull xfs fix from Chandan Babu:

 - Validate quota records recovered from the log before writing them to
   the disk.

* tag 'xfs-6.7-fixes-3' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
  xfs: dquot recovery does not validate the recovered dquot
  xfs: clean up dqblk extraction
2023-11-25 08:57:09 -08:00
Linus Torvalds
2821c393d4 Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
Pull arm64 fixes from Catalin Marinas:

 - Fix "rodata=on" not disabling "rodata=full" on arm64

 - Add arm64 make dependency between vmlinuz.efi and Image, leading to
   occasional build failures previously (with parallel building)

 - Add newline to the output formatting of the za-fork kselftest

* tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux:
  arm64: add dependency between vmlinuz.efi and Image
  kselftest/arm64: Fix output formatting for za-fork
  arm64: mm: Fix "rodata=on" when CONFIG_RODATA_FULL_DEFAULT_ENABLED=y
2023-11-25 08:43:46 -08:00
Linus Torvalds
00cff7b29b Merge tag 'for-linus-6.7a-rc3-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
Pull xen fixes from Juergen Gross:

 - A small cleanup patch for the Xen privcmd driver

 - A fix for the swiotlb-xen driver which was missing the advertising of
   the maximum mapping length

 - A fix for Xen on Arm for a longstanding bug, which happened to occur
   only recently: a structure in percpu memory crossed a page boundary,
   which was rejected by the hypervisor

* tag 'for-linus-6.7a-rc3-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip:
  arm/xen: fix xen_vcpu_info allocation alignment
  xen: privcmd: Replace zero-length array with flex-array member and use __counted_by
  swiotlb-xen: provide the "max_mapping_size" method
2023-11-25 08:32:44 -08:00
Helge Deller
4326683851 parisc: Reduce size of the bug_table on 64-bit kernel by half
Enable GENERIC_BUG_RELATIVE_POINTERS which will store 32-bit relative
offsets to the bug address and the source file name instead of 64-bit
absolute addresses. This effectively reduces the size of the
bug_table[] array by half on 64-bit kernels.

Signed-off-by: Helge Deller <deller@gmx.de>
2023-11-25 09:43:18 +01:00
Helge Deller
e5f3e299a2 parisc: Drop the HP-UX ENOSYM and EREMOTERELEASE error codes
Those return codes are only defined for the parisc architecture and
are leftovers from when we wanted to be HP-UX compatible.

They are not returned by any Linux kernel syscall but do trigger
problems with the glibc strerrorname_np() and strerror() functions as
reported in glibc issue #31080.

There is no need to keep them, so simply remove them.

Signed-off-by: Helge Deller <deller@gmx.de>
Reported-by: Bruno Haible <bruno@clisp.org>
Closes: https://sourceware.org/bugzilla/show_bug.cgi?id=31080
Cc: stable@vger.kernel.org
2023-11-25 09:43:18 +01:00
Helge Deller
fe76a1349f parisc: Use natural CPU alignment for bug_table
Make sure that the __bug_table section gets 32- or 64-bit aligned,
depending if a 32- or 64-bit kernel is being built.
Mark it non-writeable and use .blockz instead of the .org assembler
directive to pad the struct.

Signed-off-by: Helge Deller <deller@gmx.de>
Cc: stable@vger.kernel.org   # v6.0+
2023-11-25 09:43:18 +01:00
Helge Deller
c9fcb2b65c parisc: Ensure 32-bit alignment on parisc unwind section
Make sure the .PARISC.unwind section will be 32-bit aligned.

Signed-off-by: Helge Deller <deller@gmx.de>
Cc: stable@vger.kernel.org   # v6.0+
2023-11-25 09:43:17 +01:00
Helge Deller
b28fc0d873 parisc: Mark lock_aligned variables 16-byte aligned on SMP
On parisc we need 16-byte alignment for variables which are used for
locking. Mark the __lock_aligned attribute acordingly so that the
.data..lock_aligned section will get that alignment in the generated
object files.

Signed-off-by: Helge Deller <deller@gmx.de>
Cc: stable@vger.kernel.org   # v6.0+
2023-11-25 09:43:17 +01:00
Helge Deller
07eecff8ae parisc: Mark jump_table naturally aligned
The jump_table stores two 32-bit words and one 32- (on 32-bit kernel)
or one 64-bit word (on 64-bit kernel).
Ensure that the last word is always 64-bit aligned on a 64-bit kernel
by aligning the whole structure on sizeof(long).

Signed-off-by: Helge Deller <deller@gmx.de>
Cc: stable@vger.kernel.org   # v6.0+
2023-11-25 09:43:17 +01:00
Helge Deller
33f806da2d parisc: Mark altinstructions read-only and 32-bit aligned
Signed-off-by: Helge Deller <deller@gmx.de>
Cc: stable@vger.kernel.org   # v6.0+
2023-11-25 09:43:17 +01:00