Depending on MARCH_HAS_Z196_FEATURES __atomic_add_const() returns void or
the previous value before the atomic variant. Make sure that for both cases
void is returned so potential incorrect usage results in both cases in a
compile error.
Reviewed-by: Juergen Christ <jchrist@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
When the kernel stack pointer is pointing to invalid memory,
a 'Kernel stack overflow' message is printed, which is misleading.
Change the message to actually say that the stack pointer is invalid
instead.
Reviewed-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
If the vector facility is installed cpu_has_vx() is always true, if it is
not installed the result is always false, and no vector exception can
happen. Therefore remove the superfluous check.
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use pr_emerg() instead of printk() in case of a stack overflow,
providing the emergency printk level. Also slightly adjust the
printed text for pr_emerg() and panic().
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The usage of get_user() in illegal_op() is quite unusual. Make the code
more readable and get rid of unnecessary casts. The generated code is
identical before/after this change.
Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Shorten detect_diag9c() and use regular EX_TABLE program check handling.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Shorten diag500_storage_limit() and use regular EX_TABLE program check
handling.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Shorten cmma_test_essa() and use regular EX_TABLE program check handling.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The early boot code contains various open-coded inline assemblies with
exception handling. In order to handle possible exceptions each of them
changes the program check new psw, and restores it.
In order to simplify the various inline assemblies add simple exception
table support: the program check handler is called with a fully populated
pt_regs on the stack and may change the psw and register members. When the
program check handler returns the psw and registers from pt_regs will be
used to continue execution.
The program check handler searches the exception table for an entry which
matches the address of the program check. If such an entry is found the psw
address within pt_regs on the stack is replaced with a fixup address, and
execution continues at the new address.
If no entry is found the psw is changed to a disabled wait psw and
execution stops.
Before entering the C part of the program check handler the address of the
program check new psw is replaced to a minimalistic handler.
This is supposed to help against program check loops. If an exception
happens while in program check processing the register contents of the
original exception are restored and a disabled wait psw is loaded.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Setup a pt_regs structure on the stack, poplulate it in low level assembler
code, and pass it to print_pgm_check_info(). This way there is no need to
access then lowcore from print_pgm_check_info() anymore, and the function
looks like a normal program check handler function.
Acked-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Avoid confusion and rename __LC_PGM_INT_CODE since it correlates to the
pgm_code member of struct lowcore, and not the pgm_int_code member.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
A few include directives use the local search variant even though the files
to be included aren't local. Therefore use the normal system header file
variant of the include directive.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
mmap_base() has logic to ensure that the variable "gap" stays within the
range defined by "gap_min" and "gap_max". Replace this with the clamp()
macro to shorten and simplify code.
Signed-off-by: Qasim Ijaz <qasdev00@gmail.com>
Link: https://lore.kernel.org/r/20250204162508.12335-1-qasdev00@gmail.com
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
[gor@linux.ibm.com: also remove the gap_min and gap_max variables]
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Similar to x86 and loongarch add a "debug-alternative" command line
parameter, which allows for alternative debugging. The parameter
itself comes with architecture specific semantics:
"debug-alternative"
-> print debug message for every single alternative
"debug-alternative=0;2"
-> print debug message for all alternatives with type 0 and 2
"debug-alternative=0:0-7"
-> print debug message for all alternatives with type 0 which have a
facility number within the range of 0-7
"debug-alternative=0:!8;1"
-> print debug message for all alternatives with type 0, for all
facility numbers, except facility 8, and in addition print all
alternatives with type 1
A defconfig build currently results in a kernel with more than 20.000
alternatives, where the majority is for the niai alternative (spinlocks),
and the relocated lowcore alternative. The following kernel command like
options limit alternative debug output, and enable dynamic debug messages:
debug-alternative=0:!49;1:!0
earlyprintk
bootdebug
ignore_loglevel
loglevel=8
dyndbg="file alternative.c +p"
This results in output like this:
alt: [0/ 11] 0000021b9ce8680c: c0f400000089 -> c00400000000
alt: [0/ 64] 0000021b9ce87e60: c0f400000043 -> c00400000000
alt: [0/133] 0000021b9ce88c56: c0f400000027 -> c00400000000
alt: [0/ 74] 0000021b9ce89410: c0f40000002a -> c00400000000
alt: [0/ 40] 0000021b9dc3720a: 47000000 -> b280d398
alt: [0/193] 0000021b9dc37306: 47000000 -> b201d2b0
alt: [0/193] 0000021b9dc37354: c00400000000 -> d20720c0d2b0
alt: [1/ 5] 0000038d720d7bf2: c0f400000016 -> c00400000000
With
[<alternative type>/<alternative data>] <address> oldcode -> newcode
Alternative data depends on the alternative type: for type 0
(ALT_TYPE_FACILITY) data is the facility. For type 1 (ALT_TYPE_FEATURE)
data is the corresponding machine feature.
Acked-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Make decompressor_handled_param() a wrapper for
__decompressor_handled_param(). __decompressor_handled_param() now
takes two parameters: a function name and a parameter name, which do
not necessarily match.
This allows to use characters like "-", which are not allowed in
function names, for command line parameters which are handled by the
decompressor and should be ignored by the kernel.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Get rid of the cpu_has_bear jump label and convert cpu_has_bear() to a cpu
feature function using test_facility() and with that use a static branch.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Instead of having a private cpu_has_vx() implementation use the new common
cpu feature method. Move the facility detection to the decompressor so it
matches all other cpu features.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Move machine type detection to the decompressor and use static branches
to implement and use machine_is_[lpar|vm|kvm]() instead of a runtime check
via MACHINE_IS_[LPAR|VM|KVM].
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Move stsi() inline assembly to header file so it is possible to use it
also for the decompressor.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Merge stsi() and __stsi() and cleanup the inline assembly. This involves
making use of the flag output constraint. Semantically the result is
identical to before.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The exception handling for __stsi() was added in 2001 when it still was
possible to run Linux on systems without LPAR hypervisor, and therefore
without an stsi instruction. Given that this is not supported anymore
remove the exception handling from the __stsi() inline assembly.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use static branch(es) to implement and use machine_has_diag9c() instead of
a runtime check via MACHINE_HAS_DIAG9C.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use static branch(es) to implement and use machine_has_esop() instead
of a runtime check via MACHINE_HAS_ESOP.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use static branch(es) to implement and use machine_has_tx() instead of
a runtime check with MACHINE_HAS_TE.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use static branch(es) to implement and use machine_has_tlb_guest()
instead of a runtime check via MACHINE_HAS_TLB_GUEST.
Also add sclp_early_detect_machine_features() in order to allow for
feature detection from the decompressor.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Use static branch(es) to implement and use machine_has_scc() instead
of a runtime check via MACHINE_HAS_SCC.
This comes with a cleanup of early time initialization: the initial
tod_clock_base value is now passed via the bootdata mechanism, instead
of using absolute lowcore as transport vehicle from the decompressor
to the kernel.
Also the early tod clock initialization is moved to the decompressor
which allows to use a static branch with machine_has_scc() within the
kernel.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert the explicit relocated lowcore alternative type to a more
generic machine feature. This only reduces the number of alternative
types, but has no impact on code generation.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Provide infrastructure which allows to generate machine_has_<feature>()
functions, which are replacing the existing MACHINE_HAS_<feature> macros.
Such function usages generate a static branch depending on <feature>. The
static branch is patched using an alternative.
Each <feature> correlates with a bit set in the machine_features bit
field. If the corresponding bit is set, the branch will be patched. In
order to have any effect on branch patching feature bits must be set with
set_machine_features() in the decompressor before alternatives patching of
the kernel image.
It is possible to use clear_machine_feature() and test_machine_feature()
for machine features which cannot be completely detected within the
decompressor, e.g. if common code command line parameters allow to enable
or disable certain features. In such cases test_machine_feature() instead
of machine_has_feature() must be used within the kernel. This results in a
runtime check and not a static branch.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Convert MACHINE_HAS_... to cpu_has_...() which uses test_facility() instead
of testing the machine_flags lowcore member if the feature is present.
test_facility() generates better code since it results in a static branch
without accessing memory. The branch is patched via alternatives by the
decompressor depending on the availability of the required facility.
Reviewed-by: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
The linux390@de.ibm.com email address is outdated. It has been removed from
the MAINTAINERS file nearly 10 years ago with commit 64776820b4 ("s390:
remove generic email address from maintainers file").
Change MODULE_AUTHOR() accordingly and also use the more common
IBM Corporation company name.
Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Rework __clear_user() similar to raw_copy_from_user() / raw_copy_to_user()
and inline the function saving the overhead of branches.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Avoid that the compiler generates an mvcos loop for constant sizes
smaller than 4096 bytes. The mvcos instruction copies between zero and
4096 bytes (effective length) with one operation. Therefore it is not
necessary to implement a loop for sizes smaller or equal to 4096
bytes.
This reduces the kernel text size by ~50kb (defconfig, gcc 14.2.0):
add/remove: 4/5 grow/shrink: 6/471 up/down: 2294/-51700 (-49406)
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Inline copy_from_user() and copy_to_user(). With the shortened inline
assemblies of raw_copy_to_user() and raw_copy_from_user() the additional
kernel text size is acceptable, considering that this avoids function
calls on hot paths.
This increases the kernel text size by ~90kb (defconfig, gcc 14.2.0):
add/remove: 13/4 grow/shrink: 650/14 up/down: 93484/-3254 (90230)
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Implement separate raw_copy_to_user_key() and raw_copy_from_user_key()
functions, which allows to remove the open-coded operand access control
handling from the normal raw_copy_to_user() / raw_copy_from_user()
functions - they are simplified to use immediate instructions to load
hard-coded operand access control values into register zero, which saves
one instruction.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
Add specific exception handler for copy_to_user() / copy_from_user()
mvcos fault handling, which allows to shorten the inline assemblies to
three instructions.
On fault the exception handler adjusts the length used by the mvcos
instruction in a way that the instruction completes with condition code
zero, indicating the number of bytes copied with the input/output operand
'size'. This allows to calculate and return the number of bytes not copied,
if any, like required.
Loop and return value handling is changed to C so that the compiler may
optimize the code.
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
