Arm64 already uses 64-bit preempt count and the need reschedule bit is
maintained in a separate 32-bit word from the preempt count. Therefore
preempt count has enough bits to represent 16 levels of NMI nesting,
hence enable it for arm64. This saves a per-CPU variable and additional
instructions in the NMI path.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-12-boqun@kernel.org
With the changes that enable preempt count to track IRQ disabling
nesting, we don't have enough bits in 32-bit preempt count
implementation, as a result we move NMI nesting bits out of the 32-bit
preempt count. However on the architectures that can support 64-bit
preempt count implementation, we can keep the NMI nesting bits in the
32-bit preempt count and avoid maintaining NMI nesting bits outside of
the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow
architectures to select this. Note that under this Kconfig, preempt
count is maintained in a 64-bit word however preempt_count() still
remains as an int because all the effective bits still fit in
(previously we mask out NEED_RESCHED bit in preempt_count()). This
should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit
preempt count since it may introduce {over,under}flow]
Originally-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-11-boqun@kernel.org
Currently preempt_count() is always a non-negative int on all archs
(PREEMPT_NEED_RESCHED archs will mask out the MSB when returning
preempt_count()), hence the checking in __cant_migrate() is in fact just
checking whether preempt_count() is 0 or not. In a future change, we are
going to use all the 32 bits of preempt_count(), which would make
negative int values possible from preempt_count(). Therefore convert the
"> 0" comparison into a zero check to prepare for the future change.
No functional changes are intended.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-10-boqun@kernel.org
The semantics of various IRQ disabling guards match what
*_irq_{disable,enable}() provide, i.e. the interrupt disabling is
properly nested, therefore it's OK to switch to use
*_irq_{disable,enable}() primitives.
[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided
by Peter and Lyude]
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-8-boqun@kernel.org
While making changes to the refcounted interrupt patch series, at some
point on my local branch I broke something and ended up writing some kunit
tests for testing refcounted interrupts as a result. So, let's include
these tests now that we have refcounted interrupts.
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804161447.84806-7-boqun@kernel.org
Currently the nested interrupt disabling and enabling is represented by
_irqsave() and _irqrestore() APIs, which are relatively unsafe, for
example:
<interrupts are enabled as beginning>
spin_lock_irqsave(l1, flag1);
spin_lock_irqsave(l2, flag2);
spin_unlock_irqrestore(l1, flags1);
<l2 is still held but interrupts are enabled>
// accesses to interrupt-disable protected data will cause races
This is even easier to trigger with guard facilities:
unsigned long flag2;
scoped_guard(spin_lock_irqsave, l1) {
spin_lock_irqsave(l2, flag2);
}
// l2 locked but interrupts are enabled.
spin_unlock_irqrestore(l2, flag2);
(Hand-to-hand locking critical sections are not uncommon for a
fine-grained lock design)
And because of this unsafety, Rust cannot easily wrap the
interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
* local_interrupt_disable();
* local_interrupt_enable();
They work like local_irq_save() and local_irq_restore() except that 1)
the outermost local_interrupt_disable() call saves the interrupt state
into a per-CPU variable, so that the outermost local_interrupt_enable()
can restore the state, and 2) a per-CPU counter is added to record the
nest level of these calls, so that interrupts are not accidentally
enabled inside the outermost critical section.
Also add the corresponding spin_lock primitives: spin_lock_irq_disable()
and spin_unlock_irq_enable(), as a result, code as follows:
spin_lock_irq_disable(l1);
spin_lock_irq_disable(l2);
spin_unlock_irq_enable(l1);
// Interrupts are still disabled.
spin_unlock_irq_enable(l2);
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier
to design.
[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
Co-developed-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260804182657.87716-1-boqun@kernel.org
While OpenRISC currently doesn't fail to build upstream, it appears that
including <asm/smp.h> in the right headers is enough to break that -
primarily because OpenRISC's asm/smp.h header doesn't actually provide
any definition for struct cpumask. Which means the only reason we aren't
failing to build the kernel is because we've been lucky enough that
every spot including asm/smp.h already has definitions for struct
cpumask pulled in.
This became evident when trying to work on a patch series for adding
ref-counted interrupt enable/disable to the kernel, where introducing a
new interrupt_rc.h header suddenly introduced a build error on OpenRISC:
In file included from include/linux/interrupt_rc.h:17,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from drivers/irqchip/irq-ompic.c:61:
arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask'
declared inside parameter list will not be visible outside of this
definition or declaration
21 | extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
| ^~~~~~~
arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask'
declared inside parameter list will not be visible outside of this
definition or declaration
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
| ^~~~~~~
drivers/irqchip/irq-ompic.c: In function 'ompic_of_init':
>> drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of
'set_smp_cross_call' from incompatible pointer type
[-Werror=incompatible-pointer-types]
191 | set_smp_cross_call(ompic_raise_softirq);
| ^~~~~~~~~~~~~~~~~~~
| |
| void (*)(const struct cpumask *, unsigned int)
arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const
struct cpumask *, unsigned int)' but argument is of type 'void
(*)(const struct cpumask *, unsigned int)'
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
To fix this, let's take an example from the smp.h headers of other
architectures (x86, hexagon, arm64, probably more): just include
linux/cpumask.h at the top.
Signed-off-by: Lyude Paul <lyude@redhat.com>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Stafford Horne <shorne@gmail.com>
Link: https://patch.msgid.link/20260804161447.84806-5-boqun@kernel.org
In order to use preempt_count() to track the interrupt disable nesting
level, __preempt_count_{add,sub}_return() are introduced, as their names
suggest, these primitives return the new value of the preempt_count()
after changing it. The following example shows the usage of it in
local_interrupt_disable():
// increase the HARDIRQ_DISABLE bit
new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);
// if it's the first-time increment, then disable the interrupt
// at hardware level.
if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
local_irq_save(flags);
raw_cpu_write(local_interrupt_disable_state, flags);
}
Having these primitives will avoid a read of preempt_count() after
changing preempt_count() on certain architectures.
Signed-off-by: Boqun Feng <boqun@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com> # s390
Link: https://patch.msgid.link/20260804161447.84806-4-boqun@kernel.org
On PARAVIRT_SPINLOCKS=y kernels queued_spin_unlock() is dispatched
through a static_call(). Those PARAVIRT_SPINLOCKS=y kernels are quite
popular. Gating contended_release behind a static branch would leave a
NOP on the unlock hot path even, when the tracepoint is disabled.
Since the static_call() is already present, swap its target to a traced
unlock, when the tracepoint is enabled instead. When contended_release
tracepoint is disabled the target is the plain unlock (an inline store
on native x86_64), so the unlock path is unchanged and the tracepoint is
truly zero-cost.
Provide two traced variants, native_queued_spin_unlock_traced() and
pv_queued_spin_unlock_traced(), so each tail-calls its own base unlock
directly rather than recursing through the now-traced static_call().
Teach pv_is_native_spin_unlock() that the traced native variant still
counts as native.
Only PARAVIRT_SPINLOCKS=y is affected. PARAVIRT_SPINLOCKS=n keeps the
generic static-branch path.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/17fa67f9fa4cf93f1150725e89f5f916e41a9b6f.1785778551.git.d@ilvokhin.com
queued_spin_unlock() gates its contended_release trace call behind a
static branch, so a NOP sits on the unlock path even while the
tracepoint is disabled. Removing that requires replacing the unlock
implementation only while contended_release is enabled, which needs a
callback when the tracepoint is toggled.
Convert contended_release to TRACE_EVENT_FN() and add weak no-op
arch_contended_release_trace_reg()/arch_contended_release_trace_unreg()
hooks.
The default hooks are empty, so this is a no-op until an architecture
overrides them.
No functional change intended.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/1c2fcccfb584c075c02890c484f22c76a1948bf1.1785778551.git.d@ilvokhin.com
Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf
lock contention --lock-owner" cannot attribute a contended spinlock to
its holder. The waiter-side contention_begin event records that a
spinlock is contended, but not by whom. Firing contended_release in the
holder's context at unlock is the only way to capture the holder of a
contended spinlock.
Combine the contention check, trace call and release in an out-of-line
queued_spin_release_traced() so the compiler need not preserve the lock
pointer in a callee-saved register across the call.
The check in queued_spin_unlock() is paid on every unlock, even while
the tracepoint is disabled: a static-branch NOP on x86_64, and a few
more instructions to manage a stack frame elsewhere. Gate it behind
CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody
pays for a tracepoint they do not use. Sleeping locks fire
contended_release regardless.
On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g.
defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call
unlock and are wired up separately.
All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option
enabled.
_raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but
disabled. The unlock is the single 'movb'. The only instruction added to
the executed path is the 2-byte static-branch NOP. The CALL to the
traced helper and the JMP back are emitted out of line and are reached
only once the static branch is patched on:
endbr64 ; 4 bytes
xchg %ax,%ax ; 2 static-branch NOP
; (added)
movb $0x0,(%rdi) ; 3 unlock (single store)
A: decl %gs:__preempt_count ; 7
je B ; 2
jmp __x86_return_thunk ; 5
call queued_spin_release_traced ; 5 out of line, reached
; only when the
; tracepoint is on
jmp A ; 2 (added)
B: call __SCT__preempt_schedule ; 5
jmp __x86_return_thunk ; 5
Baseline is the same stream without the NOP and the out-of-line
CALL/JMP: 31 bytes vs 40 (+9 bytes).
Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all
standard configs out-of-line unlock. Architectures with inlined unlock
(s390 (always), csky and loongarch (both when !PREEMPTION)) will see a
bigger increase in binary size.
On the same path (x86_64, PARAVIRT_SPINLOCKS=n) with the tracepoint
disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no
measurable difference between baseline and patched kernels in
throughput, latency, cycles, instructions, IPC, or L1 instruction-cache
misses (kernel and total): all deltas stay within run-to-run noise.
Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on
the executed path in addition to static-branch NOP. Binary size impact
on arm64, defconfig: +932 bytes (+0.00%).
The _raw_spin_unlock()-heavy nginx workload reflects the larger hot
path: L1 instruction-cache misses rise ~1.4% (kernel and total) and
instruction count ~0.4%, consistent with the per-unlock frame.
cpu_cycles, throughput and latency show no measurable change and are
within run-to-run noise.
Architectures with fully custom qspinlock implementations (e.g.
PowerPC) are not covered by this change.
[1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com
The contended_release tracepoint needs to hook queued_spin_unlock(), but
architectures with a custom unlock define queued_spin_unlock() directly,
leaving no single generic place to add the tracing.
Introduce queued_spin_release() as the arch-overridable release
primitive and make queued_spin_unlock() a generic wrapper around it.
An architecture that only customizes the release can then override
queued_spin_release() and inherit the generic wrapper.
Rename the MIPS override to queued_spin_release() accordingly. x86
paravirt overrides queued_spin_unlock() directly and is left unchanged.
No functional change intended.
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/b8daabae6469ad72cc784a911f6cc43a6d45df3a.1785778551.git.d@ilvokhin.com
queued_spin_lock_slowpath() and queued_spin_unlock() are dispatched
through pv_ops_lock via the paravirt-ops ALTERNATIVE machinery, which
picks the target (native inline store / hypervisor call) once at boot
and cannot change at runtime.
Convert both to static_call(). The site becomes a direct call patched in
place (one byte smaller), and on native the unlock still collapses to
the inline "movb $0, (%rdi)" store, so the fast path is unchanged.
Unlike the ALTERNATIVE mechanism, a static_call() target can also be
updated at runtime via static_call_update(). This is a prerequisite for
the contended_release tracepoint, which has to swap in a traced unlock
while the system is running.
[ ilvokhin: commit message; fix PARAVIRT_SPINLOCKS=n build; teach
__static_call_validate() about the inline unlock insn; make the
slowpath site module-safe: static_call_mod() +
EXPORT_STATIC_CALL_TRAMP(); pass @lock to the callee-save unlock,
fixing a boot hang under CALL_DEPTH_TRACKING. Boot tested native + KVM
PV guest. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Co-developed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://lore.kernel.org/all/20260603120811.GW3493090@noisy.programming.kicks-ass.net/
Link: https://patch.msgid.link/9a32ae399eb804a02a31af04dcabe7e7ee4f3fdf.1785778551.git.d@ilvokhin.com
Implement a generic interface for memory barriers (full system/DMA/SMP).
The interface uses a parameter to force user to specify their intent
with barriers.
Provide `Read`, `Write`, `Full` orderings which map to the existing
`rmb()`, `wmb()` and `mb()`. Generic is used here instead of providing
individual standalone functions to reduce code duplication; for example,
the `CONFIG_SMP` check in `smp_mb` is uniformly implemented for all SMP
barriers. This could extend to `virt_mb`'s if they're introduced in the
future. It would also make it easier if new ordering types are
introduced in the future (e.g. `Acquire`, `Release`).
Signed-off-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Boqun Feng <boqun@kernel.org>
Link: https://patch.msgid.link/20260609-rust-barrier-v2-2-30fcc48e1cd0@garyguo.net
futex_private_hash_put() drops the reference to fph before evaluating
fph->mm for wake_up_var(). futex_ref_put() enables preemption again before
returning. If that put drops the final reference and the task is preempted,
another task can pivot to the replacement hash and free the old hash after
an RCU grace period. The first task then reads fph->mm from the freed
allocation when it resumes.
KASAN reports a slab-use-after-free in futex_private_hash_put(), with the
read at offset 24 in a freed kmalloc-512 allocation. The allocation and
free stacks point to futex_hash_allocate() and the RCU free path,
respectively.
Load the mm pointer while the fph reference is still held and pass the
saved value to wake_up_var(). wake_up_var() uses the pointer as a waitqueue
key and does not dereference the mm through it.
Fixes: bd54df5ea7 ("futex: Allow to resize the private local hash")
Signed-off-by: Felix Hoffmann <f3lix.dev@gmx.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20260731155024.1150011-1-f3lix.dev@gmx.de
Commit ee10b10281 ("riscv/runtime-const: Replace open-coded
placeholder with RUNTIME_MAGIC") switched the lui + addi[w] placeholder
from hand-encoded immediates to %hi()/%lo() of RUNTIME_MAGIC.
GNU Assembler folds %hi()/%lo() of an absolute expression at assembly
time. LLVM's integrated assembler since 21.1 defers it to the linker
instead, which range checks it: on riscv64 R_RISCV_HI20 must fit a
signed 20-bit field, and RUNTIME_MAGIC (0x89ABCDEF) is a positive 64-bit
value rather than a sign-extended 32-bit one, so it does not:
ld.lld: error: relocation R_RISCV_HI20 out of range: 563901 is not in
[-524288, 524287]
The sequence is patched at runtime and its instruction offsets are
recorded via ".long 1b - .", so the linker must not touch it in the
first place. Add ".option norelax", as is already done for
ALTERNATIVE() and static keys; the integrated assembler then resolves
%hi()/%lo() itself and emits no relocation, restoring the exact
encoding the open-coded placeholder produced.
Fixes: ee10b10281 ("riscv/runtime-const: Replace open-coded placeholder with RUNTIME_MAGIC")
Closes: https://github.com/ClangBuiltLinux/linux/issues/2179
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Fangrui Song <i@maskray.me>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260802174342.597092-1-i@maskray.me
KCSAN reports a data race between readers_active_check() and a
concurrently executing reader:
BUG: KCSAN: data-race in readers_active_check / percpu_down_write
race at unknown origin, with read to 0xffff9f3eb5bf5f30 of 4 bytes
by task 1271 on cpu 14:
readers_active_check+0x...
percpu_down_write+0x152/0x1f0
value changed: 0xfffffff9 -> 0xfffffff8
readers_active_check() calls per_cpu_sum(*sem->read_count), which
iterates over all CPUs and reads each CPU's per-CPU read_count
variable. Concurrently, a reader on a remote CPU is modifying its own
CPU's read_count via this_cpu_inc() / this_cpu_dec() as it enters and
exits the critical section. These are plain reads and writes to the
same per-CPU storage, hence KCSAN flags a data race.
This race is benign. readers_active_check() is called from the
percpu_down_write() wait loop (rcuwait_wait_event) after sem->block is
already set. At this point:
- New readers must immediately back out (they see block set, decrement
their counter, and wake the writer), so counters can only decrease.
- If the sum catches a reader's increment before its decrement,
readers_active_check() sees a non-zero sum and returns false. The
writer merely iterates the wait loop again -- a harmless retry.
- A false zero (observing sum == 0 while a reader is still active)
cannot happen: per_cpu_sum() reads each CPU's counter, and each
per-CPU int read is atomic on all architectures, so an active
reader's counter is always seen as non-zero.
Annotate the read with data_race() to suppress the KCSAN warning and
document the intentional nature of this unlocked access.
Signed-off-by: Sun Shaojie <sunshaojie@kylinos.cn>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260623104132.505117-1-sunshaojie@kylinos.cn
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a
macro that only gets defined by the Makefiles in the kernel.
This can be very confusing when switching between userspace
and kernelspace coding, or when dealing with uapi headers that
rather should use __ASSEMBLER__ instead. So let's standardize now
on the __ASSEMBLER__ macro that is provided by the compilers.
This is a completely mechanical patch (done with a simple "sed -i"
statement).
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260619124936.208519-1-thuth@redhat.com
register_lock_class() can return NULL when the lock class pool is
exhausted, graph_lock() fails, or key validation fails. However,
__lock_set_class() uses the return value directly in pointer arithmetic
without a NULL check:
class = register_lock_class(lock, subclass, 0);
hlock->class_idx = class - lock_classes;
If class is NULL, this computes a wild offset that corrupts
hlock->class_idx. The subsequent reacquire_held_locks() call will
invoke hlock_class() with this corrupted index, leading to a NULL or
out-of-bounds pointer dereference.
Add the missing NULL check, consistent with how __lock_acquire() already
handles this case at the same call site.
Fixes: 64aa348edc ("lockdep: lock_set_subclass - reset a held lock's subclass")
Signed-off-by: Naveen Kumar Chaudhary <naveen.osdev@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Dmitry Ilvokhin <d@ilvokhin.com>
Link: https://patch.msgid.link/h2kfw43n4527x6mgi2lwpz2rieqnfzgictpv4wr5nyfjkc47co@2r5vz4uz44db
The futex address must be naturally aligned and this is checked via
"address % size" where `address' is the supplied address and `size' is
the expected size of futex. It is guaranteed that `size' is power of two
but the compiler does not see it and creates here a `div' operation
(x86, arm, gcc-15).
We can take advantage of the pow2 property and rewrite it as
"address & (size-1)".
As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30
improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ]
to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]
or by 5.3%
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260701161736.xYYizA0e@linutronix.de
Runtime constify the read-only after init data __futex_shift(shift_32),
__futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash()
hot path to avoid referencing global variable.
This also allows __futex_queues to be allocated dynamically to
"nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES
(1 << CONFIG_NODES_SHIFT) worth of slots upfront.
Runtime constants are initialized before their first access and
runtime_const_init() provides necessary barrier to ensure subsequent
accesses are not reordered against their initialization.
No functional changes intended.
perf bench futex on a 3rd Gen EPYC (2 x 64C/128T):
+----------------+-----------+-----------+-----------+--------------+
| Benchmark | Kernel 1 | Kernel 2 | Unit | % Improvement|
| | (avg/5) | (avg/5) | | (K2 vs K1) |
+----------------+-----------+-----------+-----------+--------------+
| Wake-parallel | 0.01614 | 0.00456 | ms | +71.75% |
| Requeue | 0.26394 | 0.24644 | ms | +6.63% |
| Lock-pi | 34.0 | 57.2 | ops/sec | +68.24% |
+----------------+-----------+-----------+-----------+--------------+
Performance testing on a 144-thread Intel(R) Xeon(R) CPU E7-8890 v3 (4 NUMA nodes):
+-------------------------------------------------------------+
| perf bench futex hash -b 0 |
+----------------------+------------+------------+------------+
| Configuration | As-is | Patched | Delta |
+----------------------+------------+------------+------------+
| 1 thread, 1 futex | 6,449,632 | 6,532,004 | +1.28% |
| 144 threads, 1024 fx | 2,111,486 | 2,139,685 | +1.34% |
+----------------------+------------+------------+------------+i
[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> # MAX_NUMNODES bloat
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Link: https://patch.msgid.link/20260728052540.4728-9-kprateek.nayak@amd.com
Futex hash computation requires a mask operation with read-only after
init data that will be converted to a runtime constant in the subsequent
commit.
Introduce runtime_const_mask_32 to further optimize the mask operation
in the futex hash computation hot path.
GCC generates a:
nilf %r1,<imm32>
to tackle arbitrary 32-bit masks and the same is implemented here.
Immediate patching pattern for __runtime_fixup_mask() has been adopted
from __runtime_fixup_ptr().
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
Link: https://patch.msgid.link/20260728052540.4728-7-kprateek.nayak@amd.com
Futex hash computation requires a mask operation with read-only after
init data that will be converted to a runtime constant in the subsequent
commit.
Introduce runtime_const_mask_32 to further optimize the mask operation
in the futex hash computation hot path. Since all the current use-cases
are of the form GENMASK(n, 0), with n > 0, following sequence:
srli a0, a1, imm
slli a0, a0, imm
is used for RISC-V where imm = (31 - width) to improve instruction
density and performance.
"The RISC-V Instruction Set Manual, Volume I - Unprivileged
Architecture" [1] Sec. 2.4.1 "Integer Register-Immediate Instructions"
notes the immediate shift for SRLI and SLLI are 5 bits wide starting at
bit #10. __runtime_fixup_shift() is reused to patch the immediate shifts
for the two instructions.
If a future use case arises that needs to tackle arbitrary mask,
consider using:
lui a0, 0x12346 # upper; +0x800 then >>12 for correct rounding
addi a0, a0, 0x678 # lower 12 bits
to patch the 32-bit mask in the asm block and return "__ret & (val)"
from runtime_const_mask_32() which allows compiler to further optimize
the logical and operation. __runtime_fixup_ptr() already patches a
lui + addi sequence which can be reused when the need arises.
A possible implementation for this alternate scheme can be found at [2].
Suggested-by: Samuel Holland <samuel.holland@sifive.com>
Suggested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Assisted-by: Claude:claude-sonnet-4-5
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://docs.riscv.org/reference/isa/_attachments/riscv-unprivileged.pdf [1]
Link: https://lore.kernel.org/lkml/20260430094730.31624-6-kprateek.nayak@amd.com/ [2]
Link: https://patch.msgid.link/20260728052540.4728-6-kprateek.nayak@amd.com
Futex hash computation requires a mask operation with read-only after
init data that will be converted to a runtime constant in the subsequent
commit.
Introduce runtime_const_mask_32 to further optimize the mask operation
in the futex hash computation hot path. Since all the current use-cases
are of the form GENMASK(n, 0), with n > 0, a single:
ubfx w0, w0, #0, #widthm1 // w0 = w0 [widthm1:0]
instruction is used for amd64 to improve instruction dinsity and
performance.
"Arm A-profile A64 Instruction Set Architecture" manual, Sec.
"A64 -- Base Instructions" [1] for UBFX instruction highlights the
immediate "width" is encoded as width minus 1 in imms (Bits [15:10])
which is patched by __runtime_fixup_mask() once the mask is known.
If a future use case arises that needs to tackle arbitrary mask,
consider using:
movz w1, #lo16, lsl #0
movk w1, #hi16, lsl #16
to patch the 32-bit mask in the asm block and return "__ret & (val)"
from runtime_const_mask_32() which allows compiler to further optimize
the logical and operation. __runtime_fixup_ptr() already patches a
"movz, + movk lsl #16" sequence which can be reused when the need
arises.
A possible implementation for this alternate scheme can be found at [2].
Suggested-by: Samuel Holland <samuel.holland@sifive.com>
Suggested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://developer.arm.com/documentation/ddi0602/2026-03/Base-Instructions/ [1]
Link: https://lore.kernel.org/lkml/20260430094730.31624-4-kprateek.nayak@amd.com/ [2]
Link: https://patch.msgid.link/20260728052540.4728-4-kprateek.nayak@amd.com
The current scheme to directly patch the kernel text for runtime
constants runs into the following issue with futex adapted to using
runtime constants on arm64:
Unable to handle kernel write to read-only memory at virtual address ...
The pc points to the *p assignment in the following call chain:
futex_init()
runtime_const_init(shift, __futex_shift)
__runtime_fixup_shift()
*p = cpu_to_le32(insn);
which suggests that core_initcall() is too late to patch the kernel text
directly unlike the "d_hash_shift" which is initialized during
vfs_caches_init_early() before the protections are in place.
Use aarch64_insn_patch_text_nosync() to patch the runtime constants
instead of doing it directly to allow runtime_const_init() slightly
later into the boot.
Since aarch64_insn_patch_text_nosync() calls caches_clean_inval_pou()
internally, __runtime_fixup_caches() ends up being redundant.
runtime_const_init() are rare and the overheads of multiple calls to
caches_clean_inval_pou() instead of batching them together should be
negligible in practice.
The cpu_to_le32() conversion of instruction isn't necessary since it is
handled later in the aarch64_insn_patch_text_nosync() call-chain:
aarch64_insn_patch_text_nosync(addr, insn)
aarch64_insn_write(addr, insn)
__aarch64_insn_write(addr, cpu_to_le32(insn))
Sashiko noted that aarch64_insn_patch_text_nosync() does not expect a
lm_alias() address and Catalin suggested it is safe to drop the
lm_alias() for runtime patching since the kernel text is readable. The
address passed to fixup function is interpreted as a __le32 and
dereferenced as is to read the opcode at the patch site.
No functional changes are intended.
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com>
Link: https://patch.msgid.link/20260728052540.4728-3-kprateek.nayak@amd.com
Background:
==========
Currently lockdep does not print out the held locks of non-current
tasks that are running on some other CPU, due to the fact that
the held locks array is in flux and may be unreliable to print.
Syzkaller on the other hand found it that the analysis of locking
bugs is easier if we print this information too, because the
more locking information the merrier. In particular races are
bound to have multiple tasks running on different CPUs, and
the exclusion of their held locks information is unnecessarily
limiting.
So while it's still true that printing out their held locks
array is racy, it's not as bad as it seems.
There's 16 internal callers to lockdep_print_held_locks():
- 14 callers call it with the current task, which should be
safe out of box.
- 1 caller, debug_show_all_locks(), calls it with RCU held,
which should guarantee that 'p' cannot go away under us.
- 1 caller, debug_show_held_locks(), exposes the internal API
with the constraint that it should only be called by drivers
or platform code if the task isn't actively running - we can
assume that if it nevertheless does, it will be Their Problem™.
As for held locks being changed from under debug_show_held_locks(),
while the task cannot go away, so the held-locks array itself is
safe (although potentially non-stable), AFAICS the worst-case race
can be garbage printed out by print_lock(), not any actual crashes.
In particular:
unsigned int class_idx = hlock->class_idx;
may be stale (belong to a lock that already got released on another
CPU), but it should still be a valid class index bound by
MAX_LOCKDEP_KEYS, and thus the lock_classes_in_use bitmap use
should be safe.
The other two accesses are ::acquire_ip and ::instance:
printk(KERN_CONT "%px", hlock->instance);
print_lock_name(hlock, lock);
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);
But both are printed out as pointers, so no risk of dereference
of a dangling pointer. We may print a garbage pointer.
Also note that the check itself doesn't protect debug_show_held_locks()
from printing garbage, as there's nothing that keeps a task from
becoming runnable a nanosecond after we've run the task_is_running()
check. In fact I'd argue that it's better to make this function
*more* racy, for the simple robustness reason that we absolutely
do not want it to crash even in the racy case.
TL;DR: it should be fine to print the held locks of running
tasks too, as long as we print out the information as well
that a task is running, so that users are aware of any
racy output.
Implementation:
==============
Implement that change.
Also re-flow the function and streamline the printout into
a single statement for all cases, which changes
the 'no locks held by' / '%d lock[s] held by' phrasing that had a
dependency on English spelling of plurals, to a uniform:
locks held by bash/1234: %d
Which spells correctly for 0, 1 and higher values, and should also
be easier to parse both for humans and for scripts.
Finally, print out the last CPU a task has ran on. This is very
useful information for races and for locking bugs in particular.
This basically extends the 'on CPU#%d' message we print for
running tasks to all tasks we print.
Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Suggested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Tested-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Cc: Boqun Feng <boqun@kernel.org>
Cc: Gary Guo <gary@garyguo.net>
Cc: Mark Brown <broonie@kernel.org>
Cc: Theodore Tso <tytso@mit.edu>
Cc: Miguel Ojeda <ojeda@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Waiman Long <longman@redhat.com>
Link: https://patch.msgid.link/akoeSIQGwqd9cZwd@gmail.com
Pull x86 fix from Ingo Molnar:
- Prevent OOB access in the resctrl code while offlining
CPUs when Intel SNC (Sub-NUMA Clustering) is enabled
(Reinette Chatre)
* tag 'x86-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86,fs/resctrl: Prevent out-of-bounds access while offlining CPU when SNC enabled
Pull perf events fixes from Ingo Molnar:
- Fix a perf_event_attr::remove_on_exec bug for group events
(Taeyang Lee)
- Fix uprobes CALL emulation interaction with shadow stacks, and
add a testcase for this (David Windsor)
- Fix uprobes unregister bug (Jiri Olsa)
* tag 'perf-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
uprobes/x86: Use proper mm_struct in __in_uprobe_trampoline
selftests/x86: Add shadow stack uprobe CALL test
x86/uprobes: Keep shadow stack in sync for emulated CALLs
perf/core: Detach event groups during remove_on_exec
Pull futex fix from Ingo Molnar:
- Fix a futex-requeue deadlock detection regression (Thomas Gleixner)
* tag 'locking-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
futex/requeue: Revert "Prevent NULL pointer dereference in remove_waiter() on self-deadlock""
Pull irq fixes from Ingo Molnar:
"Misc irqchip driver fixes:
- Fix a resource leak in the RISC-V imsic-early driver (Haoxiang Li)
- Fix an OF node reference leak in the ARM gic-v3-its driver (Yuho
Choi)
- Fix a dangling handler function on module removal bug in the
TS-4800 ARM board irqchip driver (Qingshuang Fu)"
* tag 'irq-urgent-2026-07-05' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
irqchip/ts4800: Fix missing chained handler cleanup on remove
irqchip/gic-v3-its: Fix OF node reference leak
irqchip/irq-riscv-imsic-early: Fix fwnode leak on state setup failure
Pull sound fixes from Takashi Iwai:
"A standard set of driver-specific fixes and quirks accumulated since
the merge window:
ASoC:
- SOF: Sanity check to prevent OOB reads
- rsnd: Fix clock leak and double-disable issues with PM
- tas675x: Misc fixes for register fields, etc
- lpass-va-macro: Correct codec version for Qualcomm SC7280
- amd-yc: DMIC quirk for Alienware m15 R7 AMD
Others:
- us144mkii: Fix a UAF on disconnect and anchor list corruption
- HD-audio: Realtek quirks for HP models"
* tag 'sound-7.2-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound:
ASoC: rsnd: src: Add missing scu_supply clock to suspend/resume
Documentation: sound: tas675x: Fix temperature range and impedance documentation
ASoC: codecs: tas675x: Fix CHx temperature range register bit fields
ASoC: codecs: tas675x: use READ_ONCE for params to be used concurrently
ASoC: rsnd: adg: make rsnd_adg_clk_control() idempotent
ASoC: SOF: validate probe info element counts
ALSA: usx2y: us144mkii: fix work UAF on disconnect
ASoC: amd: yc: Add Alienware m15 R7 AMD to DMIC quirk table
ALSA: hda/realtek: Add quirk for HP Victus 16-e0xxx (88EE) to enable mute LED
MAINTAINERS: ASoC: SOF: add AMD reviewer for Sound Open Firmware
ASoC: codecs: lpass-va-macro: Fix LPASS Codec Version for SC7280
ALSA: us144mkii: capture_urb_complete: redundant usb_anchor_urb corrupts anchor list on each resubmission
Pull spi fixes from Mark Brown:
"A small set of fixes that came in since -rc1, we have one core fix for
shutting down target mode properly if the system suspends while it's
running plus a small set of fairly unremarkable device specific fixes.
There's also a couple of pure DT binding changes for Renesas SoCs, the
power domains one allows some SoCs to be correctly described with
existing code"
* tag 'spi-fix-v7.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi:
spi: rzv2h-rspi: Fix DMA transfer error handling for signal interruption
spi: dt-bindings: snps,dw-apb-ssi: add 'power-domains' property
spi: dt-bindings: snps,dw-apb-ssi: drop superfluous RZ/N1 entry
spi: dw: use the correct error msg if request_irq() fails
spi: dw: fix first spi transfer with dma always fallback to PIO
spi: core: Abort active target transfer on controller suspend
spi: sh-msiof: abort transfers when reset times out
Pull s390 fixes from Vasily Gorbik:
- Fix PKEY_VERIFYPROTK ioctl key type handling by removing the generic
key-length based type check with its wrong bit-size calculation, and
leaving protected key verification to the pkey handler
- Fix monwriter buffer reuse by rejecting records that change the data
length, preventing out of bounds user copy into the kernel buffer
* tag 's390-7.2-3' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux:
s390/monwriter: Reject buffer reuse with different data length
pkey: Move keytype check from pkey api to handler
Pull MIPS fixes from Thomas Bogendoerfer.
* tag 'mips-fixes_7.2_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux:
MIPS: configs: Enable the current Ingenic USB PHY symbol
MIPS: loongson64: add IRQ work based on self-IPI
MIPS: mm: Add check for highmem before removing memory block
mips: Add build salt to the vDSO
MIPS: DEC: Ensure RTC platform device deregistration upon failure
Pull smb server fixes from Steve French:
- Fix several use-after-free races in durable handle reconnect,
supersede, and oplock handling
- Avoid holding the inode oplock lock while waiting for a lease break
acknowledgement. This removes delays of up to 35 seconds when cifs.ko
closes a deferred handle in response to a lease break
- Fix malformed security descriptor handling, including an undersized
DACL allocation issue and an out-of-bounds ACE SID read
- Fix memory leaks in security descriptor and DOS attribute xattr
encoding/decoding error paths
- Fix outstanding SMB2 credit leaks on aborted requests and correct the
QUERY_INFO credit charge calculation
- Fix hard-link creation without replacement being incorrectly rejected
when the handle lacks DELETE access
- Avoid unnecessary zeroing of large SMB2 read buffers
- Add an oplock list lockdep annotation and update the documented
support status for durable handles and SMB3.1.1 compression
- Durable handle fixes to address ownership and lifetime races during
reconnect, session teardown, oplock handling, and superseding opens,
preventing stale session and file references from being used by
concurrent operations
* tag 'v7.2-rc1-smb3-server-fixes' of git://git.samba.org/ksmbd:
ksmbd: fix app-instance durable supersede session UAF
ksmbd: snapshot previous oplock state before durable checks
ksmbd: close superseded durable handles through refcount handoff
ksmbd: fix use-after-free of fp->owner.name in durable handle owner check
smb/server: do not require delete access for non-replacing links
ksmbd: don't hold ci->m_lock while waiting for a lease break ack
ksmbd: doc: update feature support status for durable handles and compression
ksmbd: annotate oplock list traversals under m_lock
ksmbd: fix outstanding credit leak on abort and error paths
ksmbd: fix credit charge calculation for SMB2 QUERY_INFO
ksmbd: avoid zeroing the read buffer in smb2_read()
ksmbd: validate num_subauth when copying ACE in set_ntacl_dacl
ksmbd: reject undersized DACLs before parsing ACEs
ksmbd: fix n.data memory leak in ksmbd_vfs_set_dos_attrib_xattr
ksmbd: Fix acl.sd_buf memory leak and invalid sd_size error handling
ksmbd: fix sd_ndr.data memory leak in ksmbd_vfs_set_sd_xattr
