mirrors/linux
2
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-08-31 03:35:32 -04:00
Code Issues Packages Projects Releases Wiki Activity

locking/qspinlock: Add contended_release tracepoint

Browse Source 
Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf
lock contention --lock-owner" cannot attribute a contended spinlock to
its holder. The waiter-side contention_begin event records that a
spinlock is contended, but not by whom. Firing contended_release in the
holder's context at unlock is the only way to capture the holder of a
contended spinlock.

Combine the contention check, trace call and release in an out-of-line
queued_spin_release_traced() so the compiler need not preserve the lock
pointer in a callee-saved register across the call.

The check in queued_spin_unlock() is paid on every unlock, even while
the tracepoint is disabled: a static-branch NOP on x86_64, and a few
more instructions to manage a stack frame elsewhere. Gate it behind
CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (default n) so nobody
pays for a tracepoint they do not use. Sleeping locks fire
contended_release regardless.

On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g.
defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call
unlock and are wired up separately.

All below are with the QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE option
enabled.

_raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but
disabled. The unlock is the single 'movb'. The only instruction added to
the executed path is the 2-byte static-branch NOP. The CALL to the
traced helper and the JMP back are emitted out of line and are reached
only once the static branch is patched on:

          endbr64                            ; 4 bytes
          xchg   %ax,%ax                     ; 2 static-branch NOP
                                             ;   (added)
          movb   $0x0,(%rdi)                 ; 3 unlock (single store)
       A: decl   %gs:__preempt_count         ; 7
          je     B                           ; 2
          jmp    __x86_return_thunk          ; 5
          call   queued_spin_release_traced  ; 5 out of line, reached
                                             ;   only when the
                                             ;   tracepoint is on
          jmp    A                           ; 2 (added)
       B: call   __SCT__preempt_schedule     ; 5
          jmp    __x86_return_thunk          ; 5

Baseline is the same stream without the NOP and the out-of-line
CALL/JMP: 31 bytes vs 40 (+9 bytes).

Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all
standard configs out-of-line unlock. Architectures with inlined unlock
(s390 (always), csky and loongarch (both when !PREEMPTION)) will see a
bigger increase in binary size.

On the same path (x86_64, PARAVIRT_SPINLOCKS=n) with the tracepoint
disabled, a _raw_spin_unlock()-heavy nginx workload [1] shows no
measurable difference between baseline and patched kernels in
throughput, latency, cycles, instructions, IPC, or L1 instruction-cache
misses (kernel and total): all deltas stay within run-to-run noise.

Unlike x86, on arm64 the frame setup code (STP, MOV and LDP) lands on
the executed path in addition to static-branch NOP. Binary size impact
on arm64, defconfig: +932 bytes (+0.00%).

The _raw_spin_unlock()-heavy nginx workload reflects the larger hot
path: L1 instruction-cache misses rise ~1.4% (kernel and total) and
instruction count ~0.4%, consistent with the per-unlock frame.
cpu_cycles, throughput and latency show no measurable change and are
within run-to-run noise.

Architectures with fully custom qspinlock implementations (e.g.
PowerPC) are not covered by this change.

[1]: https://lore.kernel.org/all/aiphFXe_TPNPxZ_n@shell.ilvokhin.com/

Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Juergen Gross <jgross@suse.com>
Link: https://patch.msgid.link/0d998e22a0c595f670cfc6725bb683323aced5cb.1785778551.git.d@ilvokhin.com
This commit is contained in:
Dmitry Ilvokhin
2026-08-04 07:15:43 +00:00
committed by Peter Zijlstra
parent 216c6c67f7
commit f7e2cb6d49
3 changed files with 63 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
include/asm-generic/qspinlock.h
View File

@@ -41,6 +41,7 @@
#include <asm-generic/qspinlock_types.h>
#include <linux/atomic.h>
#include <linux/tracepoint-defs.h>
#ifndef queued_spin_is_locked
/**
@@ -130,12 +131,32 @@ static __always_inline void queued_spin_release(struct qspinlock *lock)
#endif
#ifndef queued_spin_unlock
DECLARE_TRACEPOINT(contended_release);
extern void queued_spin_release_traced(struct qspinlock *lock);
/**
* queued_spin_unlock - unlock a queued spinlock
* @lock : Pointer to queued spinlock structure
*
* Generic tracing wrapper around the arch-overridable
* queued_spin_release().
*/
static __always_inline void queued_spin_unlock(struct qspinlock *lock)
{
/*
* Trace and release are combined in queued_spin_release_traced() so
* the compiler does not need to preserve the lock pointer across the
* function call, avoiding callee-saved register save/restore on the
* hot path. queued_spin_release() is therefore called both here and in
* queued_spin_release_traced(). Keep the two in sync.
*/
if (IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE) &&
tracepoint_enabled(contended_release)) {
queued_spin_release_traced(lock);
return;
}
queued_spin_release(lock);
}
#endif

20
kernel/Kconfig.locks
View File

@@ -243,6 +243,26 @@ config QUEUED_SPINLOCKS
def_bool y if ARCH_USE_QUEUED_SPINLOCKS
depends on SMP
config QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE
bool "Trace contended_release on queued spinlocks"
depends on QUEUED_SPINLOCKS && TRACEPOINTS
help
Fire the lock:contended_release tracepoint when a contended queued
spinlock is released, so it is possible to attribute a contended
spinlock to its holder.
Architectures that can patch the unlock site do this at no cost and
do not need this option.
Everywhere else the check is compiled into queued_spin_unlock() and
a small cost is paid on every unlock even when the tracepoint is
disabled: a static-branch NOP and possibly a few more instructions
to manage a stack frame.
Sleeping locks fire lock:contended_release regardless of this option.
If unsure, say N.
config BPF_ARCH_SPINLOCK
bool

22
kernel/locking/qspinlock.c
View File

@@ -104,6 +104,28 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock,
#define queued_spin_lock_slowpath native_queued_spin_lock_slowpath
#endif
#if !defined(queued_spin_unlock) && \
IS_ENABLED(CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE)
/*
* Out-of-line trace-and-release path for queued_spin_unlock(), used when
* the contended_release tracepoint is enabled.
*
* queued_spin_release() is duplicated here on purpose: doing the release
* in this function (rather than tracing here and releasing in the caller)
* lets queued_spin_unlock() return right after the call, so the
* tracepoint-disabled hot path never has to keep lock live across a call
* in a callee-saved register. Keep this release in sync with the one in
* queued_spin_unlock().
*/
void __lockfunc queued_spin_release_traced(struct qspinlock *lock)
{
if (queued_spin_is_contended(lock))
trace_call__contended_release(lock);
queued_spin_release(lock);
}
EXPORT_SYMBOL(queued_spin_release_traced);
#endif
#endif /* _GEN_PV_LOCK_SLOWPATH */
/**