mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-02-17 11:21:05 -05:00
64ad6470c882fcaecfa4a1da96ea94de7ca0dc80
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Fan Yu
|
0c10f9cd81 |
tools/delaytop: improve error handling for missing PSI support
Enhanced display logic to conditionally show PSI information only when successfully read, with helpful guidance for users to enable PSI support (psi=1 cmdline parameter). Link: https://lkml.kernel.org/r/20250907001417537vSx6nUsb3ILqI0iQ-WnGp@zte.com.cn Signed-off-by: Fan Yu <fan.yu9@zte.com.cn> Reviewed-by: xu xin <xu.xin16@zte.com.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Fan Yu
|
5e57515d81 |
tools/delaytop: add interactive mode with keyboard controls
The original delaytop only supported static output with limited interaction. Users had to restart the tool with different command-line options to change sorting or display modes, which disrupted continuous monitoring and reduced productivity during performance investigations. Adds real-time interactive controls through keyboard input: 1) Add interactive menu system with visual prompts 2) Support dynamic sorting changes without restarting 3) Enable toggle of memory verbose mode with 'M' key The interactive mode transforms delaytop from a static monitoring tool into a dynamic investigation platform, allowing users to adapt the view in real-time based on observed performance patterns. Link: https://lkml.kernel.org/r/20250907001338580EURha20BxWFmBSrUpS8D1@zte.com.cn Signed-off-by: Fan Yu <fan.yu9@zte.com.cn> Reviewed-by: xu xin <xu.xin16@zte.com.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Fan Yu
|
99d9c55f88 |
tools/delaytop: add memory verbose mode support
The original delaytop tool always displayed detailed memory subsystem breakdown, which could be overwhelming for users who only need high-level overview. Add flexible display control allowing users to choose their preferred information granularity. The new flexibility provides: 1) For quick monitoring: use normal mode to reduce visual clutter 2) For deep analysis: use verbose mode to see all memory subsystem details Link: https://lkml.kernel.org/r/202509070012527934u0ySb3teQ4gOYKnocyNO@zte.com.cn Signed-off-by: Fan Yu <fan.yu9@zte.com.cn> Reviewed-by: xu xin <xu.xin16@zte.com.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Fan Yu
|
0471440c80 |
tools/delaytop: add flexible sorting by delay field
Patch series "tools/delaytop: implement real-time keyboard interaction
support", v2.
Current Limitations
===================
The current delaytop implementation has two main limitations:
1) Static sorting only by CPU delay Forcing users to restart with
different parameters to analyze other resource bottlenecks.
2) Memory delay information is always expanded Causing information
overload when only high-level memory pressure monitoring is needed.
Improvements
============
1) Implemented dynamic sorting capability
- Interactive key 'o' triggers sort mode.
- Supports sorting by CPU/IO/Memory/IRQ delays.
- Memory subcategories available in verbose mode.
* c - CPU delay (default)
* i - IO delay
* m - Total memory delay
* q - IRQ delay
* s/r/t/p/w - Memory subcategories (in verbose mode)
2) Added memory display modes
- Compact view (default): shows aggregated memory delays.
- Verbose view ('M' key): breaks down into memory sub-delays.
* SWAP - swapin delays
* RCL - freepages reclaim delays
* THR - thrashing delays
* CMP - compaction delays
* WP - write-protect copy delays
Practical benefits
==================
1) Dynamic Sorting for Real-Time Bottleneck Detection System
administrators can now dynamically change sorting to identify different
types of resource bottlenecks without restarting.
2) Enhanced Usability with On-Screen Keybindings More intuitive
interactive usage with on-screen keybindings help. Reduced screen
clutter when only memory overview is needed.
Use Case
========
# ./delaytop
System Pressure Information: (avg10/avg60vg300/total)
CPU some: 0.0%/ 0.0%/ 0.0%/ 106817(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
[o]sort [M]memverbose [q]quit
Top 20 processes (sorted by cpu delay):
PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms)
------------------------------------------------------------------------
110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00
57 57 cpuhp/7 3.18 0.00 0.00 0.00
99 99 cpuhp/14 2.97 0.00 0.00 0.00
51 51 cpuhp/6 0.90 0.00 0.00 0.00
44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00
76 76 idle_inject/10 0.31 0.00 0.00 0.00
100 100 idle_inject/14 0.30 0.00 0.00 0.00
1309 1309 systemsettings 0.29 0.00 0.00 0.00
60 60 ksoftirqd/7 0.28 0.00 0.00 0.00
45 45 cpuhp/5 0.22 0.00 0.00 0.00
63 63 cpuhp/8 0.20 0.00 0.00 0.00
87 87 cpuhp/12 0.18 0.00 0.00 0.00
93 93 cpuhp/13 0.17 0.00 0.00 0.00
1265 1265 acpid 0.17 0.00 0.00 0.00
1552 1552 sshd 0.17 0.00 0.00 0.00
2584 2584 sddm-helper 0.16 0.00 0.00 0.00
1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00
1326 1326 nde-netfilter 0.14 0.00 0.00 0.00
27 27 cpuhp/2 0.13 0.00 0.00 0.00
631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00
# ./delaytop -M
System Pressure Information: (avg10/avg60vg300/total)
CPU some: 0.0%/ 0.0%/ 0.0%/ 106827(ms)
CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms)
IO full: 0.0%/ 0.0%/ 0.0%/ 2245(ms)
IO some: 0.0%/ 0.0%/ 0.0%/ 2791(ms)
IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms)
[o]sort [M]memverbose [q]quit
Top 20 processes (sorted by mem delay):
PID TGID COMMAND MEM(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms)
------------------------------------------------------------------------------------------
121732 121732 delaytop 0.01 0.00 0.00 0.00 0.00 0.01
95876 95876 top 0.00 0.00 0.00 0.00 0.00 0.00
121641 121641 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
121693 121693 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
121661 121661 systemd-userwor 0.00 0.00 0.00 0.00 0.00 0.00
1 1 systemd 0.00 0.00 0.00 0.00 0.00 0.00
2 2 kthreadd 0.00 0.00 0.00 0.00 0.00 0.00
3 3 pool_workqueue_ 0.00 0.00 0.00 0.00 0.00 0.00
4 4 kworker/R-rcu_g 0.00 0.00 0.00 0.00 0.00 0.00
5 5 kworker/R-rcu_p 0.00 0.00 0.00 0.00 0.00 0.00
6 6 kworker/R-slub_ 0.00 0.00 0.00 0.00 0.00 0.00
7 7 kworker/R-netns 0.00 0.00 0.00 0.00 0.00 0.00
9 9 kworker/0:0H-sy 0.00 0.00 0.00 0.00 0.00 0.00
11 11 kworker/u32:0-n 0.00 0.00 0.00 0.00 0.00 0.00
12 12 kworker/R-mm_pe 0.00 0.00 0.00 0.00 0.00 0.00
13 13 rcu_tasks_kthre 0.00 0.00 0.00 0.00 0.00 0.00
14 14 rcu_tasks_rude_ 0.00 0.00 0.00 0.00 0.00 0.00
15 15 rcu_tasks_trace 0.00 0.00 0.00 0.00 0.00 0.00
16 16 ksoftirqd/0 0.00 0.00 0.00 0.00 0.00 0.00
17 17 rcu_preempt 0.00 0.00 0.00 0.00 0.00 0.00
When psi is not enabled:
# ./delaytop
System Pressure Information: (avg10/avg60vg300/total)
PSI not found: check if psi=1 enabled in cmdline
This patch (of 5):
The delaytop tool only supported sorting by CPU delay, which limited its
usefulness when users needed to identify bottlenecks in other subsystems.
Users had no way to sort processes by IO, IRQ, or other delay types to
quickly pinpoint specific performance issues.
Add -s/--sort option to allow sorting by different delay types. Users can
now quickly identify bottlenecks in specific subsystems by sorting
processes by the relevant delay metric.
Link: https://lkml.kernel.org/r/20250907001101305vrTGnXaRNvtmsGkp-Ljk_@zte.com.cn
Link: https://lkml.kernel.org/r/20250907001205573L3XpsQMIQnLgDqiiKYd3H@zte.com.cn
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Reviewed-by: xu xin <xu.xin16@zte.com.cn>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Wang Yaxin <wang.yaxin@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
fan.yu9@zte.com.cn
|
d92dccd05a |
delaytop: enhance error logging and add PSI feature description
This patch improves error diagnostics and documentation for delaytop: 1) Enhanced error logging: - Added explicit error messages in critical failure paths - Implemented BOOL_FPRINT macro for robust output handling 2) PSI feature documentation: - Updated header comment to reflect PSI monitoring capability - Improved output formatting for PSI information System Pressure Information: (avg10/avg60/avg300/total) CPU some: 0.0%/ 0.0%/ 0.0%/ 345(ms) CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) IO full: 0.0%/ 0.0%/ 0.0%/ 65(ms) IO some: 0.0%/ 0.0%/ 0.0%/ 79(ms) IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) Link: https://lkml.kernel.org/r/202507281628341752gMXCMN7S-Vz_LHYHum9r@zte.com.cn Signed-off-by: Fan Yu <fan.yu9@zte.com.cn> Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> Acked-by: Yang Yang <yang.yang29@zte.com.cn> Cc: Fan Yu <fan.yu9@zte.com.cn> Cc: Jonathan Corbet <corbet@lwn.net> Cc: xu xin <xu.xin16@zte.com.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Wang Yaxin
|
6b47c9f8ee |
delaytop: add psi info to show system delay
Support showing whole delay of system by reading PSI, just like the first few lines of information output by the top command. the output of delaytop includes both system-wide delay and delay of individual tasks, providing a more comprehensive reflection of system latency status. Use case ======== bash# ./delaytop System Pressure Information: (avg10/avg60/avg300/total) CPU: full: 0.0%/ 0.0%/ 0.0%/0 some: 0.1%/ 0.0%/ 0.0%/14216596 Memory: full: 0.0%/ 0.0%/ 0.0%/34010659 some: 0.0%/ 0.0%/ 0.0%/35406492 IO: full: 0.1%/ 0.0%/ 0.0%/51029453 some: 0.1%/ 0.0%/ 0.0%/55330465 IRQ: full: 0.0%/ 0.0%/ 0.0%/0 Top 20 processes (sorted by CPU delay): PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms) --------------------------------------------------------------------------------------------- 32 32 kworker/2:0H-sy 23.65 0.00 0.00 0.00 0.00 0.00 0.00 0.00 497 497 kworker/R-scsi_ 1.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 495 495 kworker/R-scsi_ 1.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 494 494 scsi_eh_0 1.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 485 485 kworker/R-ata_s 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 574 574 kworker/R-kdmfl 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 34 34 idle_inject/3 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1123 1123 nde-netfilter 0.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 60 60 ksoftirqd/7 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 114 114 kworker/0:2-cgr 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 496 496 scsi_eh_1 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 51 51 cpuhp/6 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1667 1667 atd 0.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 45 45 cpuhp/5 0.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1102 1102 nde-backupservi 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1098 1098 systemsettings 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1100 1100 audit-monitor 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 53 53 migration/6 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1482 1482 sshd 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 39 39 cpuhp/4 0.19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Link: https://lkml.kernel.org/r/20250710135451340_5pOgpIFi0M5AE7H44W1D@zte.com.cn Co-developed-by: Fan Yu <fan.yu9@zte.com.cn> Signed-off-by: Fan Yu <fan.yu9@zte.com.cn> Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> Signed-off-by: Jiang Kun <jiang.kun2@zte.com.cn> Cc: Balbir Singh <bsingharora@gmail.com> Cc: David Hildenbrand <david@redhat.com> Cc: Peilin He <he.peilin@zte.com.cn> Cc: Qiang Tu <tu.qiang35@zte.com.cn> Cc: wangyong <wang.yong12@zte.com.cn> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Yang Yang <yang.yang29@zte.com.cn> Cc: Yunkai Zhang <zhang.yunkai@zte.com.cn> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Yaxin Wang
|
01bda05819 |
tools/accounting/delaytop: add delaytop to record top-n task delay
Problem
=======
The "getdelays" can only display the latency of a single task
by specifying a PID, but it has the following limitations:
1. single-task perspective: only supports querying the latency (CPU, I/O,
memory, etc.) of an individual task via PID and cannot provide a global
analysis of high-latency processes across the system.
2. lack of High-Latency process awareness: when the overall system
latency is high (e.g., a spike in CPU latency), there is no way to
quickly identify the top N processes contributing to the highest
latency.
3. poor interactivity: It lacks dynamic sorting and refresh
capabilities (similar to top), making it difficult to monitor latency
changes in real time.
Solution
========
To address these limitations, we introduce the "delaytop" with the
following capabilities:
1. system view: monitors latency metrics (CPU, I/O, memory, IRQ, etc.)
for all system processes
2. supports field-based sorting (e.g., default sort by CPU latency in
descending order)
3. dynamic interactive interface: focus on specific processes with
--pid; limit displayed entries with --processes 20; control monitoring
duration with --iterations;
Use case
========
bash# ./delaytop
Top 20 processes (sorted by CPU delay):
PID TGID COMMAND CPU(ms) IO(ms) SWAP(ms) RCL(ms) THR(ms) CMP(ms) WP(ms) IRQ(ms)
---------------------------------------------------------------------------------------------
26 26 kworker/1:0H 5.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
32 32 kworker/2:0H-kb 2.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00
38 38 kworker/3:0H-ev 2.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00
84 84 kworker/R-vfio- 1.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00
24 24 ksoftirqd/1 1.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19 19 idle_inject/0 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00
16 16 rcu_exp_par_gp_ 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11 11 kworker/0:1 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00
22 22 idle_inject/1 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00
3 3 pool_workqueue_ 0.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00
81 81 scsi_eh_1 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00
30 30 ksoftirqd/2 0.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00
36 36 ksoftirqd/3 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9 9 kworker/0:0-eve 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
8 8 kworker/R-netns 0.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
76 76 kworker/1:1-pm 0.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
21 21 cpuhp/1 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
4 4 kworker/R-rcu_g 0.21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12 12 kworker/u16:0-i 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1 1 init 0.18 0.00 0.00 0.00 0.00 0.00 0.08 0.00
Link: https://lkml.kernel.org/r/20250619211843633h05gWrBDMFkEH6xAVm_5y@zte.com.cn
Co-developed-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
Signed-off-by: Yaxin Wang <wang.yaxin@zte.com.cn>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Peilin He <he.peilin@zte.com.cn>
Cc: Qiang Tu <tu.qiang35@zte.com.cn>
Cc: wangyong <wang.yong12@zte.com.cn>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Yang Yang <yang.yang29@zte.com.cn>
Cc: ye xingchen <ye.xingchen@zte.com.cn>
Cc: Yunkai Zhang <zhang.yunkai@zte.com.cn>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|