mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-04-03 12:16:22 -04:00
c73a56ed3c
The subtest 'Leader sampling' some time fails on s390.
- for z/VM guest: Disable the test for z/VM guest. There is no
CPU Measurement facility to run the test successfully.
- for LPAR: Use correct event names.
A detailed analysis follows here:
Now to the debugging and investigation:
1. With command
perf record -e '{cycles,cycles}:S' -- ....
the first cycles event starts sampling.
On s390 this sets up sampling with a frequency of 4000 Hz.
This translates to hardware sample rate of 1377000 instructions per
micro-second to meet a frequency of 4000 HZ.
2. With first event cycles now sampling into a hardware buffer, an
interrupt is triggered each time a sampling buffer gets full.
The interrupt handler is then invoked and debug output shows the
processing of samples. The size of one hardware sample is 32 bytes.
With an interrupt triggered when the hardware buffer page of 4KB
gets full, the interrupt handler processes 128 samples.
(This is taken from s390 specific fast debug data gathering)
2025-11-07 14:35:51.977248 000003ffe013cbfa \
perf_event_count_update event->count 0x0 count 0x1502e8
2025-11-07 14:35:51.977248 000003ffe013cbfa \
perf_event_count_update event->count 0x1502e8 count 0x1502e8
2025-11-07 14:35:51.977248 000003ffe013cbfa \
perf_event_count_update event->count 0x2a05d0 count 0x1502e8
2025-11-07 14:35:51.977252 000003ffe013cbfa \
perf_event_count_update event->count 0x3f08b8 count 0x1502e8
2025-11-07 14:35:51.977252 000003ffe013cbfa \
perf_event_count_update event->count 0x540ba0 count 0x1502e8
2025-11-07 14:35:51.977253 000003ffe013cbfa \
perf_event_count_update event->count 0x690e88 count 0x1502e8
2025-11-07 14:35:51.977254 000003ffe013cbfa \
perf_event_count_update event->count 0x7e1170 count 0x1502e8
2025-11-07 14:35:51.977254 000003ffe013cbfa \
perf_event_count_update event->count 0x931458 count 0x1502e8
2025-11-07 14:35:51.977254 000003ffe013cbfa \
perf_event_count_update event->count 0xa81740 count 0x1502e8
3. The value is constantly increasing by the number of instructions
executed to generate a sample entry. This is the first line of the
pairs of lines. count 0x1502e8 --> 1377000
# perf script | grep 1377000 | wc -l
214
# perf script | wc -l
428
#
That is 428 lines in total, and half of the lines contain value
1377000.
4. The second event cycles is opened against the counting PMU, which
is an independent PMU and is not interrupt driven. Once enabled it
runs in the background and keeps running, incrementing silently
about 400+ counters. The counter values are read via assembly
instructions.
This second counter PMU's read call back function is called when the
interrupt handler of the sampling facility processes each sample. The
function call sequence is:
perf_event_overflow()
+--> __perf_event_overflow()
+--> __perf_event_output()
+--> perf_output_sample()
+--> perf_output_read()
+--> perf_output_read_group()
for_each_sibling_event(sub, leader) {
values[n++] = perf_event_count(sub, self);
printk("%s sub %p values %#lx\n", __func__, sub, values[n-1]);
}
The last function perf_event_count() is invoked on the second event
cylces *on* the counting PMU. An added printk statement shows the
following lines in the dmesg output:
# dmesg|grep perf_output_read_group |head -10
[ 332.368620] perf_output_read_group sub 00000000d80b7c1f values 0x3a80917 (1)
[ 332.368624] perf_output_read_group sub 00000000d80b7c1f values 0x3a86c7f (2)
[ 332.368627] perf_output_read_group sub 00000000d80b7c1f values 0x3a89c15 (3)
[ 332.368629] perf_output_read_group sub 00000000d80b7c1f values 0x3a8c895 (4)
[ 332.368631] perf_output_read_group sub 00000000d80b7c1f values 0x3a8f569 (5)
[ 332.368633] perf_output_read_group sub 00000000d80b7c1f values 0x3a9204b
[ 332.368635] perf_output_read_group sub 00000000d80b7c1f values 0x3a94790
[ 332.368637] perf_output_read_group sub 00000000d80b7c1f values 0x3a9704b
[ 332.368638] perf_output_read_group sub 00000000d80b7c1f values 0x3a99888
#
This correlates with the output of
# perf report -D | grep 'id 00000000000000'|head -10
..... id 0000000000000006, value 00000000001502e8, lost 0
..... id 000000000000000e, value 0000000003a80917, lost 0 --> line (1) above
..... id 0000000000000006, value 00000000002a05d0, lost 0
..... id 000000000000000e, value 0000000003a86c7f, lost 0 --> line (2) above
..... id 0000000000000006, value 00000000003f08b8, lost 0
..... id 000000000000000e, value 0000000003a89c15, lost 0 --> line (3) above
..... id 0000000000000006, value 0000000000540ba0, lost 0
..... id 000000000000000e, value 0000000003a8c895, lost 0 --> line (4) above
..... id 0000000000000006, value 0000000000690e88, lost 0
..... id 000000000000000e, value 0000000003a8f569, lost 0 --> line (5) above
Summary:
- Above command starts the CPU sampling facility, with runs interrupt
driven when a 4KB page is full. An interrupt processes the 128 samples
and calls eventually perf_output_read_group() for each sample to save it
in the event's ring buffer.
- At that time the CPU counting facility is invoked to read the value of
the event cycles. This value is saved as the second value in the
sample_read structure.
- The first and odd lines in the perf script output displays the period
value between 2 samples being created by hardware. It is the number
of instructions executes before the hardware writes a sample.
- The second and even lines in the perf script output displays the number
of CPU cycles needed to process each sample and save it in the event's
ring buffer.
These 2 different values can never be identical on s390.
Since event leader sampling is not possible on s390 the perf tool will
return EOPNOTSUPP soon. Perpare the test case for that.
Suggested-by: James Clark <james.clark@linaro.org>
Reviewed-by: Jan Polensky <japo@linux.ibm.com>
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Tested-by: Jan Polensky <japo@linux.ibm.com>
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Linux kernel ============ The Linux kernel is the core of any Linux operating system. It manages hardware, system resources, and provides the fundamental services for all other software. Quick Start ----------- * Report a bug: See Documentation/admin-guide/reporting-issues.rst * Get the latest kernel: https://kernel.org * Build the kernel: See Documentation/admin-guide/quickly-build-trimmed-linux.rst * Join the community: https://lore.kernel.org/ Essential Documentation ----------------------- All users should be familiar with: * Building requirements: Documentation/process/changes.rst * Code of Conduct: Documentation/process/code-of-conduct.rst * License: See COPYING Documentation can be built with make htmldocs or viewed online at: https://www.kernel.org/doc/html/latest/ Who Are You? ============ Find your role below: * New Kernel Developer - Getting started with kernel development * Academic Researcher - Studying kernel internals and architecture * Security Expert - Hardening and vulnerability analysis * Backport/Maintenance Engineer - Maintaining stable kernels * System Administrator - Configuring and troubleshooting * Maintainer - Leading subsystems and reviewing patches * Hardware Vendor - Writing drivers for new hardware * Distribution Maintainer - Packaging kernels for distros For Specific Users ================== New Kernel Developer -------------------- Welcome! Start your kernel development journey here: * Getting Started: Documentation/process/development-process.rst * Your First Patch: Documentation/process/submitting-patches.rst * Coding Style: Documentation/process/coding-style.rst * Build System: Documentation/kbuild/index.rst * Development Tools: Documentation/dev-tools/index.rst * Kernel Hacking Guide: Documentation/kernel-hacking/hacking.rst * Core APIs: Documentation/core-api/index.rst Academic Researcher ------------------- Explore the kernel's architecture and internals: * Researcher Guidelines: Documentation/process/researcher-guidelines.rst * Memory Management: Documentation/mm/index.rst * Scheduler: Documentation/scheduler/index.rst * Networking Stack: Documentation/networking/index.rst * Filesystems: Documentation/filesystems/index.rst * RCU (Read-Copy Update): Documentation/RCU/index.rst * Locking Primitives: Documentation/locking/index.rst * Power Management: Documentation/power/index.rst Security Expert --------------- Security documentation and hardening guides: * Security Documentation: Documentation/security/index.rst * LSM Development: Documentation/security/lsm-development.rst * Self Protection: Documentation/security/self-protection.rst * Reporting Vulnerabilities: Documentation/process/security-bugs.rst * CVE Procedures: Documentation/process/cve.rst * Embargoed Hardware Issues: Documentation/process/embargoed-hardware-issues.rst * Security Features: Documentation/userspace-api/seccomp_filter.rst Backport/Maintenance Engineer ----------------------------- Maintain and stabilize kernel versions: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * Backporting Guide: Documentation/process/backporting.rst * Applying Patches: Documentation/process/applying-patches.rst * Subsystem Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git for Maintainers: Documentation/maintainer/configure-git.rst System Administrator -------------------- Configure, tune, and troubleshoot Linux systems: * Admin Guide: Documentation/admin-guide/index.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Sysctl Tuning: Documentation/admin-guide/sysctl/index.rst * Tracing/Debugging: Documentation/trace/index.rst * Performance Security: Documentation/admin-guide/perf-security.rst * Hardware Monitoring: Documentation/hwmon/index.rst Maintainer ---------- Lead kernel subsystems and manage contributions: * Maintainer Handbook: Documentation/maintainer/index.rst * Pull Requests: Documentation/maintainer/pull-requests.rst * Managing Patches: Documentation/maintainer/modifying-patches.rst * Rebasing and Merging: Documentation/maintainer/rebasing-and-merging.rst * Development Process: Documentation/process/maintainer-handbooks.rst * Maintainer Entry Profile: Documentation/maintainer/maintainer-entry-profile.rst * Git Configuration: Documentation/maintainer/configure-git.rst Hardware Vendor --------------- Write drivers and support new hardware: * Driver API Guide: Documentation/driver-api/index.rst * Driver Model: Documentation/driver-api/driver-model/driver.rst * Device Drivers: Documentation/driver-api/infrastructure.rst * Bus Types: Documentation/driver-api/driver-model/bus.rst * Device Tree Bindings: Documentation/devicetree/bindings/ * Power Management: Documentation/driver-api/pm/index.rst * DMA API: Documentation/core-api/dma-api.rst Distribution Maintainer ----------------------- Package and distribute the kernel: * Stable Kernel Rules: Documentation/process/stable-kernel-rules.rst * ABI Documentation: Documentation/ABI/README * Kernel Configuration: Documentation/kbuild/kconfig.rst * Module Signing: Documentation/admin-guide/module-signing.rst * Kernel Parameters: Documentation/admin-guide/kernel-parameters.rst * Tainted Kernels: Documentation/admin-guide/tainted-kernels.rst Communication and Support ========================= * Mailing Lists: https://lore.kernel.org/ * IRC: #kernelnewbies on irc.oftc.net * Bugzilla: https://bugzilla.kernel.org/ * MAINTAINERS file: Lists subsystem maintainers and mailing lists * Email Clients: Documentation/process/email-clients.rst