The `perf stat --bpf-counters test` fails intermittently on hybrid
architectures or systems with dynamic frequency scaling (DVFS). This
happens because the test workload (`sqrtloop`) runs for a fixed 1-second
duration, and the CPU frequency can scale dynamically between idle and
maximum frequency. As the first run runs on a cold CPU and the second run
runs on a warmed-up CPU (or vice versa), the number of instructions
executed in 1 second differs by up to 2.2x, violating the comparison
tolerance.
Also, when running as root, BPF tracepoints and scheduling programs
trigger frequently. Since standard `perf stat -e instructions` measures
both user and kernel space instructions, it counts BPF helper and program
execution overheads, whereas the BPF counters themselves do not self-
measure. This introduces a large kernel-space instruction count
discrepancy between standard and BPF counters.
Fix these issues by:
1. Switching the workload to a strictly deterministic, iteration-based
workload: `awk 'BEGIN { for (i=0; i<10000000; i++) sum+=i }'`. We pin
the
workload to a single random allowed CPU using `taskset -c $CPU` via a
bash array.
2. Restricting the counted event to user-space only (`instructions:u` or
`/u`).
3. Tightening the comparison tolerance from 20% to 15%.
These modifications isolate the measurements to user-space instructions of
the deterministic loop, which executes a virtually identical number of
instructions on both runs (with less than 0.001% variation), eliminating
Dynamic Frequency Scaling (DVFS), kernel scheduling noise, and BPF helper
self-measurement overheads.
Fixes: 2c0cb9f560 ("perf test: Add a shell test for 'perf stat --bpf-counters' new option")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The `perf trace record and replay` test fails intermittently on slow or
virtualized hosts because the default recording workload (`sleep 1`)
occasionally completes without scheduling the target `nanosleep` or
`clock_nanosleep` system calls inside the recorded sample window,
resulting in the error: `Failed: cannot find *nanosleep syscall`.
Generalize the `perf_record_with_retry` helper in
`tests/shell/lib/perf_record.sh` to support a custom record command prefix
via the `PERF_RECORD_CMD` environment variable (defaulting to "perf
record").
Update `trace_record_replay.sh` to use this robust retry loop running with
`PERF_RECORD_CMD="perf trace record"` and a base workload of `sleep`. The
test will automatically retry with scaled sleep durations (from 0.01s up
to 2.0s) until the required `nanosleep` event is successfully captured.
Fixes: 15bcfb96d0 ("perf test: Add trace record and replay test")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The `python profiling with jitdump` test failed due to:
1. Target PID extraction resolving to duplicate space-separated values,
which broke the buildid-cache loops.
2. The default workload duration being too short to capture JIT stack
trampoline samples, resulting in 0 matching JIT symbols.
Fix the PID parsing by sorting and retrieving a unique single-line value.
Implement a robust retry loop starting at 1M python loop iterations and
scaling up to 100M iterations until JIT symbols are successfully captured
and verified.
Fixes: c9cd0c7e52 ("perf test: Add python JIT dump test")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The metrics value validation test requires system-wide recording (`-a`),
which can fail on systems without root permissions or where paranoid
levels restrict tracing. Add a check to skip the test if `-a` is not
supported.
Also fix false negatives during validation by updating parse error string
patterns and resolving issues in metric list generation.
Fixes: 3ad7092f51 ("perf test: Add metric value validation test")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Introduce `perf_record_with_retry` and `perf_record_cleanup` in a shared
library `tests/shell/lib/perf_record.sh` to prevent record test failures
caused by transient recording or workload delays.
Update `record.sh`, `record_lbr.sh`, `pipe_test.sh`, `kvm.sh`, and
`stat_all_pfm.sh` to use this robust record retry logic. These tests now
start with very short durations (e.g. 0.01 seconds) and scale up if the
initial recording failed to capture samples, significantly improving test
execution speed on success while remaining resilient to slow systems.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Currently, the noploop and thloop workloads only support sleep durations
in integer seconds because they parse the argument using atoi() and use
alarm() for timer signaling.
To support much shorter execution times in tests (speeding up test suites
and allowing faster retries), change the input parsing to use atof() for
double floating-point seconds. Use ualarm() for fractional durations less
than 1.0 seconds, and fall back to alarm() for durations of 1.0 second or
more.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The parallel test harness uses the carriage return delete escape sequence
`PERF_COLOR_DELETE_LINE` ("\033[A\33[2K\r") to erase and update the
"Running (X active)" progress lines.
However, if a test description is longer than the terminal width, the line
wraps around. When this happens, the cursor up escape sequence `\033[A`
only moves the cursor to the last wrapped row, leaving the top half of the
description printed on the previous line. This leads to name duplication
and output corruption spilling over multiple rows on consoles narrower
than the maximum description length (e.g., 101 columns wide).
Fix this by dynamically querying the terminal width using
`get_term_dimensions` and truncating the printed test descriptions using
the `%-*.*s` printf format. We reserve 35 characters for prefix, status,
and spacing metrics to guarantee the progress line never wraps.
Fixes: 0e036dcad4 ("perf test: Display number of active running tests")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Commit b1c5efbfd9 ("perf parse-events: Remove hard coded legacy hardware
and cache parsing") introduced a bypass to PMU filtering to prevent uncore
PMUs from being filtered out during event parsing, which was required for
resolving `duration_time` and `uncore_freq` when running with `--cputype`.
However, this bypass was active whenever `pmu_filter` was set, which also
incorrectly bypassed filtering for the `--pmu-filter` option.
Introduce a `cputype_filter` boolean flag in `parse_events_state` and
`parse_events_option_args` to distinguish filtering initiated by
`--cputype` from that initiated by `--pmu-filter`. Restrict the core-only
check in `parse_events__filter_pmu()` to when `cputype_filter` is true.
Fixes: b1c5efbfd9 ("perf parse-events: Remove hard coded legacy hardware and cache parsing")
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Ian Rogers <irogers@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The second loop for per_pidcommd is meaningful only when it doesn't have
from and to tasks. Also make sure c->Y is set before copying the comm
string otherwise it will be overwritten by next one.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The command line options can be passed without copy as their lifetime is
limited within the same function.
Acked-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The pattern of strdup() and free() is found, and I think it just can
use the original backtrace directly.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The backtrace was used by sched-switch and sched-wakeup only. No need
to call cat_backtrace() in the process_sample_event(). Let's pass NULL
backtrace and generate it from the sched events.
As it needs a pointer to the 'machine', let's save the session in the
timechart struct and use the host machine of the session instead of
passing the pointer to all handlers. It should be fine to assume the
host machine as timechart command doesn't deal with guest machines and
there's no way to get tracepoints from the guest events.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The cat_backtrace() is only called from process_sample_event() which
means the event type is always PERF_RECORD_SAMPLE. We don't need to
pass the event just because to print already known info.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The python metrics code used in places like ilist.py passes a
pmu-filter of "default_core" on non-hybrid x86/ARM/.. systems. As a
PMU like "cpu" isn't a literal name match then no PMU matches
"default_core" and the events fail to parse for the metric. Fix the
name matching and PMU lookup for "default_core" and check that it
fixes ilist.py.
Fixes: 74e2dbe7be ("perf tools: Add --pmu-filter option for filtering PMUs")
Signed-off-by: Ian Rogers <irogers@google.com>
Reviewed‑by: Qinxin Xia <xiaqinxin@huawei.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Since dynamic_events/kprobe_events files show the fetcharg debug
information as comment lines, its reader needs to ignore it.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The test case 'perf test aslr' fails on s390. The root cause of the
failure is subtest test_callchain_aslr. This test invokes command
# ./perf record -g -e task-clock:u -o /tmp/perf-test-aslr \
-- perf test -w noploop 3
to generate a call stack using event task-clock:u. On s390 this defaults
to '--call-graph dwarf' whereas on x86_64 this defaults to framepointer
(fp) format. The command
# ./perf inject --aslr -i /tmp/perf-test-aslr
now scans all SAMPLE entries recorded in the perf.data file to convert
possible addresses. This is done in aslr_tool__process_sample() looking
at sample_type bits PERF_SAMPLE_IP, PERF_SAMPLE_TID,
PERF_SAMPLE_TIME, PERF_SAMPLE_PERIOD, PERF_SAMPLE_CALLCHAIN,
PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER.
On s390 the samples do not contain FP entries
of type PERF_SAMPLE_CALLCHAIN (the bit is set in sample_type, but the
number of FP entries is 0).
The processing enters the PERF_SAMPLE_STACK_USER portion where the
data is copied to the newly constructed sample and then aborted with
this warning:
/* TODO: can this be less conservative? */
pr_debug("Dropping stack user sample as possible ASLR leak\n");
With command line option '--call-graph dwarf' the new output file
does not contain any samples at all. This leads to a missing $new_addr
value in the shell script and a failure.
Fix this and skip this subtest. Emit a hint that this subtest is
currently unsupported on all platform when option --call-graph dwarf
is selected.
Since one subtest is skipped, the complete test is reported as
skipped.
Fixes: 190c454638 ("perf test: Add inject ASLR test")
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Reviewed-by: Ian Rogers <irogers@google.com>
Suggested-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The macro kvm_trace_symbol_exit is used for providing the mappings
for the exit trap vectors and their names. Add mappings for H_FAC_UNAVAIL
and H_VIRT so that exit reasons are displayed as string instead of
vector numbers when using perf kvm stat.
Signed-off-by: Gautam Menghani <gautam@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The powerpc-hcalls.py script currently prints hypercall
statistics in a fixed sort order based on the number of
hcall occurrences, making it difficult to analyze hcalls
from different execution characteristics.
Add support for runtime-configurable sorting so users
can order hypercall statistics by count, minimum
latency, maximum latency, or average latency using
a --sort option.
Parse arguments through sys.argv to support perf script
argument passing semantics.
Example:
perf record -a -e powerpc* sleep 10
perf script -s ./powerpc-hcalls.py -i ./perf.data -- --sort=max
SORT KEY = max
hcall count min(ns) max(ns) avg(ns)
H_SEND_LOGICAL_LAN 47 7380 40148 8739
H_VIO_SIGNAL 706 880 17454 1911
H_RANDOM 1 15176 15176 15176
H_PUT_TCE_INDIRECT 4 3032 10444 4956
H_ADD_LOGICAL_LAN_BUFFER 363 1250 8716 1534
H_SEND_CRQ 8 2086 6846 3044
H_PUT_TCE 9 1284 4932 2646
H_STUFF_TCE 13 1620 3962 2358
This makes it easier to identify frequently occurring
or high-latency hypercalls depending on the analysis
being performed.
Signed-off-by: Shivani Nittor <shivani@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The name of the hcall for opcode 432 is "H_DISABLE_AND_GET". This typo
was fixed in the main file [1] in the commit 0f10228c6f ("KVM: PPC: Fix
typo on H_DISABLE_AND_GET hcall").
[1]: arch/powerpc/include/asm/hvcall.h
Signed-off-by: Gautam Menghani <gautam@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Update the hcall list with the newer hcalls in PPC.
[1]: github.com/torvalds/linux/blob/master/arch/powerpc/include/asm/hvcall.h
Signed-off-by: Gautam Menghani <gautam@linux.ibm.com>
Reviewed-by: Shivani Nittor <shivani@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
In redhat perftool testsuite, observed fail for this test:
-- [ FAIL ] -- perf_sched :: test_timehist :: --with-summary (output regexp parsing)
This led to analysis of "perf sched timehist" summary options.
# perf sched record -a -o ./perf.data -- sleep 0.1
This will record using perf sched record
perf sched timeliest has two options "-s" and "-S"
# perf sched -i ./perf.data timehist -S
-S : Captures summary also at the end
# perf sched -i ./perf.data timehist -s
-s : Captures only summary
The test saves -s result which has only summary and compares with
summary which comes at the end from -S . Since there is a difference
in these two, test fails.
Checking the behaviour change in -S and -s results, difference is:
rcu_sched[16] 2 4 0.013 0.001 0.003 0.006 33.23 0
migration/11[73] 2 1 0.006 0.006 0.006 0.006 0.00 0
migration/3[33] 2 1 0.006 0.006 0.006 0.006 0.00 0
- :216753[216753] -1 1 0.041 0.041 0.041 0.041 0.00 0
+ sleep[216753] -1 1 0.041 0.041 0.041 0.041 0.00 0
migration/8[58] 2 1 0.005 0.005 0.005 0.005 0.00 0
NetworkManager[811] 1 2 0.089 0.028 0.044 0.060 36.06 0
migration/13[83] 2 1 0.005 0.005 0.005 0.005 0.00 0
Here 216753 is pid for sleep which is a zombie process. This is
happening in latest kernel due to an update in "-S" result.
In -S, the process name appears in the results "sleep[216753]",
where as in the -s, only pid is present in the summary result
":216753[216753]".
After commit 39f473f6d0 ("perf sched timehist: decode process names
of processes in zombie state")
for -S option, if process name is using pid, it uses different way to
set it. So that we get the process name and not just Pid.
This change went in only for timehist_print_sample() function.
Add this improvement in generic place so that even -s option (which
captures summary) also will have meaningful information.
Acked-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Signed-off-by: Athira Rajeev <atrajeev@linux.ibm.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The python stubs allow passing None for threads and cpus to the
perf.parse_events() and perf.parse_metrics() bindings. However, PyArg_ParseTuple
parses None into a Py_None object, which is not a NULL pointer.
Because the C code lacked an explicit check for Py_None, it would cast
Py_None to a pyrf_thread_map/pyrf_cpu_map struct pointer and dereference it,
causing a memory corruption crash.
Fix this pre-existing issue by explicitly checking for Py_None alongside NULL
in pyrf__parse_events, pyrf__parse_metrics, and pyrf_evsel__open.
Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alice Rogers <alice.mei.rogers@gmail.com>
Cc: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Leo Yan <leo.yan@linux.dev>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Sessions are necessary to be able to use perf.data files within a
tool. Add a wrapper python type that incorporates the tool. Allow a
sample callback to be passed when creating the session. When
process_events is run this callback will be called, if supplied, for
sample events.
An example use looks like:
```
$ perf record -e cycles,instructions -a sleep 3
$ PYTHONPATH=..../perf/python python3
Python 3.13.7 (main, Aug 20 2025, 22:17:40) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import perf
>>> count=0
... def handle_sample(x):
... global count
... if count < 3:
... print(dir(x))
... count = count + 1
... perf.session(perf.data("perf.data"),sample=handle_sample).process_events()
...
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'sample_addr', 'sample_cpu', 'sample_id', 'sample_ip', 'sample_period', 'sample_pid', 'sample_stream_id', 'sample_tid', 'sample_time', 'type']
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'sample_addr', 'sample_cpu', 'sample_id', 'sample_ip', 'sample_period', 'sample_pid', 'sample_stream_id', 'sample_tid', 'sample_time', 'type']
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'sample_addr', 'sample_cpu', 'sample_id', 'sample_ip', 'sample_period', 'sample_pid', 'sample_stream_id', 'sample_tid', 'sample_time', 'type']
```
Also, add the ability to get the thread associated with a session. For
threads, allow the comm string to be retrieved. This can be useful for
filtering threads. Connect up some of the standard event handling in
psession->tool to better support queries of the machine. Also connect
up the symbols.
Signed-off-by: Ian Rogers <irogers@google.com>
Assisted-by: Gemini:gemini-3.1-pro-preview
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alice Rogers <alice.mei.rogers@gmail.com>
Cc: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Leo Yan <leo.yan@linux.dev>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
As with evlist this a no-op for most of the perf tool. The reference
count is set to 1 at allocation, the put will see the 1, decrement it
and perform the delete.
The purpose for adding the reference count is for the python code. Prior
to this change the python code would clone evsels, but this has issues
if events are opened, etc. leading to assertion failures.
With a reference count the same evsel can be used and the reference
count incremented for the python usage. To not change the python evsel
API getset functions are added for the evsel members, no set function is
provided for size as it doesn't make sense to alter this.
Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alice Rogers <alice.mei.rogers@gmail.com>
Cc: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Leo Yan <leo.yan@linux.dev>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
This a no-op for most of the perf tool. The reference count is set to
1 at allocation, the put will see the 1, decrement it and perform the
delete.
The purpose for adding the reference count is for the python code. Prior
to this change the python code would clone evlists, but this has issues
if events are opened, etc.
This change adds a reference count for the evlists and a later change
will add it to evsels. The combination is needed for the python code to
operate correctly (not hit asserts in the evsel clone), but the changes
are broken apart for the sake of smaller patches.
Assisted-by: Gemini:gemini-3.1-pro-preview
Signed-off-by: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alice Rogers <alice.mei.rogers@gmail.com>
Cc: Dapeng Mi <dapeng1.mi@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Leo Yan <leo.yan@linux.dev>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Richter <tmricht@linux.ibm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
