perf lock contention returns zero exit value even if the lock contention
BPF setup failed.
# ./perf lock con -b true
libbpf: kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled?
libbpf: failed to find '.BTF' ELF section in /lib/modules/6.13.0-rc3+/build/vmlinux
libbpf: failed to find valid kernel BTF
libbpf: kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled?
libbpf: failed to find '.BTF' ELF section in /lib/modules/6.13.0-rc3+/build/vmlinux
libbpf: failed to find valid kernel BTF
libbpf: Error loading vmlinux BTF: -ESRCH
libbpf: failed to load object 'lock_contention_bpf'
libbpf: failed to load BPF skeleton 'lock_contention_bpf': -ESRCH
Failed to load lock-contention BPF skeleton
lock contention BPF setup failed
# echo $?
0
Fix this by saving the return code for lock_contention_prepare
so that command exits with proper return code. Similarly set the
return code properly for two other functions in builtin-lock, namely
setup_output_field() and select_key().
Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20250110093730.93610-1-atrajeev@linux.vnet.ibm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
When a filtered column is not present in the sort order, profiles become
arbitrary broken. Filtered and non-filtered entries are collapsed
together, and the filtered-by field ends up with a random value (either
from a filtered or non-filtered entry). If we end up with filtered
entry/value, then the whole collapsed entry will be filtered out and will
be missing in the profile. If we end up with non-filtered entry/value,
then the overhead value will be wrongly larger (include some subset
of filtered out samples).
This leads to very confusing profiles. The problem is hard to notice,
and if noticed hard to understand. If the filter is for a single value,
then it can be fixed by adding the corresponding field to the sort order
(provided user understood the problem). But if the filter is for multiple
values, it's impossible to fix b/c there is no concept of binary sorting
based on filter predicate (we want to group all non-filtered values in
one bucket, and all filtered values in another).
Examples of affected commands:
perf report --tid=123
perf report --sort overhead,symbol --comm=foo,bar
Fix this by considering filtered status as the highest priority
sort/collapse predicate.
As a side effect this effectively adds a new feature of showing profile
where several lines are combined based on arbitrary filtering predicate.
For example, showing symbols from binaries foo and bar combined together,
but not from other binaries; or showing combined overhead of several
particular threads.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Link: https://lore.kernel.org/r/359dc444ce94d20e59d3a9e360c36fbeac833a04.1736927981.git.dvyukov@google.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The relationship between subtests and test cases is somewhat
confusing, so let's do away with the notion of sub-tests and switch to
just working with some number of test cases. Add a
test_suite__for_each_test_case as in many cases, except the special
one test case situation, the iteration can just be on all test
cases. Switch variable names to be more intention revealing of what
their value is.
This work was motivated by discussion with Kan where it was noted the
code is becoming overly indented:
https://lore.kernel.org/lkml/20241109160219.49976-1-irogers@google.com/
Unifying more of the sub-test/no-sub-tests avoids one level of
indentation in a number of places.
Signed-off-by: Ian Rogers <irogers@google.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Cc: James Clark <james.clark@linaro.org>
Link: https://lore.kernel.org/r/20250110045736.598281-2-irogers@google.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The variables to make builds silent/verbose live inside
tools/build/Makefile.build. Move those variables to the top-level
Makefile.perf to be generally available.
Committer testing:
See the SYSCALL lines, now they are consistent with the other
operations in other lines:
SYSTBL /tmp/build/perf-tools-next/arch/x86/include/generated/asm/syscalls_32.h
SYSTBL /tmp/build/perf-tools-next/arch/x86/include/generated/asm/syscalls_64.h
GEN /tmp/build/perf-tools-next/common-cmds.h
GEN /tmp/build/perf-tools-next/arch/arm64/include/generated/asm/sysreg-defs.h
PERF_VERSION = 6.13.rc2.g3d94bb6ed1d0
GEN perf-archive
MKDIR /tmp/build/perf-tools-next/jvmti/
MKDIR /tmp/build/perf-tools-next/jvmti/
MKDIR /tmp/build/perf-tools-next/jvmti/
MKDIR /tmp/build/perf-tools-next/jvmti/
GEN perf-iostat
CC /tmp/build/perf-tools-next/jvmti/libjvmti.o
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Charlie Jenkins <charlie@rivosinc.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Suzuki K Poulose <suzuki.poulose@arm.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Mike Leach <mike.leach@linaro.org>
Cc: linux-arm-kernel@lists.infradead.org
Cc: coresight@lists.linaro.org
Link: https://lore.kernel.org/r/20250114-perf_make_test-v1-1-decc1c517b11@rivosinc.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
The test failed back and forth due to the call chain being heavily
impacted by the libc, which varies across different architectures and
distros.
The libc contains the symbols for "gaih_inet" and "getaddrinfo" in some
cases, but not always. Moreover, these symbols can be either normal
symbols or dynamic symbols, making it difficult to decide the call chain
entries due to the symbols are inconsistent.
To fix the issue, this commit identifies three call chain entries are
always present. These entries are matched by iterating through the
lines in the "perf script" result. The recording attribute max-stack is
set to 4 for the possible maximum call chain depth.
After:
# perf test -vF pton
--- start ---
Pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\)
Matching: ping 285058 [025] 1219802.466939: probe_libc:inet_pton: (ffffa14b7cf0)
Pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib/aarch64-linux-gnu/libc-2.31.so|inlined\)$
Matching: ping 285058 [025] 1219802.466939: probe_libc:inet_pton: (ffffa14b7cf0)
Matching: ffffa14b7cf0 __GI___inet_pton+0x0 (/usr/lib/aarch64-linux-gnu/libc-2.31.so)
Pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$
Matching: ping 285058 [025] 1219802.466939: probe_libc:inet_pton: (ffffa14b7cf0)
Matching: ffffa14b7cf0 __GI___inet_pton+0x0 (/usr/lib/aarch64-linux-gnu/libc-2.31.so)
Matching: ffffa1488040 getaddrinfo+0xe8 (/usr/lib/aarch64-linux-gnu/libc-2.31.so)
Matching: aaaab8672da4 [unknown] (/usr/bin/ping)
---- end ----
82: probe libc's inet_pton & backtrace it with ping : Ok
Closes: https://lore.kernel.org/linux-perf-users/1728978807-81116-1-git-send-email-renyu.zj@linux.alibaba.com/
Closes: https://lore.kernel.org/linux-perf-users/Z0X3AYUWkAgfPpWj@x1/T/#m57327e135b156047e37d214a0d453af6ae1e02be
Reported-by: Guilherme Amadio <amadio@gentoo.org>
Reported-by: Jing Zhang <renyu.zj@linux.alibaba.com>
Reviewed-by: James Clark <james.clark@linaro.org>
Signed-off-by: Leo Yan <leo.yan@arm.com>
Tested-by: Thomas Richter <tmricht@linux.ibm.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20241202111958.553403-1-leo.yan@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When kernel is built without debuginfo, running 'perf record' with
--off-cpu results in segfault as below:
./perf record --off-cpu -e dummy sleep 1
libbpf: kernel BTF is missing at '/sys/kernel/btf/vmlinux', was CONFIG_DEBUG_INFO_BTF enabled?
libbpf: failed to find '.BTF' ELF section in /lib/modules/6.13.0-rc3+/build/vmlinux
libbpf: failed to find valid kernel BTF
Segmentation fault (core dumped)
The backtrace pointed to:
#0 0x00000000100fb17c in btf.type_cnt ()
#1 0x00000000100fc1a8 in btf_find_by_name_kind ()
#2 0x00000000100fc38c in btf.find_by_name_kind ()
#3 0x00000000102ee3ac in off_cpu_prepare ()
#4 0x000000001002f78c in cmd_record ()
#5 0x00000000100aee78 in run_builtin ()
#6 0x00000000100af3e4 in handle_internal_command ()
#7 0x000000001001004c in main ()
Code sequence is:
static void check_sched_switch_args(void)
{
struct btf *btf = btf__load_vmlinux_btf();
const struct btf_type *t1, *t2, *t3;
u32 type_id;
type_id = btf__find_by_name_kind(btf, "btf_trace_sched_switch",
BTF_KIND_TYPEDEF);
btf__load_vmlinux_btf() fails when CONFIG_DEBUG_INFO_BTF is not enabled.
Here bpf__find_by_name_kind() calls btf__type_cnt() with NULL btf value
and results in segfault.
To fix this, add a check to see if btf is not NULL before invoking
bpf__find_by_name_kind().
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Disha Goel <disgoel@linux.vnet.ibm.com>
Cc: Hari Bathini <hbathini@linux.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Link: https://lore.kernel.org/r/20241223135813.8175-1-atrajeev@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perftool-testsuite_probe fails in test_adding_kernel as below:
Regexp not found: "probe:inode_permission_11"
-- [ FAIL ] -- perf_probe :: test_adding_kernel :: force-adding probes ::
second probe adding (with force) (output regexp parsing)
event syntax error: 'probe:inode_permission_11'
\___ unknown tracepoint
Error: File /sys/kernel/tracing//events/probe/inode_permission_11
not found.
Hint: Perhaps this kernel misses some CONFIG_ setting to
enable this feature?.
The test does the following:
1) Adds a probe point first using:
$CMD_PERF probe --add $TEST_PROBE
2) Then tries to add same probe again without —force and expects it to
fail. Next tries to add same probe again with —force. In this case,
perf probe succeeds and adds the probe with a suffix number. Example:
./perf probe --add inode_permission
Added new event:
probe:inode_permission (on inode_permission)
./perf probe --add inode_permission --force
Added new event:
probe:inode_permission_1 (on inode_permission)
./perf probe --add inode_permission --force
Added new event:
probe:inode_permission_2 (on inode_permission)
Each time, suffix is added to existing probe name.
To get the suffix number, test cases uses:
NO_OF_PROBES=`$CMD_PERF probe -l | wc -l`
This will work if there is no other probe existing in the system. If
there are any other probes other than kernel probes or inode_permission,
( example: any probe), "perf probe -l" will include count for other
probes too.
Example, in the system where this failed, already some probes were
default added. So count became 10
./perf probe -l | wc -l
10
So to be specific for "inode_permission", restrict the probe count check
to that probe point alone using:
NO_OF_PROBES=`$CMD_PERF probe -l $TEST_PROBE| wc -l`
Similarly while removing the probe using "probe --del *", (removing all
probes), check uses:
../common/check_all_lines_matched.pl "Removed event: probe:$TEST_PROBE"
But if there are other probes in the system, the log will contain
reference to other existing probe too. Hence change usage of
check_all_lines_matched.pl to check_all_patterns_found.pl This will make
sure expecting string comes in the result
Signed-off-by: Athira Rajeev <atrajeev@linux.vnet.ibm.com>
Acked-by: Veronika Molnarova <vmolnaro@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Disha Goel <disgoel@linux.vnet.ibm.com>
Cc: Hari Bathini <hbathini@linux.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Cc: Michael Petlan <mpetlan@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/r/20250110094324.94604-1-atrajeev@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Needed to build tools/lib/bpf/ on various arches other than x86_64,
notably arm64 when using the perf tarballs generated by:
$ make help | grep perf-
perf-tar-src-pkg - Build the perf source tarball with no compression
perf-targz-src-pkg - Build the perf source tarball with gzip compression
perf-tarbz2-src-pkg - Build the perf source tarball with bz2 compression
perf-tarxz-src-pkg - Build the perf source tarball with xz compression
perf-tarzst-src-pkg - Build the perf source tarball with zst compression
$
Building with BPF support was opt-in in perf for a long time, and
testing it via the tarball main kernel Makefile targets in an
architecture other than x86_64 was an odd case.
I had noticed this at some point earlier this year while cross building
perf to some arches, including arm64, but it fell thru the cracks, see
the Link tag below.
Fix it now by adding those arch/*/include/uapi/asm/bpf_perf_event.h
files to the MANIFEST file used in building the perf source tarball.
Tested with:
perfbuilder@number:~$ time dm debian:experimental-x-arm64
1 21.60 debian:experimental-x-arm64 : Ok aarch64-linux-gnu-gcc (Debian 14.1.0-5) 14.1.0 flex 2.6.4
BUILD_TARBALL_HEAD=d31a974f6edc576f84c35be9526fec549a3b3520
$
$ git log --oneline -1 d31a974f6edc576f84c35be9526fec549a3b3520
d31a974f6edc576f (HEAD -> perf-tools-next) perf MANIFEST: Add arch/*/include/uapi/asm/bpf_perf_event.h to the perf tarball
$
That was previously failing:
perfbuilder@number:~$ grep debian:experimental-x-arm64 dm.log.old/summary
19 4.80 debian:experimental-x-arm64 : FAIL gcc version 14.1.0 (Debian 14.1.0-5)
$
perfbuilder@number:~$ grep -B6 'Error 1' dm.log.old/debian:experimental-x-arm64
In file included from /git/perf-6.12.0-rc6/tools/include/uapi/linux/bpf_perf_event.h:11,
from libbpf.c:36:
/git/perf-6.12.0-rc6/tools/include/uapi/asm/bpf_perf_event.h:2:10: fatal error: ../../arch/arm64/include/uapi/asm/bpf_perf_event.h: No such file or directory
2 | #include "../../arch/arm64/include/uapi/asm/bpf_perf_event.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [/git/perf-6.12.0-rc6/tools/build/Makefile.build:105: /tmp/build/perf/libbpf/staticobjs/libbpf.o] Error 1
perfbuilder@number:~$
Closes: https://lore.kernel.org/all/Z0UNRCRYKunbDYxP@hyperscale.parallels
Fixes: 9eea8fafe3 ("libbpf: fix __arg_ctx type enforcement for perf_event programs")
Reported-by: Michel Lind <michel@michel-slm.name>
Tested-by: Michel Lind <michel@michel-slm.name>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: 317c11923cf676437456e44a7f408d4ce589a9c0.camel@michel-slm.name
Link: https://lore.kernel.org/bpf/ZfyEgoG3JFiOs2Fs@x1/
Link: https://lore.kernel.org/r/Z0Yy5u42Q1hWoEzz@x1
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In machine__create_module(), it reads /proc/modules to get a list of
modules in the system. The file shows the start address (of text) and
the size of the module so it uses the info to reconstruct system memory
maps for symbol resolution.
But module memory consists of multiple segments and they can be
scaterred. Currently perf tools assume they are contiguous and see some
overlaps. This can confuse the tool when it finds a map containing a
given address.
As we mostly care about the function symbols in the text segment, it can
fixup the size or end address of modules when there's an overlap. We
can use maps__fixup_end() which updates the end address using the start
address of the next map.
Ideally it should be able to track other segments (like data/rodata),
but that would require some changes in /proc/modules IMHO.
Reported-by: Blake Jones <blakejones@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Daniel Gomez <da.gomez@samsung.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Luis Chamberlain <mcgrof@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Petr Pavlu <petr.pavlu@suse.com>
Cc: Sami Tolvanen <samitolvanen@google.com>
Link: https://lore.kernel.org/r/20241218220453.203069-1-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When there are more than one symbols at the same address, it needs to
choose which one is better. In choose_best_symbol() it didn't check the
type of symbols. It's possible to have labels in other symbols and in
that case, it would be better to pick the actual symbol over the labels.
To minimize the possible impact on other symbols, I only check NOTYPE
symbols specifically.
$ readelf -sW vmlinux | grep -e __do_softirq -e __softirqentry_text_start
105089: ffffffff82000000 814 FUNC GLOBAL DEFAULT 1 __do_softirq
111954: ffffffff82000000 0 NOTYPE GLOBAL DEFAULT 1 __softirqentry_text_start
The commit 77b004f4c5 tried to do the same by not giving the size
to the label symbols but it seems there's some label-only symbols in asm
code. Let's restore the original code and choose the right symbol using
type of the symbols.
Fixes: 77b004f4c5 ("perf symbol: Do not fixup end address of labels")
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: Ian Rogers <irogers@google.com>
Cc: James Clark <james.clark@linaro.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Link: http://lore.kernel.org/lkml/Z3b-DqBMnNb4ucEm@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Some version of compilers reported unaligned accesses in perf trace when
undefined-behavior sanitizer is on. I found that it uses raw data in
the sample directly and assuming it's properly aligned.
Unlike other sample fields, the raw data is not 8-byte aligned because
there's a size field (u32) before the actual data. So I added a static
buffer in syscall__augmented_args() and return it instead. This is not
ideal but should work well as perf trace is single-threaded.
A better approach would be aligning the raw data by adding a 4-byte data
before the augmented args but I'm afraid it'd break the backward
compatibility.
Committer testing:
To build with the undefined behaviour sanitizer:
$ make CC=clang EXTRA_CFLAGS=-fsanitize=undefined -C tools/perf
Checking if the resulting binary is instrumented:
root@number:~# nm ~/bin/perf | grep ubsan | wc -l
113
root@number:~# nm ~/bin/perf | grep ubsan | tail -5
000000000043d5b0 t _ZN7__ubsanL19UBsanOnDeadlySignalEiPvS0_
000000000043ce50 T _ZNK7__ubsan5Value12getSIntValueEv
000000000043cf40 T _ZNK7__ubsan5Value12getUIntValueEv
000000000043d140 T _ZNK7__ubsan5Value13getFloatValueEv
000000000043cfd0 T _ZNK7__ubsan5Value19getPositiveIntValueEv
root@number:~#
Now running something that will access timespec, as reported in the
Closes URL:
root@number:~# perf trace --max-events=1 -e *nano* sleep 1.1
trace/beauty/timespec.c:10:64: runtime error: member access within misaligned address 0x7fc583cfb2a4 for type 'struct augmented_arg', which requires 8 byte alignment
0x7fc583cfb2a4: note: pointer points here
99 99 11 00 10 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 01 e1 f5 05 00 00 00 00 00 00 00 00
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior trace/beauty/timespec.c:10:64
<SNIP>
As Namhyung said we need to make the raw_data to be 64-bit aligned,
probably we need to add a PERF_SAMPLE_ALIGNED_RAW with a 64-bit raw_size
instead of the current u32 done at kernel/events/core.c,
perf_output_sample(), that perf_output_put(handle, raw->size) where
raw->size is an u32 and then the raw_data is always 64-bit unaligned...
After the patch:
root@number:~# perf trace -e *nano* sleep 1.1
0.000 (1100.064 ms): sleep/1984224 clock_nanosleep(rqtp: { .tv_sec: 1, .tv_nsec: 100000001 }, rmtp: 0x7fff5b3fe970) = 0
root@number:~#
Closes: https://lore.kernel.org/r/Z2STgyD1p456Qqhg@google.com
Reviewed-by: Howard Chu <howardchu95@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20250102201248.790841-1-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
