Support @BUILDID or @FILE suffix for SDT events. This allows perf to add
probes on SDTs/pre-cached events on given FILE or the file which has
given BUILDID (also, this complements BUILDID.)
For example, both gcc and libstdc++ has same SDTs as below. If you
would like to add a probe on sdt_libstdcxx:catch on gcc, you can do as
below.
----
# perf list sdt | tail -n 6
sdt_libstdcxx:catch@/usr/bin/gcc(0cc207fc4b27) [SDT event]
sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
sdt_libstdcxx:rethrow@/usr/bin/gcc(0cc207fc4b27) [SDT event]
sdt_libstdcxx:rethrow@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
sdt_libstdcxx:throw@/usr/bin/gcc(0cc207fc4b27) [SDT event]
sdt_libstdcxx:throw@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
# perf probe -a %sdt_libstdcxx:catch@0cc
Added new event:
sdt_libstdcxx:catch (on %catch in /usr/bin/gcc)
You can now use it in all perf tools, such as:
perf record -e sdt_libstdcxx:catch -aR sleep 1
----
Committer note:
Doing the full sequence of steps to get the results above:
With a clean build-id cache:
[root@jouet ~]# rm -rf ~/.debug/
[root@jouet ~]# perf list sdt
List of pre-defined events (to be used in -e):
[root@jouet ~]#
No events whatsoever, then, we can add all events in gcc to the build-id
cache, doing a --add + --dry-run:
[root@jouet ~]# perf probe --dry-run --cache -x /usr/bin/gcc --add %sdt_libstdcxx:\*
Added new events:
sdt_libstdcxx:throw (on %* in /usr/bin/gcc)
sdt_libstdcxx:rethrow (on %* in /usr/bin/gcc)
sdt_libstdcxx:catch (on %* in /usr/bin/gcc)
You can now use it in all perf tools, such as:
perf record -e sdt_libstdcxx:catch -aR sleep 1
[root@jouet ~]#
It really didn't add any events, it just cached them:
[root@jouet ~]# perf probe -l
[root@jouet ~]#
We can see that it was cached as:
[root@jouet ~]# ls -la ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/
total 976
drwxr-xr-x. 2 root root 4096 Jul 13 21:47 .
drwxr-xr-x. 3 root root 4096 Jul 13 21:47 ..
-rwxr-xr-x. 4 root root 985912 Jun 22 18:52 elf
-rw-r--r--. 1 root root 303 Jul 13 21:47 probes
[root@jouet ~]# file ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/elf
/root/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/elf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2, stripped
[root@jouet ~]# cat ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/probes
%sdt_libstdcxx:throw=throw
p:sdt_libstdcxx/throw /usr/bin/gcc:0x71ffd
%sdt_libstdcxx:rethrow=rethrow
p:sdt_libstdcxx/rethrow /usr/bin/gcc:0x720b8
%sdt_libstdcxx:catch=catch
p:sdt_libstdcxx/catch /usr/bin/gcc:0x7307f
%sdt_libgcc:unwind=unwind
p:sdt_libgcc/unwind /usr/bin/gcc:0x7eec0
#sdt_libstdcxx:*=%*
[root@jouet ~]#
Ok, now we can use 'perf probe' to refer to those cached entries as:
Humm, nope, doing as above we end up with:
[root@jouet ~]# perf probe -a %sdt_libstdcxx:catch
Semantic error :* is bad for event name -it must follow C symbol-naming rule.
Error: Failed to add events.
[root@jouet ~]#
But it worked at some point, lets try not using --dry-run:
Resetting everything:
# rm -rf ~/.debug/
# perf probe -d *:*
# perf probe -l
# perf list sdt
List of pre-defined events (to be used in -e):
#
Ok, now it cached everything, even things we haven't asked it to
(sdt_libgcc:unwind):
[root@jouet ~]# perf probe -x /usr/bin/gcc --add %sdt_libstdcxx:\*
Added new events:
sdt_libstdcxx:throw (on %* in /usr/bin/gcc)
sdt_libstdcxx:rethrow (on %* in /usr/bin/gcc)
sdt_libstdcxx:catch (on %* in /usr/bin/gcc)
You can now use it in all perf tools, such as:
perf record -e sdt_libstdcxx:catch -aR sleep 1
[root@jouet ~]# perf list sdt
List of pre-defined events (to be used in -e):
sdt_libgcc:unwind [SDT event]
sdt_libstdcxx:catch [SDT event]
sdt_libstdcxx:rethrow [SDT event]
sdt_libstdcxx:throw [SDT event]
[root@jouet ~]#
And we have the events in place:
[root@jouet ~]# perf probe -l
sdt_libstdcxx:catch (on execute_cfa_program+1551@../../../libgcc/unwind-dw2.c in /usr/bin/gcc)
sdt_libstdcxx:rethrow (on d_print_subexpr+280@libsupc++/cp-demangle.c in /usr/bin/gcc)
sdt_libstdcxx:throw (on d_print_subexpr+93@libsupc++/cp-demangle.c in /usr/bin/gcc)
[root@jouet ~]#
And trying to use them at least has 'perf trace --event sdt*:*' working.
Then, if we try to add the ones in libstdc++:
[root@jouet ~]# perf probe -x /usr/lib64/libstdc++.so.6 -a %sdt_libstdcxx:\*
Error: event "catch" already exists.
Hint: Remove existing event by 'perf probe -d'
or force duplicates by 'perf probe -f'
or set 'force=yes' in BPF source.
Error: Failed to add events.
[root@jouet ~]#
Doesn't work, dups, but at least this served to, unbeknownst to the user, add
the SDT probes in /usr/lib64/libstdc++.so.6!
[root@jouet ~]# perf list sdt
List of pre-defined events (to be used in -e):
sdt_libgcc:unwind [SDT event]
sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6) [SDT event]
sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
sdt_libstdcxx:rethrow@/usr/bin/gcc(9a0730e2bcc6) [SDT event]
sdt_libstdcxx:rethrow@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
sdt_libstdcxx:throw@/usr/bin/gcc(9a0730e2bcc6) [SDT event]
sdt_libstdcxx:throw@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
[root@jouet ~]#
Now we should be able to get to the original cset comment, if we remove all
SDTs events in place, not from the cache, from the kernel, where it was set up as:
[root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/sdt_libstdcxx/
total 0
drwxr-xr-x. 5 root root 0 Jul 13 22:00 .
drwxr-xr-x. 80 root root 0 Jul 13 21:56 ..
drwxr-xr-x. 2 root root 0 Jul 13 22:00 catch
-rw-r--r--. 1 root root 0 Jul 13 22:00 enable
-rw-r--r--. 1 root root 0 Jul 13 22:00 filter
drwxr-xr-x. 2 root root 0 Jul 13 22:00 rethrow
drwxr-xr-x. 2 root root 0 Jul 13 22:00 throw
[root@jouet ~]#
[root@jouet ~]# head -2 /sys/kernel/debug/tracing/events/sdt_libstdcxx/throw/format
name: throw
ID: 2059
[root@jouet ~]#
Now to remove it:
[root@jouet ~]# perf probe -d sdt_libstdc*:*
Removed event: sdt_libstdcxx:catch
Removed event: sdt_libstdcxx:rethrow
Removed event: sdt_libstdcxx:throw
[root@jouet ~]#
Which caused:
[root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/sdt_libstdcxx/
ls: cannot access '/sys/kernel/debug/tracing/events/sdt_libstdcxx/': No such file or directory
[root@jouet ~]#
Ok, now we can do:
[root@jouet ~]# perf list sdt_libstdcxx:catch
List of pre-defined events (to be used in -e):
sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6) [SDT event]
sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
[root@jouet ~]#
So, these are not really 'pre-defined events', i.e. we can't use them with
'perf record --event':
[root@jouet ~]# perf record --event sdt_libstdcxx:catch*
event syntax error: 'sdt_libstdcxx:catch*'
\___ unknown tracepoint
Error: File /sys/kernel/debug/tracing/events/sdt_libstdcxx/catch* not found.
Hint: Perhaps this kernel misses some CONFIG_ setting to enable this feature?.
<SNIP>
[root@jouet ~]#
To have it really pre-defined we must use perf probe to get its definition from
the cache and set it up in the kernel, creating the tracepoint to _then_ use it
with 'perf record --event':
[root@jouet ~]# perf probe -a sdt_libstdcxx:catch
Semantic error :There is non-digit char in line number.
<SNIP>
Oops, there is another gotcha here, we need that pesky '%' character:
[root@jouet ~]# perf probe -a %sdt_libstdcxx:catch
Added new events:
sdt_libstdcxx:catch (on %catch in /usr/bin/gcc)
sdt_libstdcxx:catch_1 (on %catch in /usr/lib64/libstdc++.so.6.0.22)
You can now use it in all perf tools, such as:
perf record -e sdt_libstdcxx:catch_1 -aR sleep 1
[root@jouet ~]#
But then we added _two_ events, one with the name we expected, the other one
with a _ added, when doing the analysis we need to pay attention to who maps to
who.
And here is where we get to the point of this patch, which is to be able to
disambiguate those definitions for 'catch' in the build-id cache, but first we need
remove those events we just added:
[root@jouet ~]# perf probe -d %sdt_libstdcxx:catch
Oops, that didn't remove anything, we need to _remove_ that % char in this case:
[root@jouet ~]# perf probe -d sdt_libstdcxx:catch
Removed event: sdt_libstdcxx:catch
And we need to remove the other event added, i.e. I forgot to add a * at the end:
[root@jouet ~]# perf probe -d sdt_libstdcxx:catch*
Removed event: sdt_libstdcxx:catch_1
[root@jouet ~]#
Ok, disambiguating it using what is in this patch:
[root@jouet ~]# perf list sdt_libstdcxx:catch
List of pre-defined events (to be used in -e):
sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6) [SDT event]
sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
[root@jouet ~]#
[root@jouet ~]# perf probe -a %sdt_libstdcxx:catch@9a07
Added new event:
sdt_libstdcxx:catch (on %catch in /usr/bin/gcc)
You can now use it in all perf tools, such as:
perf record -e sdt_libstdcxx:catch -aR sleep 1
[root@jouet ~]# perf probe -l
sdt_libstdcxx:catch (on execute_cfa_program+1551@../../../libgcc/unwind-dw2.c in /usr/bin/gcc)
[root@jouet ~]#
Yeah, it works! But we need to try and simplify this :-)
Update: Some aspects of this simplification take place in the following
patches.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/146831793746.17065.13065062753978236612.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Make "perf probe --cache --list" show only available cached events by
checking build-id validity.
E.g. without this patch:
----
$ ./perf probe --cache --add oldevent=cmd_probe
$ make #(to update ./perf)
$ ./perf probe --cache --add newevent=cmd_probe
$ ./perf probe --cache --list
/home/mhiramat/ksrc/linux/tools/perf/perf (061e90539bac69
probe_perf:newevent=cmd_probe
/home/mhiramat/ksrc/linux/tools/perf/perf (c2e44d614e33e1
probe_perf:oldevent=cmd_probe
----
It shows both of old and new events but user can not use old one.
With this;
----
$ ./perf probe --cache -l
/home/mhiramat/ksrc/linux/tools/perf/perf (061e90539bac69
probe_perf:newevent=cmd_probe
----
This shows only new events which are on the existing binary.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/146831789417.17065.17896487479879669610.stgit@devbox
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Before this patch, if a sample is triggered inside a module not in
/lib/modules/`uname -r`/, even if the module is in buildid-cache, 'perf
report' will still be unable to find the correct symbol. For example:
# rm -rf ~/.debug/
# perf buildid-cache -a ./mymodule.ko
# perf probe -m ./mymodule.ko -a get_mymodule_val
Added new event:
probe:get_mymodule_val (on get_mymodule_val in mymodule)
You can now use it in all perf tools, such as:
perf record -e probe:get_mymodule_val -aR sleep 1
# perf record -e probe:get_mymodule_val cat /proc/mymodule
mymodule:3
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.011 MB perf.data (1 samples) ]
# perf report --stdio
[SNIP]
#
# Overhead Command Shared Object Symbol
# ........ ....... ................ ......................
#
100.00% cat [mymodule] [k] 0x0000000000000001
# perf report -vvvv --stdio
dso__load_sym: adjusting symbol: st_value: 0 sh_addr: 0 sh_offset: 0x70
symbol__new: get_mymodule_val 0x70-0x8a
[SNIP]
This is caused by dso__load() -> dso__load_sym(). In dso__load(), kmod
is true only when its file is found in some well know directories. All
files loaded from buildid-cache are treated as user programs. Following
dso__load_sym() set map->pgoff incorrectly.
This patch gives kernel modules in buildid-cache a chance to adjust
value of kmod. After dso__load() get the type of symbols, if it is
buildid, check the last 3 chars of original filename against '.ko', and
adjust the value of kmod if the file is a kernel module.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Cody P Schafer <dev@codyps.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jeremie Galarneau <jeremie.galarneau@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kirill Smelkov <kirr@nexedi.com>
Cc: Li Zefan <lizefan@huawei.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1454680939-24963-3-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add --purge FILE to remove all caches of FILE.
Since the current --remove FILE removes a cache which has
same build-id of given FILE. Since the command takes a
FILE path, it can confuse user who tries to remove cache
about FILE path.
-----
# ./perf buildid-cache -v --add ./perf
Adding 133b7b5486d987a5ab5c3ebf4ea14941f45d4d4f ./perf: Ok
# (update the ./perf binary)
# ./perf buildid-cache -v --remove ./perf
Removing 305bbd1be68f66eca7e2d78db294653031edfa79 ./perf: FAIL
./perf wasn't in the cache
-----
Actually, the --remove's FAIL is not shown, it just silently fails.
So, this patch adds --purge FILE action for such usecase.
perf buildid-cache --purge FILE removes all caches which has same FILE
path.
In other words, it removes all caches including old binaries.
-----
# ./perf buildid-cache -v --add ./perf
Adding 133b7b5486d987a5ab5c3ebf4ea14941f45d4d4f ./perf: Ok
# (update the ./perf binary)
# ./perf buildid-cache -v --purge ./perf
Removing 133b7b5486d987a5ab5c3ebf4ea14941f45d4d4f ./perf: Ok
-----
BTW, if you want to purge all the caches, remove ~/.debug/* .
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20150227045026.1999.64084.stgit@localhost.localdomain
[ s/dirname/dir_name/g to fix build on fedora14, where dirname is a global ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add new buildid cache if the update target file is not cached.
This can happen when an old binary is replaced by new one after caching
the old one. In this case, user sees his operation just failed.
But it does not look straight, since user just pass the binary "path",
not "build-id".
----
# ./perf buildid-cache --add ./perf
(update ./perf to new binary)
# ./perf buildid-cache --update ./perf
./perf wasn't in the cache
#
----
This patch adds given new binary to cache if the new binary is
not cached. So we'll not see the above error.
----
# ./perf buildid-cache --add ./perf
(update ./perf to new binary)
# ./perf buildid-cache --update ./perf
#
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20150226065440.23912.1494.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We were still using the pathname found on the MMAP event, that could not
be the one we used when recording, so use the build-id cache for that,
only falling back to use the pathname in the MMAP event if no build-ids
are available.
With this we now also are able to do secure, seamless offline annotation.
Example:
[root@doppio linux-2.6-tip]# perf report -g none -v 2> /dev/null | head -10
8.12% Xorg /usr/lib64/libpixman-1.so.0.14.0 0x0000000000026d02 B [.] pixman_rasterize_edges
4.68% firefox /usr/lib64/xulrunner-1.9.1/libxul.so 0x00000000005dbdba B [.] 0x000000005dbdba
3.70% swapper /lib/modules/2.6.34-rc6/build/vmlinux 0xffffffff81022cea ! [k] read_hpet
2.96% init /lib/modules/2.6.34-rc6/build/vmlinux 0xffffffff81022cea ! [k] read_hpet
2.73% swapper /lib/modules/2.6.34-rc6/build/vmlinux 0xffffffff8100a738 ! [k] mwait_idle_with_hints
[root@doppio linux-2.6-tip]# perf annotate -v pixman_rasterize_edges 2>&1 | grep Executing
Executing: objdump --start-address=0x000000371ce26670 --stop-address=0x000000371ce2709f -dS /root/.debug/.build-id/bd/6ac5199137aaeb279f864717d8d061477466c1|grep -v /root/.debug/.build-id/bd/6ac5199137aaeb279f864717d8d061477466c1|expand
[root@doppio linux-2.6-tip]# perf buildid-list | grep libpixman-1.so.0.14.0
bd6ac5199137aaeb279f864717d8d061477466c1 /usr/lib64/libpixman-1.so.0.14.0
[root@doppio linux-2.6-tip]#
Reported-by: Stephane Eranian <eranian@google.com>
Cc: Frédéric Weisbecker <fweisbec@gmail.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Cc: Tom Zanussi <tzanussi@gmail.com>
LKML-Reference: <new-submission>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>