The 'annotate' tool does some filtering in the entries in a DSO but
forgot to reset the cache done in dso__find_symbol(), cauxing a SEGV:
[root@zoo ~]# perf annotate netlink_poll
perf: Segmentation fault
-------- backtrace --------
perf[0x526ceb]
/lib64/libc.so.6(+0x34960)[0x7faedfbe0960]
perf(rb_erase+0x223)[0x499d63]
perf[0x4213e9]
perf[0x4bc123]
perf[0x4bc621]
perf[0x4bf26b]
perf[0x4bc855]
perf(perf_session__process_events+0x340)[0x4bddc0]
perf(cmd_annotate+0x6bb)[0x421b5b]
perf[0x479063]
perf(main+0x60a)[0x42098a]
/lib64/libc.so.6(__libc_start_main+0xf0)[0x7faedfbcbfe0]
perf[0x420aa9]
[0x0]
[root@zoo ~]#
Fix it by reseting the find cache when removing symbols.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Fixes: b685ac22b4 ("perf symbols: Add front end cache for DSO symbol lookup")
Link: http://lkml.kernel.org/n/tip-b2y9x46y0t8yem1ive41zqyp@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
A problem can occur in a statically linked perf when vmlinux can be found:
# perf probe --add sys_epoll_pwait
probe-definition(0): sys_epoll_pwait
symbol:sys_epoll_pwait file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.2.0-rc1+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.2.0-rc1+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_pwait address found : ffffffff8122bd40
Matched function: SyS_epoll_pwait
Failed to get call frame on 0xffffffff8122bd40
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code: -2)
The reason is caused by libdw that, if libdw is statically linked, it
can't load libebl_{arch}.so reliable.
In this case it is still possible to get the address from
/proc/kalksyms. However, perf tries that only when libdw returns
-EBADF.
This patch gives it another chance to utilize symbol table, even if
libdw returns an error code other than -EBADF.
After applying this patch:
# perf probe -nv --add sys_epoll_pwait
probe-definition(0): sys_epoll_pwait
symbol:sys_epoll_pwait file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.2.0-rc1+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.2.0-rc1+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_pwait address found : ffffffff8122bd40
Matched function: SyS_epoll_pwait
Failed to get call frame on 0xffffffff8122bd40
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: p:probe/sys_epoll_pwait _text+2276672
probe:sys_epoll_pwait (on sys_epoll_pwait)
You can now use it in all perf tools, such as:
perf record -e probe:sys_epoll_pwait -aR sleep 1
Although libdw returns an error (Failed to get call frame), perf tries
symbol table and finally gets correct address.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Alexei Starovoitov <ast@plumgrid.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Ahern <dsahern@gmail.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kaixu Xia <xiakaixu@huawei.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1440151770-129878-2-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Events that don't sample the timestamp have a timestamp value of -1.
Intel PT processing wasn't taking that into account.
This is particularly noticeable with Intel BTS because timestamps are
not requested by default.
Then, if the conversion of -1 to TSC results in a small number, the
processing is unaffected.
However if the conversion results in a big number, then the data is
processed prematurely before relevant sideband data like mmap events,
which in turn results in samples with unknown dsos.
Commiter note:
Since BTS wasn't upstream, I split the patch to fold the BTS part with
the patch introducing it, to avoid having this bug in the commit
history. PT was already upstream, so this patch contains that part.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1440060692-5585-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The "/proc/kcore requires CAP_SYS_RAWIO" message comes up all the time
for 'perf script' if vmlinux is not found and the user isn't root, even
when the kernel is not being traced and even though the message is only
really relevant for annotation.
Change it to pr_debug and instead put a note in the message displayed if
annotation is not possible.
Also, the file being accessed might not be /proc/kcore. Tools can be
directed to a different location using the --kallsyms option in which
case kcore is expected to be in the same directory. Adjust the message
so it is not misleading in that case.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Li Zhang <zhlcindy@linux.vnet.ibm.com>
Cc: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/r/1440065260-8802-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Patch "perf script: Don't assume evsel position of tracking events"
changed 'perf script' to use 'perf_evlist__id2evsel()'. That results
in a segfault if there is more than 1 event and there are
synthesized mmap events e.g.
$ perf record -e cycles,instructions -p$$ sleep 1
$ perf script --show-mmap-events
Segmentation fault (core dumped)
That happens because these synthesized events have an 'id' of zero
which does not match any 'evsel'.
Currently, these synthesized events use the sample type of the first
evsel.
Change 'perf_evlist__id2evsel()' to reflect that which also makes
it consistent with 'perf_evlist__event2evsel()'.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Fixes: 06b234ec26 ("perf script: Don't assume evsel position of tracking events")
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1440059205-1765-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When processing a fork event, the tools lookup the parent thread by its
tid. In a couple of cases, it is possible for that thread to have the
wrong pid.
That can happen if the data is being processed out of order, or if the
(fork) event that would have removed the erroneous thread was lost.
Assume the latter case, print a dump message, remove the erroneous
thread, create a new one with the correct pid, and keep going.
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: http://lkml.kernel.org/r/1439994561-27436-3-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for Intel Processor Trace.
Intel PT support fits within the new auxtrace infrastructure. Recording
is supporting by identifying the Intel PT PMU, parsing options and
setting up events.
Decoding is supported by queuing up trace data by cpu or thread and then
decoding synchronously delivering synthesized event samples into the
session processing for tools to consume.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-7-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for decoding an Intel Processor Trace.
Intel PT trace data must be 'decoded' which involves walking the object
code and matching the trace data packets.
The decoder requests a buffer of binary data via a get_trace()
call-back, which it decodes using instruction information which it gets
via another call-back walk_insn().
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-6-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for decoding instructions for Intel Processor Trace. The
kernel x86 instruction decoder is copied for this.
This essentially provides intel_pt_get_insn() which takes a binary
buffer, uses the kernel's x86 instruction decoder to get details of the
instruction and then categorizes it for consumption by an Intel PT
decoder.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1439450095-30122-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix the following 32-bit compilation errors:
util/annotate.c: In function ‘addr_map_symbol__account_cycles’:
util/annotate.c:643:3: error: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘u64’ [-Werror=format=]
pr_debug2("BB with bad start: addr %lx start %lx sym %lx saddr %lx\n",
^
util/annotate.c:643:3: error: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 5 has type ‘u64’ [-Werror=format=]
util/annotate.c:643:3: error: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 6 has type ‘u64’ [-Werror=format=]
These were introduced by the patch:
"perf report: Add infrastructure for a cycles histogram"
Also change the 'saddr' variable from 'unsigned long' to 'u64'
noting that theoretically we could be processing data captured
on a 64-bit machine but processing it on a 32-bit machine.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Fixes: d4957633bf ("perf report: Add infrastructure for a cycles histogram")
Link: http://lkml.kernel.org/r/1439536294-18241-1-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
"perf probe --lines sys_poll" shows only the first line of sys_poll,
because the SYSCALL_DEFINE macro:
----
SYSCALL_DEFINE*(foo,...)
{
body;
}
----
is expanded as below (on debuginfo)
----
static inline int SYSC_foo(...)
{
body;
}
int SyS_foo(...) <- is an alias of sys_foo.
{
return SYSC_foo(...);
}
----
So, "perf probe --lines sys_foo" decodes SyS_foo function and it also skips
inlined functions(SYSC_foo) inside the target function because those functions
are usually defined somewhere else.
To fix this issue, this fix checks whether the inlined function is defined at
the same point of the target function, and if so, it doesn't skip the inline
function.
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/20150812012406.11811.94691.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Currently perf evlist -F shows the number as if it's always sampling
frequency. But we now support per-event freq/period settings. So it'd
better to show more detailed info whether it's freq or period.
$ perf record -e 'cpu/config=1/,cpu/config=2,period=300000/' sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB perf.data ]
$ perf evlist -F
cpu/config=1/: sample_freq=4000
cpu/config=2,period=300000/: sample_period=300000
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1439102724-14079-2-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In some cases it's useful to characterize samples by file. This is
useful to get a higher level categorization, for example to map cost to
subsystems.
Add a srcfile sort key to perf report. It builds on top of the existing
srcline support.
Commiter notes:
E.g.:
# perf record -F 10000 usleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.016 MB perf.data (13 samples) ]
[root@zoo ~]# perf report -s srcfile --stdio
# Total Lost Samples: 0
#
# Samples: 13 of event 'cycles'
# Event count (approx.): 869878
#
# Overhead Source File
# ........ ...........
60.99% .
20.62% paravirt.h
14.23% rmap.c
4.04% signal.c
0.11% msr.h
#
The first line is collecting all the files for which srcfiles couldn't somehow
get resolved to:
# perf report -s srcfile,dso --stdio
# Total Lost Samples: 0
#
# Samples: 13 of event 'cycles'
# Event count (approx.): 869878
#
# Overhead Source File Shared Object
# ........ ........... ................
40.97% . ld-2.20.so
20.62% paravirt.h [kernel.vmlinux]
20.02% . libc-2.20.so
14.23% rmap.c [kernel.vmlinux]
4.04% signal.c [kernel.vmlinux]
0.11% msr.h [kernel.vmlinux]
#
XXX: Investigate why that is not resolving on Fedora 21, Andi says he hasn't
seen this on Fedora 22.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/1438988064-21834-1-git-send-email-andi@firstfloor.org
[ Added column length update, from 0e65bdb3f90f ('perf hists: Update the column width for the "srcline" sort key') ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When we introduce a new sort key, we need to update the
hists__calc_col_len() function accordingly, otherwise the width
will be limited to strlen(header).
We can't update it when obtaining a line value for a column (for
instance, in sort__srcline_cmp()), because we reset it all when doing a
resort (see hists__output_recalc_col_len()), so we need to, from what is
in the hist_entry fields, set each of the column widths.
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Fixes: 409a8be615 ("perf tools: Add sort by src line/number")
Link: http://lkml.kernel.org/n/tip-jgbe0yx8v1gs89cslr93pvz2@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
For perf report/script srcline currently only the base file name of the
source file is printed. This is a good default because it usually fits
on the screen.
But in some cases we want to know the full file name, for example to
aggregate hits per file.
In the later case we need more than the base file name to resolve file
naming collisions: for example the kernel source has ~70 files named
"core.c"
It's also useful as input to post processing tools which want to point
to the right file.
Add a flag to allow full file name output.
Add an option to perf report/script to enable this option.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/1438986245-15191-1-git-send-email-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This is the core patch for supporting eBPF on-the-fly compiling, does
the following work:
1. Search clang compiler using search_program().
2. Run command template defined in llvm-bpf-cmd-template option in
[llvm] config section using read_from_pipe(). Patch of clang and
source code path is injected into shell command using environment
variable using force_set_env().
Commiter notice:
When building with DEBUG=1 we get a compiler error that gets fixed with
the same approach described in commit b236512280:
perf kmem: Fix compiler warning about may be accessing uninitialized variable
The last argument to strtok_r doesn't need to be initialized, its
just a placeholder to make this routine reentrant, but gcc doesn't know
about that and complains, breaking the build, fix it by setting it to
NULL.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Alexei Starovoitov <ast@plumgrid.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Ahern <dsahern@gmail.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kaixu Xia <xiakaixu@huawei.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/n/1436445342-1402-14-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch introduces [llvm] config section with 5 options. Following
patches will use then to config llvm dynamica compiling.
'llvm-utils.[ch]' is introduced in this patch for holding all
llvm/clang related stuffs.
Example:
[llvm]
# Path to clang. If omit, search it from $PATH.
clang-path = "/path/to/clang"
# Cmdline template. Following line shows its default value.
# Environment variable is used to passing options.
#
# *NOTE*: -D__KERNEL__ MUST appears before $CLANG_OPTIONS,
# so user have a chance to use -U__KERNEL__ in $CLANG_OPTIONS
# to cancel it.
clang-bpf-cmd-template = "$CLANG_EXEC -D__KERNEL__ $CLANG_OPTIONS \
$KERNEL_INC_OPTIONS -Wno-unused-value \
-Wno-pointer-sign -working-directory \
$WORKING_DIR -c $CLANG_SOURCE -target \
bpf -O2 -o -"
# Options passed to clang, will be passed to cmdline by
# $CLANG_OPTIONS.
clang-opt = "-Wno-unused-value -Wno-pointer-sign"
# kbuild directory. If not set, use /lib/modules/`uname -r`/build.
# If set to "" deliberately, skip kernel header auto-detector.
kbuild-dir = "/path/to/kernel/build"
# Options passed to 'make' when detecting kernel header options.
kbuild-opts = "ARCH=x86_64"
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Alexei Starovoitov <ast@plumgrid.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Ahern <dsahern@gmail.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kaixu Xia <xiakaixu@huawei.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1437477214-149684-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Currently the value of a PMU config term is silently truncated if it is
too big. This is an impediment to validating the value for other
criteria later on i.e. the user provides an invalid value that gets
truncated to a valid one.
The maximum value validation is only done for the parser where the error
is passed back to the user. In other cases the silent truncation
continues so as not to affect tools that perhaps rely on it.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1437150840-31811-16-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Compute the IPC and the basic block cycles for the annotate display.
IPC is computed by counting the instructions, and then dividing the
accounted cycles by that count.
The actual IPC computation can only be done at annotate time, because we
need to parse the objdump output first to know the number of
instructions in the basic block.
The cycles/IPC are also put into the perf function annotation so that
the display code can show them.
Again basic block overlaps are not handled, with the longest winning,
but there are some heuristics to hide the IPC when the longest is not
the most common.
v2: Compute IPC correctly.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: http://lkml.kernel.org/r/1437233094-12844-6-git-send-email-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
