That field is only updated when we use the "sched:sched_stat_runtime"
tracepoint, and that is only done so far when we use the '--stat' command line
option, without it we get just zeros, confusing the users:
Without this patch:
# trace -a -s sleep 1
<SNIP>
qemu-system-x86 (9931), 468 events, 9.6%, 0.000 msec
syscall calls total min avg max stddev
(msec) (msec) (msec) (msec) (%)
---------- ------ --------- --------- --------- --------- ------
ppoll 98 982.374 0.000 10.024 29.983 12.65%
write 34 0.401 0.005 0.012 0.027 5.49%
ioctl 102 0.347 0.002 0.003 0.007 3.08%
firefox (10871), 1856 events, 38.2%, 0.000 msec
(msec) (msec) (msec) (msec) (%)
---------- ------ --------- --------- --------- --------- ------
poll 395 934.873 0.000 2.367 17.120 11.51%
recvmsg 395 0.988 0.001 0.003 0.021 4.20%
read 106 0.460 0.002 0.004 0.007 3.17%
futex 24 0.108 0.001 0.004 0.010 10.05%
mmap 2 0.041 0.016 0.021 0.026 23.92%
write 6 0.027 0.004 0.004 0.005 2.52%
After this patch that ', 0.000 msecs' gets suppressed when --stat is not
in use.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-p7emqrsw7900tdkg43v9l1e1@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Sometimes we want to sort an existing rbtree by a different key,
introduce a template for that, that needs only to be provided the
rbtree root and the number of entries in it.
To do that a new rbtree will be created with extra space for each entry,
where possibly pre-calculated keys will be stored to be used in the
resort process and also later, when using the newly sorted rbtree.
Please check the following two changesets to see it in use for resorting
stats for threads and its syscalls in 'perf trace --summary'.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-9l6e1q34lmf3wwdeewstyakg@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Without this patch, the last output doesn't have timestamp appended if
--timestamp-filename is not explicitly provided. For example:
# perf record -a --switch-output &
[1] 11224
# kill -s SIGUSR2 11224
[ perf record: dump data: Woken up 1 times ]
# [ perf record: Dump perf.data.2015122622372823 ]
# fg
perf record -a --switch-output
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.027 MB perf.data (540 samples) ]
# ls -l
total 836
-rw------- 1 root root 33256 Dec 26 22:37 perf.data <---- *Odd*
-rw------- 1 root root 817156 Dec 26 22:37 perf.data.2015122622372823
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461178794-40467-5-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
[ Updated man page, that also got an entry for --timestamp-filename ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When the kernel allows tweaking perf_event_max_stack and the event being
setup has PERF_SAMPLE_CALLCHAIN in its perf_event_attr.sample_type, tell
the user that tweaking /proc/sys/kernel/perf_event_max_stack may solve
the problem.
Before:
# echo 32000 > /proc/sys/kernel/perf_event_max_stack
# perf record -g usleep 1
Error:
The sys_perf_event_open() syscall returned with 12 (Cannot allocate memory) for event (cycles:ppp).
/bin/dmesg may provide additional information.
No CONFIG_PERF_EVENTS=y kernel support configured?
#
After:
# echo 64000 > /proc/sys/kernel/perf_event_max_stack
# perf record -g usleep 1
Error:
Not enough memory to setup event with callchain.
Hint: Try tweaking /proc/sys/kernel/perf_event_max_stack
Hint: Current value: 64000
#
Suggested-by: David Ahern <dsahern@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-ebv0orelj1s1ye857vhb82ov@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf is not able to register probe in kernel module when dwarf supprt
is not there(and so it goes for symtab). Perf passes full path of
module where only module name is required which is causing the problem.
This patch fixes this issue.
Before applying patch:
$ dpkg -s libdw-dev
dpkg-query: package 'libdw-dev' is not installed and no information is...
$ sudo ./perf probe -m /linux/samples/kprobes/kprobe_example.ko kprobe_init
Added new event:
probe:kprobe_init (on kprobe_init in /linux/samples/kprobes/kprobe_example.ko)
You can now use it in all perf tools, such as:
perf record -e probe:kprobe_init -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/kprobe_init /linux/samples/kprobes/kprobe_example.ko:kprobe_init
$ sudo ./perf record -a -e probe:kprobe_init
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.105 MB perf.data ]
$ sudo ./perf script # No output here
After applying patch:
$ sudo ./perf probe -m /linux/samples/kprobes/kprobe_example.ko kprobe_init
Added new event:
probe:kprobe_init (on kprobe_init in kprobe_example)
You can now use it in all perf tools, such as:
perf record -e probe:kprobe_init -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/kprobe_init kprobe_example:kprobe_init
$ sudo ./perf record -a -e probe:kprobe_init
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.105 MB perf.data (2 samples) ]
$ sudo ./perf script
insmod 13990 [002] 5961.216833: probe:kprobe_init: ...
insmod 13995 [002] 5962.889384: probe:kprobe_init: ...
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1461680741-12517-1-git-send-email-ravi.bangoria@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf can add a probe on kernel module which has not been loaded yet.
The current implementation finds the module name from path. But if the
filename is different from the actual module name then perf fails to
register a probe while loading module because of mismatch in the names.
For example, samples/kobject/kobject-example.ko is loaded as
kobject_example.
Before applying patch:
$ sudo ./perf probe -m /linux/samples/kobject/kobject-example.ko foo_show
Added new event:
probe:foo_show (on foo_show in kobject-example)
You can now use it in all perf tools, such as:
perf record -e probe:foo_show -aR sleep 1
$ cat /sys/kernel/debug/tracing/kprobe_events
p:probe/foo_show kobject-example:foo_show
$ insmod kobject-example.ko
$ lsmod
Module Size Used by
kobject_example 16384 0
Generate read to /sys/kernel/kobject_example/foo while recording data
with below command
$ sudo ./perf record -e probe:foo_show -a
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.093 MB perf.data ]
$./perf report --stdio -F overhead,comm,dso,sym
Error:
The perf.data.old file has no samples!
After applying patch:
$ sudo ./perf probe -m /linux/samples/kobject/kobject-example.ko foo_show
Added new event:
probe:foo_show (on foo_show in kobject_example)
You can now use it in all perf tools, such as:
perf record -e probe:foo_show -aR sleep 1
$ sudo cat /sys/kernel/debug/tracing/kprobe_events
p:probe/foo_show kobject_example:foo_show
$ insmod kobject-example.ko
$ lsmod
Module Size Used by
kobject_example 16384 0
Generate read to /sys/kernel/kobject_example/foo while recording data
with below command
$ sudo ./perf record -e probe:foo_show -a
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.097 MB perf.data (8 samples) ]
$ sudo ./perf report --stdio -F overhead,comm,dso,sym
...
# Samples: 8 of event 'probe:foo_show'
# Event count (approx.): 8
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ............
#
100.00% cat [kobject_example] [k] foo_show
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1461680741-12517-2-git-send-email-ravi.bangoria@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We get notifications for threads that gets created while we're tracing,
but for preexisting threads we may end not having synthesized them, like
when tracing a 'perf trace' session that will use '--pid' to trace some
other thread.
And besides we should probably stop synthesizing those records and
instead read thread information in a lazy way, i.e. just when we need,
like done in this patch:
Now the 'pid_t' argument in 'perf_event_open' gets translated to a COMM:
# perf trace -e perf_event_open perf stat -e cycles -p 31601
0.027 ( 0.027 ms): perf/23393 perf_event_open(attr_uptr: 0x2fdd0d8, pid: 31601 (abrt-dump-journ), cpu: -1, group_fd: -1, flags: FD_CLOEXEC)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= 3
^C
And in other syscalls containing pid_t without thread->comm_set at the
time of the formatting.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Milian Wolff <milian.wolff@kdab.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-ioeps6dlwst17d6oozc9shtk@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Don't read broken data after 'head' pointer.
Following commits will feed perf_evlist__mmap_read() with some 'head'
pointers not maintained by kernel. If 'head' pointer breaks an event, we
should avoid reading from the broken event. This can happen in backward
ring buffer.
For example:
old head
| |
V V
+---+------+----------+----+-----+--+
|..E|D....D|C........C|B..B|A....|E.|
+---+------+----------+----+-----+--+
'old' pointer points to the beginning of 'A' and trying read from it,
but 'A' has been overwritten. In this case, don't try to read from 'A',
simply return NULL.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1461637738-62722-2-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Given that the 'val' parameter is ignored for FUTEX_LOCK_PI, get rid of
the bogus deadlock detection flag in the wrapper code and avoid the
extra argument, making it resemble its unlock counterpart. And if
nothing else, we already only pass 0 anyway.
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Cc: Davidlohr Bueso <dbueso@suse.de>
Link: http://lkml.kernel.org/r/1461208447-29328-1-git-send-email-dave@stgolabs.net
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This fixes a bug caused by an unitialized callchain cursor. The crash
frist appeared in:
6f736735e3 ("perf evsel: Require that callchains be resolved before
calling fprintf_{sym,callchain}")
The callchain cursor is a struct that contains pointers, that when
uninitialized will cause unpredictable behavior (usually a crash)
when trying to append to the callchain.
The existing implementation has the following issues:
1. The callchain cursor used is not initialized, resulting in
unpredictable behavior when used.
2. The cursor is declared on the stack. Even if it is properly initalized,
the implmentation will leak memory when the function returns,
since all the references to the callchain_nodes allocated by
callchain_cursor_append will be lost when the cursor goes out of
scope.
3. Storing the cursor on the stack is inefficient. Even if memory is
properly freed when it goes out of scope, a performance penalty
will be incurred due to reallocation of callchain nodes.
callchain_cursor_append is designed to avoid these reallocations
when an existing cursor is reused.
This patch fixes the crash by replacing cursor_callchain with a reference
to the global callchain_cursor which also resolves all 3 issues mentioned
above.
How to reproduce the crash:
$ perf record --call-graph=dwarf stress -t 1 -c 1
$ perf script > /dev/null
Segfault
Signed-off-by: Chris Phlipot <cphlipot0@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Fixes: 6f736735e3 ("perf evsel: Require that callchains be resolved before calling fprintf_{sym,callchain}")
Link: http://lkml.kernel.org/r/1461119531-2529-1-git-send-email-cphlipot0@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
write_buildid() increments 'name_len' with intention to take into
account trailing zero byte. However, 'name_len' was already incremented
in machine__write_buildid_table() before. So this leads to
out-of-bounds read in do_write():
$ ./perf record sleep 0
[ perf record: Woken up 1 times to write data ]
=================================================================
==15899==ERROR: AddressSanitizer: global-buffer-overflow on address 0x00000099fc92 at pc 0x7f1aa9c7eab5 bp 0x7fff940f84d0 sp 0x7fff940f7c78
READ of size 19 at 0x00000099fc92 thread T0
#0 0x7f1aa9c7eab4 (/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/libasan.so.2+0x44ab4)
#1 0x649c5b in do_write util/header.c:67
#2 0x649c5b in write_padded util/header.c:82
#3 0x57e8bc in write_buildid util/build-id.c:239
#4 0x57e8bc in machine__write_buildid_table util/build-id.c:278
...
0x00000099fc92 is located 0 bytes to the right of global variable '*.LC99' defined in 'util/symbol.c' (0x99fc80) of size 18
'*.LC99' is ascii string '[kernel.kallsyms]'
...
Shadow bytes around the buggy address:
0x00008012bf80: f9 f9 f9 f9 00 00 00 00 00 00 03 f9 f9 f9 f9 f9
=>0x00008012bf90: 00 00[02]f9 f9 f9 f9 f9 00 00 00 00 00 05 f9 f9
0x00008012bfa0: f9 f9 f9 f9 00 03 f9 f9 f9 f9 f9 f9 00 00 00 00
Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/1461053847-5633-1-git-send-email-aryabinin@virtuozzo.com
[ Remove the off-by one at the origin, to keep len(s) == strlen(s) assumption ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo:
Build fixes:
- Fix 'perf trace' build when DWARF unwind isn't available (Arnaldo Carvalho de Melo)
- Remove x86 references from arch-neutral Build, fixing it in !x86 arches,
reported as breaking the build for powerpc64le in linux-next (Arnaldo Carvalho de Melo)
Infrastructure changes:
- Do memset() variable 'st' using the correct size in the jit code (Colin Ian King)
- Fix postgresql ubuntu 'perf script' install instructions (Chris Phlipot)
- Use callchain_param more thoroughly when checking how callchains were
configured, eventually will be the only way to look for callchain parameters
(Arnaldo Carvalho de Melo)
- Fix some issues in the 'perf test kallsyms' entry (Arnaldo Carvalho de Melo)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Add sample_reg_mask array with pt_regs registers.
This is needed for printing supported regs ( -I? option).
Signed-off-by: Madhavan Srinivasan <maddy@linux.vnet.ibm.com>
Acked-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
One of the branches leading to an error had no debug message emitted,
fix it, the new lines are:
# perf test -v kallsyms
<SNIP>
0xffffffff81001000: diff name v: xen_hypercall_set_trap_table k: hypercall_page
0xffffffff810691f0: diff name v: try_to_free_pud_page k: try_to_free_pmd_page
<SNIP>
0xffffffff8150bb20: diff name v: wakeup_expire_count_show.part.5 k: wakeup_active_count_show.part.7
0xffffffff816bc7f0: diff name v: phys_switch_id_show.part.11 k: phys_port_name_show.part.12
0xffffffff817bbb90: diff name v: __do_softirq k: __softirqentry_text_start
<SNIP>
This in turn exercises another bug, still under investigation, because those
aliases _are_ in kallsyms, with the same name...
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Fixes: ab414dcda8 ("perf test: Fixup aliases checking in the 'vmlinux matches kallsyms' test")
Link: http://lkml.kernel.org/n/tip-5fhea7a54a54gsmagu9obpr4@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Before:
# perf test -v kallsyms
<SNIP>
Maps only in vmlinux:
ffffffff81d5e000-ffffffff81ec3ac8 115e000 [kernel].init.text
ffffffff81ec3ac8-ffffffffa0000000 12c3ac8 [kernel].exit.text
ffffffffa0000000-ffffffffa000c000 0 [fjes]
ffffffffa000c000-ffffffffa0017000 0 [video]
ffffffffa0017000-ffffffffa001c000 0 [grace]
<SNIP>
ffffffffa0a7f000-ffffffffa0ba5000 0 [xfs]
ffffffffa0ba5000-ffffffffffffffff 0 [veth]
Maps in vmlinux with a different name in kallsyms:
Maps only in kallsyms:
ffff880000100000-ffff88001000b000 80000103000 [kernel.kallsyms]
ffff88001000b000-ffff880100000000 8001000e000 [kernel.kallsyms]
ffff880100000000-ffffc90000000000 80100003000 [kernel.kallsyms]
<SNIP>
ffffffffa0000000-ffffffffff600000 7fffa0003000 [kernel.kallsyms]
ffffffffff600000-ffffffffffffffff 7fffff603000 [kernel.kallsyms]
test child finished with -1
---- end ----
vmlinux symtab matches kallsyms: FAILED!
#
After:
# perf test -v 1
1: vmlinux symtab matches kallsyms :
--- start ---
test child forked, pid 7058
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.6.0-rc1+/build/vmlinux for symbols
0xffffffff81076870: diff end addr for aesni_gcm_dec v: 0xffffffff810791f2 k: 0xffffffff81076902
0xffffffff81079200: diff end addr for aesni_gcm_enc v: 0xffffffff8107bb03 k: 0xffffffff81079292
0xffffffff8107e8d0: diff end addr for aesni_gcm_enc_avx_gen2 v: 0xffffffff81083e76 k: 0xffffffff8107e943
0xffffffff81083e80: diff end addr for aesni_gcm_dec_avx_gen2 v: 0xffffffff81089611 k: 0xffffffff81083ef3
0xffffffff81089990: diff end addr for aesni_gcm_enc_avx_gen4 v: 0xffffffff8108e7c4 k: 0xffffffff81089a03
0xffffffff8108e7d0: diff end addr for aesni_gcm_dec_avx_gen4 v: 0xffffffff810937ef k: 0xffffffff8108e843
Maps only in vmlinux:
ffffffff81d5e000-ffffffff81ec3ac8 115e000 [kernel].init.text
ffffffff81ec3ac8-ffffffffa0000000 12c3ac8 [kernel].exit.text
Maps in vmlinux with a different name in kallsyms:
Maps only in kallsyms:
test child finished with -1
---- end ----
vmlinux symtab matches kallsyms: FAILED!
#
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Fixes: 8e0cf965f9 ("perf symbols: Add support for reading from /proc/kcore")
Link: http://lkml.kernel.org/n/tip-n6vrwt9t89w8k769y349govx@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
