Fix a bug in del_perf_probe_events() which returns an error (-ENOENT)
even if the probes are successfully deleted.
This happens only if the probes are on user-apps and not on kernel,
simply because it doesn't clear the previous error.
So, without this fix, we get an error even though events are being
successfully removed.
------
# ./perf probe -x ./perf del_perf_probe_events
Added new event:
probe_perf:del_perf_probe_events (on del_perf_probe_events in ...
You can now use it in all perf tools, such as:
perf record -e probe_perf:del_perf_probe_events -aR sleep 1
# ./perf probe -d \*:\*
Removed event: probe_perf:del_perf_probe_events
Error: Failed to delete events.
------
This fixes the above error.
------
# ./perf probe -d \*:\*
Removed event: probe_perf:del_perf_probe_events
------
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Richard Weinberger <richard@nod.at>
Link: http://lkml.kernel.org/r/20150527083725.23880.45209.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Show the reason of error when dso__load* fails. This shows when user
gives wrong kernel image or wrong path.
Without this, perf probe shows an obscure message:
----
$ perf probe -k ~/kbin/linux-3.x86_64/vmlinux -L vfs_read
Failed to find path of kernel module.
Error: Failed to show lines.
----
With this, perf shows appropriate error message:
----
$ perf probe -k ~/kbin/linux-3.x86_64/vmlinux -L vfs_read
Failed to find the path for kernel: Mismatching build id
Error: Failed to show lines.
----
And:
----
$ perf probe -k /non-exist/kernel/vmlinux -L vfs_read
Failed to find the path for kernel: No such file or directory
Error: Failed to show lines.
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Richard Weinberger <richard@nod.at>
Link: http://lkml.kernel.org/r/20150527083718.23880.84100.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Disallow PMU events intel_pt and intel_bts until the tools support them.
By default any PMU is selectable as an event but until the tools have
intel_pt and intel_bts support using them would result in no data being
recorded without any indication as to why.
Before the change:
$ perf record -e intel_bts// sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.008 MB perf.data ]
$ perf report --stdio
Error:
The perf.data file has no samples!
After the change:
$ perf record -e intel_bts// sleep 1
invalid or unsupported event: 'intel_bts//'
Run 'perf list' for a list of valid events
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1432295653-13989-2-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
I was assuming rb_erase() was setting things up like list_del_init, but
the fact that thread__delete() was being sucessfull is because the last
thing before deleting is to remove the thread from the
machine->dead_threads list, using list_del_init(), that has the same
effect as using rb_erase_init()...
Introduce this function so that we can use it when removing objects from
rb_trees.
Then we will be able to BUG_ON(still on a list) in destructors.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-55b16mbtndjyd7zzg8nmnamx@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf_session__peek_event() generally leverages there being a single mmap
of the perf.data file, however on 32-bit platforms when there is more
that 32MiB of data, then there are multiple mmaps, so
perf_session__peek_event() reads from the file.
In that case a couple of bugs were exposed (note how the seg. fault
appears with >32M of data):
$ perf record --per-thread -e intel_bts// ../rtit-tests/loopy 1000000
[ perf record: Woken up 13 times to write data ]
[ perf record: Captured and wrote 24.568 MB perf.data ]
$ perf script > /dev/null
$ perf record --per-thread -e intel_bts// ../rtit-tests/loopy 10000000
[ perf record: Woken up 136 times to write data ]
[ perf record: Captured and wrote 270.794 MB perf.data ]
$ perf script > /dev/null
Segmentation fault (core dumped)
The wrong address was being passed to the readn() function and the
buffer size was not being checked.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Link: http://lkml.kernel.org/r/1432040746-1755-5-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Patch "perf tools: Add location to pmu event terms" moved declarations
for parse_events_term__num() and parse_events_term__str() so that they
were no longer visible in parse-events.y. That can result in segfaults
as the arguments no longer need match the function prototype.
Move the declarations back, changing YYLTYPE pointers to
pointers-to-void because YYLTYPE is not generated until parse-events.y
is processed.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Link: http://lkml.kernel.org/r/1432040746-1755-2-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Commit 4c85935122 ("perf probe: Support
glob wildcards for function name") introduces a problem:
# /root/perf probe kmem_cache_free
Failed to find symbol kmem_cache_free in kernel
Error: Failed to add events.
The reason is the replacement of map__for_each_symbol_by_name() (by
map__for_each_symbol()). Although their names are similar,
map__for_each_symbol doesn't call map__load() and dso__sort_by_name()
before searching. The missing of map__load() causes this problem because
it search symbol before load dso map.
This patch ensures map__load() is called before using
map__for_each_symbol().
After this patch:
# /root/perf probe kmem_cache_free
Added new event:
probe:kmem_cache_free (on kmem_cache_free%return)
You can now use it in all perf tools, such as:
perf record -e probe:kmem_cache_free -aR sleep 1
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1431692084-46287-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Currently the se_cmp and se_collapse use pointer comparison,
which is ok for for testing equality of strings. It's not ok
as comparing function for rbtree insertion, because it gives
different results based on current pointer values.
We saw test 32 (hists cumulation test) failing based on different
environment setup. Having all sort functions straightened fix the
test for us.
Reported-by: Jan Stancek <jstancek@redhat.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jan Stancek <jstancek@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
There's a bug that perf report sometimes ignore some options on --stdio
output. This bug is triggered only if a related config variable is set.
For example, let's assume we have a following config file.
$ cat ~/.perfconfig
[call-graph]
print-type = graph
[hist]
percentage = absolute
Then, following perf config will not honor some options.
$ perf record -ag sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.199 MB perf.data (77 samples) ]
$ perf report -g none --stdio
# To display the perf.data header info, please use --header/--header-only options.
#
# Samples: 77 of event 'cycles'
# Event count (approx.): 25425383
#
# Overhead Command Shared Object Symbol
# ........ ............... ....................... ..............
#
16.34% swapper [kernel.vmlinux] [k] intel_idle
|
---intel_idle
cpuidle_enter_state
cpuidle_enter
cpu_startup_entry
...
With '-g none' option, it should not show callchains, but it still shows
callchains. However it works as expected on --tui output.
Similarly, '--percentage relative' option is not work and still shows a
absolute percentage values.
Looking at the source, I found that those setting were overwritten by
config variables when setup_pager() called. The setup_pager() is to
start a pager process so that it can manage long lines of output on the
stdio mode. But as it calls the perf_config() after parsing arguments,
the settings were overwritten regardless of command line options.
The reason it calls perf_config() is to find the 'pager_program' which
might be set by a config variable, I guess. However current perf code
does not provide the config variable for it, so it's just meaningless
IMHO. Eliminating the call makes the option working as expected.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Taeung Song <treeze.taeung@gmail.com>
Link: http://lkml.kernel.org/r/1431529406-6762-1-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
perf probe currently errors out if there are any tail calls to probed
functions:
[root@rhel71be]# perf probe do_fork
Failed to find probe point in any functions.
Error: Failed to add events.
Fix this by teaching perf to ignore tail calls.
Without patch:
[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Failed to find probe point in any functions.
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code: -2)
With patch:
[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Ignoring tail call from SyS_clone
Found 4 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
No kprobe blacklist support, ignored
Added new events:
Writing event: p:probe/do_fork _text+768432
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
[Ignore the error about failure to write event - this kernel is missing
a patch to resolve _text properly]
The reason to ignore tail calls is that the address does not belong to
any function frame. In the example above, the address in SyS_clone is
0xc0000000000bc27c, but looking at the debug-info:
<1><830081>: Abbrev Number: 133 (DW_TAG_subprogram)
<830083> DW_AT_external : 1
<830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone
<830087> DW_AT_decl_file : 7
<830088> DW_AT_decl_line : 1689
<83008a> DW_AT_prototyped : 1
<83008a> DW_AT_type : <0x8110eb>
<83008e> DW_AT_low_pc : 0xc0000000000bc270
<830096> DW_AT_high_pc : 0xc
<83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa)
<8300a0> DW_AT_GNU_all_call_sites: 1
<8300a0> DW_AT_sibling : <0x830178>
<snip>
<3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site)
<830148> DW_AT_low_pc : 0xc0000000000bc27c
<830150> DW_AT_GNU_tail_call: 1
<830150> DW_AT_abstract_origin: <0x82e7e1>
The frame ends at 0xc0000000000bc27c. I suppose this is why this
particular call is a "tail" call. FWIW, systemtap seems to ignore these
as well and requires users to explicitly place probes at these call
sites if necessary. I print out the caller so that users know.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Link: http://lkml.kernel.org/r/1430394151-15928-1-git-send-email-naveen.n.rao@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The name "ftrace" really refers to the function hook infrastructure. It
is not about the trace_events. Rename the max trace_event type size to
something more descriptive and appropriate.
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Indicate to check variable location range in error message when we got
failed to find the variable.
Before this patch:
$ perf probe --add 'generic_perform_write+118 bytes'
Failed to find the location of bytes at this address.
Perhaps, it has been optimized out.
Error: Failed to add events.
After this patch:
$ perf probe --add 'generic_perform_write+118 bytes'
Failed to find the location of the 'bytes' variable at this address.
Perhaps it has been optimized out.
Use -V with the --range option to show 'bytes' location range.
Error: Failed to add events.
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1431336304-16863-3-git-send-email-hekuang@huawei.com
[ Improve the error message based on lkml thread ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It is not easy for users to get the accurate byte offset or the line
number where a local variable can be probed.
With '--range' option, local variables in the scope of the probe point
are showed with a byte offset range, and can be added according to this
range information.
For example, there are some variables in the function
generic_perform_write():
<generic_perform_write@mm/filemap.c:0>
0 ssize_t generic_perform_write(struct file *file,
1 struct iov_iter *i, loff_t pos)
2 {
3 struct address_space *mapping = file->f_mapping;
4 const struct address_space_operations *a_ops = mapping->a_ops;
...
42 status = a_ops->write_begin(file, mapping, pos, bytes, flags,
&page, &fsdata);
44 if (unlikely(status < 0))
But we fail when we try to probe the variable 'a_ops' at line 42 or 44.
$ perf probe --add 'generic_perform_write:42 a_ops'
Failed to find the location of a_ops at this address.
Perhaps, it has been optimized out.
This is because the source code do not match the assembly, so a variable
may not be available in the source code line where it appears.
After this patch, we can lookup the accurate byte offset range of a
variable, 'INV' indicates that this variable is not valid at the given
point, but available in the scope:
$ perf probe --vars 'generic_perform_write:42' --range
Available variables at generic_perform_write:42
@<generic_perform_write+141>
[INV] ssize_t written @<generic_perform_write+[324-331]>
[INV] struct address_space_operations* a_ops @<generic_perform_write+[55-61,170-176,223-246]>
[VAL] (unknown_type) fsdata @<generic_perform_write+[70-307,346-411]>
[VAL] loff_t pos @<generic_perform_write+[0-286,286-336,346-411]>
[VAL] long int status @<generic_perform_write+[83-342,346-411]>
[VAL] long unsigned int bytes @<generic_perform_write+[122-311,320-338,346-403,403-411]>
[VAL] struct address_space* mapping @<generic_perform_write+[35-344,346-411]>
[VAL] struct iov_iter* i @<generic_perform_write+[0-340,346-411]>
[VAL] struct page* page @<generic_perform_write+[70-307,346-411]>
Then it is more clear for us to add a probe with this variable:
$ perf probe --add 'generic_perform_write+170 a_ops'
Added new event:
probe:generic_perform_write (on generic_perform_write+170 with a_ops)
Signed-off-by: He Kuang <hekuang@huawei.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1431336304-16863-2-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Support glob wildcards for function name when adding new probes. This
will allow us to build caches of function-entry level information with
$params.
e.g.
----
# perf probe --no-inlines --add 'kmalloc* $params'
Added new events:
probe:kmalloc_slab (on kmalloc* with $params)
probe:kmalloc_large_node (on kmalloc* with $params)
probe:kmalloc_order_trace (on kmalloc* with $params)
You can now use it in all perf tools, such as:
perf record -e probe:kmalloc_order_trace -aR sleep 1
# perf probe --list
probe:kmalloc_large_node (on kmalloc_large_node@mm/slub.c with size flags node)
probe:kmalloc_order_trace (on kmalloc_order_trace@mm/slub.c with size flags order)
probe:kmalloc_slab (on kmalloc_slab@mm/slab_common.c with size flags)
----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150508010335.24812.19972.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In addition to using refcounts for the struct thread lifetime
management, we need to protect access to machine->threads from
concurrent access.
That happens in 'perf top', where a thread processes events, inserting
and deleting entries from that rb_tree while another thread decays
hist_entries, that end up dropping references and ultimately deleting
threads from the rb_tree and releasing its resources when no further
hist_entry (or other data structures, like in 'perf sched') references
it.
So the rule is the same for refcounts + protected trees in the kernel,
get the tree lock, find object, bump the refcount, drop the tree lock,
return, use object, drop the refcount if no more use of it is needed,
keep it if storing it in some other data structure, drop when releasing
that data structure.
I.e. pair "t = machine__find(new)_thread()" with a "thread__put(t)", and
"perf_event__preprocess_sample(&al)" with "addr_location__put(&al)".
The addr_location__put() one is because as we return references to
several data structures, we may end up adding more reference counting
for the other data structures and then we'll drop it at
addr_location__put() time.
Acked-by: David Ahern <dsahern@gmail.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-bs9rt4n0jw3hi9f3zxyy3xln@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
