stackcount: Support uprobes, tracepoints, and USDT #730
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for user-space functions in `stackcount` by taking an additional `-l` command-line parameter specifying the name of the user-space library. When a user-space library is specified, `stackcount` attaches to a specific process and traces a user-space function with user-space stacks only. Regex support for uprobes (similar to what is available for kprobes) is not currently provided. Also add a couple of functions to the `BPF` object for consistency.
attach_kprobe allows a regular expression for the function name, while attach_uprobe does not. Add support in libccc for enumerating all the function symbols in a binary, and use that in the BPF module to attach uprobes according to a regular expression. For example: ```python bpf = BPF(text="...") bpf.attach_uprobe(name="c", sym_re=".*write$", fn_name="probe") ```
Modify attach_tracepoint to take a regex argument, in which case it enumerates all tracepoints matching that regex and attaches to all of them. The logic for enumerating tracepoints should eventually belong in libccc and be shared across all the tools (tplist, trace and so on).
bcc_elf would not terminate the enumeration correctly when the user-provided callback returned -1 but there were still more sections remaining in the ELF to be enumerated.
Refactored stackcount and added support for uprobes and tracepoints, which also required changes to the BPF module. USDT support still pending.
Refactor symbol listing from paging style to foreach-style with a callback function per-symbol. Even though we're now performing a callback from C to Python for each symbol, this is preferable to the paging approach because we need all the symbols in the current use case. Also refactored `stackcount` slightly; only missing support for USDT probes now.
For user-space functions, or when requested for kernel-space functions or tracepoints, group the output by process. Toggled with the -P switch, off by default (except for user-space).
Now, stackcount supports USDT tracepoints in addition to kernel functions, user functions, and kernel tracepoints. The format is the same as with the other general-purpose tools (argdist, trace): ``` stackcount -p $(pidof node) u:node:gc* stackcount -p 185 u:pthread:pthread_create ```
Add examples and man page documentation for kernel tracepoints, USDT tracepoints, and other features.
When -p is specified, don't print the comm and pid. Also, when -P is specified for kernel probes (kprobes and tracepoints), use -1 for symbol resolution so that we don't try to resolve kernel functions as user symbols. Finally, print the comm and pid at the end of the stack output and not at the beginning.
|
buildbot, test this please |
|
buildbot, retest this please |
|
LGTM. @brendangregg pls review |
|
Tested, works, is awesome. A minor note: I don't like the ordering of user stacks & PIDs: I'd prefer having the "bash [22487]" line last, not first. That way, you're reading from smallest to biggest context: what's on-CPU, its parent, and so on, until thread_start, then the parent process ID. Seems more logical that way. For folded output (stackcount is missing a -f, should add it sometime), it follows a similar order, just reversed: you have comm-PID;thread_start;...;on-CPU_func, and then when you make a flame graph you have the rectangles for each process at the bottom, and towers on top. Also, since I matched a PID with -p, I'm not sure I'd include the "bash [22487]" line in the output anyway. Redundant. I'd do that if it's a user probe without the -p. I don't feel strongly about this bit. ... Update: ok, I just broke something. I'd run my ftrace uprobe tool to check something (double check why I was seeing no pthread_create's), and then stackcount didn't work. Weird. How did I manage that? ... I did another "make install" of bcc, and it fixed it. It's reproducable: Is this just a separate issue? |
|
@brendangregg I just tested this with your |
|
@brendangregg Latest commit addresses your comments and also fixes a bug with |
|
@goldshtn thanks, this patchset LGTM. My other issue is still a mystery: libbcc.so.0 gets overwritten when I run my perf-tools uprobe program. But I'm assuming at this point it's unrelated to this patchset. Ok, using bcc to debug bcc (again): (hm, that execsnoop output should strip \n's, and probably truncate to sum limit unless a -w wide option is provided.) Ok, so it reproduces with just "/sbin/ldconfig.real -v". Oh, duh: It's redoing the symlink. I think this is only something I'd see with my dev system. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
stackcountnow supports uprobes, tracepoints, and USDT probes (using the bcc/BPF native tracepoint and USDT support). Examples and man page updated. Also fixed a minor bug in symbol enumeration (bcc).Resolves #580.
A couple of highlights to try: