forked from iovisor/bcc
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
3083c85
commit 38cef48
Showing
10 changed files
with
2,100 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
.TH offcputime 8 "2016-01-14" "USER COMMANDS" | ||
.SH NAME | ||
offcputime \- Summarize off-CPU time by kernel stack trace. Uses Linux eBPF/bcc. | ||
.SH SYNOPSIS | ||
.B offcputime [\-h] [\-p PID] [\-i INTERVAL] [\-T] [duration] | ||
.SH DESCRIPTION | ||
This program shows kernel stack traces and task names that were blocked and | ||
"off-CPU", and the total duration they were blocked: their "off-CPU time". | ||
It works by tracing when threads block and when they return to CPU, measuring | ||
both the time they were blocked and the blocked kernel stack trace and the | ||
task name. This data is summarized in the kernel using an eBPF map, and by | ||
summing the blocked time by unique stack trace and task name. | ||
|
||
The output summary will help you identify reasons why threads | ||
were blocking, and quantify the time they were blocked. This spans all types | ||
of blocking activity: disk I/O, network I/O, locks, page faults, involuntary | ||
context switches, etc. | ||
|
||
This is complementary to CPU profiling (e.g., CPU flame graphs) which shows | ||
the time spent on-CPU. This shows the time spent off-CPU, and the output, | ||
especially the -f format, can be used to generate an "off-CPU time flame graph". | ||
|
||
See https://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html | ||
|
||
The stack depth is currently limited to 20, and the stack traces are kernel | ||
mode only. Check for newer versions where either may be improved. | ||
|
||
This currently only works on x86_64. Check for future versions. | ||
.SH REQUIREMENTS | ||
CONFIG_BPF and bcc. | ||
.SH OPTIONS | ||
.TP | ||
\-h | ||
Print usage message. | ||
.TP | ||
\-v | ||
Show raw addresses. | ||
.TP | ||
\-p PID | ||
Trace this process ID only (filtered in-kernel). | ||
.TP | ||
duration | ||
Duration to trace, in seconds. | ||
.SH EXAMPLES | ||
.TP | ||
Trace all thread blocking events, and summarize (in-kernel) by kernel stack | ||
trace and total off-CPU time: | ||
# | ||
.B offcputime | ||
.TP | ||
Trace for 5 seconds only: | ||
# | ||
.B offcputime 5 | ||
.TP | ||
Trace for 5 seconds, and emit output in folded stack format (suitable for | ||
flame graphs): | ||
# | ||
.B offcputime -f 5 | ||
.TP | ||
Trace PID 185 only: | ||
# | ||
.B offcputime -p 185 | ||
.SH OVERHEAD | ||
This summarizes unique stack traces in-kernel for efficiency, allowing it to | ||
trace a higher rate of events than methods that post-process in user space. The | ||
stack trace and time data is only copied to user space once, when the output is | ||
printed. While these techniques greatly lower overhead, scheduler events are | ||
still a high frequency event, as they can exceed 1 million events per second, | ||
and so caution should still be used. Test before production use. | ||
|
||
If the overhead is still a problem, take a look at the MINBLOCK_US tunable in | ||
the code. If your aim is to chase down longer blocking events, then this could | ||
be increased to filter shorter blocking events, further lowering overhead. | ||
.SH SOURCE | ||
This is from bcc. | ||
.IP | ||
https://github.com/iovisor/bcc | ||
.PP | ||
Also look in the bcc distribution for a companion _examples.txt file containing | ||
example usage, output, and commentary for this tool. | ||
.SH OS | ||
Linux | ||
.SH STABILITY | ||
Unstable - in development. | ||
.SH AUTHOR | ||
Brendan Gregg | ||
.SH SEE ALSO | ||
stackcount(8) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
.TH stackcount 8 "2016-01-14" "USER COMMANDS" | ||
.SH NAME | ||
stackcount \- Count kernel function calls and their stack traces. Uses Linux eBPF/bcc. | ||
.SH SYNOPSIS | ||
.B stackcount [\-h] [\-p PID] [\-i INTERVAL] [\-T] [\-r] pattern | ||
.SH DESCRIPTION | ||
stackcount traces kernel functions and frequency counts them with their entire | ||
kernel stack trace, summarized in-kernel for efficiency. This allows higher | ||
frequency events to be studied. The output consists of unique stack traces, | ||
and their occurance counts. | ||
|
||
The pattern is a string with optional '*' wildcards, similar to file globbing. | ||
If you'd prefer to use regular expressions, use the \-r option. | ||
|
||
The stack depth is currently limited to 10 (+1 for the current instruction | ||
pointer). | ||
|
||
This currently only works on x86_64. Check for future versions. | ||
.SH REQUIREMENTS | ||
CONFIG_BPF and bcc. | ||
.SH OPTIONS | ||
.TP | ||
\-h | ||
Print usage message. | ||
.TP | ||
\-r | ||
Allow regular expressions for the search pattern. The default allows "*" | ||
wildcards only. | ||
.TP | ||
\-s | ||
Show address offsets. | ||
.TP | ||
\-T | ||
Include a timestamp with interval output. | ||
.TP | ||
\-v | ||
Show raw addresses. | ||
.TP | ||
\-i interval | ||
Summary interval, in seconds. | ||
.TP | ||
\-p PID | ||
Trace this process ID only (filtered in-kernel). | ||
.TP | ||
pattern | ||
A kernel function name, or a search pattern. Can include wildcards ("*"). If the | ||
\-r option is used, can include regular expressions. | ||
.SH EXAMPLES | ||
.TP | ||
Count kernel stack traces for submit_bio(): | ||
# | ||
.B stackcount submit_bio | ||
.TP | ||
Count kernel stack traces for ip_output(): | ||
# | ||
.B stackcount ip_output | ||
.TP | ||
Show symbol offsets: | ||
# | ||
.B stackcount -s ip_output | ||
.TP | ||
Show offsets and raw addresses (verbose): | ||
# | ||
.B stackcount -sv ip_output | ||
.TP | ||
Count kernel stacks for kernel functions matching tcp_send*: | ||
# | ||
.B stackcount 'tcp_send*' | ||
.TP | ||
Same as previous, but using regular expressions: | ||
# | ||
.B stackcount -r '^tcp_send.*' | ||
.TP | ||
Output every 5 seconds, with timestamps: | ||
# | ||
.B stackcount -Ti 5 ip_output | ||
.TP | ||
Only count stacks when PID 185 is on-CPU: | ||
# | ||
.B stackcount -p 185 ip_output | ||
.SH OVERHEAD | ||
This summarizes unique stack traces in-kernel for efficiency, allowing it to | ||
trace a higher rate of function calls than methods that post-process in user | ||
space. The stack trace data is only copied to user space when the output is | ||
printed, which usually only happens once. Given these techniques, I'd suspect | ||
that call rates of < 10,000/sec would incur negligible overhead (for this | ||
current version; future versions may improve this). Beyond that, | ||
there will be a point where the overhead is measurable, as this does add | ||
a number of instructions to each function call to walk and save stacks. | ||
Test before production use. You can also use funccount to get a handle on | ||
function call rates first. | ||
.SH SOURCE | ||
This is from bcc. | ||
.IP | ||
https://github.com/iovisor/bcc | ||
.PP | ||
Also look in the bcc distribution for a companion _examples.txt file containing | ||
example usage, output, and commentary for this tool. | ||
.SH OS | ||
Linux | ||
.SH STABILITY | ||
Unstable - in development. | ||
.SH AUTHOR | ||
Brendan Gregg | ||
.SH SEE ALSO | ||
stacksnoop(8), funccount(8) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
.TH stacksnoop 8 "2016-01-14" "USER COMMANDS" | ||
.SH NAME | ||
stacksnoop \- Print kernel stack traces for kernel functions. Uses Linux eBPF/bcc. | ||
.SH SYNOPSIS | ||
.B stacksnoop [\-h] [\-p PID] [\-s] [\-v] function | ||
.SH DESCRIPTION | ||
stacksnoop traces a given kernel function and for each call, prints the | ||
kernel stack back trace for that call. This shows the ancestry of function | ||
calls, and is a quick way to investigate low frequency kernel functions and | ||
their cause. For high frequency kernel functions, see stackcount. | ||
|
||
The stack depth is currently limited to 10 (+1 for the current instruction | ||
pointer). | ||
|
||
This currently only works on x86_64. Check for future versions. | ||
.SH REQUIREMENTS | ||
CONFIG_BPF and bcc. | ||
.SH OPTIONS | ||
.TP | ||
\-h | ||
Print usage message. | ||
.TP | ||
\-s | ||
Show address offsets. | ||
.TP | ||
\-v | ||
Print more fields. | ||
.TP | ||
\-p PID | ||
Trace this process ID only (filtered in-kernel). | ||
.TP | ||
function | ||
Kernel function name. | ||
.SH EXAMPLES | ||
.TP | ||
Print kernel stack traces for each call to ext4_sync_fs: | ||
# | ||
.B stacksnoop ext4_sync_fs | ||
.TP | ||
Also show the symbol offsets: | ||
# | ||
.B stacksnoop -s ext4_sync_fs | ||
.TP | ||
Show extra columns: | ||
# | ||
.B stacksnoop -v ext4_sync_fs | ||
.TP | ||
Only trace when PID 185 is on-CPU: | ||
# | ||
.B stacksnoop -p 185 ext4_sync_fs | ||
.SH FIELDS | ||
.TP | ||
TIME(s) | ||
Time of the call, in seconds. | ||
.TP | ||
STACK | ||
Kernel stack trace. The first column shows "ip" for instruction pointer, and | ||
"r#" for each return pointer in the stack. The second column is the stack trace | ||
as hexidecimal. The third column is the translated kernel symbol names. | ||
.SH OVERHEAD | ||
This can have significant overhead if frequently called functions (> 1000/s) are | ||
traced, and is only intended for low frequency function calls. This is because | ||
details including the stack trace for every call is passed to user space and | ||
processed. See stackcount for higher frequency calls, which performs in-kernel | ||
summaries. | ||
.SH SOURCE | ||
This is from bcc. | ||
.IP | ||
https://github.com/iovisor/bcc | ||
.PP | ||
Also look in the bcc distribution for a companion _examples.txt file containing | ||
example usage, output, and commentary for this tool. | ||
.SH OS | ||
Linux | ||
.SH STABILITY | ||
Unstable - in development. | ||
.SH AUTHOR | ||
Brendan Gregg | ||
.SH SEE ALSO | ||
stackcount(8) |
Oops, something went wrong.