A miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the "perf" command). Both ftrace and perf are core Linux tracing tools, included in the kernel source. Your system probably has ftrace already, and perf is often just a package add (see Prerequisites).

These tools are designed to be easy to install (fewest dependencies), provide advanced performance observability, and be simple to use: do one thing and do it well. This collection was created by Brendan Gregg (author of the DTraceToolkit).

Many of these tools employ workarounds so that functionality is possible on existing Linux kernels. Because of this, many tools have caveats (see man pages), and their implementation should be considered a placeholder until future kernel features, or new tracing subsystems, are added.

These are intended for Linux 3.2 and newer kernels. For Linux 2.6.x, see Warnings.

These tools were introduced in the USENIX LISA 2014 presentation: Linux Performance Analysis: New Tools and Old Secrets

• slides: https://www.slideshare.net/brendangregg/linux-performance-analysis-new-tools-and-old-secrets
• video: https://www.usenix.org/conference/lisa14/conference-program/presentation/gregg

Using ftrace:

• iosnoop: trace disk I/O with details including latency. Examples.
• iolatency: summarize disk I/O latency as a histogram. Examples.
• execsnoop: trace process exec() with command line argument details. Examples.
• opensnoop: trace open() syscalls showing filenames. Examples.
• killsnoop: trace kill() signals showing process and signal details. Examples.
• fs/cachestat: basic cache hit/miss statistics for the Linux page cache. Examples.
• net/tcpretrans: show TCP retransmits, with address and other details. Examples.
• system/tpoint: trace a given tracepoint. Examples.
• kernel/funccount: count kernel function calls, matching a string with wildcards. Examples.
• kernel/functrace: trace kernel function calls, matching a string with wildcards. Examples.
• kernel/funcslower: trace kernel functions slower than a threshold. Examples.
• kernel/funcgraph: trace a graph of kernel function calls, showing children and times. Examples.
• kernel/kprobe: dynamically trace a kernel function call or its return, with variables. Examples.
• user/uprobe: dynamically trace a user-level function call or its return, with variables. Examples.
• tools/reset-ftrace: reset ftrace state if needed. Examples.

Using perf_events:

• misc/perf-stat-hist: power-of aggregations for tracepoint variables. Examples.
• syscount: count syscalls by syscall or process. Examples.
• disk/bitesize: histogram summary of disk I/O size. Examples.

Using eBPF:

• As a preview of things to come, see the bcc tracing Tools section. These use bcc, a front end for using eBPF. bcc+eBPF will allow some of these tools to be rewritten and improved, and additional tools to be created.

Showing new processes and arguments:

# ./execsnoop 
Tracing exec()s. Ctrl-C to end.
   PID   PPID ARGS
 22898  22004 man ls
 22905  22898 preconv -e UTF-8
 22908  22898 pager -s
 22907  22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8
 22906  22898 tbl
 22911  22910 locale charmap
 22912  22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n
 22913  22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8
 22914  22912 grotty

Measuring block device I/O latency from queue insert to completion:

# ./iolatency -Q
Tracing block I/O. Output every 1 seconds. Ctrl-C to end.

  >=(ms) .. <(ms)   : I/O      |Distribution                          |
       0 -> 1       : 1913     |######################################|
       1 -> 2       : 438      |#########                             |
       2 -> 4       : 100      |##                                    |
       4 -> 8       : 145      |###                                   |
       8 -> 16      : 43       |#                                     |
      16 -> 32      : 43       |#                                     |
      32 -> 64      : 1        |#                                     |

[...]

Tracing the block:block_rq_insert tracepoint, with kernel stack traces, and only for reads:

# ./tpoint -s block:block_rq_insert 'rwbs ~ "*R*"'
   cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]
   cksum-11908 [000] d... 7269839.919107: 
 => __elv_add_request
 => blk_flush_plug_list
 => blk_finish_plug
 => __do_page_cache_readahead
 => ondemand_readahead
 => page_cache_async_readahead
 => generic_file_read_iter
 => new_sync_read
 => vfs_read
 => SyS_read
 => system_call_fastpath

[...]

Count kernel function calls beginning with "bio_", summarize every second:

# ./funccount -i 1 'bio_*'
Tracing "bio_*"... Ctrl-C to end.

FUNC                              COUNT
bio_attempt_back_merge               26
bio_get_nr_vecs                     361
bio_alloc                           536
bio_alloc_bioset                    536
bio_endio                           536
bio_free                            536
bio_fs_destructor                   536
bio_init                            536
bio_integrity_enabled               536
bio_put                             729
bio_add_page                       1004

[...]

There are many more examples in the examples directory. Also see the man pages.

The intent is as few as possible. Eg, a Linux 3.2 server without debuginfo. See the tool man page for specifics.

FTRACE configured in the kernel. You may already have this configured and available in your kernel version, as FTRACE was first added in 2.6.27. This requires CONFIG_FTRACE and other FTRACE options depending on the tool. Some tools (eg, funccount) require CONFIG_FUNCTION_PROFILER.