Demonstrations of tplist. tplist displays kernel tracepoints and USDT probes, including their format. It can be used to discover probe points for use with the trace and argdist tools. Kernel tracepoints are scattered around the kernel and provide valuable static tracing on block and network I/O, scheduling, power events, and many other subjects. USDT probes are placed in libraries (such as libc) and executables (such as node) and provide static tracing information that can (optionally) be turned on and off at runtime. For example, suppose you want to discover which USDT probes a particular executable contains. Just run tplist on that executable (or library): $ tplist -l basic_usdt /home/vagrant/basic_usdt basic_usdt:start_main /home/vagrant/basic_usdt basic_usdt:loop_iter /home/vagrant/basic_usdt basic_usdt:end_main The loop_iter probe sounds interesting. How many arguments are available? $ tplist '*loop_iter' -l basic_usdt -v basic_usdt:loop_iter [sema 0x601036] 2 location(s) 2 argument(s) This output indicates that the loop_iter probe is used in two locations in the basic_usdt executable, and that it has two arguments. Fortunately, the argdist and trace tools understand the probe format and can print out the arguments automatically -- you can refer to them as arg1, arg2, and so on. Try to explore with some common libraries on your system and see if they contain UDST probes. Here are two examples you might find interesting: $ tplist -l pthread # list probes in libpthread /lib64/libpthread.so.0 libpthread:pthread_start /lib64/libpthread.so.0 libpthread:pthread_create /lib64/libpthread.so.0 libpthread:pthread_join /lib64/libpthread.so.0 libpthread:pthread_join_ret /lib64/libpthread.so.0 libpthread:mutex_init ... more output truncated $ tplist -l c # list probes in libc /lib64/libc.so.6 libc:setjmp /lib64/libc.so.6 libc:longjmp /lib64/libc.so.6 libc:longjmp_target /lib64/libc.so.6 libc:memory_arena_reuse_free_list /lib64/libc.so.6 libc:memory_heap_new ... more output truncated tplist also understands kernel tracepoints, and can list their format as well. For example, let's look for all block I/O-related tracepoints: # tplist 'block*' block:block_touch_buffer block:block_dirty_buffer block:block_rq_abort block:block_rq_requeue block:block_rq_complete block:block_rq_insert block:block_rq_issue block:block_bio_bounce block:block_bio_complete block:block_bio_backmerge block:block_bio_frontmerge block:block_bio_queue block:block_getrq block:block_sleeprq block:block_plug block:block_unplug block:block_split block:block_bio_remap block:block_rq_remap The block:block_rq_complete tracepoints sounds interesting. Let's print its format to see what we can trace with argdist and trace: $ tplist -v block:block_rq_complete block:block_rq_complete dev_t dev; sector_t sector; unsigned int nr_sector; int errors; char rwbs[8]; The dev, sector, nr_sector, etc. variables can now all be used in probes you specify with argdist or trace. For debugging USDT probes, it is sometimes useful to see the exact locations and arguments of the probes, including the registers or global variables from which their values are coming from. In super-verbose mode, tplist will print this information (note the -vv): $ tplist -vv -l c *alloc* libc:memory_malloc_retry [sema 0x0] location #0 /lib64/libc.so.6 0x835c0 argument #0 8 unsigned bytes @ bp location #1 /lib64/libc.so.6 0x83778 argument #0 8 unsigned bytes @ bp location #2 /lib64/libc.so.6 0x85a50 argument #0 8 unsigned bytes @ bp libc:memory_realloc_retry [sema 0x0] location #0 /lib64/libc.so.6 0x84b90 argument #0 8 unsigned bytes @ r13 argument #1 8 unsigned bytes @ bp location #1 /lib64/libc.so.6 0x85cf0 argument #0 8 unsigned bytes @ r13 argument #1 8 unsigned bytes @ bp libc:memory_calloc_retry [sema 0x0] location #0 /lib64/libc.so.6 0x850f0 argument #0 8 unsigned bytes @ bp USAGE message: $ tplist -h usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter] Display kernel tracepoints or USDT probes and their formats. positional arguments: filter A filter that specifies which probes/tracepoints to print optional arguments: -h, --help show this help message and exit -p PID, --pid PID List USDT probes in the specified process -l LIB, --lib LIB List USDT probes in the specified library or executable -v Increase verbosity level (print variables, arguments, etc.)