Demonstrations of biotop, the Linux eBPF/bcc version. Short for block device I/O top, biotop summarizes which processes are performing disk I/O. It's top for disks. Sample output: # ./biotop Tracing... Output every 1 secs. Hit Ctrl-C to end 08:04:11 loadavg: 1.48 0.87 0.45 1/287 14547 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 14501 cksum R 202 1 xvda1 361 28832 3.39 6961 dd R 202 1 xvda1 1628 13024 0.59 13855 dd R 202 1 xvda1 1627 13016 0.59 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.00 1880 supervise W 202 1 xvda1 2 8 6.71 1873 supervise W 202 1 xvda1 2 8 2.51 1871 supervise W 202 1 xvda1 2 8 1.57 1876 supervise W 202 1 xvda1 2 8 1.22 1892 supervise W 202 1 xvda1 2 8 0.62 1878 supervise W 202 1 xvda1 2 8 0.78 1886 supervise W 202 1 xvda1 2 8 1.30 1894 supervise W 202 1 xvda1 2 8 3.46 1869 supervise W 202 1 xvda1 2 8 0.73 1888 supervise W 202 1 xvda1 2 8 1.48 By default the screen refreshes every 1 second, and shows the top 20 disk consumers, sorted on total Kbytes. The first line printed is the header, which has the time and then the contents of /proc/loadavg. For the interval summarized by the output above, the "cksum" command performed 361 disk reads to the "xvda1" device, for a total of 28832 Kbytes, with an average I/O time of 3.39 ms. Two "dd" processes were also reading from the same disk, which a higher I/O rate and lower latency. While the average I/O size is not printed, it can be determined by dividing the Kbytes column by the I/O column. The columns through to Kbytes show the workload applied. The final column, AVGms, shows resulting performance. Other bcc tools can be used to get more details when needed: biolatency and biosnoop. Many years ago I created the original "iotop", and later regretted not calling it diskiotop or blockiotop, as "io" alone is ambiguous. This time it is biotop. The -C option can be used to prevent the screen from clearing (my preference). Here's using it with a 5 second interval: # ./biotop -C 5 Tracing... Output every 5 secs. Hit Ctrl-C to end 08:09:44 loadavg: 0.42 0.44 0.39 2/282 22115 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 5993 47976 0.33 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.67 1866 svscan R 202 1 xvda1 33 132 1.24 1880 supervise W 202 1 xvda1 10 40 0.56 1873 supervise W 202 1 xvda1 10 40 0.79 1871 supervise W 202 1 xvda1 10 40 0.78 1876 supervise W 202 1 xvda1 10 40 0.68 1892 supervise W 202 1 xvda1 10 40 0.71 1878 supervise W 202 1 xvda1 10 40 0.65 1886 supervise W 202 1 xvda1 10 40 0.78 1894 supervise W 202 1 xvda1 10 40 0.80 1869 supervise W 202 1 xvda1 10 40 0.91 1888 supervise W 202 1 xvda1 10 40 0.63 22069 bash R 202 1 xvda1 1 16 19.94 9251 kworker/u16:2 W 202 16 xvdb 2 8 0.13 08:09:49 loadavg: 0.47 0.44 0.39 1/282 22231 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 13450 107600 0.35 22199 cksum R 202 1 xvda1 941 45548 4.63 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.93 24467 kworker/0:2 W 202 16 xvdb 1 64 0.28 1880 supervise W 202 1 xvda1 10 40 0.81 1873 supervise W 202 1 xvda1 10 40 0.81 1871 supervise W 202 1 xvda1 10 40 1.03 1876 supervise W 202 1 xvda1 10 40 0.76 1892 supervise W 202 1 xvda1 10 40 0.74 1878 supervise W 202 1 xvda1 10 40 0.94 1886 supervise W 202 1 xvda1 10 40 0.76 1894 supervise W 202 1 xvda1 10 40 0.69 1869 supervise W 202 1 xvda1 10 40 0.72 1888 supervise W 202 1 xvda1 10 40 1.70 22199 bash R 202 1 xvda1 2 20 0.35 482 xfsaild/md0 W 202 16 xvdb 5 13 0.27 482 xfsaild/md0 W 202 32 xvdc 2 8 0.33 31331 pickup R 202 1 xvda1 1 4 0.31 08:09:54 loadavg: 0.51 0.45 0.39 2/282 22346 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 14689 117512 0.32 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.33 1880 supervise W 202 1 xvda1 10 40 0.65 1873 supervise W 202 1 xvda1 10 40 1.08 1871 supervise W 202 1 xvda1 10 40 0.66 1876 supervise W 202 1 xvda1 10 40 0.79 1892 supervise W 202 1 xvda1 10 40 0.67 1878 supervise W 202 1 xvda1 10 40 0.66 1886 supervise W 202 1 xvda1 10 40 1.02 1894 supervise W 202 1 xvda1 10 40 0.88 1869 supervise W 202 1 xvda1 10 40 0.89 1888 supervise W 202 1 xvda1 10 40 1.25 08:09:59 loadavg: 0.55 0.46 0.40 2/282 22461 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 14442 115536 0.33 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.46 1880 supervise W 202 1 xvda1 10 40 0.87 1873 supervise W 202 1 xvda1 10 40 0.87 1871 supervise W 202 1 xvda1 10 40 0.78 1876 supervise W 202 1 xvda1 10 40 0.86 1892 supervise W 202 1 xvda1 10 40 0.89 1878 supervise W 202 1 xvda1 10 40 0.87 1886 supervise W 202 1 xvda1 10 40 0.86 1894 supervise W 202 1 xvda1 10 40 1.06 1869 supervise W 202 1 xvda1 10 40 1.12 1888 supervise W 202 1 xvda1 10 40 0.98 08:10:04 loadavg: 0.59 0.47 0.40 3/282 22576 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 14179 113432 0.34 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.39 1880 supervise W 202 1 xvda1 10 40 0.81 1873 supervise W 202 1 xvda1 10 40 1.02 1871 supervise W 202 1 xvda1 10 40 1.15 1876 supervise W 202 1 xvda1 10 40 1.10 1892 supervise W 202 1 xvda1 10 40 0.77 1878 supervise W 202 1 xvda1 10 40 0.72 1886 supervise W 202 1 xvda1 10 40 0.81 1894 supervise W 202 1 xvda1 10 40 0.86 1869 supervise W 202 1 xvda1 10 40 0.83 1888 supervise W 202 1 xvda1 10 40 0.79 24467 kworker/0:2 R 202 32 xvdc 3 12 0.26 1056 cron R 202 1 xvda1 2 8 0.30 24467 kworker/0:2 R 202 16 xvdb 1 4 0.23 08:10:09 loadavg: 0.54 0.46 0.40 2/281 22668 PID COMM D MAJ MIN DISK I/O Kbytes AVGms 22069 dd R 202 1 xvda1 250 2000 0.34 326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.40 1880 supervise W 202 1 xvda1 8 32 0.93 1873 supervise W 202 1 xvda1 8 32 0.76 1871 supervise W 202 1 xvda1 8 32 0.60 1876 supervise W 202 1 xvda1 8 32 0.61 1892 supervise W 202 1 xvda1 8 32 0.68 1878 supervise W 202 1 xvda1 8 32 0.90 1886 supervise W 202 1 xvda1 8 32 0.57 1894 supervise W 202 1 xvda1 8 32 0.97 1869 supervise W 202 1 xvda1 8 32 0.69 1888 supervise W 202 1 xvda1 8 32 0.67 This shows another "dd" command reading from xvda1. On this system, various "supervise" processes do 8 disk writes per second, every second (they are creating and updating "status" files). USAGE message: # ./biotop.py -h usage: biotop.py [-h] [-C] [-r MAXROWS] [interval] [count] Block device (disk) I/O by process positional arguments: interval output interval, in seconds count number of outputs optional arguments: -h, --help show this help message and exit -C, --noclear don't clear the screen -r MAXROWS, --maxrows MAXROWS maximum rows to print, default 20 examples: ./biotop # block device I/O top, 1 second refresh ./biotop -C # don't clear the screen ./biotop 5 # 5 second summaries ./biotop 5 10 # 5 second summaries, 10 times only