kvmexit.py: introduce a tool to show kvm exit reasons and counts

Considering virtual machines' frequent exits can cause performance
problems, introduce a tool to show kvm exit reasons and counts, so
that the most frequent exited reasons could be located, reduced, or
even avoided.

For better performance, this tool employs a percpu array and percpu
hash in bpf to store exit reason and its counts. Besides, the bcc
python provides aggregation and various custom output. For more
background, realization and examples, please see kvmexit_example.txt
and man/man8/kvmexit.8 for more reference.

Signed-off-by: Fei Li <[email protected]>

README.md

@@ -123,6 +123,7 @@ pair of .c and .py files, and some are directories of files.
 - tools/[inject](tools/inject.py): Targeted error injection with call chain and predicates [Examples](tools/inject_example.txt).
 - tools/[killsnoop](tools/killsnoop.py): Trace signals issued by the kill() syscall. [Examples](tools/killsnoop_example.txt).
 - tools/[klockstat](tools/klockstat.py): Traces kernel mutex lock events and display locks statistics. [Examples](tools/klockstat_example.txt).
+- tools/[kvmexit](tools/kvmexit.py): Display the exit_reason and its statistics of each vm exit. [Examples](tools/kvmexit_example.txt).
 - tools/[llcstat](tools/llcstat.py): Summarize CPU cache references and misses by process. [Examples](tools/llcstat_example.txt).
 - tools/[mdflush](tools/mdflush.py): Trace md flush events. [Examples](tools/mdflush_example.txt).
 - tools/[memleak](tools/memleak.py): Display outstanding memory allocations to find memory leaks. [Examples](tools/memleak_example.txt).

man/man8/kvmexit.8

@@ -0,0 +1,115 @@
+.TH kvmexit 8 "2021-07-08" "USER COMMANDS"
+.SH NAME
+kvmexit \- Display the exit_reason and its statistics of each vm exit.
+.SH SYNOPSIS
+.B kvmexit [\-h] [\-p PID [\-v VCPU | \-a] ] [\-t TID | \-T 'TID1,TID2'] [duration]
+.SH DESCRIPTION
+Considering virtual machines' frequent exits can cause performance problems,
+this tool aims to locate the frequent exited reasons and then find solutions
+to reduce or even avoid the exit, by displaying the detail exit reasons and
+the counts of each vm exit for all vms running on one physical machine.
+
+This tool uses a PERCPU_ARRAY: pcpuArrayA and a percpu_hash: hashA to
+collaboratively store each kvm exit reason and its count. The reason is there
+exists a rule when one vcpu exits and re-enters, it tends to continue to run on
+the same physical cpu as the last cycle, which is also called 'cache hit'. Thus
+we turn to use a PERCPU_ARRAY to record the 'cache hit' situation to speed
+things up; and for other cases, then use a percpu_hash.
+
+As RAW_TRACEPOINT_PROBE(kvm_exit) consumes less cpu cycles, when this tool is
+used, it firstly tries to employ raw tracepoints in modules, and if failes,
+then fall back to regular tracepoint.
+
+Limitation: In view of the hardware-assisted virtualization technology of
+different architectures, currently we only adapt on vmx in intel.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+
+This also requires Linux 4.7+ (BPF_PROG_TYPE_TRACEPOINT support).
+.SH OPTIONS
+.TP
+\-h
+Print usage message.
+.TP
+\-p PID
+Display process with this PID only, collpase all tids with exit reasons sorted
+in descending order.
+.TP
+\-v VCPU
+Display this VCPU only for this PID.
+.TP
+\-a ALLTIDS
+Display all TIDS for this PID.
+.TP
+\-t TID
+Display thread with this TID only with exit reasons sorted in descending order.
+.TP
+\-T 'TID1,TID2'
+Display threads for a union like {395490, 395491}.
+.TP
+duration
+Duration of display, after sleeping several seconds.
+.SH EXAMPLES
+.TP
+Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end:
+#
+.B kvmexit
+.TP
+Display kvm exit reasons and statistics for all threads after sleeping 6 secs:
+#
+.B kvmexit 6
+.TP
+Display kvm exit reasons and statistics for PID 1273795 after sleeping 5 secs:
+#
+.B kvmexit -p 1273795 5
+.TP
+Display kvm exit reasons and statistics for PID 1273795 and its all threads after sleeping 5 secs:
+#
+.B kvmexit -p 1273795 5 -a
+.TP
+Display kvm exit reasons and statistics for PID 1273795 VCPU 0... Hit Ctrl-C to end:
+#
+.B kvmexit -p 1273795 -v 0
+.TP
+Display kvm exit reasons and statistics for PID 1273795 VCPU 0 after sleeping 4 secs:
+#
+.B kvmexit -p 1273795 -v 0 4
+.TP
+Display kvm exit reasons and statistics for TID 1273819 after sleeping 10 secs:
+#
+.B kvmexit -t 1273819 10
+.TP
+Display kvm exit reasons and statistics for TIDS ['1273820', '1273819']... Hit Ctrl-C to end:
+#
+.B kvmexit -T '1273820,1273819'
+.SH OVERHEAD
+This traces the "kvm_exit" kernel function, records the exit reason and
+calculates its counts. Contrast with filling more vm-exit reason debug entries,
+this tool is more easily and flexibly: the bcc python logic could provide nice
+kernel aggregation and custom output, the bpf in-kernel percpu_array and
+percpu_cache further improves performance.
+
+The impact of using this tool on the host should be negligible. While this
+tool is very efficient, it does affect the guest virtual machine itself, the
+average test results on guest vm are as follows:
+ | cpu cycles
+ no TP | 1127
+ regular TP | 1277 (13% downgrade)
+ RAW TP | 1187 (5% downgrade)
+
+Host: echo 1 > /proc/sys/net/core/bpf_jit_enable
+.SH SOURCE
+This is from bcc.
+.IP
+https://github.com/iovisor/bcc
+.PP
+Also look in the bcc distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Fei Li <[email protected]>