This is a light-weight utility used for performance profiling and data analysis.
The performance data capturing is designed by profiling per-minute for one second period to capture the CPU-related Perf data and Memory/IO with 5s. Memory capture uses Ftrace to capture the page allocation callstack, and the Flamegraph is generated by byte-based. IO is to capture the blk IO insert/complete trace event. On the other hand, the system performance data(irq, mem, net, ...) are also recorded by default. And it's planned to be translated to the files with specific formats(csv, tsv...) for further analysis(ScatterPlot/Flamegraph/LineChart...etc.).
$ git clone https://github.com/bboymimi/easy-flamegraph.git
After git clone the repo, there are some dependent packages which need to be installed:
sudo apt update
sudo apt install -y binutils linux-tools-common linux-tools-$(uname -r) sysstat make zip
Then, the easy-flamegraph can be installed to the system by:
$ cd easy-flamegraph
$ sudo make install
Then, the code will be put under:
$ ls /usr/lib/easy-flamegraph/
conditions entry ezcli FlameGraph gen-flamegraph.sh lib perf-output sysinfo
The periodic sampling is implemented in the crontab config:
$ cat /etc/cron.d/easy-flamegraph-cron
# Flamegraph-mem sample in 1 minute period
*/1 * * * * root /usr/lib/easy-flamegraph/entry > /dev/null
pip3 install -r requirements.txt
sudo -E ./summary.py
Then, open the dashbaord by: "google-chrome-stable /var/log/easy-flamegraph/summary.html" or firefox
For parsing the folder:
summary.py -i easy-flamegraph/
The dashboard will be in easy-flamegraph/summary.html.
Please wait at least one minute for the profiling data to be generated. Also, please refer to the following configurations to enable specific profilig for the dashboard.
On Ubuntu Xenial, the dashboard cannot be generated as Bokeh requires Python 3.6. However, the system is running 3.5. Please use lxc container to set up the Ubuntu LTS environment (>=v4.15) for parsing the data.
The configuration file is /etc/default/easy-flamegraph.conf.
There are global settings that can apply to each subsystem (cpu/mem/io...). And the name is started with G.
e.g.
# Specify the sampling rate
G_SRATE=99
# Specify PID for perf record
G_PID=
# How many seconds perf record should run
G_PERF_PERIOD=1
The flamegraph will be generated automatically into the specific subsystem folders: /var/log/easy-flamegraph{cpu,mem,io}. However, the specific flamegraph generation needs to be enabled in the etc/default/easy-flamegraph.conf.
The easy-flamegraph info log can be found in /var/log/easy-flamegraph/log if it cannot work.
USE_CPU=true
$ google-chrome-stable /var/log/easy-flamegraph/cpu/2020-08-10_151802.cpu.t25u26.svg
By default, to decrease the overhead, the CPU profiler is triggered when the whole system average CPU utilization >= 25%.
CPU_THRESHOLD=25
The svg naming rule: t25 means the current threshold is 25%. u26 means the captured CPU utilization is 26%(>=25%).
The CPU flamegraph is generated by profiling the CPU by perf record -a --call-graph dwarf sleep 1. And this can show the whole system CPU utlization in a specific moment.
The per-cpu flamegraph is useful to debug the high loading issue in a specific CPU.
G_PER_CPU_FLAMEGRAPH=true
When the per-cpu flamegraph is enabled and the USE_CPU is true, easy-flamegraph will profile the system when the loading is higher than the threshold. By using the dashboard, the time series loading can be visualized and useful to identify the culprit. When clicking on the dot and the flamegraph popped out is None, please check if the loading is lower than the CPU_THRESHOLD or the G_PER_CPU_FLAMEGRAPH is false (only one flamegraph will be generated for the "all" system). The different flamegraph with the specific click will be popped out when clicking on different CPU utlization line. The current dashboard implementation supports at most eight (8) CPUs and needs to improve when the # of CPUs is big. The per-cpu flamegraph is still generated in /var/log/easy-flamegraph/cpu and can be viewed manually by "google-chrome-stable xxx.svg."
TODO: The BCC profile to replace the Perf as it's light weight and can reduce the user<->kernel data transfer overhead.
sudo python3 /usr/share/bcc/tools/profile -a -I -F 1999 -f 1 | /usr/lib/easy-flamegraph/FlameGraph/flamegraph.pl > /tmp/stack_for_flame.svg; google-chrome-stable /tmp/stack_for_flame.svg
USE_MEM=true
The memory flamegraph profiles the kernel space page allocation related to function __alloc_pages_nodemask. Ftrace is used as the backend as it doesn't need to install other packages and is easy to use. And it helps enormously debug the memory leakage issue.
Periodically, the automatically captured memory allocation svg files are located in: /var/log/easy-flamegraph/mem/
The dashaboard is generated by the /proc/vmstat and /proc/meminfo. And different kernel version has subtle difference. If you find any error, pleaser let me know.
TODO: Ftrace is used as the backend implementation to capture the callstack of the page_alloc perf event. This is more accurate than the Perf event as Ftrace captures the dumpstack of page allocation continuously instead of sampling the CPU used by Perf. If you use the perf probe __alloc_pages_nodemask and then perf record -a -g to profile the callstack of the __alloc_pages_nodemask, there will not be much callstack as expected. As Perf is the profiling basis tool and doesn't track every calling path. However, still need to figure out how to integrate Perf with Ftrace to lower down the profiling effort.
The memory flamegraph can be generated by the script:
$ sudo ./profile/conditions/mem/mem_ftrace
Use /home/vin/os/easy-flamegraph/profile/easy-flamegraph.conf as the config file
The memory allocation sampling will profile 5s.....
done.
[ftrace] Use the /var/log/easy-flamegraph/mem/2021-05-12_172506.mem.memtrace as the source of the FlameGraph.
###########
# The whole system perf interactive .svg graph "/var/log/easy-flamegraph/mem/2021-05-12_172506.mem.memtrace.svg" has been generated.
#
# The FlameGraph can be viewed by:
# $ google-chrome-stable /var/log/easy-flamegraph/mem/2021-05-12_172506.mem.memtrace.svg
# or
# $ firefox /var/log/easy-flamegraph/mem/2021-05-12_172506.mem.memtrace.svg
#
###########
Or it can be generated by the detailed steps: The ftrace dump log: alloc_pages_report.log
$ sudo trace-cmd record -a -e kmem:mm_page_alloc -T sleep 1
$ sudo trace-cmd report > alloc_pages_report.log
Then, the log file can be parsed by stackcollapse-tracecmd.pl
cat ~/Downloads/276737/alloc_pages_report.log | ./stackcollapse-tracecmd.pl --process_name --trace_func_name=__alloc_pages_nodemask | ./flamegraph.pl --title "__alloc_pages_nodemask callstack" --subtitle "With process name and __alloc_pages_nodemask" > with-pname-ftrace.svg
cat ~/Downloads/276737/alloc_pages_report.log | ./stackcollapse-tracecmd.pl --trace_func_name "__alloc_pages_nodemask" | ./flamegraph.pl --title "__alloc_pages_nodemask callstack" --subtitle "*Without* process name and __alloc_pages_nodemask" > no-pname-ftrace.svg
TODO: Need to implement the insert/completion of the BIOs. Also, the latency map can also be generated for each specific process.
This is the wrapper used for the FlameGraph generation. The FlameGraph is invented by Brendan Greg. For more information, please refer to: https://github.com/brendangregg/FlameGraph
At first, you need to profile the system and get the perf.data:
# To get the all CPUs profiling callstack from user space to kernel space
$ sudo perf record -a --call-graph dwarf
# To get the all CPUs profiling of kernel space callstack
$ sudo perf record -a -g
By default, gen-flamegraph.sh will read the perf.data under the current folder or you can assign the perf.data by "-i":
It's preferred to add the "sudo" as reading the /proc/kallsyms need the root privilege if the kernel debug symbol isn't installed.
$ git clone https://github.com/bboymimi/perf-utils.git
$ sudo gen-flamegraph.sh -i perf.data
Or grep the specific string you are interested to generate the FlameGraph:
$ sudo ./gen-flamegraph.sh -g ssh
Use the /home/ubuntu/perf-utils/perf.data as the source of the FlameGraph.
###########
# The perf interactive .svg graph "/home/ubuntu/perf-output/2017-08-16_09:59:32.perf.data.foldedSssh.svg" has been generated.
# The FlameGraph can be viewed by:
# $ google-chrome-stable /home/ubuntu/perf-output/2017-08-16_09:59:32.perf.data.foldedSssh.svg
# or
# $ firefox /home/ubuntu/perf-output/2017-08-16_09:59:32.perf.data.foldedSssh.svg
Sometimes, the per-cpu flamegraph is useful to identify if a specific process is hogging on a CPU. In this case, looking at the top or mpstat -P ALL 1 by looking at the average CPU usage would be misleading.
$ sudo ./gen-flamegraph.sh -p true
Read the help page to get more detail:
$ ./gen-flamegraph.sh -h
usage: ./gen-flamegraph.sh -g <grep string to make specific flamegraph> -i <perf file> -k <kernel version #>
d - drop the perf related data(include perf.data!!) and keep the .svg flamegraph file to save space
g - grep strings - to grep specific strings e.g., kworker, to make flamegraph
i - perf report file
k - kernel version - specific kernel version number
o - output directory - the output directory to save the .svg/script file
s - symfs - to assign the directory to search for the debug symbol of kernel modules
t - tar the /home/vin/os/easy-flamegraph/perf-output/
p - generate the flamegraph for each CPU
subtitle - the subtitle of the framegraph
title - the title of the framegraph