Installing ebpf_exporter on Amazon Linux 2

[wkneewalden]

Attempted installation on Amazon Linux 2 with the following commands using bcc install from source:

sudo yum-config-manager --enable epel
sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr python-cachetools gcc gcc-c++ make zlib-devel elfutils-libelf-devel
sudo yum install -y luajit luajit-devel
sudo yum install -y http:https://repo.iovisor.org/yum/extra/mageia/cauldron/x86_64/netperf-2.7.0-1.mga6.x86_64.rpm
pip3 install pyroute2
sudo yum install -y ncurses-devel
sudo yum install -y clang llvm llvm-devel llvm-static clang-devel clang-libs
git clone https://github.com/iovisor/bcc
pushd .
mkdir bcc/build; cd bcc/build
cmake3 ..
time make
sudo make install
popd
sudo yum -y install kernel-devel-$(uname -r)
sudo mount -t debugfs debugfs /sys/kernel/debug
sudo yum install golang
cd ~
git clone https://github.com/cloudflare/ebpf_exporter
cd ebpf_exporter/
make build

Result was an error in 'make build' step shown below, and this was the same result using the bcc package install from Amazon Linux 2 (sudo amazon-linux-extras install BCC), keeping in mind I was not able to install netperf listed above from rpm:

github.com/aquasecurity/libbpfgo
# github.com/aquasecurity/libbpfgo
In file included from ../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:5                                                                                                         :0:
./libbpfgo.h:17:10: fatal error: bpf/bpf.h: No such file or directory
 #include <bpf/bpf.h>
          ^~~~~~~~~~~
compilation terminated.

The locations of the missing library, along with another one, are here:

[root /]# find / -name bpf.h
/usr/include/linux/bpf.h
/usr/include/bcc/compat/linux/bpf.h
/usr/src/kernels/5.15.117-73.143.amzn2.x86_64/include/linux/bpf.h
/usr/src/kernels/5.15.117-73.143.amzn2.x86_64/include/net/netns/bpf.h
/usr/src/kernels/5.15.117-73.143.amzn2.x86_64/include/uapi/linux/bpf.h
/home/ec2-user/bcc/libbpf-tools/bpftool/include/uapi/linux/bpf.h
/home/ec2-user/bcc/libbpf-tools/bpftool/libbpf/include/uapi/linux/bpf.h
/home/ec2-user/bcc/libbpf-tools/bpftool/libbpf/src/bpf.h
/home/ec2-user/bcc/src/cc/libbpf/include/uapi/linux/bpf.h
/home/ec2-user/bcc/src/cc/libbpf/src/bpf.h
/home/ec2-user/go/pkg/mod/github.com/iovisor/[email protected]/elf/include/uapi/linux/bpf.h

[root /]# find / -name libbpf.h
/usr/include/bcc/libbpf.h
/home/ec2-user/bcc/libbpf-tools/bpftool/libbpf/src/libbpf.h
/home/ec2-user/bcc/src/cc/libbpf/src/libbpf.h
/home/ec2-user/bcc/src/cc/libbpf.h
/home/ec2-user/go/pkg/mod/github.com/iovisor/[email protected]/elf/include/libbpf.h

However, when I try to modify libbpfgo.h to point bpf.h (and libbpf.h) from another existing location such as inside /home/ec2-user/go, the build has a very long list of errors.

[bobrik]

You don't need bcc for this at all. You do need libbpf and libelf with static archives (libbpf.a and libelf.a). It doesn't look like amazon linux has those or maybe I just can't find them.

You can still build a dynamically linked version:

ivan@vm:~/projects/ebpf_exporter$ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) amazonlinux
bash-5.2# yum install golang make elfutils-libelf-devel kernel-libbpf-devel
bash-5.2# make GO_LDFLAGS=

Let me know if that works for you.

[bobrik]

I added make build-dynamic in #241 to make dynamic builds easier.

[wkneewalden]

Thanks for the response and suggestions. Could not locate 'kernel-libbpf-devel' from yum repo, so I used 'libbpf-devel' in the the following set of commands on a new instance:

sudo yum install golang make elfutils-libelf-devel libbpf-devel
git clone https://github.com/cloudflare/ebpf_exporter
cd ebpf_exporter/
make build-dynamic

The previous issue appears resolved, however a new set of errors appeared in the dynamic build, was not sure if this is related to using 'libbpf-devel' instead:

# github.com/aquasecurity/libbpfgo
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:30:9: could not determine kind of name for C.LIBBPF_MAJOR_VERSION
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:35:9: could not determine kind of name for C.LIBBPF_MINOR_VERSION
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:277:55: could not determine kind of name for C.LIBBPF_STRICT_AUTO_RLIMIT_MEMLOCK
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:278:55: could not determine kind of name for C.LIBBPF_STRICT_MAP_DEFINITIONS
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:276:55: could not determine kind of name for C.LIBBPF_STRICT_NO_OBJECT_LIST
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:275:55: could not determine kind of name for C.LIBBPF_STRICT_SEC_NAME
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:715:9: could not determine kind of name for C.bpf_map__lookup_elem
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:524:15: could not determine kind of name for C.bpf_map_create
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:986:7: could not determine kind of name for C.bpf_object__next_map
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:1008:7: could not determine kind of name for C.bpf_object__next_program
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:1316:21: could not determine kind of name for C.bpf_program__type
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:2111:16: could not determine kind of name for C.libbpf_probe_bpf_map_type
../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:2119:16: could not determine kind of name for C.libbpf_probe_bpf_prog_type
cgo:
gcc errors for preamble:
In file included from ../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:5:0:
./libbpfgo.h: In function 'init_perf_buf':
./libbpfgo.h:86:12: error: 'struct perf_buffer_opts' has no member named 'sz'
     pb_opts.sz = sizeof(struct perf_buffer_opts);
            ^
./libbpfgo.h:88:10: error: too many arguments to function 'perf_buffer__new'
     pb = perf_buffer__new(map_fd, page_cnt, perfCallback, perfLostCallback, (void *) ctx, &pb_opts);
          ^~~~~~~~~~~~~~~~
In file included from ./libbpfgo.h:18:0,
                 from ../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:5:
/usr/include/bpf/libbpf.h:682:1: note: declared here
 perf_buffer__new(int map_fd, size_t page_cnt,
 ^~~~~~~~~~~~~~~~
In file included from ../go/pkg/mod/github.com/aquasecurity/[email protected]/libbpfgo.go:5:0:
./libbpfgo.h: At top level:
./libbpfgo.h:135:82: error: parameter 2 ('order') has incomplete type
                                                       enum bpf_cgroup_iter_order order,
                                                                                  ^~~~~
./libbpfgo.h: In function 'bpf_iter_attach_opts_new':
./libbpfgo.h:148:10: error: 'union bpf_iter_link_info' has no member named 'cgroup'
     linfo->cgroup.order = order;
          ^~
./libbpfgo.h:149:10: error: 'union bpf_iter_link_info' has no member named 'cgroup'
     linfo->cgroup.cgroup_fd = cgroup_fd;
          ^~
./libbpfgo.h:150:10: error: 'union bpf_iter_link_info' has no member named 'cgroup'
     linfo->cgroup.cgroup_id = cgroup_id;
          ^~
./libbpfgo.h:151:10: error: 'union bpf_iter_link_info' has no member named 'task'
     linfo->task.tid = tid;
          ^~
./libbpfgo.h:152:10: error: 'union bpf_iter_link_info' has no member named 'task'
     linfo->task.pid = pid;
          ^~
./libbpfgo.h:153:10: error: 'union bpf_iter_link_info' has no member named 'task'
     linfo->task.pid_fd = pid_fd;

[bobrik]

It looks like your distro has a libbpf that is too old. I would recommend to either (in order of preference):

1. Build in docker: here's how we do it.
2. Use precompiled binaries we provide for every rehease: here's v2.2.0.
3. Build and install latest libbpf yourself and build a static binary then: here's how we build libbpf.

[wkneewalden]

I tried option 3 with this set of commands for install to work. Thanks a lot for the assistance, and appreciate taking the time to add the feature to simplify the install. I will test the examples and see if any other issues arise.

sudo yum install golang make elfutils-libelf-devel
sudo bash
mkdir /build && git clone --branch v1.2.0 --depth 1 https://github.com/libbpf/libbpf.git /build/libbpf && make -j $(nproc) -C /build/libbpf/src BUILD_STATIC_ONLY=y LIBSUBDIR=lib install install_uapi_headers && tar -czf /build/libbpf.tar.gz /usr/lib/libbpf.a /usr/lib/pkgconfig/libbpf.pc /usr/include/bpf /usr/include/linux/bpf.h /usr/include/linux/bpf_common.h /usr/include/linux/btf.h
exit
git clone https://github.com/cloudflare/ebpf_exporter
cd ebpf_exporter/
sudo bash
make build-dynamic
chown ec2-user:ec2-user ebpf_exporter
exit

[wkneewalden]

Confirmed that examples install with command on instructions as ec2-user and exporter can run biolatency example. That should be everything.