CentOS7.9 - Kernel 3.10.0-1160.88.1.el7.x86_64 - Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value

[aarvee11]

• Unable to start the ebpf_exporter

# docker run --rm -it --privileged -p 9435:9435 -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cap-add=SYS_ADMIN --ulimit memlock=1073741824 cloudflare_ebpf_exporter:latest --config.dir=examples --config.names=timers --debug
2023/11/27 06:20:57 Using on-demand resolution for cgroups (fanotify not available)
2023/11/27 06:20:57 libbpf [debug]: libbpf: loading examples/timers.bpf.o
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(3) tp_btf/timer_start, size 232, link 0, flags 6, type=1
2023/11/27 06:20:57 libbpf [debug]: libbpf: sec 'tp_btf/timer_start': found program 'timer_start' at insn offset 0 (0 bytes), code size 29 insns (232 bytes)
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(4) .reltp_btf/timer_start, size 48, link 27, flags 40, type=9
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(5) license, size 4, link 0, flags 3, type=1
2023/11/27 06:20:57 libbpf [debug]: libbpf: license of examples/timers.bpf.o is GPL
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(6) .maps, size 32, link 0, flags 3, type=1
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(17) .BTF, size 1153, link 0, flags 0, type=1
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(19) .BTF.ext, size 252, link 0, flags 0, type=1
2023/11/27 06:20:57 libbpf [debug]: libbpf: elf: section(27) .symtab, size 408, link 1, flags 0, type=2
2023/11/27 06:20:57 libbpf [debug]: libbpf: looking for externs among 17 symbols...
2023/11/27 06:20:57 libbpf [debug]: libbpf: collected 0 externs total
2023/11/27 06:20:57 libbpf [debug]: libbpf: map 'timer_starts_total': at sec_idx 6, offset 0.
2023/11/27 06:20:57 libbpf [debug]: libbpf: map 'timer_starts_total': found type = 1.
2023/11/27 06:20:57 libbpf [debug]: libbpf: map 'timer_starts_total': found key [8], sz = 8.
2023/11/27 06:20:57 libbpf [debug]: libbpf: map 'timer_starts_total': found value [8], sz = 8.
2023/11/27 06:20:57 libbpf [debug]: libbpf: map 'timer_starts_total': found max_entries = 1024.
2023/11/27 06:20:57 libbpf [debug]: libbpf: sec '.reltp_btf/timer_start': collecting relocation for section(3) 'tp_btf/timer_start'
2023/11/27 06:20:57 libbpf [debug]: libbpf: sec '.reltp_btf/timer_start': relo #0: insn #7 against 'timer_starts_total'
2023/11/27 06:20:57 libbpf [debug]: libbpf: prog 'timer_start': found map 0 (timer_starts_total, sec 6, off 0) for insn #7
2023/11/27 06:20:57 libbpf [debug]: libbpf: sec '.reltp_btf/timer_start': relo #1: insn #15 against 'timer_starts_total'
2023/11/27 06:20:57 libbpf [debug]: libbpf: prog 'timer_start': found map 0 (timer_starts_total, sec 6, off 0) for insn #15
2023/11/27 06:20:57 libbpf [debug]: libbpf: sec '.reltp_btf/timer_start': relo #2: insn #20 against 'timer_starts_total'
2023/11/27 06:20:57 libbpf [debug]: libbpf: prog 'timer_start': found map 0 (timer_starts_total, sec 6, off 0) for insn #20
2023/11/27 06:20:57 libbpf [warn]: libbpf: Error in bpf_object__probe_loading():Permission denied(13). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.
2023/11/27 06:20:57 libbpf [warn]: libbpf: failed to load object 'examples/timers.bpf.o'
2023/11/27 06:20:57 Error attaching exporter: error loading bpf object from "examples/timers.bpf.o" for config "timers": failed to load BPF object: permission denied

• Security Limits Config

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3090729
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

• Kernel details

# uname -r
3.10.0-1160.88.1.el7.x86_64

• BPF Enabled

# grep BPF /boot/config-3.10.0-1160.88.1.el7.x86_64
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y

Can anyone please help?

[aarvee11]

Tried on an Ubuntu machine and the issue remains:

# docker run -it --ulimit memlock=-1 ebpf_exporter:latest --config.dir=examples --config.names=syscalls
2023/11/27 09:19:27 Using on-demand resolution for cgroups (fanotify not available)
2023/11/27 09:19:27 libbpf [warn]: libbpf: Error in bpf_object__probe_loading():Operation not permitted(1). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value.
2023/11/27 09:19:27 libbpf [warn]: libbpf: failed to load object 'examples/syscalls.bpf.o'
2023/11/27 09:19:27 Error attaching exporter: error loading bpf object from "examples/syscalls.bpf.o" for config "syscalls": failed to load BPF object: operation not permitted

Limits:

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 2061314
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2061314
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Kernel and OS Details

# uname -r
5.4.0-164-generic

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.5 LTS
Release:	20.04
Codename:	focal

BPF Status:

# grep BPF /boot/config-$(uname -r)
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m

[bobrik]

Please see this section: https://github.com/cloudflare/ebpf_exporter#examples

3.10.0-1160.88.1.el7.x86_64

Your kernel version is 10 years old, so there's no surprise here.

[aarvee11]

@bobrik Am facing the issue even on 5.4.0-164-generic kernel. Anyways, will give it a shot again and come back