Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FC37: cannot load module: failed to find valid kernel BTF (compiled make build-binary binary does not work) #245

Closed
Rudd-O opened this issue Aug 6, 2023 · 10 comments
Closed

FC37: cannot load module: failed to find valid kernel BTF (compiled make build-binary binary does not work) #245

Rudd-O opened this issue Aug 6, 2023 · 10 comments

Comments

@Rudd-O
Copy link

Rudd-O commented Aug 6, 2023

With kernel 6.4.7-100.fc37.x86_64 and relevant devel packages installed, I cannot load biolatency:

Aug 06 23:27:51 roxanne.dragonfear ebpf_exporter[160320]: 2023/08/06 23:27:51 libbpf [warn]: libbpf: failed to find valid kernel BTF
Aug 06 23:27:51 roxanne.dragonfear ebpf_exporter[160320]: 2023/08/06 23:27:51 libbpf [warn]: libbpf: Error loading vmlinux BTF: -3
Aug 06 23:27:51 roxanne.dragonfear ebpf_exporter[160320]: 2023/08/06 23:27:51 libbpf [warn]: libbpf: failed to load object '/usr/libexec/ebpf_exporter/biolatency.bpf.o'
Aug 06 23:27:51 roxanne.dragonfear ebpf_exporter[160320]: 2023/08/06 23:27:51 Error attaching exporter: error loading bpf object from "/usr/libexec/ebpf_exporter/biolatency.bpf.o" for config "biolatency": failed to load BPF object: no such process
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: ebpf_exporter.service: Main process exited, code=exited, status=1/FAILURE
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: ebpf_exporter.service: Failed with result 'exit-code'.
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: ebpf_exporter.service: Scheduled restart job, restart counter is at 5.
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: Stopped ebpf_exporter.service - Prometheus exporter for eBPF-based metrics.
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: ebpf_exporter.service: Start request repeated too quickly.
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: ebpf_exporter.service: Failed with result 'exit-code'.
Aug 06 23:27:51 roxanne.dragonfear systemd[1]: Failed to start ebpf_exporter.service - Prometheus exporter for eBPF-based metrics.

As you can see, the BTF file does exist in /sys/kernel:

newfstatat(AT_FDCWD, "/usr/libexec/ebpf_exporter", {st_mode=S_IFDIR|0755, st_size=26, ...}, 0) = 0
openat(AT_FDCWD, "/usr/libexec/ebpf_exporter/biolatency.yaml", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/libexec/ebpf_exporter/biolatency.bpf.o", O_RDONLY|O_CLOEXEC) = 3
access("/proc/version_signature", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/libexec/ebpf_exporter/biolatency.bpf.o", O_RDONLY|O_CLOEXEC) = 7
newfstatat(7, "", {st_mode=S_IFREG|0644, st_size=43720, ...}, AT_EMPTY_PATH) = 0
access("/sys/kernel/btf/vmlinux", R_OK) = 0
openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 7
newfstatat(7, "", {st_mode=S_IFREG|0444, st_size=5635378, ...}, AT_EMPTY_PATH) = 0
newfstatat(7, "", {st_mode=S_IFREG|0444, st_size=5635378, ...}, AT_EMPTY_PATH) = 0
access("/boot/vmlinux-6.4.7-100.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory)
access("/lib/modules/6.4.7-100.fc37.x86_64/vmlinux-6.4.7-100.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory)
access("/lib/modules/6.4.7-100.fc37.x86_64/build/vmlinux", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/modules/6.4.7-100.fc37.x86_64/kernel/vmlinux", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/debug/boot/vmlinux-6.4.7-100.fc37.x86_64", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/debug/boot/vmlinux-6.4.7-100.fc37.x86_64.debug", R_OK) = -1 ENOENT (No such file or directory)
access("/usr/lib/debug/lib/modules/6.4.7-100.fc37.x86_64/vmlinux", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 7
2023/08/06 23:31:28 libbpf [warn]: libbpf: failed to find valid kernel BTF
2023/08/06 23:31:28 libbpf [warn]: libbpf: Error loading vmlinux BTF: -3
2023/08/06 23:31:28 libbpf [warn]: libbpf: failed to load object '/usr/libexec/ebpf_exporter/biolatency.bpf.o'
2023/08/06 23:31:28 Error attaching exporter: error loading bpf object from "/usr/libexec/ebpf_exporter/biolatency.bpf.o" for config "biolatency": failed to load BPF object: no such process

This is a regression from 1.x which worked flawlessly.
@bobrik
Copy link
Contributor

bobrik commented Aug 7, 2023

What version are you running? Is this packaged by you or somebody else? Could you run the latest with --debug?

The failed to find valid kernel BTF message is coming from here:

There should also be a message like this:

2023/08/06 23:49:39 libbpf [debug]: libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0

relevant devel packages installed

There aren't any devel packages needed at runtime for v2.

@Rudd-O
Copy link
Author

Rudd-O commented Aug 7, 2023 •
edited
Loading

What version are you running? Is this packaged by you or somebody else?

ebpf_exporter latest master source compiled by me. Kernel is whatever Fedora ships. Debug log here:

[root@roxanne ~]# /usr/bin/ebpf_exporter --debug --config.dir /usr/libexec/ebpf_exporter --config.names biolatency
2023/08/07 20:49:51 libbpf [debug]: libbpf: loading /usr/libexec/ebpf_exporter/biolatency.bpf.o
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(2) raw_tp/block_rq_insert, size 160, link 0, flags 6, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec 'raw_tp/block_rq_insert': found program 'block_rq_insert' at insn offset 0 (0 bytes), code size 20 insns (160 bytes)
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(3) .relraw_tp/block_rq_insert, size 32, link 15, flags 40, type=9
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(4) raw_tp/block_rq_issue, size 160, link 0, flags 6, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec 'raw_tp/block_rq_issue': found program 'block_rq_issue' at insn offset 0 (0 bytes), code size 20 insns (160 bytes)
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(5) .relraw_tp/block_rq_issue, size 32, link 15, flags 40, type=9
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(6) raw_tp/block_rq_complete, size 1544, link 0, flags 6, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec 'raw_tp/block_rq_complete': found program 'block_rq_complete' at insn offset 0 (0 bytes), code size 193 insns (1544 bytes)
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(7) .relraw_tp/block_rq_complete, size 128, link 15, flags 40, type=9
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(8) .maps, size 64, link 0, flags 3, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(9) license, size 4, link 0, flags 3, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: license of /usr/libexec/ebpf_exporter/biolatency.bpf.o is GPL
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(10) .BTF, size 35163, link 0, flags 0, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(12) .BTF.ext, size 1980, link 0, flags 0, type=1
2023/08/07 20:49:51 libbpf [debug]: libbpf: elf: section(15) .symtab, size 816, link 16, flags 0, type=2
2023/08/07 20:49:51 libbpf [debug]: libbpf: looking for externs among 34 symbols...
2023/08/07 20:49:51 libbpf [debug]: libbpf: collected 1 externs total
2023/08/07 20:49:51 libbpf [debug]: libbpf: extern (kcfg) #0: symbol 28, off 0, name LINUX_KERNEL_VERSION
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'start': at sec_idx 8, offset 0.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'start': found type = 1.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'start': found key [8], sz = 8.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'start': found value [18], sz = 8.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'start': found max_entries = 10000.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'bio_latency_seconds': at sec_idx 8, offset 32.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'bio_latency_seconds': found type = 1.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'bio_latency_seconds': found key [502], sz = 16.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'bio_latency_seconds': found value [18], sz = 8.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'bio_latency_seconds': found max_entries = 7140.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 'biolate.kconfig' (global data): at sec_idx 15, offset 0, flags 480.
2023/08/07 20:49:51 libbpf [debug]: libbpf: map 2 is "biolate.kconfig"
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_insert': collecting relocation for section(2) 'raw_tp/block_rq_insert'
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_insert': relo #0: insn #0 against 'LINUX_KERNEL_VERSION'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_insert': found extern #0 'LINUX_KERNEL_VERSION' (sym 28) for insn #0
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_insert': relo #1: insn #14 against 'start'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_insert': found map 0 (start, sec 8, off 0) for insn #14
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_issue': collecting relocation for section(4) 'raw_tp/block_rq_issue'
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_issue': relo #0: insn #0 against 'LINUX_KERNEL_VERSION'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_issue': found extern #0 'LINUX_KERNEL_VERSION' (sym 28) for insn #0
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_issue': relo #1: insn #14 against 'start'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_issue': found map 0 (start, sec 8, off 0) for insn #14
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': collecting relocation for section(6) 'raw_tp/block_rq_complete'
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #0: insn #6 against 'start'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 0 (start, sec 8, off 0) for insn #6
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #1: insn #141 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #141
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #2: insn #149 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #149
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #3: insn #154 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #154
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #4: insn #167 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #167
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #5: insn #175 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #175
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #6: insn #180 against 'bio_latency_seconds'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 1 (bio_latency_seconds, sec 8, off 32) for insn #180
2023/08/07 20:49:51 libbpf [debug]: libbpf: sec '.relraw_tp/block_rq_complete': relo #7: insn #188 against 'start'
2023/08/07 20:49:51 libbpf [debug]: libbpf: prog 'block_rq_complete': found map 0 (start, sec 8, off 0) for insn #188
2023/08/07 20:49:51 libbpf [debug]: libbpf: Unsupported BTF_KIND:19
2023/08/07 20:49:51 libbpf [debug]: libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': -22
2023/08/07 20:49:51 libbpf [warn]: libbpf: failed to find valid kernel BTF
2023/08/07 20:49:51 libbpf [warn]: libbpf: Error loading vmlinux BTF: -3
2023/08/07 20:49:51 libbpf [warn]: libbpf: failed to load object '/usr/libexec/ebpf_exporter/biolatency.bpf.o'
2023/08/07 20:49:51 Error attaching exporter: error loading bpf object from "/usr/libexec/ebpf_exporter/biolatency.bpf.o" for config "biolatency": failed to load BPF object: no such process

@bobrik
Copy link
Contributor

bobrik commented Aug 7, 2023

This seem to be the issue: Unsupported BTF_KIND:19. Here 19 is BTF_KIND_ENUM64 and support for it was added here:

What libbpf version do you have installed when you compile ebpf_exporter?

@bobrik
Copy link
Contributor

bobrik commented Aug 7, 2023

We provide pre-built statically binaries built with a tested version of libbpf:

Does it work if you try that? Testing it would remove the libbpf version discrepancy possibility.

@Rudd-O
Copy link
Author

Rudd-O commented Aug 8, 2023

Dammit. Yes, that binary works correctly. Perhaps it's the CFLAGS I'm using when building the modules or the binary?

@Rudd-O
Copy link
Author

Rudd-O commented Aug 8, 2023

I have tried unsetting CFLAGS during the RPM build. Nothing. Same error.

@Rudd-O
Copy link
Author

Rudd-O commented Aug 8, 2023 •
edited
Loading

Actually, even a simple make build-binary and then a cd examples ; make ; cd .. will produce a non-functional binary. And some of the examples won't even build anyway (biolatency built in Fedora 37 builds, but doesn't work as you can see).

The problem is definitely the binary, as my locally built modules work with the static binary you furnished.

@Rudd-O
Copy link
Author

Rudd-O commented Aug 8, 2023

Gah. I can't build statically on Fedora anyway because they dropped libelf static years ago:

https://www.spinics.net/lists/fedora-devel/msg275554.html

Nothing seems to require these packages, but that might be simply be
because they are static libraries, so there aren't any runtime
requirements.

Duh, that file was needed by software developers.

@Rudd-O Rudd-O changed the title FC37: cannot load module: failed to find valid kernel BTF FC37: cannot load module: failed to find valid kernel BTF (compiled make build-binary binary does not work) Aug 8, 2023
@bobrik
Copy link
Contributor

bobrik commented Aug 8, 2023

Looks like FC37 comes with libbpf v0.8.0, which explains the error:

[root@ed8b688f4215 ebpf_exporter]# yum info libbpf-devel.aarch64
Last metadata expiration check: 0:00:23 ago on Tue Aug  8 00:43:09 2023.
Available Packages
Name         : libbpf-devel
Epoch        : 2
Version      : 0.8.0
Release      : 2.fc37
Architecture : aarch64
Size         : 83 k
Source       : libbpf-0.8.0-2.fc37.src.rpm
Repository   : fedora
Summary      : Development files for libbpf
URL          : https://github.com/libbpf/libbpf
License      : LGPLv2 or BSD
Description  : The libbpf-devel package contains libraries header files for
             : developing applications that use libbpf

If you really want to build it yourself, you can either use the docker image from this repo or build in FC38.

@bobrik bobrik mentioned this issue Aug 8, 2023
Merged
@bobrik
Copy link
Contributor

bobrik commented Aug 8, 2023

I opened #247 to make this sort of failure scenario obvious.

@bobrik bobrik closed this as completed Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bobrik @Rudd-O