Location of the kernel headers #397

ismail · 2016-02-18T07:05:34Z

I am trying to run bcc on openSUSE Linux and there is a problem with the kernel header path it seems:

> ./hello_world.py
In file included from <built-in>:316:
<command line>:2:10: fatal error: './include/linux/kconfig.h' file not found
#include "./include/linux/kconfig.h"
         ^
1 error generated.
Traceback (most recent call last):
  File "./hello_world.py", line 11, in <module>
    BPF(text='void kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); }').trace_print()
  File "/usr/local/lib/python2.7/site-packages/bcc/__init__.py", line 126, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

The file exists at /usr/src/linux/include/linux/kconfig.h , I am wondering if there is a special setup I should be doing before using bcc.

The text was updated successfully, but these errors were encountered:

drzaeus77 · 2016-02-18T07:10:17Z

On most systems, I find the kernel headers in /lib/modules/$(uname -r)/build, and therefore the bcc sources (specifically kbuild_helper.{cc,h}) are implemented to search at that location. Usually the build file is a symlink to the architecture specific location. In SUSE, is there a kernel-headers package that can be installed in addition to the kernel image?

ismail · 2016-02-18T07:38:41Z

There is kernel-default-devel which provides /lib/modules/$(uname -r)/build but the header structure is very different than what I expect:

> ls -al /lib/modules/$(uname -r)/build/include/
total 52
drwxr-xr-x   4 root root  4096 Feb  5 19:03 ./
drwxr-xr-x   5 root root  4096 Feb  8 10:19 ../
drwxr-xr-x 743 root root 36864 Feb  5 19:03 config/
drwxr-xr-x   3 root root  4096 Feb  5 19:03 generated/

ismail · 2016-02-18T07:51:44Z

This is obviously a SUSE related problem, I am closing the bug. Will try to ask our kernel devs.

drzaeus77 · 2016-02-18T07:53:44Z

:) I imagine there is a reasonable answer. My hope is that you aren't required to install the full kernel sources. Let us know when you find out.

lcp · 2016-02-18T10:17:16Z

I have a quick fix.
https://build.opensuse.org/package/view_file/home:gary_lin:iovisor/bcc/bcc-suse-kernel-headers.patch?expand=1

ismail · 2016-02-18T10:50:06Z

@lcp thats awesome! Maybe it can be upstreamed :-)

lcp · 2016-02-19T03:41:54Z

In SLE/openSUSE, the kernel headers are actually in two packages: kernel-default-devel and kernel-devel. kernel-default-devel actually requires kernel-default, so those two packages will be installed altogether. The headers in /lib/modules/$(version)/build are from kernel-default-devel and those in /lib/modules/$(version)/source are from kernel-devel, and build and source link to different directories:

build -> /usr/src/linux-4.4.0-3-obj/x86_64/default
source -> /usr/src/linux-4.4.0-3

The headers are separated on purpose. From the descriptions of the packages, kernel-default-devel contains the files "necessary for building kernel modules against the default flavor of the kernel" while kernel-devel contains the files "required for development of external kernel modules."

My patch is to add the "source" paths so that clang can find the corresponding headers. I found that fedora links source to build, so the patch probably won't affect them.

@drzaeus77 Do you think it's fine to upstream my patch or just leave it in SUSE?

drzaeus77 · 2016-02-19T04:06:08Z

The descriptions of those two packages look very similar to me :) Unfortunately, the other two datapoints in my informal survey (arch linux and ubuntu) show that the presence of the source symlink is not common enough. Both of those rely solely on the build path. Since this is arch specific though, it probably makes sense as an overridable compile flag. We can expose it through cmake.

lcp · 2016-02-22T03:52:07Z

They're similar but still different. There are several flavors of kernels in SLE/openSUSE, e.g. default and vanilla for x86_64, and the kernel-"flavor"-devel packages contain the headers different from the flavors, so the "generated" headers are in kernel-"flavor"-devel, and the rest are in kernel-devel. Thus, in our case, we need both "build" and "source" in the include paths.

I thought "build" and "source" are created by "make modules_install" and other distro would keep them, but it seems not true :-(

finelli · 2016-03-07T13:04:09Z

Hello, trying to build bcc on a Debian / Jessie, and getting the same issue:

In file included from <built-in>:316:
<command line>:2:10: fatal error: './include/linux/kconfig.h' file not found
#include "./include/linux/kconfig.h"

I tried linking the missing directory, but then it breaks with a different error

/virtual/include/bcc/helpers.h:20:10: fatal error: 'uapi/linux/bpf.h' file not found

Has anybody succeeded in compiling it under Jessie ? I got the same errors if I try to run debian/rules binary, is there something that I am missing ?

drzaeus77 · 2016-03-07T15:37:29Z

No, I haven't tried compiling under jessie.

Let's assume for a moment that the symlink was properly done, and so the first error is worked around. The second error seems to suggest that the kernel sources don't include the bpf headers. Which kernel version are you symlinking to?

finelli · 2016-03-07T16:57:36Z

Hello, thank for your answer. That'what I did:

# cd /lib/modules/$(uname -r)/build
# ln -s /usr/src/linux-headers-4.3.0-0.bpo.1-common/include/linux linux

The files are present:

# locate bpf.h | grep uapi
/usr/src/linux-headers-4.3.0-0.bpo.1-common/include/uapi/linux/bpf.h
/usr/src/linux-headers-4.3.0-0.bpo.1-common/include/uapi/linux/netfilter/xt_bpf.h
/usr/src/linux-headers-4.3.0-0.bpo.1-common/include/uapi/linux/tc_act/tc_bpf.h
# uname -a
Linux kygerlitor.garibaldi.biodec.com 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.5-1~bpo8+1 (2016-02-23) x86_64 GNU/Linux

is the problem that the headers are under linux-headers-4.3.0-0.bpo.1-common and not under linux-headers-4.3.0-0.bpo.1-amd64 (just wildguessing) ?

drzaeus77 · 2016-03-07T17:06:12Z

No, I don't think -amd64 will change anything, the output of uname -r is all that matters.

Is there also a symlink from build to linux? Can you share the output of ls -l /lib/modules/$(uname -r)/? When bcc runs, it will set up itself to compile the virtual main.c as if it were a kernel module, by setting the path to /lib/modules/$(uname -r)/build and then doing things like -Iinclude and -I./include/uapi.

finelli · 2016-03-07T17:39:22Z

this is the output of /lib/modules/4.3.0-0.bpo.1-amd64

# ls -l
total 3836
lrwxrwxrwx  1 root root     42 Feb 24 00:56 build -> /usr/src/linux-headers-4.3.0-0.bpo.1-amd64
drwxr-xr-x 11 root root    109 Mar  6 23:50 kernel
-rw-r--r--  1 root root 937795 Mar  7 12:32 modules.alias
-rw-r--r--  1 root root 903464 Mar  7 12:32 modules.alias.bin
-rw-r--r--  1 root root   3655 Feb 24 00:56 modules.builtin
-rw-r--r--  1 root root   5066 Mar  7 12:32 modules.builtin.bin
-rw-r--r--  1 root root 364752 Mar  7 12:32 modules.dep
-rw-r--r--  1 root root 506651 Mar  7 12:32 modules.dep.bin
-rw-r--r--  1 root root    402 Mar  7 12:32 modules.devname
-rw-r--r--  1 root root 123638 Feb 24 00:56 modules.order
-rw-r--r--  1 root root    181 Mar  7 12:32 modules.softdep
-rw-r--r--  1 root root 472516 Mar  7 12:32 modules.symbols
-rw-r--r--  1 root root 584164 Mar  7 12:32 modules.symbols.bin
lrwxrwxrwx  1 root root     43 Feb 24 00:56 source -> /usr/src/linux-headers-4.3.0-0.bpo.1-common
drwxr-xr-x  3 root root     17 Mar  7 12:32 updates

drzaeus77 · 2016-03-07T17:44:50Z

Ah, interesting. The symlinks are different, so yes it seems that the headers picked up will be different, unless /usr/src/linux-headers-4.3.0-0.bpo.1-amd64 and /usr/src/linux-headers-4.3.0-0.bpo.1-common are linked together. Where did the build symlink come from (as in which package, try dpkg -S /lib/modules/linux-headers-4.3.0-0.bpo.1-amd64/build), are the headers missing from there, and can you possibly switch build to point to source? I don't know what the "right" thing to do on a jessie system would be.

4ast · 2016-03-07T17:55:36Z

build != source for out-of-source build. Currently I workaround that by symlinking stuff manually. The better fix would be on bcc side to give both /lib/../source and /lib/../build directories to -I

finelli · 2016-03-07T17:59:11Z

# dpkg -S /lib/modules/linux-headers-4.3.0-0.bpo.1-amd64
dpkg-query: no path found matching pattern /lib/modules/linux-headers-4.3.0-0.bpo.1-amd64

drzaeus77 · 2016-03-07T18:06:28Z

@4ast which directory would have been used for a kmod build (M=??), that was supposed to be the only requirement. Having two M= doesn't make sense for kmod, and nor does it make sense for us. There should be a 'right way' without needing ./sources.

4ast · 2016-03-07T18:09:27Z

it's one M=. I think ko's makefile is looking for 'source' link inside 'build' dir.

drzaeus77 · 2016-03-07T18:25:29Z

That seems unlikely, on my system there is no ./source link and modules build just fine. I haven't read any makefile sources to confirm that, though...

drzaeus77 · 2016-03-07T18:30:56Z

@finelli sorry, I asked for the wrong command. It should have been dpkg -S /lib/modules/4.3.0-0.bpo.1-amd64/build. For good measure, it would be useful to also see dpkg -S /usr/src/linux-headers-4.3.0-0.bpo.1-amd64 and dpkg -S /usr/src/linux-headers-4.3.0-0.bpo.1-common.

4ast · 2016-03-07T18:33:01Z

Makefile:1132

finelli · 2016-03-07T22:18:53Z

As follows:

# dpkg -S /lib/modules/4.3.0-0.bpo.1-amd64/build
linux-headers-4.3.0-0.bpo.1-amd64: /lib/modules/4.3.0-0.bpo.1-amd64/build
# dpkg -S /usr/src/linux-headers-4.3.0-0.bpo.1-amd64
linux-headers-4.3.0-0.bpo.1-amd64: /usr/src/linux-headers-4.3.0-0.bpo.1-amd64
# dpkg -S /usr/src/linux-headers-4.3.0-0.bpo.1-common
linux-headers-4.3.0-0.bpo.1-common: /usr/src/linux-headers-4.3.0-0.bpo.1-common

Some linux distributions structure the /lib/modules directories differently, causing complexities. Add cmake overrides to be able to compile different behavior. If your distro sets up `/lib/modules/$(uname -r)/{source,build}` with header files split between the two (debian does this), then add -DBCC_KERNEL_HAS_SOURCE_DIR=1 to the cmake command line. If your distro just has something other than build/, but things are still in one subdirectory, then add -DBCC_KERNEL_MODULES_SUFFIX=foo to the cmake command line. Also, fix one implicit declaration warning introduced by the new bpf_get_stackid() helper. Fixes: #397 Signed-off-by: Brenden Blanco <[email protected]>

drzaeus77 · 2016-03-08T07:46:43Z

The mentioned pull request was tested on debian but not suse. Please let me know how the fix looks on your respective systems.

lcp · 2016-03-08T08:37:37Z

Adding "-DBCC_KERNEL_HAS_SOURCE_DIR=1" to cmake works for me.

finelli · 2016-03-08T10:29:57Z

Confirmed: adding "-DBCC_KERNEL_HAS_SOURCE_DIR=1" to cmake works for me too (not tried the debian/rules part, not sure how to add that option to the build)

finelli · 2016-03-08T10:44:00Z

Well, a little too fast: the hello_world example runs, but other examples are broken:

# ../examples/tracing/task_switch.py 
In file included from /home/m/software/bcc/examples/tracing/task_switch.c:2:
In file included from include/linux/sched.h:32:
include/linux/cputime.h:4:10: fatal error: 'asm/cputime.h' file not found
#include <asm/cputime.h>
         ^
1 error generated.

drzaeus77 · 2016-03-08T16:33:53Z

Ok, the pull request is updated, hopefully the new version includes all of the right bits.

finelli · 2016-03-16T10:39:36Z

I confirm that no example gives error. Thanks.

mailinglists35 · 2016-04-17T01:06:35Z

@finelli how did you solve the issue? I am running on jessie with binary packages from http:https://52.8.15.63/apt/ as instructed in INSTALL.md - did you manually build instead?

libbcc:amd64/trusty 0.1.8-1 uptodate
libbcc-examples:amd64/trusty 0.1.8-1 uptodate
python-bcc:all/trusty 0.1.8-1 uptodate

finelli · 2016-04-18T10:05:08Z

Yes, I built them, I am not using the above packages

ismail · 2016-08-05T13:04:18Z

This fails for me again with git master (on openSUSE Tumbleweed as my original bug report):

seferogullari /usr/local/share/bcc/tools > sudo ./ext4slower
In file included from :317:
:5:10: fatal error: './include/linux/kconfig.h' file not found
#include "./include/linux/kconfig.h"
^
1 error generated.
Traceback (most recent call last):
File "./ext4slower", line 305, in
b = BPF(text=bpf_text)
File "/usr/local/lib/python2.7/site-packages/bcc/init.py", line 166, in init
raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

ismail closed this as completed Feb 18, 2016

ismail reopened this Feb 18, 2016

drzaeus77 mentioned this issue Mar 8, 2016

Add option to change kernel build search paths #430

Merged

4ast closed this as completed in #430 Mar 8, 2016

drzaeus77 mentioned this issue Aug 16, 2016

update debian packaging #657

Merged

finelli mentioned this issue Aug 29, 2016

Fix debian package build with docker container #679

Closed

alban mentioned this issue Oct 10, 2016

Run-time detection of kernel headers location for containers (follow up of #397) #743

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Location of the kernel headers #397

Location of the kernel headers #397

ismail commented Feb 18, 2016

drzaeus77 commented Feb 18, 2016

ismail commented Feb 18, 2016

ismail commented Feb 18, 2016

drzaeus77 commented Feb 18, 2016

lcp commented Feb 18, 2016

ismail commented Feb 18, 2016

lcp commented Feb 19, 2016

drzaeus77 commented Feb 19, 2016

lcp commented Feb 22, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 8, 2016

lcp commented Mar 8, 2016

finelli commented Mar 8, 2016

finelli commented Mar 8, 2016

drzaeus77 commented Mar 8, 2016

finelli commented Mar 16, 2016

mailinglists35 commented Apr 17, 2016

finelli commented Apr 18, 2016

ismail commented Aug 5, 2016

Location of the kernel headers #397

Location of the kernel headers #397

Comments

ismail commented Feb 18, 2016

drzaeus77 commented Feb 18, 2016

ismail commented Feb 18, 2016

ismail commented Feb 18, 2016

drzaeus77 commented Feb 18, 2016

lcp commented Feb 18, 2016

ismail commented Feb 18, 2016

lcp commented Feb 19, 2016

drzaeus77 commented Feb 19, 2016

lcp commented Feb 22, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

drzaeus77 commented Mar 7, 2016

4ast commented Mar 7, 2016

finelli commented Mar 7, 2016

drzaeus77 commented Mar 8, 2016

lcp commented Mar 8, 2016

finelli commented Mar 8, 2016

finelli commented Mar 8, 2016

drzaeus77 commented Mar 8, 2016

finelli commented Mar 16, 2016

mailinglists35 commented Apr 17, 2016

finelli commented Apr 18, 2016

ismail commented Aug 5, 2016