PERF_COUNT_SW_BPF_OUTPUT feature unsupported

[brendangregg]

My first problem was needing Linux 4.4 or higher.

I'm now trying 4.5-rc2, but it's still not working. I'm probably messing up the builds:

/mnt/src/bcc/examples/tracing# uname -a
Linux bgregg-build-i-a353d777 4.5.0-rc2-virtual+ #1 SMP Wed Feb 10 19:16:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

/mnt/src/bcc/examples/tracing# ./trace_perf_output.py 
PERF_COUNT_SW_BPF_OUTPUT feature unsupported

/mnt/src/bcc/examples/tracing# strace -feperf_event_open ./trace_perf_output.py 
syscall_321(0, 0x7ffe7e95b7f0, 0x30, 0, 0x4, 0x7ffe7e95b8f0) = 0x3
syscall_321(0, 0x7ffe7e95b7f0, 0x30, 0x8, 0x2, 0x280) = 0x4
syscall_321(0x5, 0x7ffe7e95c7a0, 0x30, 0x40500, 0, 0x2) = 0x5
perf_event_open(0x7ffe7e95c500, -1, 0, -1, 0x8 /* PERF_FLAG_??? */) = 8
PERF_COUNT_SW_BPF_OUTPUT feature unsupported

Other ways to debug this?

I did rebuild bcc, and even copied over the new /usr/include/linux/perf_event.h (containing PERF_COUNT_SW_BPF_OUTPUT), which hadn't happened automatically (even after "make headers_install").

[drzaeus77]

You're definitely digging in the right direction. This is the one case where kernel headers are required for the libbpf.c build, which is where that error is coming from. The runtime clang/bpf portion of the build should work just fine, since that happens from a /lib/modules/$(uname -r)/build context.

I don't actually know the right way to ship this support, unless we actually steal the value from the kernel header file and do a runtime feature detection. I don't think we should reasonably ask users who want this feature bit to make headers_install (and I think that should have worked for you).

[brendangregg]

Fixed:

/mnt/src/bcc# diff -u src/cc/libbpf.c.orig src/cc/libbpf.c
--- src/cc/libbpf.c.orig    2016-02-10 20:21:07.545578507 +0000
+++ src/cc/libbpf.c 2016-02-10 20:22:51.031989091 +0000
@@ -349,7 +349,7 @@
 }

 void * bpf_open_perf_buffer(perf_reader_raw_cb raw_cb, void *cb_cookie, int pid, int cpu) {
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,4,0)
+#if 1
   int pfd;
   struct perf_event_attr attr = {};
   struct perf_reader *reader = NULL;

/mnt/src/bcc/examples/tracing# ./trace_perf_output.py 
Tracing sys_write, try `dd if=/dev/zero of=/dev/null`
Tracing... Hit Ctrl-C to end.
[6] 2616163.617931: 12345678
[2] 2616163.617929: 12345678
[1] 2616163.618891: 12345678
[0] 2616163.604057: 12345678
[5] 2616163.830302: 12345678
[4] 2616163.973848: 12345678
[7] 2616163.845879: 12345678

[brendangregg]

Maybe I'm still messing up my kernel build, and some version file has < 4.4, breaking the bcc build. Certainly seems to be >= 4.4:

/mnt/src/bcc# uname -a
Linux bgregg-build-i-a353d777 4.5.0-rc2-virtual+ #1 SMP Wed Feb 10 19:16:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[drzaeus77]

In the previous case, you probably had a mix of perf_event.h from new kernel and version.h from old kernel. It's still hacky. Should I just steal the value of PERF_COUNT_SW_BPF_OUTPUT and stick it in bcc somewhere?

[brendangregg]

/mnt/src/bcc# cat /usr/include/linux/version.h 
#define LINUX_VERSION_CODE 199947
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

Let me just, er, urm, there should be a tool for this, but..

/mnt/src/bcc# perl -e 'print ((3 << 16) + (13 << 8) + 0)'
199936

Ok, it's roughly 3.13 vintage. This file is not getting updated.

[drzaeus77]

You must be on Ubuntu 14.04. They'll uprev linux-libc-dev to 4.4 in 16.04

[brendangregg]

Yes, Ubuntu 14.04. So none of these update version.h?

make modules_install
make install
make headers_install

[drzaeus77]

It may depend on what path you give to make headers_install (/usr/local?) and what the CMAKE_* variables are set to for include path. Regardless, I don't think it is necessary to put other users through this same pain to get this feature. I'll make it runtime.

[4ast]

Instead of
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,X,0)
everywhere in .h and libbpf.c
we do a build time check and pass it into cmake flags ?
Is that what you meant?

On Wed, Feb 10, 2016 at 9:49 PM, Brenden Blanco [email protected]
wrote:

It may depend on what path you give to make headers_install (/usr/local?)
and what the CMAKE_* variables are set to for include path. Regardless, I
don't think it is necessary to put other users through this same pain to
get this feature. I'll make it runtime.

—
Reply to this email directly or view it on GitHub
#363 (comment).

[drzaeus77]

No, I mean using uname() at runtime like is done in b_frontend_action.cc for PROG_ARRAY feature.

[4ast]

uname won't work.
we have 4.0 with bpf bits backported.
I'm hearing other distros have it backported as well.
Using version is not going to work.

On Thu, Feb 11, 2016 at 10:11 AM, Brenden Blanco [email protected]
wrote:

No, I mean using uname() at runtime like is done in b_frontend_action.cc
for PROG_ARRAY feature.

—
Reply to this email directly or view it on GitHub
#363 (comment).

[drzaeus77]

For this use case, how about just try the syscall and return the failure. Maybe with a helper printf to hint to the user that they may be using an older kernel if it was rejected.

[4ast]

I think cmake can detect it by trying to build a dummy.c that uses
particular enum from bpf.h

On Thu, Feb 11, 2016 at 10:17 AM, Brenden Blanco [email protected]
wrote:

For this use case, how about just try the syscall and return the failure.
Maybe with a helper printf to hint to the user that they may be using an
older kernel if it was rejected.

—
Reply to this email directly or view it on GitHub
#363 (comment).

[drzaeus77]

But cmake isn't necessarily running at the same time that the library runs, or on the same machine, or with the same kernel.

What if I:

1. build bcc
2. install new kernel and reboot
3. run bcc
4. fail

Why do we want to fail at step 4? If we know how to support future kernels in the library intelligently, shouldn't we do so in order to reduce forced recompilations?

[4ast]

On Thu, Feb 11, 2016 at 10:27 AM, Brenden Blanco [email protected]
wrote:

But cmake isn't necessarily running at the same time that the library
runs, or on the same machine, or with the same kernel.

What if I:

1. build bcc
2. install new kernel and reboot
3. run bcc
4. fail

Why do we want to fail at step 4? If we know how to support future kernels
in the library intelligently, shouldn't we do so in order to reduce forced
recompilations?

I think doing syscall for all possible features at run-time at the
beginning of every py script is not going to be efficient.
Due to steps 1 and 2 the libbcc should have all features compiled-in.
If syscall fails at run-time we can just print error.
so yes, I guess we should just kill #if from sources and improve run-time
error reports.

[drzaeus77]

@brendangregg please close if this works for you

[brendangregg]

Yes, thanks, works.