14 tests fail on ubuntu 20.04

[DiffeoInvariant]

I've compiled from source as described in INSTALL.md, and installed netperf for tests. The build itself succeeds (with some mostly-harmless warnings from LLVM), but make test gives the following summary:

Running tests...
Test project /home/diffeoinvariant/datacenter-profiling/bcc/build
      Start  1: style-check
 1/40 Test  #1: style-check ......................   Passed    0.01 sec
      Start  2: c_test_static
 2/40 Test  #2: c_test_static ....................   Passed    0.28 sec
      Start  3: test_libbcc
 3/40 Test  #3: test_libbcc ......................***Failed   18.37 sec
      Start  4: py_test_stat1_b
 4/40 Test  #4: py_test_stat1_b ..................***Failed    0.19 sec
      Start  5: py_test_bpf_log
 5/40 Test  #5: py_test_bpf_log ..................***Failed    0.09 sec
      Start  6: py_test_stat1_c
 6/40 Test  #6: py_test_stat1_c ..................***Failed    0.18 sec
      Start  7: py_test_xlate1_c
 7/40 Test  #7: py_test_xlate1_c .................***Failed    0.17 sec
      Start  8: py_test_call1
 8/40 Test  #8: py_test_call1 ....................***Failed    0.20 sec
      Start  9: py_test_trace1
 9/40 Test  #9: py_test_trace1 ...................   Passed    0.65 sec
      Start 10: py_test_trace2
10/40 Test #10: py_test_trace2 ...................   Passed    2.18 sec
      Start 11: py_test_trace3_c
11/40 Test #11: py_test_trace3_c .................***Failed    0.07 sec
      Start 12: py_test_trace4
12/40 Test #12: py_test_trace4 ...................   Passed    1.93 sec
      Start 13: py_test_trace_maxactive
13/40 Test #13: py_test_trace_maxactive ..........   Passed    1.22 sec
      Start 14: py_test_probe_count
14/40 Test #14: py_test_probe_count ..............   Passed    6.69 sec
      Start 15: py_test_debuginfo
15/40 Test #15: py_test_debuginfo ................***Failed    0.07 sec
      Start 16: py_test_brb
16/40 Test #16: py_test_brb ......................***Failed    0.02 sec
      Start 17: py_test_brb2
17/40 Test #17: py_test_brb2 .....................***Failed    0.04 sec
      Start 18: py_test_clang
18/40 Test #18: py_test_clang ....................   Passed   80.33 sec
      Start 19: py_test_histogram
19/40 Test #19: py_test_histogram ................   Passed    3.75 sec
      Start 20: py_array
20/40 Test #20: py_array .........................***Failed    2.30 sec
      Start 21: py_uprobes
21/40 Test #21: py_uprobes .......................   Passed    6.74 sec
      Start 22: py_uprobes_2
22/40 Test #22: py_uprobes_2 .....................   Passed    0.38 sec
      Start 23: py_test_stackid
23/40 Test #23: py_test_stackid ..................***Failed    0.05 sec
      Start 24: py_test_tracepoint
24/40 Test #24: py_test_tracepoint ...............   Passed    2.60 sec
      Start 25: py_test_perf_event
25/40 Test #25: py_test_perf_event ...............   Passed    0.57 sec
      Start 26: py_test_utils
26/40 Test #26: py_test_utils ....................   Passed    0.05 sec
      Start 27: py_test_percpu
27/40 Test #27: py_test_percpu ...................   Passed    2.29 sec
      Start 28: py_test_dump_func
28/40 Test #28: py_test_dump_func ................   Passed    0.18 sec
      Start 29: py_test_disassembler
29/40 Test #29: py_test_disassembler .............   Passed    0.29 sec
      Start 30: py_test_tools_smoke
30/40 Test #30: py_test_tools_smoke ..............***Failed    0.03 sec
      Start 31: py_test_tools_memleak
31/40 Test #31: py_test_tools_memleak ............   Passed   25.32 sec
      Start 32: py_test_usdt
32/40 Test #32: py_test_usdt .....................   Passed    4.41 sec
      Start 33: py_test_usdt2
33/40 Test #33: py_test_usdt2 ....................   Passed    3.29 sec
      Start 34: py_test_usdt3
34/40 Test #34: py_test_usdt3 ....................   Passed    1.32 sec
      Start 35: py_test_license
35/40 Test #35: py_test_license ..................   Passed    3.16 sec
      Start 36: py_test_free_bcc_memory
36/40 Test #36: py_test_free_bcc_memory ..........   Passed    0.34 sec
      Start 37: py_test_rlimit
37/40 Test #37: py_test_rlimit ...................   Passed    0.51 sec
      Start 38: py_test_lpm_trie
38/40 Test #38: py_test_lpm_trie .................***Failed    0.05 sec
      Start 39: py_ringbuf
39/40 Test #39: py_ringbuf .......................   Passed    0.05 sec
      Start 40: py_queuestack
40/40 Test #40: py_queuestack ....................   Passed    0.51 sec

65% tests passed, 14 tests failed out of 40

Total Test time (real) = 170.89 sec

The following tests FAILED:
	  3 - test_libbcc (Failed)
	  4 - py_test_stat1_b (Failed)
	  5 - py_test_bpf_log (Failed)
	  6 - py_test_stat1_c (Failed)
	  7 - py_test_xlate1_c (Failed)
	  8 - py_test_call1 (Failed)
	 11 - py_test_trace3_c (Failed)
	 15 - py_test_debuginfo (Failed)
	 16 - py_test_brb (Failed)
	 17 - py_test_brb2 (Failed)
	 20 - py_array (Failed)
	 23 - py_test_stackid (Failed)
	 30 - py_test_tools_smoke (Failed)
	 38 - py_test_lpm_trie (Failed)

For what it's worth, the affect of whatever is causing these tests to fail is for none of the tools to work, and presumably bcc itself; for example,

diffeoinvariant@DiffeoInvariant ~/d/b/build (master)> /usr/share/bcc/tools/profile -f
could not open bpf map: stack_traces, error: Operation not permitted
Traceback (most recent call last):
  File "/usr/share/bcc/tools/profile", line 277, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 364, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

I've seen this exception arise in other issues that may have been related to not having the latest linux headers or a new enough kernel, so I made sure everything was up-to-date and that I had all the relevant headers installed before compiling.

Anyone have any ideas about what could be going wrong?

[willfindlay]

Are you running the failing tools with root privileges? Do you have the Lockdown LSM enabled?

[DiffeoInvariant]

Running the tools as root appears to fix the issues there, so thanks for that! FWIW, I do not have the Lockdown LSM enabled (I'm pretty sure, at least; cat /sys/kernel/security/lockdown gives output of [none] integrity confidentiality). The tests still fail though (and sudo make test makes no difference), and I'm not sure what's going on there. Since the tools do appear to work though, maybe it's not an issue.

[willfindlay]

I know some of the tests can be flaky (test_tools_smoke.py is particularly flaky iirc). For other tests, it may just be compatibility issues with whatever version of the Linux kernel, libbpf, etc. that you are running.

[ryandgoulding]

FWIW, I experienced the following similar failures:

sudo make test

...

80% tests passed, 9 tests failed out of 45

Total Test time (real) = 511.58 sec

The following tests FAILED:
	  3 - test_libbcc (Failed)
	  4 - py_test_stat1_b (Failed)
	  6 - py_test_stat1_c (Failed)
	  7 - py_test_xlate1_c (Failed)
	  8 - py_test_call1 (Failed)
	 16 - py_test_brb (Failed)
	 23 - py_test_stackid (Failed)
	 38 - py_test_lpm_trie (Failed)
	 45 - lua_test_standalone (Failed)
Errors while running CTest
make: *** [Makefile:118: test] Error 8

To confirm, Lockdown LSM is disabled:

cat /sys/kernel/security/lockdown
[none] integrity confidentiality

Kernel Version:

uname -a
Linux ryan-VirtualBox 5.8.0-53-generic #60~20.04.1-Ubuntu SMP Thu May 6 09:52:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

@willfindlay do you know if there are there plans to eventually support Ubuntu 20.04 LTS? I am completely new to eBPF/bcc but investigating from a purely research oriented standpoint. I will just downgrade to Ubuntu 18.04 LTS for now, as that appears to be working in the bcc-test.yml GitHub Action.