docker + tests: Run tests using python3, refactor Dockerfile

In iovisor#3707, I added ubuntu-20.04 to the list of OS's bcc-test uses to run
tests. I expected that this would result in the tests running on both
18.04 and 20.04. Unfortunately this change was effectively a no-op as
the tests are run in a Docker container and the os field in bcc-test.yml
dictates the type/version of the _host_ OS, not the container's OS.

So it's not necessary to run on both, just run on 20.04. It will be
necessary to modify Dockerfile.test to use both 18.04 and 20.04. To
prepare for this, move all python tests to use python3 interpreter, as
20.04 doesn't have python2 pip readily available.

Also, refactor Dockerfile.tests a bit so that it's possible to provide
ubuntu version and shortname as build input.

This commit does not result in the docker test container working/running
both 18.04 and 20.04, rather lays groundwork for future commits to do
so.

Signed-off-by: Dave Marchevsky <[email protected]>

.github/workflows/bcc-test.yml

@@ -11,7 +11,7 @@ jobs:
  runs-on: ${{ matrix.os }}
  strategy:
  matrix:
- os: [ubuntu-18.04, ubuntu-20.04] # 18.04.3 release has 5.0.0 kernel 
+ os: [ubuntu-20.04]
  env:
  - TYPE: Debug
  PYTHON_TEST_LOGFILE: critical.log

Dockerfile.tests

@@ -1,17 +1,23 @@
-FROM ubuntu:18.04
+ARG UBUNTU_VERSION="18.04"
+FROM ubuntu:${UBUNTU_VERSION}
 
 ARG LLVM_VERSION="8"
 ENV LLVM_VERSION=$LLVM_VERSION
 
+ARG UBUNTU_SHORTNAME="bionic"
+
 RUN apt-get update && apt-get install -y curl gnupg &&\
  llvmRepository="\n\
-deb http:https://apt.llvm.org/bionic/ llvm-toolchain-bionic main\n\
-deb-src http:https://apt.llvm.org/bionic/ llvm-toolchain-bionic main\n\
-deb http:https://apt.llvm.org/bionic/ llvm-toolchain-bionic-${LLVM_VERSION} main\n\
-deb-src http:https://apt.llvm.org/bionic/ llvm-toolchain-bionic-${LLVM_VERSION} main\n" &&\
+deb http:https://apt.llvm.org/${UBUNTU_SHORTNAME}/ llvm-toolchain-${UBUNTU_SHORTNAME} main\n\
+deb-src http:https://apt.llvm.org/${UBUNTU_SHORTNAME}/ llvm-toolchain-${UBUNTU_SHORTNAME} main\n\
+deb http:https://apt.llvm.org/${UBUNTU_SHORTNAME}/ llvm-toolchain-${UBUNTU_SHORTNAME}-${LLVM_VERSION} main\n\
+deb-src http:https://apt.llvm.org/${UBUNTU_SHORTNAME}/ llvm-toolchain-${UBUNTU_SHORTNAME}-${LLVM_VERSION} main\n" &&\
  echo $llvmRepository >> /etc/apt/sources.list && \
  curl -L https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
 
+ARG DEBIAN_FRONTEND="noninteractive"
+ENV TZ="Etc/UTC"
+
 RUN apt-get update && apt-get install -y \
  util-linux \
  bison \
@@ -40,7 +46,6 @@ RUN apt-get update && apt-get install -y \
  iproute2 \
  python3 \
  python3-pip \
- python-pip \
  ethtool \
  arping \
  netperf \
@@ -50,8 +55,7 @@ RUN apt-get update && apt-get install -y \
  libtinfo5 \
  libtinfo-dev
 
-RUN pip3 install pyroute2 netaddr dnslib cachetools
-RUN pip install pyroute2==0.5.18 netaddr==0.8.0 dnslib==0.9.14 cachetools==3.1.1
+RUN pip3 install pyroute2==0.5.18 netaddr==0.8.0 dnslib==0.9.14 cachetools==3.1.1
 
 # FIXME this is faster than building from source, but it seems there is a bug
 # in probing libruby.so rather than ruby binary

src/python/CMakeLists.txt

@@ -2,7 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (the "License")
 
 if(NOT PYTHON_CMD)
- set(PYTHON_CMD "python")
+ set(PYTHON_CMD "python3")
 endif()
 
 if(EXISTS "/etc/debian_version")

tests/python/test_array.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_attach_perf_event.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright 2021, Athira Rajeev, IBM Corp.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_bpf_log.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_brb.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_brb2.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_call1.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_clang.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_debuginfo.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Sasha Goldshtein
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_disassembler.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Clevernet
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_dump_func.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_flags.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_free_bcc_memory.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_usdt.py
 #

tests/python/test_histogram.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_license.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) 2018 Clevernet, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_lpm_trie.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) 2017 Facebook, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_lru.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_map_batch_ops.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_map_batch_ops.py
 #

tests/python/test_map_in_map.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_map_in_map.py
 #

tests/python/test_percpu.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_perf_event.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) 2016 PLUMgrid
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_probe_count.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Suchakra Sharma <[email protected]>
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_queuestack.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_ringbuf.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_rlimit.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_usdt.py
 #

tests/python/test_shared_table.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) 2016 Facebook, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_stackid.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_stat1.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_tools_memleak.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from unittest import main, skipUnless, TestCase
 from utils import kernel_version_ge

tests/python/test_tools_smoke.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Sasha Goldshtein, 2017
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_trace2.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_trace3.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_trace4.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_trace_maxactive.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_tracepoint.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Sasha Goldshtein
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_uprobes.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_uprobes2.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_uprobe2.py
 #

tests/python/test_usdt.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_usdt.py
 #

tests/python/test_usdt2.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_usdt2.py
 #

tests/python/test_usdt3.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # USAGE: test_usdt3.py
 #

tests/python/test_utils.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) Catalysts GmbH
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/python/test_xlate1.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Copyright (c) PLUMgrid, Inc.
 # Licensed under the Apache License, Version 2.0 (the "License")
 

tests/wrapper.sh.in

@@ -8,7 +8,7 @@ name=$1; shift
 kind=$1; shift
 cmd=$1; shift
 
-PYTHONPATH=@CMAKE_BINARY_DIR@/src/python/bcc-python
+PYTHONPATH=@CMAKE_BINARY_DIR@/src/python/bcc-python3
 LD_LIBRARY_PATH=@CMAKE_BINARY_DIR@:@CMAKE_BINARY_DIR@/src/cc
 
 ns=$name