pwru is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities. It allows fine-grained introspection of kernel state to facilitate debugging network connectivity issues.

The following example shows where the packets of a curl request are dropped after installing an IP tables rule:

pwru requires >= 5.3 kernel to run. For --output-skb >= 5.9 kernel is required.

The following kernel configuration is required.

You can use zgrep $OPTION /proc/config.gz to validate whether option is enabled.

You can download the statically linked executable for x86_64 and amd64 from the release page.

$ pwru --help
Usage of ./pwru:
      --filter-dst-ip string      filter destination IP addr
      --filter-dst-port uint16    filter destination port
      --filter-func string        filter kernel functions to be probed by name (exact match, supports RE2 regular expression)
      --filter-mark uint32        filter skb mark
      --filter-netns uint32       filter netns inode
      --filter-proto string       filter L4 protocol (tcp, udp, icmp, icmp6)
      --filter-src-ip string      filter source IP addr
      --filter-src-port uint16    filter source port
      --kernel-btf string         specify kernel BTF file
      --kmods strings             list of kernel modules names to attach to
      --output-limit-lines uint   exit the program after the number of events has been received/printed
      --output-meta               print skb metadata
      --output-skb                print skb
      --output-stack              print stack
      --output-tuple              print L4 tuple
      --per-cpu-buffer int        per CPU buffer in bytes (default 4096)
      --timestamp string          print timestamp per skb ("current", "relative", "none") (default "none")
      --version                   show pwru version and exit

If multiple filters are specified, all of them have to match in order for a packet to be traced.

The --filter-func switch does an exact match on function names i.e. --filter-func=foo only matches foo(); for a wildcarded match, try --filter-func=".*foo.*" instead.

Docker images for pwru are published at https://hub.docker.com/r/cilium/pwru.

An example how to run pwru with Docker:

docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/tracing:/sys/kernel/debug/tracing cilium/pwru --filter-dst-ip=1.1.1.1

The following example shows how to run pwru on a given node:

NODE=node-foobar
kubectl run pwru \
    --image=cilium/pwru:latest \
    --privileged=true \
    --attach=true -i=true --tty=true --rm=true \
    --overrides='{"apiVersion":"v1","spec":{"nodeSelector":{"kubernetes.io/hostname":"'$NODE'"}, "hostNetwork": true, "hostPID": true}}' \
    -- --filter-dst-ip=1.1.1.1 --output-tuple

If you have Vagrant installed, you can run the above example with the following commands.

1. In a terminal (terminal 1), bring up the Vagrant box:

   $ vagrant up

   This will take a few minutes to download and provision the box.

2. Connect to the Vagrant box:

   $ vagrant ssh

3. Build pwru:

   $ cd /pwru
   $ make

4. Run pwru:

   $ sudo ./pwru --filter-dst-ip=1.1.1.1 --filter-dst-port=80 --filter-proto=tcp --output-tuple

5. In a new terminal (terminal 2), connect to the Vagrant box:

   $ vagrant ssh

6. In terminal 2, run curl to generate some traffic to 1.1.1.1:

   $ curl 1.1.1.1

   Observe the output of pwru in terminal 1.

7. In terminal 2, add an iptables rule to block traffic to 1.1.1.1:

   $ sudo iptables -t filter -I OUTPUT 1 -m tcp --proto tcp --dst 1.1.1.1/32 -j DROP

8. In terminal 2, run curl to generate some traffic to 1.1.1.1:

   $ curl 1.1.1.1

   Observe the output of pwru in terminal 1.

9. To clean up, press Ctrl+C to terminate pwru in terminal 1, exit both shells, and run:

   $ vagrant destroy

• Go >= 1.16
• LLVM/clang >= 1.12

make

Alternatively, you can build in the Docker container:

make release

pwru is an open source project licensed under GPLv2. Everybody is welcome to contribute. Contributors are required to follow the Contributor Covenant Code of Conduct and must adhere to the Developer Certificate of Origin by adding a Signed-off-by line to their commit messages.

The detective gopher is based on the Go gopher designed by Renee French.