add kubernetes namespace, container name decoder

[huaizong]

when ebpf_exporter run in kubernetes, maybe we need a decoder convert pid to kubernetes namespace and kubernetes container name

it maybe implement by

• use kubernetes apiserver as example, we find the pid ** 2443 **

[root@whz-ebpf-0 ~]# ps aux | grep apiserver
root       2443  6.3 11.6 590860 451252 ?       Ssl  Apr07 111:24 kube-apiserver --allow-privileged=true --apiserver-count=1 --audit-log-maxage=30 --audit-log-maxbackup=1 --audit-log-maxsize=100 --audit-log-path=/var/log/audit/kube-apiserver-audit.log --audit-policy-file=/etc/kubernetes/audit-policy/apiserver-audit-policy.yaml --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --endpoint-reconciler-type=lease --insecure-port=0 --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP --runtime-config=admissionregistration.k8s.io/v1alpha1 --service-node-port-range=30000-32767 --storage-backend=etcd3 --advertise-address=10.0.4.15 --client-ca-file=/etc/kubernetes/ssl/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem --etcd-certfile=/etc/ssl/etcd/ssl/node-whz-ebpf-0.mstor.k8s.m.com.pem --etcd-keyfile=/etc/ssl/etcd/ssl/node-whz-ebpf-0.mstor.k8s.m.com-key.pem --etcd-servers=https://10.0.4.15:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/sa.pub --service-cluster-ip-range=10.233.0.0/18 --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
root      60599  0.0  0.0 112716   964 pts/3    S+   15:38   0:00 grep --color=auto apiserver

• parse container id from cgroup

[root@whz-ebpf-0 ~]# cat /proc/2443/cgroup | head
11:memory:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
10:blkio:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
9:hugetlb:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
8:perf_event:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
7:freezer:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
6:pids:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
5:net_prio,net_cls:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
4:devices:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
3:cpuacct,cpu:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8
2:cpuset:/kubepods/burstable/podf01e25d978dc0ba2839122a1c9fa9570/61acab8bf97fe903ea4099ff642d7adf1b67b4cce8aaeb72f52f2b36300de3d8

• parse kubernetes info label use docker inspect client

[root@whz-ebpf-0 ~]# docker inspect 61acab8bf97 | grep kubernetes.pod
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.pod.name": "kube-apiserver-whz-ebpf-0.mstor.k8s.m.com",
                "io.kubernetes.pod.namespace": "kube-system",
                "io.kubernetes.pod.uid": "f01e25d978dc0ba2839122a1c9fa9570",

• if process not run by kubernetes , we just return "unknown" namespace, "unknown" container name

[zuzzas]

@bobrik
Would you be interested in accepting something like this into the project?

There was an attempt by @huaizong, but it went unnoticed. Perhaps, you'd like to discuss some other format of a decoder?

[bobrik]

If anyone still wants to have it, please reopen (preferably with a PR).

These days we can expose cgroup (#70):

ebpf_exporter_cgroup_sched_migrations_total{cgroup="/sys/fs/cgroup/system.slice/docker-41804ad6484a1000228b0fceeeec3e5465c697492de62c3324aa466beced4e3b.scope"} 5

I think this can be joined with metrics from cadvisor, which can expose k8s pod/name/namespace and cgroup path.