Microk8s enable Cilium crashes

[AllardKrings]

Summary

If I try to enable cilium on Microk8s on a 6-node Kubernetes cluster on 6 Raspberry Pi’s it crashes and disables the whole cluster

What Should Happen Instead?

Cilium is enabled on Microk8s

Reproduction Steps

1. install microk8s on ubuntu lunar lobster on all 6 Raspberry PI 4 8Gb

2. create cluster of 6 nodes

3. enable cilium

4. kubectl get pod -n kube-system
   NAME READY STATUS RESTARTS AGE
   calico-kube-controllers-6c99c8747f-khqjk 0/1 Terminating 0 18m
   cilium-operator-bf4676d78-p8j2s 1/1 Running 0 11m
   coredns-7745f9f87f-nvphj 0/1 ContainerCreating 0 11m
   coredns-7745f9f87f-gn9jk 0/1 Terminating 0 18m
   cilium-zxsg9 0/1 CrashLoopBackOff 6 (3m25s ago) 11m
   cilium-gg67h 0/1 CrashLoopBackOff 6 (3m18s ago) 11m
   cilium-49gb8 0/1 CrashLoopBackOff 6 (3m6s ago) 11m
   cilium-q49cx 0/1 CrashLoopBackOff 6 (2m50s ago) 11m
   cilium-b6gws 0/1 CrashLoopBackOff 6 (2m43s ago) 11m
   cilium-q6r65 0/1 CrashLoopBackOff 6 (2m42s ago) 11m

Can you suggest a fix?

Are you interested in contributing with a fix?

[jerousseau]

You can try using the last cilium version : microk8s enable cilium v1.13.4
Default version "v1.10" was not working for me

[AllardKrings]

Hello jerousseau, thank you for your tried that but it only works on a single node, not in a cluster Kind regards Allard Op ma 10 jul. 2023 om 11:46 schreef jerousseau ***@***.***>

…

You can try using the last cilium version : microk8s enable cilium v1.13.4 Default version "v1.10" was not working for me — Reply to this email directly, view it on GitHub <#4060 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AET5SGICBW2FQWWIODCMYMLXPPFQDANCNFSM6AAAAAAZ4VSHPU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ktsakalozos]

With the 1.28 release (mid August) we plan to allow multi-node cilium based clusters. This work is on latest edge for now.

snap install microk8s --classic --channel=latest/edge

[AllardKrings]

That is great news! I will wait for that! Op wo 19 jul. 2023 om 10:16 schreef Konstantinos Tsakalozos < ***@***.***>

…

With the 1.28 release (mid August) we plan to allow multi-node cilium based clusters. This work is on latest edge for now. snap install microk8s --classic --channel=latest/edge — Reply to this email directly, view it on GitHub <#4060 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AET5SGPLNLGF27AYIDXVJ3TXQ6JV3ANCNFSM6AAAAAAZ4VSHPU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[KhalilSantana]

As of today, running MicroK8S 1.29.0 and Cilium v1.13.4 still fails with a CrashLoopBackoff on all arm64 nodes while the master (x86) runs fine.

$ sudo microk8s kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS             RESTARTS      AGE
kube-system   cilium-operator-7f8c9dcf9f-ghwsx   1/1     Running            1 (20m ago)   20m
kube-system   cilium-hlpcg                       1/1     Running            1 (20m ago)   20m
kube-system   coredns-864597b5fd-w2km5           1/1     Running            1 (20m ago)   20m
kube-system   hubble-relay-569fbd8b8c-rspwf      1/1     Running            0             19m
kube-system   cilium-xq566                       0/1     CrashLoopBackOff   7 (68s ago)   14m
kube-system   cilium-8w9gw                       0/1     CrashLoopBackOff   7 (31s ago)   13m

Inspecting one of the crashing pods cilium-8w9gw doesn't provide much useful information, but it seems Envoy is at fault:

$ sudo microk8s kubectl logs -n kube-system cilium-8w9gw -c cilium-agent
[...]
level=info msg="Cilium 1.13.4 4061cdfc 2023-06-14T03:37:39+00:00 go version go1.19.10 linux/arm64" subsys=daemon
level=fatal msg="Envoy: Binary \"cilium-envoy\" cannot be executed" error="signal: aborted" subsys=envoy-manager

Is it possible to toggle Envoy off to workaround this issue?