IPv6 support / dual stack support

[unixfox]

Currently, when running kilo in full mode (not the addon mode) only IPv4 addresses get assigned to the running pods.

Could a dual stack support be added to the project so that pods get IPv6 addresses and IPv4 addresses at the same time just like calico already does it with its dual stack support: https://docs.projectcalico.org/networking/dual-stack?

[squat]

Yes, amazing suggestion 🎉 this should also be quite easy :))

[unixfox]

Yes, amazing suggestion 🎉 this should also be quite easy :))

Oh that's interesting. What do you mean by quite easy? Is wireguard already doing all the work for providing IPv6 subnets?

[unixfox]

@squat I'm not a Golang developer but I'm interested to learn it in order to add the IPv6 support on kilo.

Would you mind giving me some advices on how to proceed? Is it straightforward and most of the code can easily be adapted to IPv4 & IPv6 at the same time? What are the files that are handling the IPv4 subnets in kilo?

[unixfox]

@squat Could you at least tell me if you don't have any tips to give me?

[squat]

Hi! Sorry, this issue fell off of my radar!
I got pretty deep into POCing this feature since I thought it would be simple but hit a roadblock.

Here is what I found:

1. Adding dual stack to Kilo is mostly a question of accepting a slice of subnets where one pod subnet is expected
2. This change is not so hard.
3. The one tricky place is in the Routes() method of the Topology struct, which generated all the routes a node needs for a given topology
4. The tricky thing is that we need a different physical interface for every subnet; in other words, right now, Kilo expects to have 1 physical interface for node-node traffic within a location; if we have dual stack, then we need one interface for IPv4 and one for IPv6
5. If we are encapsulating traffic, then we need to use an encapsulation type that supports IPv6; Kilo uses IPIP, which does not support IPv6
6. We could start by setting the restriction: if you want dual stack, then we don't support encapsulation
7. WireGuard supports IPv6 over IPv4 and vice versa, so running in full-mesh is also pretty easy
8. The simplest starting point would be to say: we support dual stack but only in full mesh topology

Ok, so a potential plan going forwards:

1. Dual stack but only in full mesh
2. Dual stack but only without encapsulation
3. Dual stack with any topology and any encapsulation

WDYT?

[unixfox]

I'm not an expert in network tunneling, but it seems like IP6IP6 exist as an alternative of IPIP but for IPv6.
From I can see there are two available encapsulations in kilo, IPIP and flannel. Flannel doesn't support IPv6, so no need to support dual stack in add-on mode.

If it's possible to have a dual stack encapsulation, IPv4 with IPIP and IPv6 with IP6IP6, then I think that's the best plan.
But if it's not possible then full mesh seems like another good option because as you said WireGuard support natively IPv4 and IPv6.

[squat]

Great, then let's see about first getting inter node encapsulation with IPv6inipv6 working. That would be the first important step and could be a standalone PR 👍

Btw, the three steps I listed were not meant to be alternatives but rather a plan for progressively implementing full support 💪

Do you think you would be interested in trying to implement the new encapsulation?

[unixfox]

You are right let's implement this progressively. Dual stack with only a full mesh at first is still great.
I can try to implement the IP6IP6 encapsulation but this will take a long time due to my zero knowledge in Golang. Anyway I'll see what I can do and if I succeed to make it work I'll make a PR.

[unixfox]

@squat Hello, I haven't really had the time to learn Golang yet, but I found that calico is using the BIRD protocol instead of IPIP for encapsulation.

Maybe kilo could start using it if it's too difficult to use IP6IP6?

[squat]

Hi :) AFAIK BIRD is a daemon for BGP; in other words, it's a daemon for announcing and configuring routes between different nodes. Calico can operate in BIRD mode, which means that all IP packets are sent un-encapsulated on the wire according to the routes that are discovered and configured using BIRD. Operating Kilo in this mode would be equivalent to step 2 of our play: Dual stack but only without encapsulation. In our case, we don't really need BIRD (yet) because all of Kilo's routes can be determined statically.

My understanding is that Calico has plugable encapsulation backends, including IPIP, VXLAN, WireGuard, and others. I would be curious to take a closer look and see what encapsulation they default to for dual-stack, or if this is only supported in BIRD mode.

[unixfox]

Hi :) AFAIK BIRD is a daemon for BGP; in other words, it's a daemon for announcing and configuring routes between different nodes. Calico can operate in BIRD mode, which means that all IP packets are sent un-encapsulated on the wire according to the routes that are discovered and configured using BIRD. Operating Kilo in this mode would be equivalent to step 2 of our play: Dual stack but only without encapsulation. In our case, we don't really need BIRD (yet) because all of Kilo's routes can be determined statically.

My understanding is that Calico has plugable encapsulation backends, including IPIP, VXLAN, WireGuard, and others. I would be curious to take a closer look and see what encapsulation they default to for dual-stack, or if this is only supported in BIRD mode.

Thank you for the reply!

I'm not really sure, but it seems like they don't support any encapsulation for IPv6, see: https://github.com/projectcalico/libcalico-go/issues/996
So if I understand correctly they use BIRD for announcing pod IPv6 subnets and announcing the IPv6 routes.

And wireguard is not supported in IPv6 for Calico.

[squat]

👍 very good find! I wasn't aware of this.

Looks like between our steps 2 and 3 we will likely need to spend some time working on a new encapsulation method, maybe using vxlan.

[withinboredom]

@squat with k8s 1.21 natively supporting dual-stack in beta, I was just curious how far along you got? It'd probably be some great publicity if you were one of only a few CNI providers to support this feature.

[unixfox]

@squat with k8s 1.21 natively supporting dual-stack in beta, I was just curious how far along you got? It'd probably be some great publicity if you were one of only a few CNI providers to support this feature.

Even better, it's scheduled to be in stable for Kubernetes 1.23: kubernetes/website#30538

[tetricky]

I'm curious where things are with this, now k8s has reached stable dual stack. I've been running kilo for a year over ipv4. I need to rebuild my cluster for IPv6. Kilo appears to have stalled on development, and I an considering calico to avoid future feature and maintenance problems. The concern also being that the last release is somewhat adrift of the subsequent k8s development.

[unixfox]

I'm curious where things are with this, now k8s has reached stable dual stack. I've been running kilo for a year over ipv4. I need to rebuild my cluster for IPv6. Kilo appears to have stalled on development, and I an considering calico to avoid future feature and maintenance problems. The concern also being that the last release is somewhat adrift of the subsequent k8s development.

Calico doesn't support ipv6 in wireguard though: projectcalico/calico#4492

[unixfox]

By the way, I just discovered an alternative to kilo which support IPv6: https://www.talos.dev/v1.0/guides/kubespan/

It doesn't have all the feature sets that kilo has but it provides the basic idea of interconnecting servers from different cloud providers through a wireguard link.

[squat]

That's very cool 👍 KubeSpan is able to do this because it always builds a full WireGuard mesh. This is the easiest approach because it simplifies lots of things about the cluster topology.

The goal of the current dual-stack WIP for Kilo is also to only support full-mesh topologies. After this will come non-full-mesh topologies but without encapsulation.

[unixfox]

Just discovered that now flannel support wireguard in dual stack: https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard.

That's pretty neat! It's like KubeSpan, it creates a full mesh. But at least flannel is very easy to install compared to kubespan.

[unixfox]

Calico just released their support for Wireguard on IPv6: projectcalico/calico#4492 (comment)