Support running kg as a peer

[SerialVelocity]

Hi Squat!

It would be great if it was possible to run kg as a peer. Basically, I want a process watching kubernetes for updates and auto-updating my wireguard interface with the new routes.

I've hacked it into the codebase here:
master...SerialVelocity:kg-peer
but it isn't complete or very pretty code!

[squat]

Hm I quite like this idea. This would mean that any time you want a non-k8s-node to join a cluster, the only thing you have to do is run kg, rather than make the interfaces and register routes etc yourself.

I think the one missing piece from your PoC is that the Kubernetes nodes need to be able to discover your local host somehow. To do this, we could have the kg on the host to create a Peer to reflect itself. We’ll also need a way for this peer to reserve an address in the kilo subnet, or otherwise specify its own address in a different subnet. This would need to be an additional flag.

[SerialVelocity]

Yes, I added the peer manually for now and just selected the first IP (I didn't even change the /32 subnet!)

[squat]

Let me know if you’d like to PR it :) I’d be happy to iterate in it afterwards

[SerialVelocity]

Hm I quite like this idea. This would mean that any time you want a non-k8s-node to join a cluster, the only thing you have to do is run kg, rather than make the interfaces and register routes etc yourself.

You do still have to run kube-proxy otherwise you can't resolve service IPs though.

[stv0g]

You do still have to run kube-proxy otherwise you can't resolve service IPs though.

Maybe just setting a route for the service CIDR towards an in-cluster node/kube-proxy would be sufficient?

[squat]

Maybe just setting a route for the service CIDR towards an in-cluster node/kube-proxy would be sufficient?

That's very true, this will be sufficient. We can absolutely point the entire service CIDR at a single location leader. In fact, this is the same advice our docs currently give for routing service IPs from VPN peers. It will produce suboptimal routes whenever the pods in a service are scheduled in more than one location, however this might be a reasonable first iteration.

[SerialVelocity]

Yeah, that didn't work for me because when that node reboots/crashes/becomes unstable, you lose connectivity. kube-proxy is super easy to run though so it wasn't really an issue.