Any known problems with GCP Compute Engine

[Jonathan-Eid]

I cannot get master to worker communication if both nodes are in GCP and set to the same location, trying to get logs from worker nodes times out and the pods can't communicate with the kube apiserver. I tried putting the worker nodes in a different subnet as well and set their locations unique. When I do the latter, I can connect to the leader of that new subnet, but the same problems arise with any followers. What's going on? External AWS nodes connected to our GCP master work fine.

[squat]

Hi @Jonathan-Eid is this a GKE cluster? And are you running Kilo in add-on mode or as the only CNI for the cluster? If it's add-on mode then this is probably a limitation of the compatibility with the other CNI and you'll need to run Kilo in full-mesh mode to get full connectivity.

[Jonathan-Eid]

We're not running on GKE, we setup vanilla K8s on fresh GCP Instances with Kilo as the main CNI, we're not running it on add-on mode. Please let me know if there's any more information I should send.

[Jonathan-Eid]

Here I'm calling tracepath from the master to two different pod ips

pod ip 192.168.7.3 is a pod on the worker that is the follower of the location i setup for workers, it hits the wg ip of the its leader @ 10.4.0.3 then times out

pod ip 192.168.8.2 is a pod on the leader of the location i setup for workers, it hits the wg ip and then resolves to the pod ip

[Jonathan-Eid]

Seems like a GCP firewall issue actually, things started working when i opened up all ports and ip sources.

Do you have recommendations on which ports and ip sources i need to open on the masters and workers?

I opened up their external and private subnet ips to each other on the kubelet api port, kilo port, kube server api port, wasn't sure what i was missing honestly

[Jonathan-Eid]

I allowed the ipip protocol on the firewall between nodes, now things are working normally

[squat]

Glad you got it working 💫