no internet connection within pod on K3s --no-flannel

[segator]

Installed k3s on a clean environment disabling default flannel. then installing kilo-k3s.yaml.

between nodes i can ping nodes, wg is properly configured and have connectivity between all 3 nodes.

But seems there is no internet connection inside the pods deployed on workers

interface: kilo0
  public key: xxxxx
  private key: (hidden)
  listening port: 51820

peer: xxxxxxxxxxxxxxxxx
  endpoint: xxxxxxxxxx
  allowed ips: 10.42.1.0/24, 172.17.0.1/32, 10.4.0.3/32
  latest handshake: 20 seconds ago
  transfer: 988 B received, 1.25 KiB sent
  persistent keepalive: every 5 seconds

peer: xxxxxxxxxxxxxx
  endpoint: xxxxxxxx
  allowed ips: 10.42.2.0/24, 10.17.100.16/32, 10.4.0.2/32
  transfer: 0 B received, 592 B sent
  persistent keepalive: every 5 seconds

[squat]

hi @segator could you share some details about the topology of your cluster? From the look of this WireGuard configuration, it seems like you have three nodes in a full mesh. Is that right?

[segator]

Hi, sorry to late answer, what exactly you want to know?

2 nodes behind NAT and 1 node with static public IP and ports opened.

[squat]

Hi @segator I was curious what the Kilo mesh topology was. Are they in a full mesh? Or are they using --mesh -granularity=location and some have the same location annotation / topology label?

[segator]

all of them have separate locations labels

[squat]

Have you also set the persistent-keepalive annotation on the two nodes behind NAT?

[segator]

25s of persistent keepalive, I have connection, the problem is the pods deployed does not have internet connection but if I try to ping from wg peer to peer works.

[squat]

ah, sorry, 🙈 I misread the issue

[squat]

hmm can you please share the output of iptables-save from one of the hosts? this sounds suspiciously like a firewall issue, like for some reason IP packets are not being forwarded between some devices

[segator]

this is from the k3s control plane server
all the servers are having the same internet connection issue.
this one is based on Hetzner and workers are libvirt servers

# Generated by iptables-save v1.8.4 on Fri Apr 15 12:15:59 2022
*mangle
:PREROUTING ACCEPT [741227:398957643]
:INPUT ACCEPT [731866:398369647]
:FORWARD ACCEPT [9361:587996]
:OUTPUT ACCEPT [855772:306972122]
:POSTROUTING ACCEPT [859186:307252481]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT
# Completed on Fri Apr 15 12:15:59 2022
# Generated by iptables-save v1.8.4 on Fri Apr 15 12:15:59 2022
*filter
:INPUT ACCEPT [5368:1881332]
:FORWARD DROP [48:2460]
:OUTPUT ACCEPT [6271:2079900]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:KILO-IPIP - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
:f2b-sshd - [0:0]
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A INPUT -p ipencap -m comment --comment "Kilo: jump to IPIP chain" -j KILO-IPIP
-A INPUT -p ipencap -m comment --comment "Kilo: reject other IPIP traffic" -j DROP
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5432 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A KILO-IPIP -s 172.17.0.1/32 -m comment --comment "Kilo: allow IPIP traffic" -j ACCEPT
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.43.231.62/32 -p tcp -m comment --comment "argocd/argocd-server:https has no endpoints" -m tcp --dport 443 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.97.231/32 -p tcp -m comment --comment "argocd/argocd-server-metrics:metrics has no endpoints" -m tcp --dport 8083 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.177.240/32 -p tcp -m comment --comment "argocd/argocd-metrics:metrics has no endpoints" -m tcp --dport 8082 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.231.62/32 -p tcp -m comment --comment "argocd/argocd-server:http has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.117.197/32 -p tcp -m comment --comment "argocd/argocd-applicationset-controller:webhook has no endpoints" -m tcp --dport 7000 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 180.76.183.164/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 43.154.40.72/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 106.13.121.235/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
COMMIT
# Completed on Fri Apr 15 12:15:59 2022
# Generated by iptables-save v1.8.4 on Fri Apr 15 12:15:59 2022
*nat
:PREROUTING ACCEPT [56:2884]
:INPUT ACCEPT [5:244]
:OUTPUT ACCEPT [41:2460]
:POSTROUTING ACCEPT [41:2460]
:DOCKER - [0:0]
:KILO-NAT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-4MPVXOHO6HYNTNEN - [0:0]
:KUBE-SEP-6XYFZRLUX6ERXTJZ - [0:0]
:KUBE-SEP-7IMI3U3GE6BQ3QWM - [0:0]
:KUBE-SEP-7RDGRZRM5PWNFLMW - [0:0]
:KUBE-SEP-AYQHR2EDKZHZZZID - [0:0]
:KUBE-SEP-CLWNZPHX2ETPLEO7 - [0:0]
:KUBE-SEP-CWGXMBDDT5XLDPGC - [0:0]
:KUBE-SEP-F6WETHG7EKURUS5H - [0:0]
:KUBE-SEP-OWMZ74IA4GELYNIM - [0:0]
:KUBE-SEP-P63V6CAGDOP36DLF - [0:0]
:KUBE-SEP-Q6265JSKI7VKO532 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-3MN7Q5WEBLVAXORV - [0:0]
:KUBE-SVC-B3N5Q5M4GYMHJ5EB - [0:0]
:KUBE-SVC-CUVJNDXHQEILW4E2 - [0:0]
:KUBE-SVC-KZKWQ6OUXCZC3R2R - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-QUERJKMLSZAETC2J - [0:0]
:KUBE-SVC-S3NG4EFDCNWS3YQS - [0:0]
:KUBE-SVC-UEX2T42OGTNVXVWJ - [0:0]
:KUBE-SVC-WMHZRWPBLB277GS6 - [0:0]
:KUBE-SVC-YSMKAOLNJHL3CGZS - [0:0]
:KUBE-SVC-Z4ANX4WAEWEBLCTM - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5432 -j MASQUERADE
-A POSTROUTING -s 10.42.0.0/24 -m comment --comment "Kilo: jump to KILO-NAT chain" -j KILO-NAT
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 5432 -j DNAT --to-destination 172.17.0.2:5432
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.2.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.17.101.219/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.1.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.17.100.102/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.0.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -m comment --comment "Kilo: NAT remaining packets" -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SEP-4MPVXOHO6HYNTNEN -s 10.42.1.3/32 -m comment --comment "argocd/argocd-dex-server:grpc" -j KUBE-MARK-MASQ
-A KUBE-SEP-4MPVXOHO6HYNTNEN -p tcp -m comment --comment "argocd/argocd-dex-server:grpc" -m tcp -j DNAT --to-destination 10.42.1.3:5557
-A KUBE-SEP-6XYFZRLUX6ERXTJZ -s 10.42.2.3/32 -m comment --comment "argocd/argocd-notifications-controller-metrics:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-6XYFZRLUX6ERXTJZ -p tcp -m comment --comment "argocd/argocd-notifications-controller-metrics:metrics" -m tcp -j DNAT --to-destination 10.42.2.3:9001
-A KUBE-SEP-7IMI3U3GE6BQ3QWM -s 10.42.0.3/32 -m comment --comment "kube-system/metrics-server:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-7IMI3U3GE6BQ3QWM -p tcp -m comment --comment "kube-system/metrics-server:https" -m tcp -j DNAT --to-destination 10.42.0.3:4443
-A KUBE-SEP-7RDGRZRM5PWNFLMW -s 10.42.1.3/32 -m comment --comment "argocd/argocd-dex-server:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-7RDGRZRM5PWNFLMW -p tcp -m comment --comment "argocd/argocd-dex-server:http" -m tcp -j DNAT --to-destination 10.42.1.3:5556
-A KUBE-SEP-AYQHR2EDKZHZZZID -s 10.42.1.3/32 -m comment --comment "argocd/argocd-dex-server:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-AYQHR2EDKZHZZZID -p tcp -m comment --comment "argocd/argocd-dex-server:metrics" -m tcp -j DNAT --to-destination 10.42.1.3:5558
-A KUBE-SEP-CLWNZPHX2ETPLEO7 -s 10.42.2.4/32 -m comment --comment "argocd/argocd-repo-server:server" -j KUBE-MARK-MASQ
-A KUBE-SEP-CLWNZPHX2ETPLEO7 -p tcp -m comment --comment "argocd/argocd-repo-server:server" -m tcp -j DNAT --to-destination 10.42.2.4:8081
-A KUBE-SEP-CWGXMBDDT5XLDPGC -s 138.201.91.44/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-CWGXMBDDT5XLDPGC -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 138.201.91.44:6443
-A KUBE-SEP-F6WETHG7EKURUS5H -s 10.42.0.2/32 -m comment --comment "kube-system/coredns-coredns:udp-53" -j KUBE-MARK-MASQ
-A KUBE-SEP-F6WETHG7EKURUS5H -p udp -m comment --comment "kube-system/coredns-coredns:udp-53" -m udp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SEP-OWMZ74IA4GELYNIM -s 10.42.0.2/32 -m comment --comment "kube-system/coredns-coredns:tcp-53" -j KUBE-MARK-MASQ
-A KUBE-SEP-OWMZ74IA4GELYNIM -p tcp -m comment --comment "kube-system/coredns-coredns:tcp-53" -m tcp -j DNAT --to-destination 10.42.0.2:53
-A KUBE-SEP-P63V6CAGDOP36DLF -s 10.42.2.4/32 -m comment --comment "argocd/argocd-repo-server:metrics" -j KUBE-MARK-MASQ
-A KUBE-SEP-P63V6CAGDOP36DLF -p tcp -m comment --comment "argocd/argocd-repo-server:metrics" -m tcp -j DNAT --to-destination 10.42.2.4:8084
-A KUBE-SEP-Q6265JSKI7VKO532 -s 10.42.1.4/32 -m comment --comment "argocd/argocd-redis:tcp-redis" -j KUBE-MARK-MASQ
-A KUBE-SEP-Q6265JSKI7VKO532 -p tcp -m comment --comment "argocd/argocd-redis:tcp-redis" -m tcp -j DNAT --to-destination 10.42.1.4:6379
-A KUBE-SERVICES -d 10.43.198.127/32 -p tcp -m comment --comment "kube-system/metrics-server:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-Z4ANX4WAEWEBLCTM
-A KUBE-SERVICES -d 10.43.26.33/32 -p udp -m comment --comment "kube-system/coredns-coredns:udp-53 cluster IP" -m udp --dport 53 -j KUBE-SVC-3MN7Q5WEBLVAXORV
-A KUBE-SERVICES -d 10.43.131.202/32 -p tcp -m comment --comment "argocd/argocd-repo-server:metrics cluster IP" -m tcp --dport 8084 -j KUBE-SVC-WMHZRWPBLB277GS6
-A KUBE-SERVICES -d 10.43.26.33/32 -p tcp -m comment --comment "kube-system/coredns-coredns:tcp-53 cluster IP" -m tcp --dport 53 -j KUBE-SVC-S3NG4EFDCNWS3YQS
-A KUBE-SERVICES -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:grpc cluster IP" -m tcp --dport 5557 -j KUBE-SVC-UEX2T42OGTNVXVWJ
-A KUBE-SERVICES -d 10.43.206.53/32 -p tcp -m comment --comment "argocd/argocd-notifications-controller-metrics:metrics cluster IP" -m tcp --dport 9001 -j KUBE-SVC-KZKWQ6OUXCZC3R2R
-A KUBE-SERVICES -d 10.43.131.202/32 -p tcp -m comment --comment "argocd/argocd-repo-server:server cluster IP" -m tcp --dport 8081 -j KUBE-SVC-QUERJKMLSZAETC2J
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:metrics cluster IP" -m tcp --dport 5558 -j KUBE-SVC-CUVJNDXHQEILW4E2
-A KUBE-SERVICES -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:http cluster IP" -m tcp --dport 5556 -j KUBE-SVC-B3N5Q5M4GYMHJ5EB
-A KUBE-SERVICES -d 10.43.56.158/32 -p tcp -m comment --comment "argocd/argocd-redis:tcp-redis cluster IP" -m tcp --dport 6379 -j KUBE-SVC-YSMKAOLNJHL3CGZS
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-3MN7Q5WEBLVAXORV ! -s 10.42.0.0/16 -d 10.43.26.33/32 -p udp -m comment --comment "kube-system/coredns-coredns:udp-53 cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-3MN7Q5WEBLVAXORV -m comment --comment "kube-system/coredns-coredns:udp-53" -j KUBE-SEP-F6WETHG7EKURUS5H
-A KUBE-SVC-B3N5Q5M4GYMHJ5EB ! -s 10.42.0.0/16 -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:http cluster IP" -m tcp --dport 5556 -j KUBE-MARK-MASQ
-A KUBE-SVC-B3N5Q5M4GYMHJ5EB -m comment --comment "argocd/argocd-dex-server:http" -j KUBE-SEP-7RDGRZRM5PWNFLMW
-A KUBE-SVC-CUVJNDXHQEILW4E2 ! -s 10.42.0.0/16 -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:metrics cluster IP" -m tcp --dport 5558 -j KUBE-MARK-MASQ
-A KUBE-SVC-CUVJNDXHQEILW4E2 -m comment --comment "argocd/argocd-dex-server:metrics" -j KUBE-SEP-AYQHR2EDKZHZZZID
-A KUBE-SVC-KZKWQ6OUXCZC3R2R ! -s 10.42.0.0/16 -d 10.43.206.53/32 -p tcp -m comment --comment "argocd/argocd-notifications-controller-metrics:metrics cluster IP" -m tcp --dport 9001 -j KUBE-MARK-MASQ
-A KUBE-SVC-KZKWQ6OUXCZC3R2R -m comment --comment "argocd/argocd-notifications-controller-metrics:metrics" -j KUBE-SEP-6XYFZRLUX6ERXTJZ
-A KUBE-SVC-NPX46M4PTMTKRN6Y ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-CWGXMBDDT5XLDPGC
-A KUBE-SVC-QUERJKMLSZAETC2J ! -s 10.42.0.0/16 -d 10.43.131.202/32 -p tcp -m comment --comment "argocd/argocd-repo-server:server cluster IP" -m tcp --dport 8081 -j KUBE-MARK-MASQ
-A KUBE-SVC-QUERJKMLSZAETC2J -m comment --comment "argocd/argocd-repo-server:server" -j KUBE-SEP-CLWNZPHX2ETPLEO7
-A KUBE-SVC-S3NG4EFDCNWS3YQS ! -s 10.42.0.0/16 -d 10.43.26.33/32 -p tcp -m comment --comment "kube-system/coredns-coredns:tcp-53 cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SVC-S3NG4EFDCNWS3YQS -m comment --comment "kube-system/coredns-coredns:tcp-53" -j KUBE-SEP-OWMZ74IA4GELYNIM
-A KUBE-SVC-UEX2T42OGTNVXVWJ ! -s 10.42.0.0/16 -d 10.43.243.38/32 -p tcp -m comment --comment "argocd/argocd-dex-server:grpc cluster IP" -m tcp --dport 5557 -j KUBE-MARK-MASQ
-A KUBE-SVC-UEX2T42OGTNVXVWJ -m comment --comment "argocd/argocd-dex-server:grpc" -j KUBE-SEP-4MPVXOHO6HYNTNEN
-A KUBE-SVC-WMHZRWPBLB277GS6 ! -s 10.42.0.0/16 -d 10.43.131.202/32 -p tcp -m comment --comment "argocd/argocd-repo-server:metrics cluster IP" -m tcp --dport 8084 -j KUBE-MARK-MASQ
-A KUBE-SVC-WMHZRWPBLB277GS6 -m comment --comment "argocd/argocd-repo-server:metrics" -j KUBE-SEP-P63V6CAGDOP36DLF
-A KUBE-SVC-YSMKAOLNJHL3CGZS ! -s 10.42.0.0/16 -d 10.43.56.158/32 -p tcp -m comment --comment "argocd/argocd-redis:tcp-redis cluster IP" -m tcp --dport 6379 -j KUBE-MARK-MASQ
-A KUBE-SVC-YSMKAOLNJHL3CGZS -m comment --comment "argocd/argocd-redis:tcp-redis" -j KUBE-SEP-Q6265JSKI7VKO532
-A KUBE-SVC-Z4ANX4WAEWEBLCTM ! -s 10.42.0.0/16 -d 10.43.198.127/32 -p tcp -m comment --comment "kube-system/metrics-server:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SVC-Z4ANX4WAEWEBLCTM -m comment --comment "kube-system/metrics-server:https" -j KUBE-SEP-7IMI3U3GE6BQ3QWM
COMMIT

[segator]

Regarding topology. Not sure why appear workers with an IPV6 endpoint (I replaced sensitive data)
Worker 1 and 2 are behind CGNAT, master have public static IP, master is discovering endpoints and sharing between workers
as seen in node labels

 kilo.squat.ai/discovered-endpoints: {"xxxxxxxxx":{"IP":"x.x.x.x","Port":51820,"Zone":""},"xxxxxx.....
kilo.squat.ai/granularity: location

all nodes have this labels with different value as they are located in diferents "regions"

topology.kubernetes.io/region=fsn1-dc14
kilo.squat.ai/location=fsn1-dc14

only master (node with static public IP) have this set

kilo.squat.ai/force-endpoint: mypublicendpoint:51280

only workers have set

kilo.squat.ai/persistent-keepalive: 5 

[squat]

Thanks @segator this is really helpful.

I'm a bit curious about the default DROP rule on the FORWARD chain. Would you mind trying something?
Can you add the --iptables-forward-rules=true flag to the Kilo daemonset and let me know if this changes anything?

[segator]

Hi @squat, it seems to work only on libvirt machines, not hetzner machine (The iptables-save I shown you before is hetzner one)
Or maybe is not related is a hetzner machine but is the control plane?

[squat]

Ha interesting! The plot thickens!

Can you describe how the nodes are different? Are the Herner node and the VMs running the same or different OS? are they configured similarly? Is there some other process managing the firewall on the Hetzner node? (Please do another iptables-save to confirm that the FORWARD chain now defaults to ALLOW)

[segator]

First of all, Thanks for taking the time to help on my specific case @squat , I guess you have things better to do :)
All machines are

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

The machines are deployed via terraform so the configuration is completely identical between machines, except master I run k3s as control plane instead of worker.

Regarding firewall in hetzner, if I deploy k3s with flannel instead of kilo then we have internet connectivity.

I notice now I have internet connection but DNS is not resolving in Hetzner machine, coreDNS is running on Hetzner machine, but workers still have connectivity and DNS resolving.
I even tried to define a fixed resolv-conf file in k3s configuration with 8.8.8.8, to ignore host configuration, but same :(

[segator]

Umh The issue seems the IP of DNS server 10.43.0.10 hetzner VM can not connect, but other machines they can,
If I try inside a pod to replace /etc/resolv.conf 10.43.0.10 to the serviceIP of coreDNS 10.43.26.33 then it works, any idea?

[segator]

Trying to force coreDNS serviceIP to 10.43.0.10 seems now I can resolve DNS but I still have rejected connections, DNS and Ping to any IP works but curl, wget.. etc not yet, domain is resolved but connection dropped

[squat]

Hi @segator im a bit confused. By convention, the DNS ServiceIP tends to automatically be 10.x.0.10 (10.43.0.10 in the case of K3s).
Are you saying that the cluster IP of the DNS service is something else and that /etc/resolve.conf doesn't match the real clusterIP?

That would explain why your node cannot resolve DNS. But in that case, none of the nodes should have been able to resolve DNS. Or was /etc/resolv.conf only wrong on one node?

[segator]

Seems the issue is when installing K3s disabling coreDNS and installing it with helm.

The funny part is this issue only ocurs when having Kilo, if I do the same but with flannel for exemple then there is no problem to install coreDNS as helm instead of K3s

[squat]

Interesting 🤔 would you mind sharing the flannel daemonset yaml that does work?

[squat]

So, to summarize, Kilo works if you use the stock coreDNS but not if you disable it and later install your own?

[segator]

Correct

[squat]

Ack please share the flannel config if you can 🎉

[squat]

Ping @segator :)

[segator]

Hey! I think you can close the issue, I tought it was perfectly working with flannel but it's not.
I didn't discovered the problem yet but is an issue between Hetzner + K3s.
When disabling k3s coredns stop working if installed coredns via helm independenly of the CNI installed, so in any way is related with kilo!!

Thanks!!