service ip is unreachable from pod

[recursivetree]

Environmental Info:
K3s Version:

k3s version v1.29.5+k3s1 (4e53a323)
go version go1.21.9

Node(s) CPU architecture, OS, and Version:

Linux pi-node-2 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

Cluster Configuration:

1 server

Describe the bug:

I've just initialized a dual stack cluster using the following command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --cluster-init --cluster-cidr=fd93:c40c:7613::/56,10.42.0.0/16 --service-cidr=fd95:8606:8f22::/112,10.43.0.0/16

Afterwards, I've deployed a service and some pods:

> kubectl get services
kubernetes           ClusterIP   fd95:8606:8f22::1      <none>        443/TCP    23m
postgresql-service   ClusterIP   fd95:8606:8f22::1c82   <none>        5432/TCP   17m

Inside a pod, I can get the service's ip using the POSTGRESQL_SERVICE_SERVICE_HOST env variable:

> echo $POSTGRESQL_SERVICE_SERVICE_HOST
fd95:8606:8f22::1c82

However, that IP of the service is unreachable:

> ping fd95:8606:8f22::1c82
PING fd95:8606:8f22::1c82(fd95:8606:8f22::1c82) 56 data bytes
^C
--- fd95:8606:8f22::1c82 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1025ms
> ip -6 route
fd93:c40c:7613::/64 dev eth0 proto kernel metric 256 pref medium
fd93:c40c:7613::/56 via fd93:c40c:7613::1 dev eth0 metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd93:c40c:7613::1 dev eth0 metric 1024 pref medium

Connecting using the IP od a pod behind the service works. Since the service cidr doesn't appear in the routing tables, I suspect it didn't get added?

Expected behavior:

The pod behind the service is reachable

Actual behavior:

the pod behind the service is unreachable

[recursivetree]

Update: I've tried to run the same setup using ipv4 addresses for services by changing the order of the address families in the cluster init command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --cluster-init --cluster-cidr=10.42.0.0/16,fd93:c40c:7613::/56 --service-cidr=10.43.0.0/16,fd95:8606:8f22::/112

It seems to work with that. However, I'd still prefer to use ipv6 internally.

[brandond]

I don't know that pings (ICMPv6) are supposed to work against service clusterIPs? Remember that service clusterIPs don't actually exist anywhere, by default they are just iptables port forwards to the pod endpoints, managed by kube-proxy. Some CNIs have kube-proxy replacements that work differently, and kube-proxy itself has ipvs mode which changes things a bit.

Can you try hitting the service on its TCP port?

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.