Load balancer crashing with IPv6 disabled

[lifo9]

Environmental Info:

k3s version v1.29.6+k3s1 (83ae095a)
go version go1.21.11

Node(s) CPU architecture, OS, and Version:

Linux hostname 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Cluster Configuration:

Single node.

Describe the bug:

With IPv6 disabled, svclb-traefik crashes on ContainerCreating with:

/proc/sys/net/ipv6/conf/all/forwarding: no such file or directory

It is caused by securityContext setting:

securityContext:
    sysctls:
    - name: net.ipv6.conf.all.forwarding
      value: "1"

of rancher/klipper container.

It can be fixed by deleting the mentioned net.ipv6.conf.all.forwarding from svclb-traefik daemonset.

Steps To Reproduce:

• Installed K3s simply by downloading a binary
• No special configuration
• Host has IPv6 disabled - ipv6.disable=1 in GRUB_CMDLINE_LINUX and the following sysctl settings set to 1:

net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.default.disable_ipv6
net.ipv6.route.flush

Expected behavior:

Load balancer does not crash.

Actual behavior:

Load balancer crashes.

Additional context / logs:

[brandond]

This comes from #9963 - which was tested with ipv6 disabled vis sysctl, but apparently not with ipv6.disable=1 in the kernel command line, which I guess removes the net.ipv6 sysctls entirely? I'm confused though since you mention that you have other ipv6-related sysctls set - net.ipv6.conf.all.disable_ipv6=1 for example. Which of these these actually exist on your host with that kernel command line flag set?

Are you able to remove that from your kernel command line as a workaround? The sysctl will come back whenever the loadbalancer daemonset is updated.

[lifo9]

Thanks for the suggestion, but I'd like to keep IPv6 completely disabled for security reasons (smaller attack surface).
Wouldn't it be better to apply that sysctl rule conditionally - check whether IPv6 is enabled?

EDIT:
You're right, these sysctl rules are redundant when disabling IPv6 in kernel.
But AFAIK, disabling it via kernel is more reliable as the IPv6 module is not loaded at all.

[brandond]

Wouldn't it be better to apply that sysctl rule conditionally - check whether IPv6 is enabled?

Thats what we did previously, but it lead to different problems, as discussed in the issue linked from that PR. I'll do some thinking about better ways to address this. It is unfortunate that we can't ask the container runtime to ignore sysctls that don't exist on a particular node.

[brandond]

I will also note that I have not seen presence of IPv6 on a node as an indicator of additional attack surface; most folks are moving TOWARDS IPv6, not away from it.

[mlhynfield]

Just a quick note since I just happened upon this issue with a new cluster install, there are some cases where IPv6 is disabled at the kernel level by automation or system administration outside the control of an engineer tasked with installing a K3s cluster with load balancer capabilities; in my case I had to scrap the installation and start again with a downgraded version to achieve the level of resilience/stability we required.

[provenvelocity]

I also have this issue upgrading our K3s cluster. Basically, I will disable IPV6 at the cluster. But what is weird is that I set the right settings, and by the time your code makes the daemonset load balancer service, it makes IPV6 and IPV4 entries.

Even with all the setting below are set it still tries to make a ipv6 entry... in the service load balancer

This is the ingress-nginx:

controller:
  kind: DaemonSet
  sysctls:
    "net.ipv4.ip_forward": "1"
  allowSnippetAnnotations: true
  ingressClassResource:
    default: true
  extraArgs:
    default-ssl-certificate: "ingress-nginx/devsecops-tls-secret"
  service:
    ipFamilyPolicy: SingleStack
    externalTrafficPolicy: Local
    external:
      ipFamilies:
      - IPv4
    ipFamilies:
    - IPv4
  admissionWebhooks:
    port: 2443

[provenvelocity]

Also, I can not control my kernel settings.... well I can but they will reset

[fmoral2]

validated on
#10567 (comment)
#10568 (comment)
#10569 (comment)
#10570 (comment)