Run Control Plane components as nonroot user 65532

[AleksandarSavchev]

How to categorize this PR?

/area security quality
/kind enhancement

What this PR does / why we need it:
This PR modifies Control Plane components kube-apiserver, kube-controller-manager and kube-scheduler to run as nonroot user 65532 (id of nonroot user in distroless images ref). The aim of this PR is to adhere to the principles of least privilege and reduce potential attack surface.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Control plane components `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` now run as `nonroot` user and group `65532`.

[AleksandarSavchev]

/cc @dimityrmirchev

[dimityrmirchev]

/lgtm

[gardener-prow]

LGTM label has been added.

Git tree hash: 9bb9b3bb5f4c786b63b88a5231f60e65fb01e004

[AleksandarSavchev]

Converted to draft in order to research setting runAsGroup: 65532 as well.

[ScheererJ]

Did you check with HA vpn? In the HA-mode of vpn, there are vpn side-car containers added to kube-apiserver. Do they also work with this user change?

[AleksandarSavchev]

Did you check with HA vpn? In the HA-mode of vpn, there are vpn side-car containers added to kube-apiserver. Do they also work with this user change?

I have not checked the HA scenarios. I will look into them, if the vpn side-car containers are problematic the easiest fix would be to set their containers to run as root.

[AleksandarSavchev]

After validating locally the vpn containers cannot run as nonroot and would error with the following: panic: open /tmp/acquired-ip: permission denied.
I tested adding the nonroot user to the image used in this Dockerfile and got the following error: /network-connection.sh: line 47: /proc/sys/net/ipv4/tcp_keepalive_time: Permission denied ref. The /proc/sys/net/ipv4 files have 0644 permissions and their owners cannot be changed from root.
I have decided to leave the vpn containers running as root.

[AleksandarSavchev]

/test pull-gardener-e2e-kind

[AleksandarSavchev]

/test pull-gardener-e2e-kind

[AleksandarSavchev]

/test pull-gardener-e2e-kind-migration
/test pull-gardener-e2e-kind-migration-ha-single-zone

[dimityrmirchev]

/lgtm

[gardener-prow]

LGTM label has been added.

Git tree hash: b1930fd5bae2bab580140ff89d8782dfc423ec23

[ialidzhikov]

What happens when a Pod has a securityContext set at Pod level and then some container overwrite it? Does the container securityContext is considered only or does kubernetes internally merges the container securityContext with the pod securityContext (where container securityContext values take precedence)?
I am asking because if a merge happens under the hood, then the merged securityContext would be

securityContext:
  runAsNonRoot: false
  runAsUser: 0
  runAsGroup: 65532
  fsGroup: 65532

which I think is not what we want.
The docs write

Security settings that you specify for a Container apply only to the individual Container, and they override settings made at the Pod level when there is overlap.

but I am not 100% sure.

[ialidzhikov]

It seems that a merge happens under the hood indeed.

A small isolated setup to prove the merge:

1. Create the following yaml:

apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    runAsGroup: 65532
    fsGroup: 65532
  containers:
  - image: alpine:3.13.2
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"
    imagePullPolicy: IfNotPresent
    name: alpine
    securityContext:
      runAsNonRoot: false
      runAsUser: 0
  restartPolicy: Always

2. Verify the merge

% k exec -ti alpine -- sh
/ # id
uid=0(root) gid=65532 groups=1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video),65532

Again, maybe this is not what we want? Maybe it is better to move the Pod level securityContext to the kube-apiserver container?

[AleksandarSavchev]

Yes, the Pod's securityContext setting will be used by every container and only the specified settings will be overridden by the container's securityContext. It is important to note that fsGroup can be set only at the Pod level settings so the only question is if we should explicitly set runAsGroup at the Container level.

I do not think that it is required since we already set root as the user which grants full permissions on all files in the container.

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dimityrmirchev, ialidzhikov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ialidzhikov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment