Improve out-of-the-box compatibility with clusters running PodSecurityPolicy

[chadlwilson]

Background

We run our EKS clusters with locked-down PodSecurityPolicy by default; that is we drop all privileges and change the default EKS policy to one that is essentially unprivileged.

kubernetes-external-secrets does not appear to require any particular privileges that are from the set.

This makes two changes to allow the chart/image to work out of the box without configuration on the user side.

1. changes the Dockerfile to indicate the user as uid 1000 rather than node, which is required to indicate K8S that it's not running as root (when running under Docker, at least) - consistent with the base node images

   Minimal workaround on 3.2.0 without the change in this PR is a values.yaml override of

   securityContext:
     runAsUser: 1000
   

2. changes the defaults to include a securityContext that specifies the non root requirement.

This second change is a little more debatable and I am interested in feedback however it seems a sensible idea to give an indication to chart users the actual requirements of kubernetes-external-secrets pods so they can craft custom policies if necessary; and understand any possible additional attack surface required for running a security-sensitive tool like this in their cluster (which I would argue is minimal given the pod requirements).

A possible follow-up step would be to allow to specify the container securityContext rather than just the pod-level; which would allow us to specify

securityContext:
  privileged: false
  allowPrivilegeEscalation: false
   capabilities:
    drop: ["ALL"]
  readOnlyRootFilesystem: true

This has been tested to work fine and automatically select a relatively unprivileged "secure-by-default" PSP like the below

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: unprivileged
spec:
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret

[Flydiverny]

Thanks! Setting runAsNonRoot by default seems good! :)