Kubernetes External Secrets allows you to use external secret management systems (e.g., AWS Secrets Manager) to securely add secrets in Kubernetes. Read more about the design and motivation for Kubernetes External Secrets on the GoDaddy Engineering Blog.

The project extends the Kubernetes API by adding a ExternalSecrets object using Custom Resource Definition and a controller to implement the behavior of the object itself.

An ExternalSecret declares how to fetch the secret data, while the controller converts all ExternalSecrets to Secrets. The conversion is completely transparent to Pods that can access Secrets normally.

1. ExternalSecrets are added in the cluster (e.g., kubectl apply -f external-secret-example.yml)
2. Controller fetches ExternalSecrets using the Kubernetes API
3. Controller uses ExternalSecrets to fetch secret data from external providers (e.g, AWS Secrets Manager)
4. Controller upsert Secrets
5. Pods can access Secrets normally

Alternatively, the included charts/kubernetes-external-secrets can be used to create the kubernetes-external-secrets resources and Deployment on a Kubernetes cluster using the Helm package manager.

helm install --name kubernetes-external-secrets \
--set env.AWS_REGION='<aws-region>' \
charts/kubernetes-external-secrets

Tip: A namespace can be specified by the Helm option '--namespace kube-external-secrets'

To uninstall/delete the kubernetes-external-secrets deployment:

helm delete kubernetes-external-secrets

The following table lists the configurable parameters of the kubernetes-external-secrets chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

helm install --name kubernetes-external-secrets \
--set env.POLLER_INTERVAL_MILLISECONDS='300000' \
--set podAnnotations."iam\.amazonaws\.com/role"='Name-Of-IAM-Role-With-SecretManager-Access' \
charts/kubernetes-external-secrets

If not running on EKS you will have to use an IAM user (in lieu of a role). Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars in the session/pod. You can use envVarsFromSecret in the helm chart to create these env vars from existing k8s secrets

Additionally, you can specify a roleArn which will be assumed before retrieving the secret. You can limit the range of roles which can be assumed by this particular namespace by using annotations on the namespace resource. The annotation value is evaluated as a regular expression and tries to match the roleArn.

kind: Namespace
metadata:
  name: iam-example
  annotations:
    iam.amazonaws.com/permitted: "arn:aws:iam::123456789012:role/.*"

Add your secret data to your backend. For example, AWS Secrets Manager:

aws secretsmanager create-secret --name hello-service/password --secret-string "1234"

AWS Parameter Store:

aws ssm put-parameter --name "/hello-service/password" --type "String" --value "1234"

and then create a hello-service-external-secret.yml file:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-service
secretDescriptor:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  data:
    - key: hello-service/password
      name: password

or

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-service
secretDescriptor:
  backendType: systemManager
  data:
    - key: /hello-service/password
      name: password

The following IAM policy allows a user or role to access parameters matching prod-*.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:us-west-2:123456789012:parameter/prod-*"
    }
  ]
}

The IAM policy for Secrets Manager is similar (see docs):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes128-1a2b3c",
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes192-4D5e6F",
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:aes256-7g8H9i"
      ]
    }
  ]
}

Save the file and run:

kubectl apply -f hello-service-external-secret.yml

Wait a few minutes and verify that the associated Secret has been created:

kubectl get secret hello-service -o=yaml

The Secret created by the controller should look like:

apiVersion: v1
kind: Secret
metadata:
  name: hello-service
type: Opaque
data:
  password: MTIzNA==

kubernetes-external-secrets supports both AWS Secrets Manager and AWS System Manager.

kubernetes-external-secrets supports both JSON objects ("Secret key/value" in the AWS console) or strings ("Plaintext" in the AWS console). Using JSON objects is useful when you need to atomically update multiple values. For example, when rotating a client certificate and private key.

When writing an ExternalSecret for a JSON object you must specify the properties to use. For example, if we add our hello-service credentials as a single JSON object:

aws secretsmanager create-secret --region us-west-2 --name hello-service/credentials --secret-string '{"username":"admin","password":"1234"}'

We can declare which properties we want from hello-service/credentials:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-service
secretDescriptor:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/test-role
  data:
    - key: hello-service/credentials
      name: password
      property: password
    - key: hello-service/credentials
      name: username
      property: username

kubernetes-external-secrets exposes the following metrics over a prometheus endpoint:

Minikube is a tool that makes it easy to run a Kubernetes cluster locally.

Start minikube and the daemon. This creates the CustomerResourceDefinition, and starts to process ExternalSecrets:

minikube start

npm run nodemon

Localstack mocks AWS services locally so you can test without connecting to AWS.

Run localstack in a seperate terminal window

npm run localstack

Start minikube as above

minikube start

Run the daemon with localstack

npm run local

Add secrets using the AWS cli (example)

 aws --endpoint-url=http:https://localhost:4584 secretsmanager create-secret --name hello-service/password --secret-string "1234"