on
Federate Kubernetes with AWS IAM using OIDC
Introduction
At reecetech, we want to consume AWS services from the pods running in our Kubernetes clusters. The Kubernetes clusters are predominantly hosted on-prem in VMware, although there are some worker nodes that run on EC2 instances. In order to use AWS services, the pods are required to be authenticated with AWS.
We used OpenID connect to federate the identity pods have in our Kubernetes clusters (RBAC) to AWS IAM.
This allows the pods to be able to assume a role in AWS using the AWS STS AssumeRoleWithWebIdentity API call.
There are open-source projects (Kube2Iam, KIAM) that provide similar functionality, however,
we were motivated to try OpenID connect based federation so that we did not have to maintain an extra piece of software to provide the functionality.
We also could have used our Hashicorp Vault infrastructure to broker IAM roles, but again,
we were motivated to try and not be running any extra side-car pods.
This will make things easier as most AWS SDK’s will automatically make the AssumeRoleWithWebIdentity API call
when they detect the presence of the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE.
Having this federated functionality available means reecetech can stop using AWS API keys injected into the pods via Kubernetes secrets. No more worrying about API key rotation or managing hundreds of AWS IAM users.
At the time of writing (December 2020), there is a lack of information on the Internet regarding Kubernetes to AWS federation via OpenID connect. We hope that this article can help disseminate knowledge about this type of federation more widely.
How to
Configure Kubernetes for federation
apiserver
Note: We assume you’ve already got, or are able to generate some signing keys.
The Kubernetes apiserver needs to have some extra flags added to the runtime process.
This is usually done in the kubeadm-config configuration map.
Changes in the configuration map are applied to the apiserver pods on the next time a master node joins the cluster (using kubeadm join).
There are 6 flags that need to be set. At the time of writing, we have tested on Kubernetes 1.18 and 1.19.
api-audiences: kubernetes.svc.default # doesn't matter what it is
feature-gates: ServiceAccountIssuerDiscovery=true
service-account-api-audiences: cluster.identifier.kubernetes.domain # need to tell AWS this, make it unique to the cluster
service-account-issuer: https://apiserver.domain/cluster/identifier # need to tell AWS this
service-account-jwks-uri: https://apiserver.domain/cluster/identifier/openid/v1/jwks
service-account-signing-key-file: /etc/kubernetes/pki/sa.key # your signing key (this flag can be repeated)
Publishing to S3 (optional)
We do not expose our apiserver pods to the Internet.
Instead, to allow AWS to access the URLs given in service-account-issuer and service-account-jwks-uri we created a public S3 bucket,
and synchronise the data from those two URLs to the S3 bucket.
Those flags in our clusters look something like this:
service-account-issuer: https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier
service-account-jwks-uri: https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier/openid/v1/jwks
To keep S3 up-to-date with the data being served from the apiservers, we use our CI system to periodically:
- cURL the URLs
- store the result in a git repository
- synchronise the data to S3
RBAC
The apiserver applies RBAC rules to these new endpoints configured in the flags, so you also need to allow anonymous read-access to the endpoints.
Here’s the cluster role binding to apply:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: oidc-reviewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:service-account-issuer-discovery
subjects:
- kind: Group
name: system:unauthenticated
Configure AWS for federation
Configuring AWS for the federation is done via API calls or the GUI. Unfortunately it’s not supported in CloudFormation.
Follow the AWS documentation, and if you’re using the GUI, it should look something like this:
Configure the AWS IAM role
Within AWS there needs to be an IAM role for the pods to assume. Ideally, each different application would have its own role, following the principle of granting least privileges.
The IAM role has two parts: what the role can do, and, who can assume the role.
Here is a CloudFormation template that would create a read-only role, that can be assumed by our newly federated pods:
---
AWSTemplateFormatVersion: 2010-09-09
Description: An example stack that creates an IAM role assumable by OpenID connect federation
Parameters:
OidcIdentifier:
Type: String
Description: The OpenID connect identifier that AWS is configured with for the federated cluster
Default: apiserver.domain/cluster/identifier
PodNamespace:
Type: String
Description: The Kubernetes namespace that the pods have their service account in
Default: default
PodServiceAccount:
Type: String
Description: The Kubernetes service account that the pods will be using
Resources:
BlogExampleRole:
Type: AWS::IAM::Role
Properties:
RoleName: blog-example-iam-role
ManagedPolicyArns:
- arn:aws:iam::aws:policy/ReadOnlyAccess
MaxSessionDuration: 3600 # 1 hour
AssumeRolePolicyDocument: !Sub |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRoleWithWebIdentity",
"Principal": {
"Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${OidcIdentifier}"
},
"Condition": {
"StringEquals": {
"${OidcIdentifier}:sub": "system:serviceaccount:${PodNamespace}:${PodServiceAccount}"
}
}
}
]
}
After reading the above example, you should be crying “why, oh why, do you have JSON in YAML!?”.
The reason is that we need to have a dynamic key in the StringEquals condition,
and that’s not possible without resorting to using a string for the entire AssumeRolePolicyDocument.
Credit to gjenkins8 on Github.
Note the granular access granted.
Only pods that use the service account given in the StringEquals can assume the role.
Configure the pods
Environment variables
To configure our pods to have the AWS related environment variables available at run-time, we need to add data to the pod’s configuration map.
In the example below, the pod is part of a replica set, that is created by a deployment. We have trimmed the example down to the additional lines required to set the AWS relevant environment variables.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "blog-example-config-map"
data:
AWS_DEFAULT_REGION: ap-southeast-2
AWS_ROLE_ARN: "arn:aws:iam::000000000000:role/blog-example-iam-role" # matches `RoleName` in the CloudFormation template
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/k8s-to-iam/serviceaccount/token # matches the mount in the pod spec
Service account
Within Kubernetes there needs to be an service account for the pods to use - else they are allocated default.
Ideally, each different application would have its own service account. Recall that we recommend each application has its own IAM role - in this case there should be a 1:1 mapping between the IAM role and the Kubernetes service account for an application. This is again following the principle of granting least privileges.
You are not required to bind the service account to any Kubernetes role.
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: "blog-example-service-account"
Projected mount
To configure our pods to have the OpenID connect authentication data available at run-time, we need to add a projected mount to the pod specification.
In the example below, the pod is part of a replica set, that is created by a deployment. We have trimmed the example down to the additional lines required to provide the projected mount.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: "blog-example"
labels:
app: "blog-example"
spec:
replicas: 1
selector:
matchLabels:
app: "blog-example"
template:
metadata:
labels:
app: "blog-example"
spec:
serviceAccountName: "blog-example-service-account"
containers:
- name: "blog-example"
image: centos:7
command: ["sleep"]
args: ["86400"]
envFrom:
- configMapRef:
name: "blog-example-config-map"
volumeMounts:
- mountPath: "/var/run/secrets/k8s-to-iam/serviceaccount/"
name: aws-token
volumes:
- name: aws-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 43200
audience: "cluster.identifier.kubernetes.domain" # matches `service-account-api-audiences` flag on the apiserver
It is worth noting that in our environment, we noticed an issue when running under version 1.18.6 of Kubernetes where the token
was mounted into the container with the owner root and the permissions bitmask of 0600 (read/write owner, no access to anybody else).
As far as we could tell, the volume permissions were set correctly to 0644 (or 420 decimal - thanks JSON). For reasons unknown to us,
upon upgrading the cluster to version 1.19.5 as part of unrelated maintenance the token was suddenly mounted with correct permissions.
Reviewing the release notes didn’t point to any GitHub issue that might explain the change in functionality, so it’s possible 1.18.6 will work correctly in other environments, but if you do encounter this particular issue it is worth upgrading.
Testing / Demo
With all this configured correctly, you should now be able to assume a role from a pod in your Kubernetes cluster.
We can see that the JWT is mounted at /var/run/secrets/k8s-to-iam/serviceaccount/token and is world readable:
$ ls -la /var/run/secrets/k8s-to-iam/serviceaccount/token
lrwxrwxrwx 1 root root 12 Jan 29 08:37 /var/run/secrets/k8s-to-iam/serviceaccount/token -> ..data/token
$ ls -la /var/run/secrets/k8s-to-iam/serviceaccount/..data/token
-rw-r--r-- 1 root root 1073 Jan 29 08:37 /var/run/secrets/k8s-to-iam/serviceaccount/..data/token
$ cat /var/run/secrets/k8s-to-iam/serviceaccount/token
eyJhb[..snip..]hWSQ
Decoding this JWT, we can see that the subject is our pod’s service account, and the audience is as expected:
{
"aud": [
"cluster.identifier.kubernetes.domain"
],
"exp": 1608288436,
"iat": 1608245236,
"iss": "https://oidc-bucket.s3-ap-southeast-2.amazonaws.com/cluster/identifier",
"kubernetes.io": {
"namespace": "my-namespace",
"pod": {
"name": "blog-example-9fc8c46f8-vj8b4",
"uid": "c1d2fa28-5da5-47b2-b623-dc6f0833b428"
},
"serviceaccount": {
"name": "blog-example-service-account",
"uid": "0588771a-cd56-420a-9cec-787a792f7567"
}
},
"nbf": 1608245236,
"sub": "system:serviceaccount:my-namespace:blog-example-service-account"
}
Finally, we should be able to use the AWS CLI to verify we’re able to assume the role correctly:
# yum install python3 # as root
# pip3 install -U awscli # must be newer than 1.16.232
# useradd foo && su -p foo # demo non-root user
$ unset HOME
$ aws --version
aws-cli/1.18.222 Python/3.6.8 Linux/3.10.0-1160.el7.x86_64 botocore/1.19.62
$ aws sts get-caller-identity
{
"UserId": "AROAXXXXXXXXXXXXXXXXX:botocore-session-1608249397",
"Account": "000000000000",
"Arn": "arn:aws:iam::000000000000:role/blog-example-iam-role/botocore-session-1608249397"
}
Final notes
Key rotation
At the time of writing, we have not tested rotating the signing key.
However, the apiserver flag service-account-signing-key-file can be given multiple times to facilitate key rotation.
When given multiple times, then the resulting JWKS (JSON Web Key Sets) contains multiple keys.
This allows a new key to be rolled out across multiple apiservers without restarting them all at once.
amazon-eks-pod-identity-webhook
It’s worth mentioning AWS provide a webhook to do the configuring the pod part of this article (environment variables & projected mount). The webhook is provided on Github.
We don’t use the webhook due to the way it creates a certificate authority, and it means running more software. We have achieved OpenID connect federation between Kubernetes and AWS by configuration only - no extra pods.
Conclusion
We find that using OpenID connect federation between (on-prem) Kubernetes and AWS to be a powerful and secure method for making AWS services available to our Kubernetes pods.