Integration with AWS IAM Roles for ServiceAccounts

[arruzk]

It is a second attempt to do it (first is #176). With recent changes it is easier.

• AWS SDK version was updated
• securityContext parameter in a helm chart was added

It should fix this issue #161

Sorry @farshad-hobsons if you are working on it now.
@Flydiverny @silasbw could you please review it?

[imaffe]

@arruzk Sorry for the late response, but I downloaded the latest helm version with 2.1.0 docker image, turned fsgroup to 65534 and annotation configured the service account annotation to the IAM role i want.

And the environment variable of this pod is

(Which is set by the EKS IRSA)
But the logs shows it is still using the EC2(nodes') iam role, and is not working . (Sorry I cannot post the logs cuz it might be sensitive)

Have you tested if it is really working with the IRSA ? Thanks so much

[arruzk]

@imaffe just checked now. Everything is working for me.
my test values.yaml (helm chart version is kubernetes-external-secrets-2.2.0)

env:
  AWS_REGION: eu-west-1
# created in terraform
serviceAccount:
  name: secrets-manager
  create: false
image:
  tag: latest
  pullPolicy: Always
securityContext:
  fsGroup: 65534

From pod env:

    - name: AWS_ROLE_ARN
      value: arn:aws:iam::11111111111:role/staging-ir-1-sm-reader-role
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Try to log in into the pod (with exec) and do
cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[dudicoco]

@arruzk I am also experiencing the same issue as @imaffe.
I was able to exec into the container and cat the token.

Any ideas?

[dudicoco]

@arruzk when testing this did you verify the role which is being assumed does not have the node in its trusted relationships? Is it possible the container was actually using the node's role + kiam in your tests?

[imaffe]

@arruzk I am also experiencing the same issue as @imaffe.
I was able to exec into the container and cat the token.

Any ideas?

I've fixed it, and I checked the trust relationship in the EKS cluster page, my problem was the serviceaccount name defined doesn't align with the trusted IAM role arn in the trusted relationships. The service account name is not the same as the app name, I think it's their helm chart didn't follow the conventions.

[Flydiverny]

Service account name can be overriden by providing serviceAccount.name in helm values. ( https://github.com/godaddy/kubernetes-external-secrets/blob/master/charts/kubernetes-external-secrets/values.yaml#L39) and how its set can be seen here https://github.com/godaddy/kubernetes-external-secrets/blob/master/charts/kubernetes-external-secrets/templates/_helpers.tpl#L36-L41

[Flydiverny]

If there's specific steps to get it all working it would be great if someone wants to do a PR to update the docs :)

[dudicoco]

Thanks @imaffe, turns out my namespace wasn't aligned in the trusted relationships of the IAM role.
@Flydiverny doesn't seem like there are special steps to get it working except for the usual steps needed to configure pod IAM role OIDC token, was user error on our part.
securityContext with fsGroup: 65534 is needed though, I'm not sure if it's documented?

Thanks for the help guys!

[mo-saeed]

Anybody tried it with EKS? the pod is always trying to use the node_group role although I have all configured right and this file /var/run/secrets/eks.amazonaws.com/serviceaccount/token exists in the pod

[dudicoco]

@mo-saeed yes we are using it with EKS, take a look at my previous comments in this issue.