Issue with auth/aws/login when behind a kiam sts provider in kubernetes

[caleyg]

versions:
aws_region us-east-1
hvac 0.10.0
vault server version Version 0.10.4
vault client version Vault v1.2.3

vault_aws_auth_backend_role role:

$ terraform state show vault_aws_auth_backend_role.service_ro\[\"some-role\"\]
# vault_aws_auth_backend_role.service_ro["some-role"]:
resource "vault_aws_auth_backend_role" "service_ro" {
    allow_instance_migration        = false
    auth_type                       = "iam"
    backend                         = "aws"
   bound_account_ids               = []
    bound_ami_ids                   = []
    bound_ec2_instance_ids          = []
    bound_iam_instance_profile_arns = []
    bound_iam_principal_arns        = [
        "arn:aws:iam::REDACTED:role/some/role/name",
    ]
{...}

This issue seems to occuring when using both the hvac.api.auth_methods.aws.Aws.iam_login and client.auth_aws_iam methods and in a kubernetes environment behind a kiam service (https://github.com/uswitch/kiam)

Essentially when passing in credentials to the above methods, hvac seems to call out to STS and STS responds with The security token included in the request is invalid.

At first I thought it could be that hvac is making a different call to sts than what boto3 is trying to do, but then I can use vault cli to perform the same action and vault cli can actually complete the request. See below:

vault_cmd = f'vault login -address={vault_addr} -method=aws -token-only role={vault_role}'
auth_token = subprocess.check_output(vault_cmd, shell=True, universal_newlines=True)
print(auth_token)

which yields a token which as been redacted below:

INTENTIONALLY REDACTED
CompletedProcess(args='vault login -address=https://some.dev.vault.net -method=aws -token-only role=some-role', returncode=0)

After digging around in the vault cli code, it seems that vault cli is using botocore to construct a credentials object and having the library sign it with sigv4 signing, and then AWS is giving back credentials that then Vault Client can use to auth against the Vault Server backend yielding a token for the session.

Below is more or less some simple code demonstrating the event as well as the error

===python===
session = boto3.Session()
credentials = session.get_credentials()
print(credentials.access_key)
print(credentials.secret_key)
print(credentials.token)
print(session.client('sts').get_caller_identity())
os.environ['REQUESTS_CA_BUNDLE'] = '/etc/ssl/certs/ca-certificates.crt'
client = hvac.Client(url='some-vault')
client.auth.aws.iam_login(credentials.access_key, credentials.secret_key,role='some-role')

===output===
access-key-from-boto3
secret-key-from-boto3
some-long-token-from-boto3
{'UserId': 'some-kiam-access-key:kiam-kiam', 'Account': '*****', 'Arn': 'arn:aws:sts::****:assumed-role/<domain>/kiam-kiam', 'ResponseMetadata': {'RequestId': 'some-request-id-uuid', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'some-request-id-uuid', 'content-type': 'text/xml', 'content-length': '460', 'date': 'Thu, 26 Mar 2020 19:21:15 GMT'}, 'RetryAttempts': 0}}
Traceback (most recent call last):
  File "docker_entrypoint.py", line 387, in <module>
    main()
  File "docker_entrypoint.py", line 378, in main
    client.auth.aws.iam_login(credentials.access_key, credentials.secret_key,role='some-role')
  File "/usr/local/lib/python3.7/dist-packages/hvac/api/auth_methods/aws.py", line 600, in iam_login
    json=params,
  File "/usr/local/lib/python3.7/dist-packages/hvac/adapters.py", line 174, in login
    response = self.post(url, **kwargs).json()
  File "/usr/local/lib/python3.7/dist-packages/hvac/adapters.py", line 103, in post
    return self.request('post', url, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/hvac/adapters.py", line 272, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/usr/local/lib/python3.7/dist-packages/hvac/utils.py", line 32, in raise_for_error
    raise exceptions.InvalidRequest(message, errors=errors)
hvac.exceptions.InvalidRequest: error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidClientTokenId</Code>
    <Message>The security token included in the request is invalid.</Message>
  </Error>
  <RequestId>some-request-id-uuid</RequestId>
</ErrorResponse>

Lastly we thought it might be related to bound_account_ids but we tried setting that on the role and nothing really changed.

I'll have to use python to subprocess out to vault cli to get a token in memory for now, but if this can be looked into that would be great. We would really like a purely pythonic way to retrieve the vault token from vault using the auth/aws/login endpoint.

[ramsayza]

We've just hit this too exact same setup and version:

        session = boto3.Session()
        credentials = session.get_credentials()
        print ("Got Credentials ")
        time.sleep(10)
        _vault_client.auth.aws.iam_login(credentials.access_key, credentials.secret_key, credentials.token)

--errors out--
  <Error>
    <Type>Sender</Type>
    <Code>InvalidClientTokenId</Code>
    <Message>The security token included in the request is invalid.</Message>
  </Error>
  <RequestId>3b8e0294-70ef-4da3-9cda-e49640fb3033</RequestId>
</ErrorResponse>