SDK config differs from AWS cli tool #384

mtibben · 2015-09-17T07:02:25Z

The AWS CLI tool stores the configured region in ~/.aws/config, however the golang SDK does not read it. For example, the config file might specify the region with

[default]
region = us-east-1

So when using the env var AWS_SHARED_CREDENTIALS_FILE=~/.aws/config, the default region actually comes from AWS_REGION instead. I understand the credentials file is different to the config file, but still it would be nice if the SDK and CLI were able to share config

Secondly, I noticed the SDK reads AWS_PROFILE and AWS_REGION instead of AWS_DEFAULT_PROFILE and AWS_DEFAULT_REGION as the cli tool does.

Again I can understand why things have been done this way - but is it necessary to have these differing ways to configure AWS tools? It would be easier if there was consistency.

The text was updated successfully, but these errors were encountered:

jasdel · 2015-09-18T22:45:42Z

Hi @mtibben thanks for posting this issue. Correct the Go SDK only supports the credentials file not the AWS CLI's config file. I'll mark this as a feature request were we could add support for the CLI specific configuration options.

wt · 2015-10-16T18:00:34Z

Here's the doc for the CLI tool:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

Any chance that those env vars could be consulted as well?

jasdel · 2015-10-16T19:24:14Z

Other than the CLI config, the only environment vars the Go SDK doesn't support which the CLI is AWS_DEFAULT_REGION and AWS_DEFAULT_PROFILE in the SDKs these are AWS_REGION, AWS_PROFILE.

It should be noted though that adding support for these two additional environment configuration values would introduce breaking changes into the SDK.

phoolish · 2015-12-07T15:55:05Z

I might add that it would be helpful to have cross account role support as defined in this blog post.

jasdel · 2015-12-08T14:47:51Z

Thanks for the suggestion @phoolish This is a nice feature of the CLI and would add nice functionality. I think you're able to do this now with the Go SDK but it requires a bit of manual process. Having this built in would be helpful. We're always looking for new features and PR, if you'd like to take a look at how this could be added to the SDK we'd be more than glad to review it. If you do, please create an additional issue so we can track that feature separately.

phoolish · 2015-12-08T22:08:05Z

Don't know too much go, but I was looking for a holiday project.

smile-on · 2016-01-08T19:04:50Z

@jasdel This issue with sdk ignoring shared config may be very annoying to anyone who uses aws sdk in the project that more than just a toy. It seams natural for integration tests to use connectivity config stored at the server in shared location and that would run into the limitation issue.

I'd like to make simple patch to aws-sdk-go that would allows sdk to read region from shared config if no value is given explicitly in code (as mentioned in #489). This is much simpler functionality and easy to test scope that this #384. The most importantly it helps in running practical integration test that share both credentials and config with CLI tools on the server. Would such simple patch be considered as a pool request or there is a plan for something bigger?

jasdel · 2016-01-08T23:54:46Z

@smile-on Thanks for looking into this. We haven't added because we need to determine, if the SDKs were to support the CLI's config file, dhow far to go with the integration. Generally the region and credentials are helpful because it is an obvious configuration value which applies to all the SDK's and CLI. But the majority of the CLI's config file is specific to the CLI and its features.

In addition any change we make to how the SDK sources its configuration will need to backwards compatible. This most likely will mean loading a configuration from an additional source will need to be an opt-in process. Either through a flag in code or or environmental variable so that it does not automatically enable unexpectedly.

Bowbaq · 2016-03-04T02:07:51Z

Ran into this a few days ago, so I wrote a provider that reads from the CLI config file

temujin9 · 2016-07-02T00:07:41Z

@jasdel Have any decisions been made on priority for this? Its lack blocks a rather large chunk of my upcoming work.

jasdel · 2016-07-07T20:00:35Z

Hi i'm starting to look at implementing this feature. Support for this feature would need to be opt so not to unexpectedly break user application's. Users should be able to opt into this feature vai an env var, and via in code configuration.

temujin9 · 2016-07-08T18:16:26Z

https://github.com/Bowbaq/profilecreds has worked as a workaround: I would recommend it as a stepping-off point.

Adds opt-in support for loading the SDK's configuration from the shared config file (~/.aws/config) in addition to the current shared configuration file (~/.aws/credentials). This option can be enabled with the env var `AWS_SDK_LOAD_CONFIG` set to a truthy value, or creating a session with the `session.NewFromSharedConfig()` helper instead of `session.New()`. In addition to loading from the shared config file the SDK will also now load region from the shared credentials file when the share config is enabled. This changes also treats the syntax of both shared config and credentials as the same. See the session package for more information and examples. Adds shared config support for aws#384

jasdel · 2016-07-18T20:58:33Z

Thanks for the link @mtibben I've created a PR #761 which is the initial step to update the SDK to load configuration from the shared config file (~~/.aws/config) in addition to the shared credentials file (~~/.aws/credentials). I'm working on the Assume Role work now, and will update the PR once that work is ready.

Adds opt-in support for loading the SDK's configuration from the shared config file (~/.aws/config) in addition to the current shared configuration file (~/.aws/credentials). This option can be enabled with the env var `AWS_SDK_LOAD_CONFIG` set to a truthy value, or creating a session with the `session.NewFromSharedConfig()` helper instead of `session.New()`. In addition to loading from the shared config file the SDK will also now load region from the shared credentials file when the share config is enabled. This changes also treats the syntax of both shared config and credentials as the same. See the session package for more information and examples. Adds shared config support for aws#384

jasdel · 2016-07-26T20:47:16Z

Hi all I've created PR #761 which adds support for reading the shared config file to the SDK, in addition to assume role from a profile in the SDK's default credential chain.

I'm looking to gather any feedback about the change.

Adds opt-in support for loading the SDK's configuration from the shared config file (~/.aws/config) in addition to the current shared configuration file (~/.aws/credentials). This option can be enabled with the env var `AWS_SDK_LOAD_CONFIG` set to a truthy value, or creating a session with the `session.NewFromSharedConfig()` helper instead of `session.New()`. In addition to loading from the shared config file the SDK will also now load region from the shared credentials file when the share config is enabled. This changes also treats the syntax of both shared config and credentials as the same. See the session package for more information and examples. Adds shared config support for aws#384

mtibben · 2016-08-01T11:25:54Z

Hey thanks @jasdel, to be honest I can't remember the exact use-case I was having issues with when reporting this bug, but these days I'm using https://github.com/99designs/aws-vault which automatically sets all the potential environment variables like AWS_REGION and AWS_DEFAULT_REGION based on the ~/.aws/config. This has been working quite nicely for some time.

This work is certainly appreciated however, hopefully it leads to a smoother UX when using the SDK! Thanks!

aws/session: Add support for shared config (~/.aws/config) file Fix #384

jasdel · 2016-08-02T18:31:00Z

Thanks for the update, and link @mtibben! I've merged in the change that implements the shared config with assume role support.

Let us know if you have any issues or additional feedback.

References: #384, #743, #761, #783, #784, #785

Ports v1 SDK's retry utilities to follow a consistent code path. aws.IsErrorRetryable is the primary entry point to determine if a request is retryable. Also ports support for service/kinesis to retry service specific API errors . Corrects sdk's behavior by not retrying errors with status code 501. Related to aws#2744, aws#2774, aws#2751 and aws#1826 Fixes aws#372; Fixes aws#145

Services --- * Synced the V2 SDK with latest AWS service API definitions. SDK Enhancements --- * `aws/endpoints`: Expose DNSSuffix for partitions ([aws#369](aws/aws-sdk-go-v2#369)) * Exposes the underlying partition metadata's DNSSuffix value via the `DNSSuffix` method on the endpoint's `Partition` type. This allows access to the partition's DNS suffix, e.g. "amazon.com". * Fixes [aws#347](aws/aws-sdk-go-v2#347) * `private/protocol`: Add support for parsing fractional timestamp ([aws#367](aws/aws-sdk-go-v2#367)) * Fixes the SDK's ability to parse fractional unix timestamp values and adds tests. * Fixes [aws#365](aws/aws-sdk-go-v2#365) * `aws/ec2metadata`: Add marketplaceProductCodes to EC2 Instance Identity Document ([aws#374](aws/aws-sdk-go-v2#374)) * Adds `MarketplaceProductCodes` to the EC2 Instance Metadata's Identity Document. The ec2metadata client will now retrieve these values if they are available. * Related to: [aws#2781](aws#2781) * `aws`: Adds configurations to the default retryer ([aws#375](aws/aws-sdk-go-v2#375)) * Provides more customization options for retryer by adding a constructor for default Retryer which accepts functional options. Adds NoOpRetryer to support no retry behavior. Exposes members of default retryer. * Updates the underlying logic used by the default retryer to calculate jittered delay for retry. Handles int overflow for retry delay. * Fixes [aws#370](aws/aws-sdk-go-v2#370) * `aws` : Refactors request retry behavior path logic ([aws#384](aws/aws-sdk-go-v2#384)) * Retry utilities now follow a consistent code path. aws.IsErrorRetryable is the primary entry point to determine if a request is retryable. * Corrects sdk's behavior by not retrying errors with status code 501. Adds support for retrying the Kinesis API error, LimitExceededException. * Fixes [aws#372](aws/aws-sdk-go-v2#372), [aws#145](aws/aws-sdk-go-v2#145) SDK Bugs --- * `aws`: Fixes bug in calculating throttled retry delay ([aws#373](aws/aws-sdk-go-v2#373)) * The `Retry-After` duration specified in the request is now added to the Retry delay for throttled exception. Adds test for retry delays for throttled exceptions. Fixes bug where the throttled retry's math was off. * Fixes [aws#45](aws/aws-sdk-go-v2#45) * `aws` : Adds missing sdk error checking when seeking readers ([aws#379](aws/aws-sdk-go-v2#379)) * Adds support for nonseekable io.Reader. Adds support for streamed payloads for unsigned body request. * Fixes [aws#371](aws/aws-sdk-go-v2#371) * `service/s3` : Fixes unexpected EOF error by s3manager ([aws#386](aws/aws-sdk-go-v2#386)) * Fixes bug which threw unexpected EOF error when s3 upload is performed for a file of maximum allowed size * Fixes [aws#316](aws/aws-sdk-go-v2#316) * `private/model` : Fixes generated API Reference docs links being invalid ([387](aws/aws-sdk-go-v2#387)) * Fixes [aws#327](aws/aws-sdk-go-v2#327)

jasdel added the feature-request A feature should be added or improved. label Sep 18, 2015

jasdel mentioned this issue Dec 23, 2015

Feature Request: Support for sts:assumerole/source_profile #472

Closed

smile-on mentioned this issue Jan 7, 2016

default region set in ~/.aws/config is ignored by call ec2.DescribeInstances() #bug #489

Closed

chriswgerber mentioned this issue Jan 25, 2016

Update Packer to Support STSCreds for AWS Builder hashicorp/packer#3109

Closed

jeanlaurent mentioned this issue Feb 12, 2016

Support AWS_DEFAULT_PROFILE #554

Closed

jasdel mentioned this issue Mar 3, 2016

Allow configurable region endpoints map #576

Closed

jasdel mentioned this issue Mar 14, 2016

NoCredentialProviders: no valid providers in chain coreos/coreos-kubernetes#94

Closed

jasdel added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Jul 7, 2016

jasdel mentioned this issue Jul 18, 2016

aws/session: Add support for shared config (~/.aws/config) file #761

Merged

jasdel closed this as completed in #761 Aug 2, 2016

jasdel added a commit that referenced this issue Aug 2, 2016

Merge pull request #761 from jasdel/feature/SharedConfig

beae13f

aws/session: Add support for shared config (~/.aws/config) file Fix #384

jasdel removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Aug 2, 2016

jasdel added a commit that referenced this issue Aug 2, 2016

Tag release v1.3.0

eb0af7a

References: #384, #743, #761, #783, #784, #785

mtibben mentioned this issue Sep 1, 2016

Unrelated changes are made during push 99designs/iamy#13

Closed

mumoshu mentioned this issue Oct 3, 2016

Error: failed to compress TLS assets: AccessDeniedException on "kube-aws validate" coreos/coreos-kubernetes#704

Closed

garthk mentioned this issue May 11, 2018

rdslogs doesn't respect AWS_DEFAULT_PROFILE environment variable honeycombio/rdslogs#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SDK config differs from AWS cli tool #384

SDK config differs from AWS cli tool #384

mtibben commented Sep 17, 2015

jasdel commented Sep 18, 2015

wt commented Oct 16, 2015

jasdel commented Oct 16, 2015

phoolish commented Dec 7, 2015

jasdel commented Dec 8, 2015

phoolish commented Dec 8, 2015

smile-on commented Jan 8, 2016

jasdel commented Jan 8, 2016

Bowbaq commented Mar 4, 2016

temujin9 commented Jul 2, 2016

jasdel commented Jul 7, 2016

temujin9 commented Jul 8, 2016

jasdel commented Jul 18, 2016

jasdel commented Jul 26, 2016

mtibben commented Aug 1, 2016

jasdel commented Aug 2, 2016

SDK config differs from AWS cli tool #384

SDK config differs from AWS cli tool #384

Comments

mtibben commented Sep 17, 2015

jasdel commented Sep 18, 2015

wt commented Oct 16, 2015

jasdel commented Oct 16, 2015

phoolish commented Dec 7, 2015

jasdel commented Dec 8, 2015

phoolish commented Dec 8, 2015

smile-on commented Jan 8, 2016

jasdel commented Jan 8, 2016

Bowbaq commented Mar 4, 2016

temujin9 commented Jul 2, 2016

jasdel commented Jul 7, 2016

temujin9 commented Jul 8, 2016

jasdel commented Jul 18, 2016

jasdel commented Jul 26, 2016

mtibben commented Aug 1, 2016

jasdel commented Aug 2, 2016