Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"ENI: RequestLimitExceeded: Request limit exceeded." when creating many EC2 instances at the same time #3374

Closed
2 tasks done
tklikifi opened this issue Jun 15, 2020 · 1 comment · Fixed by #3465 or #3479
Closed
2 tasks done

"ENI: RequestLimitExceeded: Request limit exceeded." when creating many EC2 instances at the same time #3374

tklikifi opened this issue Jun 15, 2020 · 1 comment · Fixed by #3465 or #3479
Labels
bug This issue is a bug.

Comments

@tklikifi
Copy link

Confirm by changing [ ] to [x] below:

Describe the question
I'm trying to create hundreds of EC2 instances with multiple network interfaces each. I'm using terraform v0.12.26, terraform-provider-aws v2.66.0 with aws-sdk-go v1.32.1. If I use terraform option -parallelism with value larger than 5, I get these errors:

  • Error: Failure updating ENI: RequestLimitExceeded: Request limit exceeded.
  • Error: Failure to assign Private IPs: RequestLimitExceeded: Request limit exceeded.

I'm using "max_retries" value 25 in terraform-provider-aws configuration.

aws-sdk-go should take care of API retries using its defaultRetryer, but there is some code that is specific to these two case service/ec2/customizations.go:27. Is there a reason for this code or am I doing something wrong?

I created custom version of aws-sdk-go and removed the lines 27-36 and I don't get these errors any more. I'd like to understand if there is a good reason for using lower retry values for these operations.
@tklikifi tklikifi added guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Jun 15, 2020
@diehlaws diehlaws self-assigned this Jun 24, 2020
@diehlaws diehlaws added bug This issue is a bug. and removed guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Jun 24, 2020
@diehlaws
Copy link
Contributor

Hi @tklikifi, thanks for reaching out to us about this. Unfortunately this was a miss in the customization you linked to for EC2 - this should be taking the MaxRetries value for the session's config into account. We'll work on fixing this shortly.

jasdel added a commit to jasdel/aws-sdk-go that referenced this issue Aug 7, 2020
@jasdel
service/ec2: Fix max retries with client customizations
b7902f5 
Fixes aws#3374 by correcting the EC2 API client's customization for
ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations
to use the aws.Config.MaxRetries value if set. Previously the API client's
customizations would ignore MaxRetries specified in the SDK's
aws.Config.MaxRetries field.
@jasdel jasdel mentioned this issue Aug 7, 2020
Merged
jasdel added a commit to jasdel/aws-sdk-go that referenced this issue Aug 7, 2020
@jasdel
service/ec2: Fix max retries with client customizations
804c113 
Fixes aws#3374 by correcting the EC2 API client's customization for
ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations
to use the aws.Config.MaxRetries value if set. Previously the API client's
customizations would ignore MaxRetries specified in the SDK's
aws.Config.MaxRetries field.
jasdel added a commit to jasdel/aws-sdk-go that referenced this issue Aug 11, 2020
@jasdel
service/ec2: Fix max retries with client customizations
8b922c2 
Fixes aws#3374 by correcting the EC2 API client's customization for
ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations
to use the aws.Config.MaxRetries value if set. Previously the API client's
customizations would ignore MaxRetries specified in the SDK's
aws.Config.MaxRetries field.
jasdel added a commit to jasdel/aws-sdk-go that referenced this issue Aug 11, 2020
@jasdel
service/ec2: Fix max retries with client customizations
fa56621 
Fixes aws#3374 by correcting the EC2 API client's customization for
ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations
to use the aws.Config.MaxRetries value if set. Previously the API client's
customizations would ignore MaxRetries specified in the SDK's
aws.Config.MaxRetries field.
jasdel added a commit that referenced this issue Aug 12, 2020
@jasdel
service/ec2: Fix max retries with client customizations (#3465)
10a6e5a 
Fixes #3374 by correcting the EC2 API client's customization for ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations to use the aws.Config.MaxRetries value if set. Previously the API client's customizations would ignore MaxRetries specified in the SDK's aws.Config.MaxRetries field.
aws-sdk-go-automation pushed a commit that referenced this issue Aug 12, 2020
Release v1.34.3 (2020-08-12)
3386d60 
===

### Service Client Updates
* `service/cloud9`: Updates service API and documentation
  * Add ConnectionType input parameter to CreateEnvironmentEC2 endpoint. New parameter enables creation of environments with SSM connection.
* `service/comprehend`: Updates service documentation
* `service/ec2`: Updates service API and documentation
  * Introduces support for IPv6-in-IPv4 IPsec tunnels. A user can now send traffic from their on-premise IPv6 network to AWS VPCs that have IPv6 support enabled.
* `service/fsx`: Updates service API and documentation
* `service/iot`: Updates service API, documentation, and paginators
  * Audit finding suppressions: Device Defender enables customers to turn off non-compliant findings for specific resources on a per check basis.
* `service/lambda`: Updates service API and examples
  * Support for creating Lambda Functions using 'java8.al2' and 'provided.al2'
* `service/transfer`: Updates service API, documentation, and paginators
  * Adds security policies to control cryptographic algorithms advertised by your server, additional characters in usernames and length increase, and FIPS compliant endpoints in the US and Canada regions.
* `service/workspaces`: Updates service API and documentation
  * Adds optional EnableWorkDocs property to WorkspaceCreationProperties in the ModifyWorkspaceCreationProperties API

### SDK Enhancements
* `codegen`: Add XXX_Values functions for getting slice of API enums by type.
  * Fixes [#3441](#3441) by adding a new XXX_Values function for each API enum type that returns a slice of enum values, e.g `DomainStatus_Values`.
* `aws/request`: Update default retry to retry "use of closed network connection" errors ([#3476](#3476))
  * Fixes [#3406](#3406)

### SDK Bugs
* `private/protocol/json/jsonutil`: Fixes a bug that truncated millisecond precision time in API response to seconds. ([#3474](#3474))
  * Fixes [#3464](#3464)
  * Fixes [#3410](#3410)
* `codegen`: Export event stream constructor for easier mocking ([#3473](#3473))
  * Fixes [#3412](#3412) by exporting the operation's EventStream type's constructor function so it can be used to fully initialize fully when mocking out behavior for API operations with event streams.
* `service/ec2`: Fix max retries with client customizations ([#3465](#3465))
  * Fixes [#3374](#3374) by correcting the EC2 API client's customization for ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations to use the aws.Config.MaxRetries value if set. Previously the API client's customizations would ignore MaxRetries specified in the SDK's aws.Config.MaxRetries field.
Merged
aws-sdk-go-automation added a commit that referenced this issue Aug 12, 2020
@aws-sdk-go-automation
Release v1.34.3 (2020-08-12) (#3479)
20cd465 
Release v1.34.3 (2020-08-12)
===

### Service Client Updates
* `service/cloud9`: Updates service API and documentation
  * Add ConnectionType input parameter to CreateEnvironmentEC2 endpoint. New parameter enables creation of environments with SSM connection.
* `service/comprehend`: Updates service documentation
* `service/ec2`: Updates service API and documentation
  * Introduces support for IPv6-in-IPv4 IPsec tunnels. A user can now send traffic from their on-premise IPv6 network to AWS VPCs that have IPv6 support enabled.
* `service/fsx`: Updates service API and documentation
* `service/iot`: Updates service API, documentation, and paginators
  * Audit finding suppressions: Device Defender enables customers to turn off non-compliant findings for specific resources on a per check basis.
* `service/lambda`: Updates service API and examples
  * Support for creating Lambda Functions using 'java8.al2' and 'provided.al2'
* `service/transfer`: Updates service API, documentation, and paginators
  * Adds security policies to control cryptographic algorithms advertised by your server, additional characters in usernames and length increase, and FIPS compliant endpoints in the US and Canada regions.
* `service/workspaces`: Updates service API and documentation
  * Adds optional EnableWorkDocs property to WorkspaceCreationProperties in the ModifyWorkspaceCreationProperties API

### SDK Enhancements
* `codegen`: Add XXX_Values functions for getting slice of API enums by type.
  * Fixes [#3441](#3441) by adding a new XXX_Values function for each API enum type that returns a slice of enum values, e.g `DomainStatus_Values`.
* `aws/request`: Update default retry to retry "use of closed network connection" errors ([#3476](#3476))
  * Fixes [#3406](#3406)

### SDK Bugs
* `private/protocol/json/jsonutil`: Fixes a bug that truncated millisecond precision time in API response to seconds. ([#3474](#3474))
  * Fixes [#3464](#3464)
  * Fixes [#3410](#3410)
* `codegen`: Export event stream constructor for easier mocking ([#3473](#3473))
  * Fixes [#3412](#3412) by exporting the operation's EventStream type's constructor function so it can be used to fully initialize fully when mocking out behavior for API operations with event streams.
* `service/ec2`: Fix max retries with client customizations ([#3465](#3465))
  * Fixes [#3374](#3374) by correcting the EC2 API client's customization for ModifyNetworkInterfaceAttribute and AssignPrivateIpAddresses operations to use the aws.Config.MaxRetries value if set. Previously the API client's customizations would ignore MaxRetries specified in the SDK's aws.Config.MaxRetries field.
@diehlaws diehlaws removed their assignment Aug 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
2 participants
@tklikifi @diehlaws