Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add WARM_IP_TARGET support #125

Merged
merged 1 commit into from
Jul 17, 2018
Merged

Add WARM_IP_TARGET support #125

merged 1 commit into from
Jul 17, 2018

Conversation

liwenwu-amazon
Copy link
Contributor

@liwenwu-amazon liwenwu-amazon commented Jul 7, 2018 •
edited
Loading

Issue #114:

Description of changes

This PR allows user to use a OS environment variable WARM_IP_TARGET to configure the number of pre-warming IP addresses.

You can modify amazon-vpc-cni.yaml to include WARM_IP_TARGET value


env:
          - name: AWS_VPC_K8S_CNI_LOGLEVEL
            value: DEBUG
          - name: MY_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "WARM_IP_TARGET"  <-------- define pre-warming IP
             value: "1"   <--------   This must be a integer, e.g. 1 IP address

If WARM_IP_TARGET is NOT defined, ipamD will fallback to current allocation behavior.

Today, by-default, if number of available IP addresses is below ENI's IP address, ipamD will allocate a new ENI. For example, for a t2.medium worker node, if the number of available IP address is below 5, ipamD will allocate a new ENI, allocate IP addresses(5 addresses) on this new ENI.

Caveat

If WARM_IP_TARGET is set too low and more Pods are expected to be scheduled on the node, ipamD will invoke more EC2 API calls to allocate IP addresses. For example, for a m4.4xlarge worker node, each ENI can have up to 30 addresses. If "WARM_IP_TARGET" is set to 1 and if there are more than 30 pods scheduled on the node, there will be 30 EC2 AssignPrivateIpAddresses() API call for each ENI compare to 1 EC2 AssignPrivateIpAddresses() API call using default behavior (as today), and in worst case for a node, 240 EC2 AssignPrivateIpAddresses() API calls compare to 8 EC2 AssignPrivateIpAddresses() API calls using default behavior(as today). If the cluster is large and contains lots of worker nodes, this can cause ipamD running into EC2 AssignPrivateIpAddresses() API call throttling problems.

Tests

  • define WARM_IP_TARGET=1 and uses t2.medium worker node where each ENI can have up to 5 Pods IP addresses

    • allocate 2 Pods, and verify there is 1 more available IP addresses
    • allocate 5 pods, and verify 1 ENI allocated and there is only 1 more IP addressed pre-warmed

  • regression test and make sure default behavior still works. Do NOT define WARM_IP_TARGET, uses t2.medium worker

    • allocate 2 Pods, and verify ipamD will allocate 1 extra ENI and have 10 IP addresses pre-warmed

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@liwenwu-amazon liwenwu-amazon added this to the v1.1 milestone Jul 11, 2018
nckturner
nckturner suggested changes Jul 16, 2018
View reviewed changes
ipamd/ipamd.go Outdated
err := c.awsClient.AllocAllIPAddress(eni.ID)
var err error
if warmIPTargetDefined {
err = c.awsClient.AllocIPAddresses(eni.ID, curIPTarget)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be separate functions?

ipamd/ipamd.go
@@ -664,3 +700,35 @@ func (c *IPAMContext) eniIPPoolReconcile(ipPool map[string]*datastore.AddressInf
return nil

}

func getWarmIPTarget() int {
inputStr, found := os.LookupEnv("WARM_IP_TARGET")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of WARM_IP_TARGET which I don't think is very descriptive, I propose NUM_WARM_IPS, or MAX_WARM_IPS each of which signifies a number rather than a resource.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity how is this supposed to work if you assigned a value of 5? Right from the start there will be 5 secondary IPs available for pods to use, but let's say 3 pods are started on that particular machine... Does the pool of secondary IPs get expanded back out to 5, if so does it allocate three new addresses one at a time? Alternatively does the pool of available IPs become 2, then when those 2 IPs are eventually used then a new block of 5 IPs will be attached when needed. The naming for the former might be NUM_WARM_IPS while the later might be IP_ALLOCATION_BLOCK_COUNT or something along those lines.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if WARM_IP_TARGET is 5,

  • at beginning, the available IPs is 5
  • after 3 Pods assigned, ipamD will try to allocate 3 more IPs in the ipamD controller loops. ipamD will try to use 1 EC2 API if it is possible. If it requires allocating 1 more ENIs, or EC2 control plane can not allocate 3 IP address at that time, ipamD controller loops will continuously retry to reach the target of 5 IPs

There are 2 reasons I choose WARM_IP_TARGET

  • There is another configure knob WARM_ENI_TARGET which works similar but at ENI level.

  • ...TARGET means the desired value, and ipamD controller loops will try continuously to reach this TARGET.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see ipPoolMonitorInterval is set to run every 5 seconds to check to see if it needs to increase the IP pool. Is that value perhaps set too low? Or if many pods are launched simultaneously will it all fall within the same time window to not cause excessive EC2 API calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is default behavior (no WARM_IP_TARGET is configured), in worst case, in 5 second interval, there are 2 EC2 APIs, one for ENI creation and one for allocating all IP address on the ENI. If maximum pods are scheduled, the number API calls is 2 * num-of-ENIs.

If WARM_IP_TARGET is configured too low, e.g. 1 You are right, it can cause excessive EC2 API call. For example, for a m4.4xlarge, it is 240 APIs compare to 16 APIs using default behavior.

pkg/awsutils/awsutils.go Outdated
@@ -826,6 +829,36 @@ func (cache *EC2InstanceMetadataCache) GetENILimit() (int, error) {
return eniLimit, nil
}

// Allocate alloactes numIPs of IP address on a eni
func (cache *EC2InstanceMetadataCache) AllocIPAddresses(eniID string, numIPs int) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function should be combined with AllocAllIPAddress().

nckturner
nckturner approved these changes Jul 17, 2018
View reviewed changes
Copy link
Contributor

@nckturner nckturner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liwenwu-amazon liwenwu-amazon merged commit 2d0a6d9 into aws:master Jul 17, 2018
@alfredkrohmer
Copy link

It would be nice to have any kind of documentation for that feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sdavids13 sdavids13 sdavids13 left review comments

@nckturner nckturner nckturner approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.1
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@liwenwu-amazon @alfredkrohmer @sdavids13 @nckturner