Graceful AWS scaling event on Kubernetes using lifecycle hooks
lifecycle-manager is a service that can be deployed to a Kubernetes cluster in order to make AWS autoscaling events more graceful using draining
Certain termination activities such as AZRebalance or TerminateInstanceInAutoScalingGroup API calls can cause autoscaling groups to terminate instances without having them properly drain first.
This can cause apps to experience errors when they are abruptly terminated.
lifecycle-manager uses lifecycle hooks from the autoscaling group (via SQS) to pre-drain the instances for you.
In addition to node draining, lifecycle-manager also tries to deregister the instance from any discovered ALB target group, this helps with pre-draining for the ALB instances prior to shutdown in order to avoid in-flight 5xx errors on your ALB - this feature is currently supported for aws-alb-ingress-controller.
-
Follow the AWS docs to create an SQS queue named
lifecycle-manager-queue, a notification role, and a lifecycle-hook on your autoscaling group pointing to the created queue. -
Deploy lifecycle-manager to your cluster:
kubectl create namespace lifecycle-manager
kubectl apply -f https://raw.githubusercontent.com/keikoproj/lifecycle-manager/master/examples/lifecycle-manager.yaml- Kill an instance in your scaling group and watch it getting drained:
$ aws autoscaling terminate-instance-in-auto-scaling-group --instance-id i-0d3ba307bc6cebeda --region us-west-2 --no-should-decrement-desired-capacity
{
"Activity": {
"ActivityId": "5285b629-6a18-0a43-7c3c-f76bac8205f0",
"AutoScalingGroupName": "my-scaling-group",
"Description": "Terminating EC2 instance: i-0d3ba307bc6cebeda",
"Cause": "At 2019-10-02T02:44:11Z instance i-0d3ba307bc6cebeda was taken out of service in response to a user request.",
"StartTime": "2019-10-02T02:44:11.394Z",
"StatusCode": "InProgress",
"Progress": 0,
"Details": "{\"Subnet ID\":\"subnet-0bf9bc85fEXAMPLE\",\"Availability Zone\":\"us-west-2c\"}"
}
}
$ kubectl logs lifecycle-manager
time="2019-10-02T02:44:05Z" level=info msg="starting lifecycle-manager service v0.2.0"
time="2019-10-02T02:44:05Z" level=info msg="region = us-west-2"
time="2019-10-02T02:44:05Z" level=info msg="queue = https://sqs.us-west-2.amazonaws.com/00000EXAMPLE/lifecycle-manager-queue"
time="2019-10-02T02:44:05Z" level=info msg="polling interval seconds = 10"
time="2019-10-02T02:44:05Z" level=info msg="drain timeout seconds = 300"
time="2019-10-02T02:44:05Z" level=info msg="drain retry interval seconds = 30"
time="2019-10-02T02:44:05Z" level=info msg="spawning sqs poller"
time="2019-10-02T02:44:12Z" level=info msg="spawning event handler"
time="2019-10-02T02:44:12Z" level=info msg="hook heartbeat timeout interval is 60, will send heartbeat every 30 seconds"
time="2019-10-02T02:44:12Z" level=info msg="draining node ip-10-10-10-10.us-west-2.compute.internal"
time="2019-10-02T02:44:42Z" level=info msg="sending heartbeat for event with instance 'i-0d3ba307bc6cebeda' and sleeping for 30 seconds"
time="2019-10-02T02:44:45Z" level=info msg="completed drain for node 'ip-10-10-10-10.us-west-2.compute.internal'"
time="2019-10-02T02:44:45Z" level=info msg="deregistering i-0d3ba307bc6cebeda from arn:aws:elasticloadbalancing:us-west-2:00000EXAMPLE:targetgroup/targetgroup-9b26c8689f3b53a1ef0/53e66aede612f044"
time="2019-10-02T02:44:45Z" level=info msg="sending heartbeat for event with instance 'i-0d3ba307bc6cebeda' and sleeping for 30 seconds"
time="2019-10-02T02:45:15Z" level=info msg="setting lifecycle event as completed with result: 'CONTINUE'"{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeLifecycleHooks",
"autoscaling:CompleteLifecycleAction",
"autoscaling:RecordLifecycleActionHeartbeat",
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTargetGroups"
],
"Resource": "*"
}- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: ["extensions", "apps"]
resources: ["daemonsets"]
verbs: ["get"]Please see CHANGELOG.md.
Please see CONTRIBUTING.md.
Please see DEVELOPER.md.