Terraform plan to deploy an autoscaling CircleCI Runner cluster.

• Terraform (>= 0.13.6)
• CircleCI CLI (>= 0.1.11924)

The diagram below shows how the infrastructure deployed by thisTerraform plan works.

Some general notes about how the plan works:

• An Auto Scaling group (ASG) will be deployed with a launch template for automatically installing Runner on Server.
  • The ASG will start with min/max values specified by the user when supplying variables for the Terraform plan.
• A Lambda function will also be deployed to check the CircleCI API for the current queue depth for the resource class represented by the servers running in the ASG.
  • This function will retrieve the queue depth value and post the value to Cloudwatch.
  • If a user-specified queue depth is exceeded, Cloudwatch will notify EC2 Auto Scaling to trigger a scale-out event.
  • When the queue depth is below a certain threshold for a certain (configurable) amount of time, Cloudwatch will notify EC2 Auto Scaling to trigger a scale-in event.

The scale-out process is shown by the numbers on the diagram, which represent these steps that occur in order:

1. A cron-based Cloudwatch alarm triggers a Lambda function that will check the current queue depth.
2. The Lambda function retrieves the user-supplied CircleCI API token and resource class name from AWS Secrets Manager using a KMS key.
3. The Lambda function requests the current queue depth from the CircleCI Runner API.
4. The queue depth value is sent to a Cloudwatch metric.
5. A Cloudwatch alarm monitors the metric.
6. If the metric exceeds a specified threshold, it notifies the Auto Scaling group, which then executes a scaling action according to its policies.

1. Create a namespace, resource class, and runner token as described in the documentation.
2. Open terraform.tfvars.example and replace required values
3. Check optional values to ensure the runners are configured appropriately for your use case
4. (Optional, but strongly recommended) Add a remote state backend to store your terraform state
5. Run terraform plan and inspect proposed changes
6. Run terraform apply to apply changes
7. Verify that runners are registered with your CircleCI org by using the command circleci runner instance list <your-org-namespace-here>

Optional: If you would like to do a sandbox deploy to test the Terraform plan using CircleCI, follow these steps:

1. Enter the necessary values in terraform.tfvars.example and save your changes
2. Run the following bash command: base64 terraform.tfvars.example > base64_tfvars.txt
3. Store the output in a CircleCI context or project-level variable named BASE64_TFVARS.

• aws_security_group.circleci_runner
• aws_security_group_rule.allow_inbound_ssh
• aws_security_group_rule.allow_outbound
• aws_placement_group.circleci_runner
• aws_autoscaling_group.circleci_runner
• aws_launch_template.circleci_runner
• aws_iam_role.queue_depth_lambda_role
• aws_iam_policy.queue_depth_lambda_role
• aws_iam_role_policy_attachment.queue_depth_lambda_role
• aws_lambda_function.queue_depth
• aws_cloudwatch_log_group.queue_depth_lambda
• aws_secretsmanager_secret.queue_depth_lambda_secrets
• aws_secretsmanager_secret_version.queue_depth_lambda_secrets
• aws_kms_key.queue_depth_lambda_secrets
• aws_kms_alias.queue_depth_lambda_secrets
• aws_cloudwatch_metric_alarm.scale_out
• aws_cloudwatch_metric_alarm.scale_in
• aws_autoscaling_policy.scale_out
• aws_autoscaling_policy.scale_in
• aws_cloudwatch_event_rule.run_queue_depth_lambda
• aws_cloudwatch_event_target.run_queue_depth_lambda
• aws_lambda_permission.allow_cloudwatch