Skip to content

Latest commit

 

History

History
 
 

deploy_ecs

README.md

Deploying to AWS ECS Using Docker Compose

This directory contains annotated files for deploying Dagster with an EcsRunLauncher to AWS ECS using Docker Compose. You can use these as a reference for configuring your own Dagster ECS deployment.

Prerequisites

  1. Install Docker
  2. Create an AWS account
  3. Install the AWS CLI
  4. Configure IAM permissions
  5. Create a Docker ECS context:
docker context create ecs dagster-ecs
  1. Create ECR Repositories for our images:
aws ecr create-repository --repository-name deploy_ecs/dagit
aws ecr create-repository --repository-name deploy_ecs/daemon
aws ecr create-repository --repository-name deploy_ecs/user_code
  1. Log in to your ECR Registry (ensure that the $AWS_REGION environment variable is set to your registry's AWS region):
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text | cut -f1)
export REGISTRY_URL=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REGISTRY_URL

Build and Push Images

Our docker-compose.yaml builds all of its images from local multi-stage Dockerfile. To expose these images to ECS, we first need to build them and then push them to the ECR Repositories we created:

  1. docker compose build
  2. docker compose push

Deploying Dagster

docker --context dagster-ecs compose --project-name dagster up

The first time you run this command, Docker will provision the necessary resources to run your stack on ECS. --context dagster-ecs tells Docker to run the containers on the Docker ECS context that we created. --project-name dagster tells Docker how to name the stack.

One of the resources Docker creates is an AWS ELB Load Balancer. You can access Dagit on port 3000 of the Load Balancer's URL. To find the Load Balancer's URL, look for the Load Balancer tagged with our project name.

Subsequent runs of this command will execute a rolling update of the stack.

Destroying

Once you're done experimenting with this reference deployment, you should destroy the AWS resources it creates.

docker --context dagster-ecs compose --project-name dagster down

This will stop your running services and tear down any infrastructure Docker provisioned on your behalf.