airflow.md

eks/airflow walkthrough

A guide for using CNDI to deploy a GitOps enabled Airflow cluster on Kubernetes in Amazon Web Services

overview 🔭

This walkthough uses cndi to customize and deploy our eks/airflow Template. In just a few minutes we will be able to deploy a new Kubernetes cluster to AWS that has been optimally configured for Airflow, including GitOps with Secrets management, TLS and High Availibility right out-of-the-box. This framework will enable quick iteration of infrastructure, applications and manifests in a GitHub workflow you are already comfortable with.

prerequisites ✅

You will need the following things to get up and running with cndi successfully:

• An AWS cloud account: cndi will deploy infrastructure within AWS

• Your cloud credentials: cndi will leverage your amazon web services's AWS ACCESS KEY ID and AWS_SECRET_ACCESS_KEY credentials to deploy resources.

• A Domain Name: Because the eks/airflow template sets up TLS certificates, we need to have a domain on which to apply them. We also need access to the domain registrar so we can add a couple CNAME records there for our cluster Ingresses.

• A GitHub account: cndi helps you manage the state of your infrastructure using a GitOps workflow, so you'll need a GitHub account with a valid GitHub Personal Access Token.

• Here's a guide of how to set up your Amazon Web Services account including roles and permissions

download cndi ⬇️

Run the following command within your terminal to download and install cndi:

# this will download the correct binary for your OS
curl -fsSL https://raw.githubusercontent.com/polyseam/cndi/main/install.sh | sh

create your cndi repository 📂

CNDI is designed around a GitOps workflow, so all of your cluster configuration and infrastructure will be stored as code within a git repo, let's create that now!

gh repo create my-cndi-cluster --private --clone && cd my-cndi-cluster

creating cluster config with cndi templates using the interactive cli 🛠️

Now that we have a repo, let's use cndi to generate all of our Infrastructure as Code and Cluster Configuration:

cndi init --interactive

You will get an interactive prompt where you'll name your project, then one to specify the CNDI template you want.

For this project select the eks/airflow Template.

? Pick a template
   aws/basic
   gce/basic
   avm/basic
 ❯ eks/airflow
   avm/airflow
   gce/airflow

Below is the list of all of the interactive prompt values that should be supplied for this project:

• Cndi Project Name: name of project
• Template: list of templates to choose from

---

• GitHub Username: a user's handle on GitHub.
• GitHub Repository URL: the url for the GitHub repository that will hold all cluster configuration
• GitHub Personal Access Token: the access token CNDI will use to access your repo for cluster creation and synchronization

---

• AWS Access key ID: access keys are long-term credentials for an IAM user
• AWS Secret Access key: access keys are long-term credentials for an IAM user
• AWS Region: region where the infastructure is being created

---

• Git Username for Airflow DAG Storage: a user's handle on GitHub used to synchronize Airflow DAGs
• Git Password for Airflow DAG Storage: a personal access token used to synchronize Airflow DAGs
• Git Repo for Airflow DAG Storage: url for repo where your Airflow DAGs will be stored

---

• Domain name you want ArgoCD to be accessible on: domain where ArgoCD will be hosted
• Domain name you want Airflow to be accessible on: domain where Airflow will be hosted

---

• Email address you want to use for lets encrypt: an email for lets encrypt to use when generating certificates
• Username you want to use for airflow cnpg database: username you want to use for airflow database
• Password you want to use for airflow cnpg database: password you want to use for airflow database
• Name of the postgresql database you want to use for airflow cnpg database: name of the postgresql database you want to use for airflow cnpg database

This process will generate a cndi_config.yaml file, and cndi directory at the root of your repository containing all the necessary files for the configuration. It will also store all the values in a file called .env at the root of your repository.

The structure of the generated CNDI project will be something like:

├── 📁 cndi
│   ├── 📁 cluster_manifests
│   │   ├── 📁 applications
|   |   |   ├── cnpg.yaml 
│   │   │   └── airflow.application.yaml
│   │   ├── argo-ingress.yaml
│   │   ├── cert-manager-cluster-issuer.yaml
│   │   ├── git-credentials-secret.yaml
│   │   └── etc
│   └── 📁 terraform
│       ├── airflow-nodes.cndi-node.tf.yaml
│       └── etc 
├── cndi_config.yaml
├── .env
├── .gitignore
├── .github
└── README.md

For a breakdown of all of these files, checkout the outputs section of the repo's main README.

upload environment variables to GitHub ⬆️

GitHub Actions is responsible for calling the cndi run command to deploy our cluster, so it is important that our secrets are available in the actions runtime. However we don't want these to be visible in our source code, so we will use GitHub Actions Secrets to store them. The gh CLI makes this very easy.

gh secret set -f .env
# if this does not complete the first time, try running it again!

deploy your templated cluster configration 🚀

Once all the config is created and environment variables are uploaded to GitHub, add, commit and push the config to your GitHub repository:

git add .
git status # take a quick look and make sure these are all files you want to push
git commit -m "initial commit"
git push --set-upstream origin main

You should now see the cluster configuration has been uploaded to GitHub:

Now, open your web browser and navigate to your project on GitHub. Click on the Actions tab, then click on the job that was triggered from your latest commit.

You will see something like the image below, which shows that GitHub has successfully run the workflow.

It is common for cndi run to take a fair amount of time, as is the case with most Terraform and cloud infrastructure deployments.

Once cndi run has been completed, at the end of the run will be a link to resource groups, where you can view resources deployed by CNDI for this project.

Click on your project resource group

Click on your on eks cluster control plane

attach the load balancer to your domain 🌐

At the end of the cndi run there is also an output called public host, which is the DNS (CNAME) of the load Balancer thats attached to your EKS instances.

• Copy public host
• Go to your custom domain,
• Create an CNAME record to route traffic to the load balancer IP address public host for Airflow and Argocd at the domain you provided.

Open the domain name you've assigned for ArgoCD in your browser to see the Argo Login page.

• (Optional if you dont have an domain name) Here's a guide of how to connect to your EKS Kubernetes Cluster once its deployed and Port Forward Argocd and the Airflow Web Server

To log in, use the username admin and the password which is the value of the ARGOCD_ADMIN_PASSWORD in the .env located in your CNDI project folder

Notice that the cluster_manifests in the GitHub repository matches config in the ArgoCD UI

└── 📁 cndi
   └── 📁 cluster_manifests
       ├── 📁 applications
       │   └── airflow.application.yaml
       ├────── git-credentials-secret.yaml
       ├────── cert-manager-cluster-issuer.yaml
       ├────── argo-ingress.yaml
       └────── etc

Verify all applications and manifests in the GitHub repository are present and their status is healthy in the ArgoCD UI

verify that Airflow is accessible on the chosen domain 🧐

After setting up your Airflow application on the chosen domain, it is necessary to verify that Airflow is accessible. To do this, the user can simply go to the chosen domain and see if they can see Airflow's login page. The default username is admin and the password is admin. If the page is accessible, then the user can log in and begin using Airflow. If not, the user should go back and make sure the previous steps were done correctly.

verify Airflow is connected to the private DAG repository 🧐

Verify that Airflow is connected to the private DAG repository. If correct, the private DAGs should be visible on the Airflow UI. If not,you should go back and make sure that the private DAG repository is properly connected to Airflow with the correct credentials:

and you are done! ⚡️

You now have a fully-configured 3-node Kubernetes cluster with TLS-enabled Airflow and Argocd

modifying the cluster! 🛠️

To add another a node to the cluster:

• Go to the cndi_config.yaml
• In the infrastructure.cndi.nodes section, add increase the number of nodes and save the file
• Run cndi ow
• Commit changes
• Push your code changes to the repository

destroying resources in the cluster! 💣

If you just want to take down any of your individual applications:

• Delete that application or manifest from your cndi_config.yaml
• Run cndi ow
• Commit changes
• Push your code changes to the repository

If you want to take down the entire cluster run:

cndi destroy