Apache Airflow Tutorial Series YouTube
- Introduction and Local Installation
- Get Airflow running in Docker
- Airflow Core Concepts in 5 mins
- Airflow Task Lifecycle and Basic Architecture
- Airflow DAG with BashOperator
- Airflow DAG with PythonOperator and XComs
- Airflow TaskFlow API
- Airflow Catchup and Backfill
- Schedule Airflow DAG with Cron Expression
- Airflow Connection and PostgresOperator
- Add Python Dependencies via Airflow Docker Image Extending and Customizing
- AWS S3 Key Sensor Operator
- Airflow Hooks S3 PostgreSQL
Here are the steps to take to get airflow 2.0 running with docker on your machine.
- Clone this repo
- Create dags, logs and plugins folder inside the project directory
mkdir ./dags ./logs ./plugins
- Set user permissions for Airflow to your current user
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
- Install docker desktop application if you don't have docker running on your machine
- Launch airflow by docker-compose
docker-compose up -d
- Check the running containers
docker ps
- Open browser and type https://0.0.0.0:8080 to launch the airflow webserver