This repository contains the code for building a robust data pipeline using Apache Airflow. The pipeline focuses on extracting, transforming, loading (ETL), and analyzing data related to UK earnings and hours.

Follow these steps to set up the Airflow environment and run the data pipeline:

• Docker installed on your machine.

1. Clone this repository:

   git clone https://github.com/kishorechk/airflow-tutorial.git

2. Navigate to the project directory:

   cd airflow-data-pipeline

3. Build and start the Docker containers:

   docker-compose up --build

4. Access the Airflow web UI at http:https://localhost:8080 (default credentials: admin/admin).

The Directed Acyclic Graph (DAG) for the data pipeline is organized into four main stages:

• Cleanup Data Directory: BashOperator to clean up the 'data' directory.
• Download File: BashOperator to download the data file from the provided URL.
• Unzip File: BashOperator to unzip the downloaded file.

• Read XLS All Employees: PythonOperator to read the Excel file containing data for all employees.

• Save Employees: PythonOperator to save the transformed data into the PostgreSQL database.

• Generate Salary Bar Chart: PythonOperator to generate a bar chart for the top 20 job types, depicting the number of jobs and mean salary.
• Generate Annual Percentage Change Line Chart: PythonOperator to generate a line chart for the annual percentage change and mean salary of the top 20 job types.

• Pandas: Used for data manipulation and cleaning.
• Matplotlib, Seaborn: Used for data visualization.