run_tasks_manually.md

Running tasks manually

Context

Instead of using the one script (workflow.py) that executes the Luigi Workflow Orchestration as shown in Step 2 of the main README.md, this documentation will break down the tasks in the script so that it can be run individually and manually.

Running Luigi tasks manually

Step 1. ExtractLoadAirportData

• Execute python script to extract airport data from website link and load into database:

  # In the root of project directory:  
  $  python ./extract_load/airports.py

Step 2. DbtDeps

• Install the dbt dependencies:

  # If not in the dbt directory:  
  $ cd ./dbt/
  
  # In the ./dbt directory:  
  $ dbt deps

  Specifically, we will be using macros from dbt_util in our custom macros that are ultimately used in the SQL queries to forms our final analytics tables.

Step 3. DbtSeedAirports

• Use dbt to easily seed CSV files stored locally:

  # In the ./dbt directory:  
  $ dbt seed --profiles-dir ./

  This alternative to using SQL Alchemy in Python scripts to upload to the databases.

  In this step, dbt will upload the ./dbt/data/raw_airports.csv to the database.

Step 4. DbtRunAirports

• Run dbt that cleans the Airport data to be used in later steps to scrape arrival data (using airport iata/icao code):

  # In the ./dbt directory:  
  $ dbt run --profiles-dir ./ --model tag:cleaned_airports

  In this step, dbt will compile and execute the SQL Query to create:
  • base_airports table.

Step 5. ScrapeLoadArrivalData

• Execute python script to extract arrival data from website link and load into database:

  # In the root of project directory:  
  $ python ./extract_load/arrivals.py

  The script in this step will query the table in the database created in Step 2.4 above to obtain the airport codes (iata/icao) and use them to loop through the arrival data website.

  Note:

Step 6. DbtSeedArrivals

• Use dbt to easily seed CSV files stored locally:

  # In the ./dbt directory:  
  $ dbt seed --profiles-dir ./

  In this step, dbt will upload the ./dbt/data/raw_arrivals.csv to the database.

Step 7. DbtRunAnalysis

• Run dbt that cleans the Arrival data and Transform the tables to its Factual Tables for analytics:

  # In the ./dbt directory:  
  $ dbt run --profiles-dir ./ --exclude tag:cleaned_airports

  In this step, dbt will compile and execute the SQL Query to create:
  • base_arrivals__malaysia table.
  • stg_airports__malaysia_distances table.
  • fct_airports__malaysia_distances_km table.
  • fct_arrivals__malaysia_summary table.