In this cloud specialization module we'll be implementing an ETL (Extract,Transform,Load) pipeline through a Google cloud function written in GOLANG. We'll be using Terraform as IAC tool to provision and use following GCP resources for our ETL pipeline.
- Google Cloud Storage.
- Google CLoud Function.
- Google Cloud Scheduler.
- GCP BigQuery
- Extract 100 csv records per function invocation.
- Transform records which meet the transformation criteria.
- Insert the records into the BigQuery Table.
- Enable periodic function invocation using Google Cloud Scheduler.
1 - Clone the xldp repo to your system and navigate to the directory relevant to this module.
git clone https://github.com/X-CBG/xldp.git
cd xldp/cloud_specializations/s204/noman2 - Download the csv file from this link.
3 - Navigate to the CloudFunction directory and create a zip archive (ETL.zip) of the cloud function source code files.
cd CloudFunction/
zip ETL.zip main.go go.mod && cd ..4 - Navigate to the Terraform folder. initialize terraform in the directory.
cd Terraform && terraform init5 - Open the directory in any editor for example VScode and update the values of sourcepath and csv_local_path
variables in variables.tf according to the absolute path of the csv file and ETL.zip on your system. Futher, please update the value of project in main.tf to reflect the ProjectID of your project in your GCP account.
6 - Authenticate your gcloud cli with GCP.
gcloud auth login
#you will be prompted with a window in your default browser, check the boxes as per requirements and your gcloud CLI will be authenticated with GCP automatically.7 - Apply the terraform configuration to setup the infrastructure resources
terraform apply --auto-approveLogin into your gcp account and verify that the following resources with the mentioned names have been created inside your project.
| Resource | Name |
|---|---|
| Cloud Storage Bucket | for_s204_xgrid |
| Cloud Storage Object | covid-csv-s204.csv |
| Cloud Function | etl_function |
| Cloud Scheduler | invoke_ETL |
| BigQuery Dataset | covidDataset |
| BigQuery Table | covid-table |
You may verify the functionality by manually triggering the cloud fucntion http endpoint from your browser.
https://us-east1-your-projectID.cloudfunctions.net/etl_function/?a=your-projectID&&b=covidDataset&&c=covid-table
** Note: Please replace "your-projectID" with the projectID of your project in your GCP accout.
You can now verify the functionality of the ETL pipeline by previewing the BigQuery table in the dataset which will now be including the first 100 records from the csv after the first invocation.