This repository contains an example project for building Data Lineage for data lakes using AWS Glue, Amazon Neptune and Spline Agent.

Remarks: This setup works with AWS Glue Data Permissions Model and does not support Lake Formation Permission Model.

First, we need to understand joins in Spark

• Application: contains a main function
• Job: when Action performs on RDD, a "job" is created and submitted into Spark
• Stage: a job be divided into "stage" based on the shuffle boundary
• Task: a stage be further divided into "tasks" based on number of partition in the RDD. So then Tasks are the smallest unit of work for Spark

• API: HTTP API powered by an API Gateway and lambda functions
• Frontend: Vue.js application
• Cloud services: AWS
• IaC: Terraform

Diagram shows architect and data model

1. Data Flow

2. Lineage Data Model in Spark Job

brew install terraform
git clone https://github.com/aws-samples/data-lineage-for-data-lake-example.git
cd data-lineage-for-data-lake-example

# download spline agent jar
wget https://repo1.maven.org/maven2/za/co/absa/spline/agent/spark/spark-3.1-spline-agent-bundle_2.12/0.6.1/spark-3.1-spline-agent-bundle_2.12-0.6.1.jar -O ./asset/lib/spark-3.1-spline-agent-bundle_2.12-0.6.1.jar

terraform init

terraform apply

To build and test the lineage visual application locally:

• update the lineage backend address in src/lineage-visual/src/main.js

axios.defaults.baseURL =
  "https://xxx.execute-api.<aws-region>.amazonaws.com/dev";

cd src/lineage-visual
npm install
npm run serve

• Run the first Glue Job

aws glue start-job-run --job-name "RawToCurated_employee_optimize"

• Open the lineage visual application
• Run the second Glue Job

aws glue start-job-run --job-name "CuratedToAggregated_employee"

• Refresh the lineage visual application