As the world of data management continues to evolve, organizations are constantly seeking new technologies and approaches to help them unlock the full potential of their data. One such approach is the data lakehouse, a pattern that combines features of both data warehouses and data lakes to create a unified data platform.
Create docker images:
docker build -t presto:0.272.1 ./Dockerfiles/presto
docker build -t cluster-apache-spark:3.1.1 Dockerfiles/sparkDocker compose:
docker-compose upInstall dependencies:
docker exec -it master bash /opt/workspace/dependencies/packages_installer.sh -
Create a bucket named "datalake" in Minio (localhost:9001)
-
Create "CarParts" database in Postgres (using CarParts.sql in sql folder)
-
Extract data from Postgres and ingest to Minio
docker exec -it master spark-submit --jars /opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar,/opt/spark/jars/hadoop-aws-3.2.0.jar,/opt/spark/jars/postgresql-42.3.5.jar \
--master spark:https://master:7077 \
--deploy-mode client \
--packages io.delta:delta-core_2.12:1.0.1 \
--conf "spark.executor.extraClassPath=/opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar:/opt/spark/jars/hadoop-aws-3.2.0.jar:/opt/spark/jars/postgresql-42.3.5.jar" \
--conf "spark.driver.extraClassPath=/opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar:/opt/spark/jars/hadoop-aws-3.2.0.jar:/opt/spark/jars/postgresql-42.3.5.jar" \
/opt/workspace/postgres_to_s3.py- Transform data in Minio
docker exec -it master spark-submit --jars /opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar,/opt/spark/jars/hadoop-aws-3.2.0.jar,/opt/spark/jars/postgresql-42.3.5.jar \
--master spark:https://master:7077 \
--deploy-mode client \
--packages io.delta:delta-core_2.12:1.0.1 \
--conf "spark.executor.extraClassPath=/opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar:/opt/spark/jars/hadoop-aws-3.2.0.jar:/opt/spark/jars/postgresql-42.3.5.jar" \
--conf "spark.driver.extraClassPath=/opt/spark/jars/aws-java-sdk-bundle-1.11.375.jar:/opt/spark/jars/hadoop-aws-3.2.0.jar:/opt/spark/jars/postgresql-42.3.5.jar" \
/opt/workspace/clean_data.py-
Create external table in Presto to query data in Minio
-
Connect to Metabase to using BI
