Developed an ETL pipeline for real-time ingestion of stock market data from the stock-market-data-manage.onrender.com API. Engineered the system to store data in Parquet format for optimized query processing and incorporated data quality checks to ensure accuracy prior to visualization.
aws-lambda
aws-s3
data-engineering
aws-kinesis
aws-glue
data-engineering-pipeline
aws-glue-crawler
aws-grafana
aws-glue-data-catalog
-
Updated
Jun 25, 2024 - Python