It a real time simulation of stock market using. kafka and aws services Architecture Diagram for the end to end pipe line

Steps:

1. Create an Amazon EC2 free tier t2 micro instance and install Kafka on it.
2. run the Kafka Zookeper
3. Start the Kafka Server
4. Create a topic
5. Run Producer
6. Run Consumer
7. Create jupyter notebooks to simulate Stock market data using the CSV file given. Kafka Producer and Kafka Consumer respectively
8. Create A S3 bucket. and allow puclic access to the bucket
9. Create AWS CLI on local machine to give access to to upload on S3.
10. start uploading files into s3
11. once enough files are uploaded crawl the the S3 bucket with crawler from AWS Glue.
12. Create a database
13. Access the database with Athena

Examples :- 1. 2. 3.