A set of services to circumvent s3 egress costs and instead download files via kinesis using cutting edge technologies like serverless programming and docker
- Transfer prices are high when downloading from object stores like AWS S3
- A single Gb costs 0.09$
- Kinesis streams don’t have egress costs
- A kinesis stream can produce 1Mb/shard/s
- That costs 0.017$ per hour which gives 3.6Gb/h (0.408$ per day)
- This means we can save up to 7.368$ per day per shard
In the name of cost saving, local development can be done without using any external services and is totally free!!!
Install SAM Cli from AWS.
Boot up s3 and kinesis (local) via docker-compose up -d
Start SAM local api gateway
cd file-splitter && sam local start-api -p 5454 --docker-network cloud-fun_cloud
Start the downloader service via idea
Run ./setup.sh