Investigate Horizon High Memory Issues

[aditya1702]

Partners and users on Discord have reported very high memory consumption from Horizon when the state rebuild happens.

Tasks

Beta Give feedback

1. Setup a dev environment in EC2 for testing
2. Recreate the high memory consumption of Horizon
3. Memory profile Horizon during state rebuild
4. Optimise Horizon performance based on memory profiling results.

[aditya1702]

Tamir's new change has significantly improved the RAM usage and also the state rebuild times. The time decreased from ~4hrs to ~1.1hrs. While memory usage dropped from ~25GB to ~5GB.