Account Pool Monitoring

[eschwartz]

I would like to have better visibility into the state of my DCE Account pool.
Specifically, I would like to how many accounts are in each AccountStatus, and be notified if too many are NotReady, or not enough are Ready.

This would allow us to:

• dive into reset CodeBuild failures, if too many accounts are failing to reset
• add more accounts to the pool, if demand for leases grows over time.

As a business analyst type user, I may also be interested to see time-series graphs of lease usage (how many account are Status=Leased)

---

I'm thinking we can put together a simple Lambda that runs every hour, and queries DynDB on the AccountStatus GSI. Then write the results to three different custom CW event metrics (one for each status).
We can configure alarms against these events, and maybe create a CW Dashboard for quick viewing.

[kapilt]

re monitoring, I would suggest monitoring any cloud build status. accurate state metrics are useful as a higher up business level metric, but a cloud build failure at a lower level would represent an actionable failure.

[marinatedpork]

This is added in #271