[serve] Add metrics for Serve controller reconcile loop health

[edoakes]

We should have a monotonically-increasing counter as well as latency metrics. These should be reported under system metrics on the dashboard.

[edoakes]

Also:

• Counter for number of times it's been restarted across the cluster.
• Log time it takes to finish recovering from checkpoint.

[edoakes]

@GeneDer let's try to prioritize this for 2.7 release

[akshay-anyscale]

#38000
#38040

[shrekris-anyscale]

Log time it takes to finish recovering from checkpoint.

@edoakes Currently, the controller recovers from checkpoints by reading cached information from the GCS and setting the goal state to that info. Is the time it takes to finish recovering from the checkpoint equal to the time it takes for the control loop to finish reconciling the current state to match the goal state? Or is it only the time it takes to update the goal state?

[edoakes]

@shrekris-anyscale see the main controller loop, deployment state returns a bit when all replicas are finished recovering (should be fast in most cases, if it's slow might cause problems). We might already have such a log line, but let's make sure it's there w/ a time.

[shrekris-anyscale]

Oh got it, I added a log statement to this PR.

[edoakes]

@alanwguo -- @shrekris-anyscale has added a bunch of useful system-level metrics for the controller. Could you or someone else working on observability add these to the system tab of the serve dashboard at some point soon?

[edoakes]

ping @alanwguo

[edoakes]

ping @alanwguo

[alanwguo]

@architkulkarni , can you take a look at this one after your current task?