Figure out success rate classification in telemetry

[siggy]

In #536 we introduced grpc_status_code and status_code to enable success rate calculations.
A naive success rate calculation would look something like:

(
  sum(irate(response_total{grpc_status_code="0"}[20s])) + 
  sum(irate(response_total{status_code="200"}[20s]))
) / 
sum(irate(response_total{}[20s]))

Note that status_code="200" doesn't count non-200 successes. Assuming we really want to count status_code<400, a couple options come to mind:

1. add a success=true|false label to the proxy's response_total metric
2. add a prometheus relabel_config to relabel all metrics with status_code's < 400 (this may require enumerating every integer [0...400)

Related to #627

[olix0r]

I think you need to do something like:

  sum(irate(response_total{status_code="200"}[20s])) -
  sum(irate(response_total{grpc_status_code!="0"}[20s]))

because all grpc messages have a 200 response code

[olix0r]

The plan for this more generally is that routes will have some configurable classification logic, so this should be pushed down into the proxy.

[hawkw]

Bear in mind also that responses with no status_code label and no grpc_status_code label are likely to be some kind of protocol-level error, which we don't currently make explicit (see #599)

[olix0r]

I think the proxy should add a classification label to response metrics. The possible values of which are currently success or failure.

[hawkw]

Question regarding gRPC statuses: it seems like some gRPC status codes (e.g. "Not Found", "Already Exists", "Permission Denied", etc) indicate errors on the client side rather than the server side. If we count the analogous HTTP 4xx error codes as "successes", it seems like some non-zero gRPC status codes might be counted as successes as well?

[hawkw]

@siggy should #639 close this?

[siggy]

@hawkw yup thanks!

fixed in #639.