Track last update time in HostMetrics

[ellisjoe]

No description provided.

[ellisjoe]

Our current metrics are exponentially weighted moving averages which never reach a value of 0 and leave us unable to determine when a host has not been used recently. Exposing the lastUpdate time in HostMetrics would allow us to filter out hosts that have not been used recently.

[iamdanfox]

So I'm not gonna block this cos it's pretty inoffensive, just kinda wish we didn't have to do some comparisons & logic further down the line... seems like if we have to factor in the last-update time then the exponentially weighted meters aren't really what the primitives we want.

[gracew]

+1 about potentially reevaluating the primitives

[uschi2000]

how would you want to use this?

[ellisjoe]

@iamdanfox @gracew agree that the primitive is wrong for our use case. Personally in favor of merging this and figuring out the correct primitive as we move forward (I'd bargain it's not going to be easy to agree on).

@uschi2000 I want to do something like:

Instant fiveMinutesAgo = Instant.now().minus(5, TimeUnit.MINUTES);
OkHttpClients.hostMetrics().stream()
    .filter(metrics -> metrics.lastUpdated().isAfter(fiveMinutesAgo))
    .filter(metrics -> isHealthy(metrics))
    .collect(toList);

Currently I have no way to determine if a host has been used in the last 5 minutes with the exposed metrics.

[uschi2000]

if you agree this is the wrong primitive and yet we merge it, will be ever be able to remove this? who's going to call this?

[markelliot]

the thing we might consider doing here is remembering k meters, one for each of the last 5/k minutes, and then rolling off the oldest meter every 5/k minutes. I think k=10 would be plenty for the set of use-cases we care about.

An alternative to this would be a Metrics meter implementation that does this under the covers.

[ellisjoe]

The thing I'd like to provide by exposing per host metrics is enough information to determine the state of the remote host/service. "state" in my case is "does a host/service pair look like it is unhealthy based on the responses it's given me in the past 5 minutes". As long as we continue to provide a mechanism to retrieve that information we should be able to deprecate and remove this when we agree on a better alternative.

@markelliot In my particular use case the thing I want is ZLEMA which removes old values from the moving average (I think this is basically what you're suggesting?). Main concern with that at the moment is that Dropwizard metrics only suports EWMA

[uschi2000]

nit: assuming we're updating this much more frequently than we're querying it, maybe remember epoch millis and create the Instant in lastUpdate()?

[ellisjoe]

done