-
-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce cardinality of Cinder volume status metrics #143
Reduce cardinality of Cinder volume status metrics #143
Conversation
Hey all. Great project. We've been using this on some fairly large deployments and have found that some of the metrics produce a very large number of unique time series which hurts performance. I'm interested on feedback on the right approach here - would you rather that the original metrics were kept for backwards compatibility, and perhaps marked for deprecation, or could we target some changes to reduce cardinality at a new major release? |
Hi @dougszumski, improvement of performance it is great, but we use those metrics for alerting. It is quite useful see in alert id of volume and project id for decrease time of investigation especialy when volume already deleted. |
Thanks @alexeymyltsev - I will have a think about the best way forward that won't break any existing use cases. |
Thank you for this contribution. One option is to split the metrics as volume_status_counter (or alike) and volume_status metrics, then you can use --disable-metric=cinder-volume_status to avoid collecting volume details. Also, we merged the --disable-slow-metrics (disabled by default), that will avoid collecting metrics that have been identified as potentially slow in large deployments. Therefore, if we think volume_status might be problematic, we can flag it as slow as well. |
Hey @dougszumski I'd like to include this in the next release, based on our chat here last week, are you planning to follow up on this patch? Thanks! |
Thanks @niedbalski! I will try to follow up today, but otherwise early next week. |
It's looking like early next week now.. |
2415bb1
to
3199bde
Compare
@niedbalski does that look any better? I just gave it a quick test on a live system. Here's some example data without --disable-slow-metrics:
|
@dougszumski yes, LGTM, 2 minor nitpicks: 1) Can you rebase your changes with current master? 2) Can you add the 'volume_status' as a slow metric in the corresponding README.md? Thanks for the work on this! |
Cinder volume status metrics contain highly variable labels, such as volume and tenant ids. For Prometheus Server and other TSDBs such as InfluxDB, it is best to avoid highly variable labels to limit the total number of unique timeseries. However, in smaller scale deployments this isn't always a problem and some users like the additional labels to enrich notifications. In this patch we therefore add support for optionally disabling the volume_status metrics via the slow metrics flag and add a new, simpler metric for keeping track of cinder volumes statuses which shouldn't cause issues in larger deployments.
3199bde
to
fecdf27
Compare
LGTM, merging. |
Cinder volume status metrics contain highly variable labels, such as volume and tenant ids. For Prometheus Server and other TSDBs such as InfluxDB, it is best to avoid highly variable labels to limit the total number of unique timeseries. However, in smaller scale deployments this isn't always a problem and some users like the additional labels to enrich notifications. In this patch we therefore add support for optionally disabling the volume_status metrics via the slow metrics flag and add a new, simpler metric for keeping track of cinder volumes statuses which shouldn't cause issues in larger deployments.
Cinder volume status metrics contain highly variable labels, such as
volume and tenant ids. For Prometheus Server and other TSDBs such as
InfluxDB, it is best to avoid highly variable labels to limit the total
number of unique timeseries.
This patch removes the old, highly variable metrics and converts
them to simpler metrics tracking the number of volumes in particular
states.