Reduce cardinality of Cinder volume status metrics

[dougszumski]

Cinder volume status metrics contain highly variable labels, such as
volume and tenant ids. For Prometheus Server and other TSDBs such as
InfluxDB, it is best to avoid highly variable labels to limit the total
number of unique timeseries.

This patch removes the old, highly variable metrics and converts
them to simpler metrics tracking the number of volumes in particular
states.

[dougszumski]

Hey all. Great project. We've been using this on some fairly large deployments and have found that some of the metrics produce a very large number of unique time series which hurts performance.

I'm interested on feedback on the right approach here - would you rather that the original metrics were kept for backwards compatibility, and perhaps marked for deprecation, or could we target some changes to reduce cardinality at a new major release?

[alexeymyltsev]

Hi @dougszumski, improvement of performance it is great, but we use those metrics for alerting. It is quite useful see in alert id of volume and project id for decrease time of investigation especialy when volume already deleted.

[dougszumski]

Thanks @alexeymyltsev - I will have a think about the best way forward that won't break any existing use cases.

[niedbalski]

@dougszumski

Thank you for this contribution. One option is to split the metrics as volume_status_counter (or alike) and volume_status metrics, then you can use --disable-metric=cinder-volume_status to avoid collecting volume details.

Also, we merged the --disable-slow-metrics (disabled by default), that will avoid collecting metrics that have been identified as potentially slow in large deployments. Therefore, if we think volume_status might be problematic, we can flag it as slow as well.

[niedbalski]

Hey @dougszumski

I'd like to include this in the next release, based on our chat here last week, are you planning to follow up on this patch?

Thanks!

[dougszumski]

Thanks @niedbalski! I will try to follow up today, but otherwise early next week.

[dougszumski]

It's looking like early next week now..

[dougszumski]

@niedbalski does that look any better? I just gave it a quick test on a live system. Here's some example data without --disable-slow-metrics:

openstack_cinder_volume_status{bootable="true",id="e9a1c9d4-3022-4c63-873c-89d9c3a5564f",name="",size="20",status="in-use",tenant_id="ae2fd17124ac4962b5162c41733ed536",volume_type="__DEFAULT__"} 5
openstack_cinder_volume_status{bootable="true",id="ff7789f3-cd3d-475f-a9b5-311ddbf135e3",name="",size="20",status="in-use",tenant_id="698519617c0f4c4da742c4cb7706d769",volume_type="__DEFAULT__"} 5
openstack_cinder_volume_status{bootable="true",id="ffaa4e8e-d1f7-4875-9c39-85317c000278",name="",size="20",status="in-use",tenant_id="ae2fd17124ac4962b5162c41733e8536",volume_type="__DEFAULT__"} 5
# HELP openstack_cinder_volume_status_counter volume_status_counter
# TYPE openstack_cinder_volume_status_counter gauge
openstack_cinder_volume_status_counter{status="attaching"} 0
openstack_cinder_volume_status_counter{status="available"} 830
openstack_cinder_volume_status_counter{status="awaiting-transfer"} 0
openstack_cinder_volume_status_counter{status="backing-up"} 0
openstack_cinder_volume_status_counter{status="creating"} 0
openstack_cinder_volume_status_counter{status="deleting"} 0
openstack_cinder_volume_status_counter{status="detaching"} 2
openstack_cinder_volume_status_counter{status="downloading"} 0
openstack_cinder_volume_status_counter{status="error"} 0
openstack_cinder_volume_status_counter{status="error_backing-up"} 0
openstack_cinder_volume_status_counter{status="error_deleting"} 0
openstack_cinder_volume_status_counter{status="error_extending"} 0
openstack_cinder_volume_status_counter{status="error_restoring"} 0
openstack_cinder_volume_status_counter{status="extending"} 0
openstack_cinder_volume_status_counter{status="in-use"} 214
openstack_cinder_volume_status_counter{status="maintenance"} 0
openstack_cinder_volume_status_counter{status="reserved"} 1
openstack_cinder_volume_status_counter{status="restoring-backup"} 0
openstack_cinder_volume_status_counter{status="retyping"} 0
openstack_cinder_volume_status_counter{status="uploading"} 0

[niedbalski]

@dougszumski yes, LGTM, 2 minor nitpicks: 1) Can you rebase your changes with current master? 2) Can you add the 'volume_status' as a slow metric in the corresponding README.md?

Thanks for the work on this!

[niedbalski]

LGTM, merging.