Support Skipping Bad Metrics Queries Rather Crashing PAD

[Carmezim]

Thank you for the great work and very useful model.

I was wondering it could be interesting to support handling some bad or faulty queries that cause the app to crash.

E.g.:

HTTP Status Code 400 (b'{"status":"error","errorType":"bad_data","error":"invalid parameter \\"query\\": 1:218: parse error: unexpected identifier \\"s\\" in label matching, expected \\",\\" or \\"}\\""}'

or

Max retries exceeded with url

etc.

The whole app crashes and restarts training from the beginning.

[Carmezim]

This should be possible to handle through PrometheusConnect retry param (urllib3.util.Retry)

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sesheta]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[sesheta]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[sesheta]

@sesheta: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.