App and retraining never start

[laburnumT]

Hi all,

I attempted to use the prometheus anomaly detector to detect anomalies in my metrics. I set it up to track just one metric (quite a few targets though). However both times I ran the program it seemed to train just fine, but it seems to hang after finishing the initial training run. I don't get the message "Initializing Tornado Web App", nor the "Will retrain model every x minutes". Nothing happens after the retraining period passes, and no metrics ever get exposed on :8080/metrics.

Running the program, but on just one instance of the metrics (which concludes in ~3 minutes), works without issue, and none of the above mentioned problems/symptoms occur. One theory I have is that the training took longer than the retraining interval, but I do not know why that would be an issue.

I have attached a copy of the logs (without the direct training data, but I can upload that if necessary).

obfus.log

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sesheta]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[sesheta]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[sesheta]

@sesheta: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.