AttributeError: 'StanModel' object has no attribute 'fit_class'

[vlam1]

Hi,

I encountered this error after our PAD container was restarted, and wondering if I can get some help resolving it. This seems like a dependency issue that was introduced recently in the quay.io/aicoe/prometheus-anomaly-detector:latest image. There are other PAD containers that are up and running fine for 9 days now with the same configs, only the restarted containers are going into crashloopbackoff (I'm guessing it's because the actual latest image was pulled on the restart).

Error traceback:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/envs/PAD/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/envs/PAD/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "app.py", line 146, in train_individual_model
    predictor_model.train(
  File "/model.py", line 45, in train
    self.model.fit(self.metric.metric_values)
  File "/opt/conda/envs/PAD/lib/python3.8/site-packages/prophet/forecaster.py", line 1171, in fit
    self.params = self.stan_backend.fit(stan_init, dat, **kwargs)
  File "/opt/conda/envs/PAD/lib/python3.8/site-packages/prophet/models.py", line 264, in fit
    self.stan_fit = self.model.optimizing(**args)
  File "/opt/conda/envs/PAD/lib/python3.8/site-packages/pystan/model.py", line 542, in optimizing
    fit = self.fit_class(data, seed)
AttributeError: 'StanModel' object has no attribute 'fit_class'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 174, in <module>
    train_model(initial_run=True, data_queue=predicted_model_queue)
  File "app.py", line 164, in train_model
    result = p.map(training_partial, PREDICTOR_MODEL_LIST)
  File "/opt/conda/envs/PAD/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/envs/PAD/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
AttributeError: 'StanModel' object has no attribute 'fit_class'

I appreciate the help.

[4n4nd]

@vlam1 could you please try using quay.io/aicoe/prometheus-anomaly-detector:ubi in one of your containers to see if it works?
I was able to test it, just want one more person to try before I update it in the repo.

[vlam1]

@4n4nd yup, I just confirmed the update fixed the issue.

[4n4nd]

Awesome. Thanks!

[vlam1]

Thank you @4n4nd! 😃