healthcheck high load

[visibilityspots]

I had to force my nomad based piaware docker setup down to the image with label 3.8.0 due to the health check in version 3.8.1. The node it was running on got a really high load making it unstable.

I'll dig into it once I find some time to debug the issue but thought I already give a headsup.

[mikenye]

Hi Jan,

Sorry this is causing you problems.

You can just disable the health check, this might be a better option for you.

With docker run add --no-healthcheck.

With docker-compose add the following to the piaware service:

healthcheck:
      disable: true

Keep me posted!

[mikenye]

Hi @visibilityspots,

Any updates?

[visibilityspots]

Finally found some time to further investigate the issue. It's definitely the healthcheck which is causing the high load every 30s as you can see in the image:

The peaks are topping over 600Mhz during the healthcheck..

Unfortunately nomad doesn't support the disabling of the HEALTHCHECK from a docker image.. So or I have to stick with 3.8.0 or I have to rebuild a custom image based on yours without the healthcheck :(

Besides the fact nomad can't disable health checks I'm not sure if a healthcheck which causes such a higher load compared to the actual service is advisable?

[mikenye]

Yes, this is not ideal. I will rewrite the healthcheck.

In the meantime, I've created you mikenye/piaware:latest_no_healthcheck and mikenye/piaware:v3.8.1_no_healthcheck.

Feel free to use these. Once I've fixed the healthcheck I'll get you to test and revert to the normal stream if that's OK.

Thanks.

[mikenye]

Hi @visibilityspots, can you pull and try out mikenye/piaware:development? This contains a reworked healthcheck using bash instead of python.

If this solves your problem, I'll merge into master.

Thanks.

[visibilityspots]

Hi,

I tried the development tagged image but the issue still pops up now only every 300s :)

[mikenye]

Annoying. I hoped that moving the healthcheck from python to bash would alleviate the problem.

Looking at the healthcheck.sh script, I can't figure out what could be contributing to the high load. My best guess is running piaware-status.

Are you able to run piaware-status manually to see whether this causes a CPU spike?

[mikenye]

Would you like me to maintain an additional "latest_nohealthcheck" version of the image?

[visibilityspots]

if that's possible that would be great to keep the noheathcheck version alive. I double checked with the piaware-status but that particular command didn't really spiked the CPU...

[mikenye]

Hi @visibilityspots,

It would be my pleasure. I'll update my nightly build scripts to build the nohealthcheck version nightly.

[mikenye]

Hi @visibilityspots,

This has been completed. I changed the name of the tag to _nohealthcheck, so you should be able to pull latest_nohealthcheck, and this will be built nightly from master branch.

Cheers!