-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nomad coping with OS resources limits #6557
Comments
Another example I just documented in #6607 is kernel tunables for bridge networking, which took up a lot of time to figure out in #6580 (comment) In that scenario we want a particular set of tunables for network namespace / Connect support, but probably don't want to enforce them because it'll break the QEMU task driver:
(These aren't strictly "resource limitations" but can be set at runtime on clients and change outside of Nomad's control, so I suspect the solutions are going to be similar.) |
When aiming to run many allocations on a single client, it is easy to overlook adjusting the OS resource limits and have nomad hits these limits. As of 0.10.0, Nomad may start user allocations but partially crash before persisting their state, resulting into many leaked processes that Nomad no longer manages. When a client gets to that state, the best option is to destroy client.
Some sample flags for linux:
/proc/sys/kernel/pid_max
and/proc/sys/kernel/threads-max
or sysctl. These are global limits to possible running pids/threads in whole system. Nomad will fail to spin up more tasks, but worse may succeed at spinning up the tasks but then crash on spinning up monitoring threadsTasksMax
: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html ; or in cgroups filesystem (e.g./sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max
)Hitting these limits is damaging to many other critical services, even if Nomad remained healthy. Docker has some known issues and some guidance for tweaking these values[1][2].
We can try to address this by many means; here is a sample of possible actions:
[1] https://success.docker.com/article/how-to-reserve-resource-temporarily-unavailable-errors-due-to-tasksmax-setting
[2] docker/for-linux#73
The text was updated successfully, but these errors were encountered: