Too many open files

[bborysenko]

After working few days we stuck with following error:

[root@node1 ~]# docker logs ca67959573aa
...
time="2018-03-01T13:25:29Z" level=warning msg="error during connect: Get http:https://%2Fvar%2Frun%2Fdocker.sock/v1.29/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22prometheus_prometheus%22%3Atrue%7D%7D: dial unix /var/run/docker.sock: socket: too many open files"

[root@node1 ~]# docker top ca67959573aa
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                18296               18281               2                   Feb28               ?                   06:15:11            /app/main discover -i 5 -o /swarm-endpoints/swarm-endpoints.json -p prometheus_prometheus

[root@node1 ~]# lsof -n -p 18296 | wc -l
65540

[bborysenko]

Finally switch to https://github.com/jmendiara/prometheus-swarm-discovery.

[ugurarpaci]

@bborysenko correct me if I am wrong but the one that you reference is using labels on service level, not the container level labels. Therefore prometheus cannot discover the underlying containers therefore It will not scrape.