Retry logic if a worker is down

[cloudnautique]

When you add multiple Docker daemons from multiple hosts, if one of the hosts drop out you end up with a 1/n build failures due to time out.

It would be ideal, if there were retry logic after the time out to drop the build back on the queue. Additional optimization would be to keep track of workers that are unresponsive and eject them from the pool. Just to save on build startup time.

[bradrydzewski]

This is fixed in the latest version which uses agent <> server communication. The agents implement a heartbeat, and if enough time passes without any communication, the build is pushed back to the front of the queue for re-processing.

So I think we have this failure scenario covered now.