docker-compose down fails because networks have active endpoints

[SunMar]

Hi,

It seems like #218 is still an issue (or returned). I'm using Dinghy 4.6.1 and http proxy 2.5 with a docker-compose file version 3.2. Not sure what information I can provide that'd be of use.

[ryansch]

It's a race condition. Run docker-compose down again and it'll succeed the second time.

[SunMar]

Yes, I figured that out, but that still leaves a failing docker-compose down in my scripts.

Do you know the cause and what the difficulty is in resolving the race condition?

On a side note I actually have the proxy disabled in my ~/.dinghy/preferences.yml, but apparently it's still used (for other features?).

[codekitchen]

The best we can do is listen for docker events and leave the network as soon as we see docker-compose stop the container, but there's no way to guarantee that we'll see that event and leave the network before docker-compose tries to drop the network as well. The fix would be to hook into docker-compose directly and force it to pause between those two steps while we do our thing, but I'm not aware of any plans to add that kind of functionality to docker-compose -- they've resisted similar proposals in the past.

That said, the fix in #218 has worked surprisingly well so far, so just to verify: are you seeing the exact same network <network name> has active endpoints error? And are you seeing any other relevant output on top of that? Can you estimate how often this happens on docker-compose down?

On that side note: the proxy container still runs because it's used for DNS as well, it just doesn't bind the ports. I really need to rename that container or something to make it more clear.

[SunMar]

Yes the error I'm seeing is:

ERROR: network *** id *** has active endpoints

It does indeed not always happen, I'll see if I can find a pattern or get an idea on how often it happens. Something that may be of influence and I forgot to mention is that I have two networks defined in my docker-compose.yml so it has to leave both for a successful down, that probably increases the chance of it failing.

I understand the difficulty here and that unfortunately it's not easily resolved.

[codekitchen]

Hm yeah I don't think I've ever set up a compose file with two networks, but it would definitely increase the chances of hitting the race condition on each down command.

[SunMar]

How do you have dinghy-http-proxy enter all the networks that are created right now and do the port bindings? I guess the join-networks script does that, but from where is that running? I'm trying to familiarize myself a bit with dinghy, see if I can think of a useful suggestion :).

[codekitchen]

Heh I'm all for that, thanks for looking :) You're right that it's all done in join-networks.go, which is called from the reload-nginx script, which is called by docker-gen whenever a docker event is received because of the Procfile config.

[SunMar]

Hi, I never got the chance to really look into it and due to various reasons also to do with some other problem I had with my computer I stopped using dinghy for a while. Lately I've been using it again and this problem seems to have disappeared. I don't know if it's because of changes to dinghy or my computer, but seeing as this ticket is still open I'll close it :).