dinghy halt && dinghy up result in error

[mediaessenz]

Starting "dinghy up" the first time works without problem.
If I stop dinghy by typing "dinghy halt" or "dinghy stop" the system stopps also without any issues.
But if I try to restart it again with "dinghy up" I always get this error:

Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded
Traceback (most recent call last):
	9: from /usr/local/bin/_dinghy_command:12:in `<main>'
	8: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/base.rb:440:in `start'
	7: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor.rb:359:in `dispatch'
	6: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/invocation.rb:126:in `invoke_command'
	5: from /usr/local/Cellar/dinghy/4.6.5/cli/thor/lib/thor/command.rb:27:in `run'
	4: from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:93:in `up'
	3: from /usr/local/Cellar/dinghy/4.6.5/cli/cli.rb:271:in `start_services'
	2: from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:25:in `up'
	1: from /usr/local/Cellar/dinghy/4.6.5/cli/dinghy/machine.rb:126:in `system'
/usr/local/Cellar/dinghy/4.6.5/cli/dinghy/system.rb:18:in `system': Failure calling `docker-machine start dinghy` (System::Failure)

I already tried to use docker-machine --debug start dinghy to get more details about the problem, but I do not understand it.

Before the upper error is thrown, this message repeats 10 times:

(dinghy) Calling .GetSSHHostname
(dinghy) DBG | executing: /usr/local/bin/prlctl list dinghy --output status --no-header
(dinghy) DBG | executing: /usr/local/bin/prlctl list -i dinghy
(dinghy) DBG | Found lease: 10.211.55.32 for MAC: 001C4208D8F8, expiring at 1571651690, leased for 1800 s.
(dinghy) DBG |
(dinghy) DBG | Found IP lease: 10.211.55.32 for MAC address 001C4208D8F8
(dinghy) DBG |
(dinghy) Calling .GetSSHPort
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHKeyPath
(dinghy) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/alex/.docker/machine/machines/dinghy/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -o IdentitiesOnly=yes -i /Users/alex/.docker/machine/machines/dinghy/id_rsa -p 22] /usr/local/bin/ssh <nil>}
About to run SSH command:
if ! type netstat 1>/dev/null; then ss -tln; else netstat -tln; fi
SSH cmd err, output: <nil>: Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 :::22                   :::*                    LISTEN

The only way to get my development system running again is to destroy the dinghy machine and create a new one. This is really bad, because important datas of my containers gets destroyed as well.

I use parallels pro 15.1.0 on macOS 10.15 (Catalina)

[codekitchen]

Check the https://github.com/Parallels/docker-machine-parallels project for any open related issues, it sounds like the driver is having trouble SSHing to the VM to verify that it is alive and running.

In the past, similar issues with Parallels have been fixed with a reboot, which might be good thing to try -- a lot easier than destroying and re-creating the VM while we hopefully track this down.

[mediaessenz]

Could not find any issue at docker-machine-parallels.
Restarting my computer has also no effect.
Any further ideas?

[ryansch]

For what it's worth I'm also running Parallels 15.1.0 on macOS Catalina with no issues.

[mediaessenz]

Interesting. How did you install docker & Co?
In my case I used brew, which also has an issue with docker-compose at the moment:
Homebrew/homebrew-core#45687

[ryansch]

I used ansible and brew but then ran the following after setup:

brew uninstall docker-compose
pip3 install docker-compose

For reference here's my ansible setup: https://github.com/outstand/dash/blob/master/ansible/mac.yml

[mediaessenz]

Thanks for sharing.
Meanwhile I fixed the brew docker-compose problem by adding some symlinks descripted in this comment: Homebrew/homebrew-core#45687 (comment)

[mediaessenz]

@ryansch is the ranger os image you are using here https://github.com/outstand/dash/blob/9de4f58956a86fb803743fdba17e76ee2e3fb610/docker/dinghy-preferences.yml#L7
better than the default one?
Are there any special configuration needed, if I want to use it?

[codekitchen]

hm, that stinks that it persists even across restarts. Do you run into the same issue if you create a VM using docker-machine without dinghy at all? Something like:

docker-machine create -d parallels testmachine
docker-machine start testmachine
# connect to docker and do some stuff
docker-machine stop testmachine
docker-machine start testmachine

Are you using your parallels install successfully for any other VMs?

If you aren't particularly tied to using parallels, our experience has been that the xhyve driver seems to be the most stable one overall, you could switch to that. I'd love to see this fixed if possible though.

[mediaessenz]

I have an other machine (not docker, windows 10) running without any issues.

The upper test you described (creating a new docker-machine) results to the same error after stop and start the new genrated testmachine:

Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

[mediaessenz]

BTW.: I have the exact same problems on two macs (iMac + MacBook Pro) after updating them to macOS Catalina

[mediaessenz]

I have solved the problem partly:
After deinstalling the Pulse Secure VPN Client and restarting my mac I'm at least able to start a previous (before shutdown) stopped dinghy docker machine ones.

Unfortunately dinghy halt && dinghy up is still not working (same error like before).

[mediaessenz]

Too soon looked forward to :-(
After restarting my computer again, I have the same problem again.
It is to cry for

[codekitchen]

Darn. Unfortunately there's not anything I can do to fix this, your direct docker-machine test shows that it's definitely an issue with the driver or with docker-machine itself. I'd recommend filing an issue with the https://github.com/Parallels/docker-machine-parallels project, showing those docker-machine steps and the full debug log output leading up to the error.

[ryansch]

@mediaessenz It might be interesting to see what the VM is doing when it gets stuck. Have you tried opening Parallels to see what gets logged to the console?

[mediaessenz]

@ryansch The first (but working) start shows several errors:

...
unable to write 'random state'
...
unable to write 'random state'
...
Device "eth1" does not exists.
...
unable to write 'random state'
...
unable to write 'random state'

The second (not working) start (after dinghy halt && dinghy up) shows only one warning:

warning: unable to find partition with the swap label (boot2dockerswap) or TYPE=swap (so Docker will likely complain about swap)
- this could also mean TCL already mounted it! (see 'free' or '/proc/swaps')

Do you see the same messages in your working system?

[mediaessenz]

I opened an issue at docker-machine-parallels now: Parallels/docker-machine-parallels#83

[mediaessenz]

A short update:
I got a brand new iMac yesterday and started to set up the system to my needs.
Because of the problems with restarting a docker machine I described here, I made the desicion not to use a timemachine backup of my old mac.
I installed only some basic stuff (browser, iterm, parallels desktop 15 and brew) before I used brew to install docker, docker-compose, docker-machine and docker-machine-parallels.
After this I created a new docker-machine and still have the same problem like before.
The machine comes up the first time without problems and end in the same error described up after stop and trying to start it again.
Also the messages shown in the parallels window I described up are the same.

[mediaessenz]

FYI: It seems that this issue only exists under newer boot2docker versions like 19.03.5:
Parallels/docker-machine-parallels#83 (comment)

@ryansch: Could it be that the ranger os image you use in your setup based on an older kernel version (~4.9)? That would explain why you have no problems.

For all the other poor souls out there:
Just remove your ~/.docker/machine/cache/boot2docker.iso, add this line to your ~/.dingy/preferences.yml under the :preferences:create: section (in line with provider: parallels):

boot2docker_url: https://github.com/boot2docker/boot2docker/releases/download/v18.06.1-ce/boot2docker.iso

and rebuild your dinghy vm again: dinghy destroy && dinghy create

[codekitchen]

Nice, it's great news that you got it tracked down. You can also pass --boot2docker-url directly to dinghy create.

We can leave this issue open until it is fixed upstream, so it's a bit easier for others to find.