0.14 sometimes hangs before running any actions

[lberki]

Stack trace looks like this:

https://gist.github.com/mattklein123/070fbfe5c49d52b0457e9c694c148380

It appears that we unconditionally run docker info on every build, and if that hangs, we also hang. There is a fix at HEAD (4b80f24) but that doesn't help 0.14 by itself.

[mattklein123]

Hi,

I'm happy to help debug this further. Can you let me know what additional info I can provide? I suspect it's something like in certain cases the docker process is waiting to read from stdin or something like that?

[mattklein123]

klein@localhost:~$ docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:30 2018
  OS/Arch:      linux/amd64
  Experimental: false
mklein@localhost:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 18.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 22
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-127-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.779GiB
Name: localhost
ID: XIOC:BKJF:YGML:WTCP:33T7:7JZO:M3BU:AEBA:GNX7:RLWA:4SML:MI4T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: mattklein123
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

For reference.

[lfpino]

Assigning back to @lberki and @aehlig (Bazel sheriff) to further debug this if needed.

[hlopko]

Since the 0.14.1 release is out, can this be closed?

[mattklein123]

If you want? Though don't you have to figure out why it hangs?

[hlopko]

I believe the fix was to hide docker sandbox behind a flag (c909639)

@lberki @dslomov is that correct?

[mattklein123]

That's a workaround, not a fix. What if I want to use the feature?

[hlopko]

Fair point, then I'd change the title of the bug and lower the priority. Assignees? Wdyt?

[lberki]

It'll be taken care of by @philwo when he's back at work.

[ittaiz]

Awesome. We definitely need this feature...

…

On Mon, 25 Jun 2018 at 18:14 lberki ***@***.***> wrote: It'll be taken care of by @philwo <https://github.com/philwo> when he's back at work. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5336 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFwPgm4IbNqN-BY48qtkDHtU_H-OCks5uAP5egaJpZM4Uc2Pz> .

[philwo]

@mattklein123 @ittaiz If docker info hangs, this sounds more like a Docker issue.

I checked the code and I don't think it ever waits for stdin. The "info" command makes a single API request to the Docker daemon, decodes the result and prints it (https://github.com/moby/moby/blob/master/client/info.go).

Common reasons for hanging commands that do network requests are a wrongly configured hostname / DNS setup. I notice that your "Name" output in docker info gives "localhost", which is at least a bit weird. It should instead return the actual hostname of your machine.

Can you still reproduce the docker info hang with Bazel and/or without Bazel on your machine?

[philwo]

There are two things we can do on Bazel's side:

1. Delay initialization of the Docker module so that users who aren't actually using the Docker sandbox are not affected by any delays.
2. Run docker info with a timeout of 10 seconds (or so) and if it still didn't finish by then, abort and print an error (and possibly whatever docker info printed to stdout/stderr, if anything).

There's nothing Bazel can do to fix or workaround the hang itself, though.

[mattklein123]

docker info runs just fine on my machine. It was something with how Bazel was calling it through Java. That's all I can offer you.

[jmmv]

Given the age of this report... and that things might have changed... I'll have to close due to the apparent difficulty in reproducing this. Please let me know if think this is still a problem.