docker pull should have a --verbose option to see all of the network calls #336

jefflage · 2017-07-14T15:24:58Z

Many organizations are now applying more aggressive outbound blocking of network requests, every to the point of blocking all outbound traffic unless specifically whitelisted. It would be good if there were a command line flag available on docker pull such that you can force it to print out the host/ip and port of any outbound requests. This would make it easier to work with security teams to get all of the necessary destinations correctly whitelisted.

The text was updated successfully, but these errors were encountered:

dnephin · 2017-07-19T16:07:53Z

Wouldn't tcpdump, wireshark, etc be a lot more reliable and the standard way of identifying these requests?

thaJeztah · 2017-07-21T14:24:18Z

I don't think this should be a command-line option; the client doesn't pull the image, it's the daemon doing so. So if configuration changes are needed, those likely need to be made on the daemon host.

For the daemon host, there is already an option to obtain this information; enable debug mode on dockerd (start dockerd --debug or set "debug": true in the daemon.json configuration), and the requests are printed in the daemon logs;

Here's what's shown if both --disable-legacy-registry=false and --insecure-registry=some-domain.example.com is set;

docker pull some-domain.example.com/foobar:latest

DEBU[0011] Calling POST /v1.30/images/create?fromImage=some-domain.example.com%2Ffoobar&tag=latest
DEBU[0011] Trying to pull some-domain.example.com/foobar from https://some-domain.example.com v2
WARN[0011] Error getting v2 registry: Get https://some-domain.example.com/v2/: dial tcp some-domain.example.com:443: getsockopt: connection refused
INFO[0011] Attempting next endpoint for pull after error: Get https://some-domain.example.com/v2/: dial tcp some-domain.example.com:443: getsockopt: connection refused
DEBU[0011] Trying to pull some-domain.example.com/foobar from http:https://some-domain.example.com v2
INFO[0011] Attempting next endpoint for pull after error: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.13.3</center>\r\n</body>\r\n</html>\r\n"
DEBU[0011] Trying to pull some-domain.example.com/foobar from https://some-domain.example.com v1
DEBU[0011] attempting v1 ping for registry endpoint https://some-domain.example.com/v1/
DEBU[0011] Fallback from error: Get https://some-domain.example.com/v1/_ping: dial tcp some-domain.example.com:443: getsockopt: connection refused
INFO[0011] Attempting next endpoint for pull after error: Get https://some-domain.example.com/v1/_ping: dial tcp some-domain.example.com:443: getsockopt: connection refused
DEBU[0011] Trying to pull some-domain.example.com/foobar from http:https://some-domain.example.com v1
DEBU[0011] [registry] Calling GET http:https://some-domain.example.com/v1/repositories/foobar/images
ERRO[0012] Not continuing with pull after error: Error: image foobar:latest not found

Does that solve your use case?

jefflage · 2017-07-24T17:13:09Z

If this option will show all of the hosts that are communicated with in the course of resolving and retrieving an image, then yes it will.

Alternately, if there is admin documentation somewhere that lists all domains that would need to be whitelisted for docker to work effectively behind a restrictive firewall, that would also work.

thaJeztah · 2017-07-24T17:57:59Z

I'm not sure such a list exists; the layers themselves may be distributed through different hosts, based on the location that the pull originates from. However, the best source of information (for Docker Hub) would be [email protected] - I don't have that information available 😅

foxundermoon · 2018-04-26T12:12:46Z

+1

s50600822 · 2018-11-21T05:33:50Z

+1
See my case where I needed it on docker/for-mac#1878 comment.
I actually need to see the request header to verify why the Java DockerClient fails to auth in a few edge cases.

PardhuMadipalli · 2020-08-10T07:30:14Z

I find this is a very useful feature if implemented. I faced some issues where I could login to docker registry, but the image pull fails, although I could pull the image from other nodes. The error message says I don't have a pull access or the repository does not exist. I could not figure what the actual issue is.

ag-TJNII · 2020-09-08T15:49:07Z

enable debug mode on dockerd (start dockerd --debug or set "debug": true in the daemon.json configuration), and the requests are printed in the daemon logs

I'm not seeing this behavior. With debug logging on I'm only seeing generic "pulling" messages but not the actual API calls. I'm troubleshooting registry behavior that doesn't match the documentation so I do need to see the exact API calls. Documentation on how to enable debug at this level gets a bit hard to follow, is there a log level higher than "debug"?

sebastian-philipp · 2020-12-04T11:38:28Z

with the reduced rate limit, this is even more annoying: We now have mandatory local proxy registries and having no simple way to verify this is bad.

JiangHongTiao · 2021-06-11T06:50:20Z

Hi guys, I would need also a bit more detailed output.
I'm trying to pull image from our local artifactory, it seems the image is downloaded, however then I get error: Unknown blob.

My output of pull:

docker pull company.com/product/images/postgres12:empty-1.2
empty-1.2: Pulling from product/images/postgres12
89d9c30c1d48: Pulling fs layer
c7bb53530e14: Pulling fs layer
87d24f6e3877: Pulling fs layer
623dc2a6d06d: Pulling fs layer
c202de0fa856: Pulling fs layer
b74a7f2415c8: Pulling fs layer
cf53f01279c5: Pulling fs layer
b12915506035: Pulling fs layer
32b47b2ae54e: Pulling fs layer
19765a6f2d9d: Pulling fs layer
223b2b3ee6d3: Pulling fs layer
623dc2a6d06d: Waiting
b12915506035: Waiting
c202de0fa856: Waiting
b74a7f2415c8: Waiting
cf53f01279c5: Waiting
19765a6f2d9d: Waiting
223b2b3ee6d3: Waiting
32b47b2ae54e: Waiting
c7bb53530e14: Verifying Checksum
c7bb53530e14: Download complete
87d24f6e3877: Verifying Checksum
87d24f6e3877: Download complete
89d9c30c1d48: Downloading
c202de0fa856: Download complete
cf53f01279c5: Verifying Checksum
cf53f01279c5: Download complete
b74a7f2415c8: Verifying Checksum
b74a7f2415c8: Download complete
b12915506035: Verifying Checksum
b12915506035: Download complete
19765a6f2d9d: Verifying Checksum
19765a6f2d9d: Download complete
32b47b2ae54e: Verifying Checksum
32b47b2ae54e: Download complete
223b2b3ee6d3: Verifying Checksum
223b2b3ee6d3: Download complete
623dc2a6d06d: Verifying Checksum
623dc2a6d06d: Download complete
unknown blob

Is there any way how can I get more detailed output? My daemon.json looks like:

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "experimental": false,
  "features": {
    "buildkit": true
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

thaJeztah · 2021-06-11T10:37:27Z

Look in the logs for the daemon, they may contain more details about the failure. "unknown blob" sounds like a possible issue with the image in the registry though

JiangHongTiao · 2021-07-08T15:54:09Z

sorry for late response. Yes, there was probably a bug in our artifactory, our IT has applied some workaround and it has started working.

jokester · 2022-10-13T11:11:51Z

An improper workaround is to write a yaml and docker compose pull the image. Sometimes this gives better message than docker pull

aejnsn · 2023-02-17T04:29:58Z

@thaJeztah That doesn't show the entirety of the request's content. I want to diagnose a container registry and behavior of the CLI, I have yet to find a way to view request or response data for pulling an image.

kudodenko · 2023-08-28T19:47:59Z

+1

cloudwitch · 2023-10-25T21:43:35Z

We have an issue where we're getting EOF errors and have been unable to recreate the issue anywhere but our Jenkins worker nodes when running pipelines. It's a intermittent issue, so we would love to be able to have extra debugging enabled until we figure it out.

It's a weird one, from what we can tell, nothing changed in the environment, and is unable to be replicated anywhere reliably.

QGB · 2024-04-02T08:42:42Z

root@ubuntu:/home/qgb# docker pull manhinhang/futu-opend-docker:ubuntu-8.0.4008
ubuntu-8.0.4008: Pulling from manhinhang/futu-opend-docker
58690f9b18fc: Already exists 
b51569e7c507: Already exists 
da8ef40b9eca: Already exists 
fb15d46c38dc: Already exists 
d42fab5f7fe5: Pulling fs layer

hangs

la3rence · 2024-06-12T11:13:37Z

I'am also wanna a -v like kubectl does. But we can check the log files instead:
https://stackoverflow.com/questions/30969435/where-is-the-docker-daemon-log

GordonTheTurtle added the area/distribution label Jul 14, 2017

dfang mentioned this issue Apr 5, 2023

verbose log for docker pull #4161

Open

daohu527 mentioned this issue Nov 10, 2023

docker/scripts/dev_start.sh : Failed to pull docker image, unauthorized: authentication required ApolloAuto/apollo#15190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker pull should have a --verbose option to see all of the network calls #336

docker pull should have a --verbose option to see all of the network calls #336

jefflage commented Jul 14, 2017

dnephin commented Jul 19, 2017

thaJeztah commented Jul 21, 2017

jefflage commented Jul 24, 2017 •

edited

Loading

thaJeztah commented Jul 24, 2017

foxundermoon commented Apr 26, 2018

s50600822 commented Nov 21, 2018

PardhuMadipalli commented Aug 10, 2020

ag-TJNII commented Sep 8, 2020 •

edited

Loading

sebastian-philipp commented Dec 4, 2020 •

edited

Loading

JiangHongTiao commented Jun 11, 2021

thaJeztah commented Jun 11, 2021

JiangHongTiao commented Jul 8, 2021 •

edited

Loading

jokester commented Oct 13, 2022

aejnsn commented Feb 17, 2023

kudodenko commented Aug 28, 2023

cloudwitch commented Oct 25, 2023

QGB commented Apr 2, 2024

la3rence commented Jun 12, 2024

docker pull should have a --verbose option to see all of the network calls #336

docker pull should have a --verbose option to see all of the network calls #336

Comments

jefflage commented Jul 14, 2017

dnephin commented Jul 19, 2017

thaJeztah commented Jul 21, 2017

jefflage commented Jul 24, 2017 • edited Loading

thaJeztah commented Jul 24, 2017

foxundermoon commented Apr 26, 2018

s50600822 commented Nov 21, 2018

PardhuMadipalli commented Aug 10, 2020

ag-TJNII commented Sep 8, 2020 • edited Loading

sebastian-philipp commented Dec 4, 2020 • edited Loading

JiangHongTiao commented Jun 11, 2021

thaJeztah commented Jun 11, 2021

JiangHongTiao commented Jul 8, 2021 • edited Loading

jokester commented Oct 13, 2022

aejnsn commented Feb 17, 2023

kudodenko commented Aug 28, 2023

cloudwitch commented Oct 25, 2023

QGB commented Apr 2, 2024

la3rence commented Jun 12, 2024

jefflage commented Jul 24, 2017 •

edited

Loading

ag-TJNII commented Sep 8, 2020 •

edited

Loading

sebastian-philipp commented Dec 4, 2020 •

edited

Loading

JiangHongTiao commented Jul 8, 2021 •

edited

Loading