[1.7.3] Docker logs error

[hrstoyanov]

When running:

kamal redeploy -P --version=latest -v

I got this:

Start container with version latest using a 7s readiness delay (or reboot if already running)...
  INFO [8ef3c5ee] Running docker container ls --all --filter name=^inswebapp-web-latest$ --quiet on ts1.possum-mountain.ts.net
  INFO [8ef3c5ee] Finished in 0.035 seconds with exit status 0 (successful).
  INFO [c98935cf] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.gitlab.com/ins_projects/code/webapp:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#inswebapp-web-}; done on ts1.possum-mountain.ts.net
  INFO [c98935cf] Finished in 0.054 seconds with exit status 0 (successful).
  INFO [87c90a77] Running /usr/bin/env mkdir -p .kamal/cords/inswebapp-web-72d866c1cb5a6b9428f53b8cb0db714f ; touch .kamal/cords/inswebapp-web-72d866c1cb5a6b9428f53b8cb0db714f/cord on ts1.possum-mountain.ts.net
  INFO [87c90a77] Finished in 0.029 seconds with exit status 0 (successful).
  INFO First web container is unhealthy on ts1.possum-mountain.ts.net, not booting any other roles
  INFO [3a09e198] Running docker container ls --all --filter name=^inswebapp-web-latest$ --quiet | xargs docker logs 2>&1 on ts1.possum-mountain.ts.net
Releasing the deploy lock...
  Finished all in 6.4 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host ts1.possum-mountain.ts.net: docker exit status: 123
docker stdout: "docker logs" requires exactly 1 argument.
See 'docker logs --help'.

Usage:  docker logs [OPTIONS] CONTAINER

[acidtib]

instead of using latest on the --version try setting it to the git sha of the built image

[hrstoyanov]

I don't use git SHA version when I build my images, which I build outside Kamal, just tag then with date:

clear; kamal redeploy -P --version=2024-07-04 -v

I get the same problems as with "latest":

Pull app image...
 DEBUG [bca67cbc] Running /usr/bin/env echo [2024-07-05T21:09:00Z] [root] Pulled image with version 2024-07-04 >> .kamal/inswebapp-audit.log on ts1.possum-mountain.ts.net
 DEBUG [bca67cbc] Command: /usr/bin/env echo [2024-07-05T21:09:00Z] [root] Pulled image with version 2024-07-04 >> .kamal/inswebapp-audit.log
 DEBUG [bca67cbc] Finished in 0.115 seconds with exit status 0 (successful).
  INFO [444112a5] Running docker image rm --force registry.gitlab.com/ins_projects/code/webapp:2024-07-04 on ts1.possum-mountain.ts.net
 DEBUG [444112a5] Command: docker image rm --force registry.gitlab.com/ins_projects/code/webapp:2024-07-04
 DEBUG [444112a5]       Untagged: registry.gitlab.com/ins_projects/code/webapp:2024-07-04
  INFO [444112a5] Finished in 0.035 seconds with exit status 0 (successful).
  INFO [e157c194] Running docker pull registry.gitlab.com/ins_projects/code/webapp:2024-07-04 on ts1.possum-mountain.ts.net
 DEBUG [e157c194] Command: docker pull registry.gitlab.com/ins_projects/code/webapp:2024-07-04
 DEBUG [e157c194]       2024-07-04: Pulling from ins_projects/code/webapp
 DEBUG [e157c194]       Digest: sha256:28c9cb3c01838a63974e0f6cf22f9e7988e784d3e94c898b6d8712f149727a29
 DEBUG [e157c194]       Status: Downloaded newer image for registry.gitlab.com/ins_projects/code/webapp:2024-07-04
 DEBUG [e157c194]       registry.gitlab.com/ins_projects/code/webapp:2024-07-04
  INFO [e157c194] Finished in 1.563 seconds with exit status 0 (successful).
  INFO [cf7161d4] Running docker inspect -f '{{ .Config.Labels.service }}' registry.gitlab.com/ins_projects/code/webapp:2024-07-04 | grep -x inswebapp || (echo "Image registry.gitlab.com/ins_projects/code/webapp:2024-07-04 is missing the 'service' label" && exit 1) on ts1.possum-mountain.ts.net
 DEBUG [cf7161d4] Command: docker inspect -f '{{ .Config.Labels.service }}' registry.gitlab.com/ins_projects/code/webapp:2024-07-04 | grep -x inswebapp || (echo "Image registry.gitlab.com/ins_projects/code/webapp:2024-07-04 is missing the 'service' label" && exit 1)
 DEBUG [cf7161d4]       inswebapp
  INFO [cf7161d4] Finished in 0.039 seconds with exit status 0 (successful).
  INFO [3821a21a] Running /usr/bin/env mkdir -p .kamal on ts1.possum-mountain.ts.net
 DEBUG [3821a21a] Command: /usr/bin/env mkdir -p .kamal
  INFO [3821a21a] Finished in 0.037 seconds with exit status 0 (successful).
  INFO [40101147] Running /usr/bin/env mkdir -p .kamal/locks on ts1.possum-mountain.ts.net
 DEBUG [40101147] Command: /usr/bin/env mkdir -p .kamal/locks
  INFO [40101147] Finished in 0.031 seconds with exit status 0 (successful).
Acquiring the deploy lock...
 DEBUG [3e6e81eb] Running /usr/bin/env mkdir .kamal/locks/inswebapp && echo "TG9ja2VkIGJ5OiAgYXQgMjAyNC0wNy0wNVQyMTowOTowMloKVmVyc2lvbjog
MjAyNC0wNy0wNApNZXNzYWdlOiBBdXRvbWF0aWMgZGVwbG95IGxvY2s=
" > .kamal/locks/inswebapp/details on ts1.possum-mountain.ts.net
 DEBUG [3e6e81eb] Command: /usr/bin/env mkdir .kamal/locks/inswebapp && echo "TG9ja2VkIGJ5OiAgYXQgMjAyNC0wNy0wNVQyMTowOTowMloKVmVyc2lvbjog
MjAyNC0wNy0wNApNZXNzYWdlOiBBdXRvbWF0aWMgZGVwbG95IGxvY2s=
" > .kamal/locks/inswebapp/details
 DEBUG [3e6e81eb] Finished in 0.033 seconds with exit status 0 (successful).
Detect stale containers...
  INFO [a5eaabd7] Running docker ps --filter label=service=inswebapp --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#inswebapp-web-}; done on ts1.possum-mountain.ts.net
 DEBUG [a5eaabd7] Command: docker ps --filter label=service=inswebapp --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#inswebapp-web-}; done
  INFO [a5eaabd7] Finished in 0.043 seconds with exit status 0 (successful).
  INFO [6f3318c3] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.gitlab.com/ins_projects/code/webapp:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#inswebapp-web-}; done on ts1.possum-mountain.ts.net
 DEBUG [6f3318c3] Command: /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.gitlab.com/ins_projects/code/webapp:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#inswebapp-web-}; done
  INFO [6f3318c3] Finished in 0.055 seconds with exit status 0 (successful).
Start container with version 2024-07-04 using a 7s readiness delay (or reboot if already running)...
  INFO [0e09745b] Running docker container ls --all --filter name=^inswebapp-web-2024-07-04$ --quiet on ts1.possum-mountain.ts.net
 DEBUG [0e09745b] Command: docker container ls --all --filter name=^inswebapp-web-2024-07-04$ --quiet
  INFO [0e09745b] Finished in 0.033 seconds with exit status 0 (successful).
  INFO [607807e5] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.gitlab.com/ins_projects/code/webapp:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#inswebapp-web-}; done on ts1.possum-mountain.ts.net
 DEBUG [607807e5] Command: /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=registry.gitlab.com/ins_projects/code/webapp:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=inswebapp --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#inswebapp-web-}; done
  INFO [607807e5] Finished in 0.061 seconds with exit status 0 (successful).
 DEBUG [f33f2619] Running /usr/bin/env echo [2024-07-05T21:09:02Z] [root] [web] Booted app version 2024-07-04 >> .kamal/inswebapp-audit.log on ts1.possum-mountain.ts.net
 DEBUG [f33f2619] Command: /usr/bin/env echo [2024-07-05T21:09:02Z] [root] [web] Booted app version 2024-07-04 >> .kamal/inswebapp-audit.log
 DEBUG [f33f2619] Finished in 0.042 seconds with exit status 0 (successful).
  INFO [ffa3f9b9] Running docker run --detach --restart unless-stopped --name inswebapp-web-2024-07-04 --hostname ts1.possum-mountain.ts.net-7ffc3394e3b1 -e KAMAL_CONTAINER_NAME="inswebapp-web-2024-07-04" -e KAMAL_VERSION="2024-07-04" --env-file .kamal/env/roles/inswebapp-web.env --log-opt max-size="10m" --label service="inswebapp" --label role="web" --label destination registry.gitlab.com/ins_projects/code/webapp:2024-07-04 on ts1.possum-mountain.ts.net
 DEBUG [ffa3f9b9] Command: docker run --detach --restart unless-stopped --name inswebapp-web-2024-07-04 --hostname ts1.possum-mountain.ts.net-7ffc3394e3b1 -e KAMAL_CONTAINER_NAME="inswebapp-web-2024-07-04" -e KAMAL_VERSION="2024-07-04" --env-file .kamal/env/roles/inswebapp-web.env --log-opt max-size="10m" --label service="inswebapp" --label role="web" --label destination registry.gitlab.com/ins_projects/code/webapp:2024-07-04
 DEBUG [ffa3f9b9]       docker: open .kamal/env/roles/inswebapp-web.env: no such file or directory.
See 'docker run --help'.
  INFO First web container is unhealthy on ts1.possum-mountain.ts.net, not booting any other roles
  INFO [1668e77a] Running docker container ls --all --filter name=^inswebapp-web-2024-07-04$ --quiet | xargs docker logs 2>&1 on ts1.possum-mountain.ts.net
 DEBUG [1668e77a] Command: docker container ls --all --filter name=^inswebapp-web-2024-07-04$ --quiet | xargs docker logs 2>&1
 DEBUG [1668e77a]       "docker logs" requires exactly 1 argument.
See 'docker logs --help'.

Usage:  docker logs [OPTIONS] CONTAINER

Fetch the logs of a container
Releasing the deploy lock...
 DEBUG [8127c534] Running /usr/bin/env rm .kamal/locks/inswebapp/details && rm -r .kamal/locks/inswebapp on ts1.possum-mountain.ts.net
 DEBUG [8127c534] Command: /usr/bin/env rm .kamal/locks/inswebapp/details && rm -r .kamal/locks/inswebapp
 DEBUG [8127c534] Finished in 0.028 seconds with exit status 0 (successful).
  Finished all in 2.3 seconds
  ERROR (SSHKit::Command::Failed): Exception while executing on host ts1.possum-mountain.ts.net: docker exit status: 123
docker stdout: "docker logs" requires exactly 1 argument.
See 'docker logs --help'.

Usage:  docker logs [OPTIONS] CONTAINER

Fetch the logs of a container
docker stderr: Nothing written

/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/command.rb:97:in `exit_status='
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/netssh.rb:186:in `execute_command'
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
<internal:kernel>:90:in `tap'
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:66:in `capture'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/sshkit_with_ext.rb:9:in `capture_with_info'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app/boot.rb:3:in `capture_with_info'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app/boot.rb:91:in `close_barrier'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app/boot.rb:22:in `rescue in run'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app/boot.rb:19:in `run'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app.rb:21:in `block (4 levels) in boot'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app.rb:20:in `each'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/cli/app.rb:20:in `block (3 levels) in boot'
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
/usr/local/bundle/gems/sshkit-1.22.2/lib/sshkit/backends/abstract.rb:31:in `run'
/usr/local/bundle/gems/kamal-1.7.3/lib/kamal/sshkit_with_ext.rb:117:in `block (2 levels) in execute'

[hrstoyanov]

Btw, here is my config:

service: inswebapp
image: ins_projects/code/webapp
servers:
    web:
      hosts:
       - ts1.possum-mountain.ts.net: dev
      traefik: false

registry:
  server: registry.gitlab.com
  username:
     - CONTAINER_REGISTRY_USER
  password:
    - CONTAINER_REGISTRY_PASSWORD

[stepri]

I had the same issue. I copied the latest kamal docker command it gave me this error: docker: open .kamal/env/roles/env.env: no such file or directory.. Fix the issue by running

kamal env push

[hrstoyanov]

@stepri @djmb
I think the issue might be that kamal uses xarg where it should be using xarg -r - see here. I have not have a chance to re-test but will soon ...