docker system prune -af --volumes does not completely clear out overlay2 folder

[emike922]

Description

We run a lot of docker containers in our CI/CD, the majority using the docker run --rm option if that matters. We also build a few images and perform a docker system prune -af --volumes each night. Every couple of months we end up with 0 bytes remaining disk space due to /var/lib/docker/overlay2 retaining data despite the nightly cleanup. The only way to recover the disk space without breaking docker that I could find so far is a complete reinstall.

For reference:

$ docker system prune --all --force --volumes
Total reclaimed space: 0B
$ docker ps --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$ docker volume ls
DRIVER    VOLUME NAME

# du -ch /var/lib/docker --max-depth=1
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
67G     /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
11M     /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
106M    /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
67G     /var/lib/docker
67G     total

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   46G   13G  79% /

# ls -lh /var/lib/docker/overlay2/
total 5.6M
drwx--x--- 5 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e
drwx--x--- 4 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e-init
drwx--x--- 4 root root 4.0K Oct 31 09:03 00656fdea360c00da5d9549adcb5f968ab19280e6dde08892f6d26901ba84cca
drwx--x--- 4 root root 4.0K Oct 30 12:47 0081118c886142094d4bfdac3051331b04a5937bc0cd9db5f4d292a793d54a33
drwx--x--- 4 root root 4.0K Oct 19 14:41 0124757a1161c7b9bc1f4a725a5ef90c7d0b9598d31e53c8c59780fc686c2ca4
...
drwx--x--- 4 root root 4.0K Nov  7 08:47 x32c6n0iual0i5x438yhm91zc
drwx--x--- 4 root root 4.0K Nov  6 08:49 x9g9q67ib6r4pr799rkjcacir
drwx--x--- 4 root root 4.0K Nov  9 07:45 yo8bov1h8hqzcpt3ubdsj3de1

# ls -lh /var/lib/docker/overlay2/ | wc -l
1388

# apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
# rm -rf /var/lib/docker
# rm -rf /var/lib/containerd
# apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   20G   40G  33% /

Reproduce

1. Intensive use of docker run --rm, docker rmi and docker build over weeks
2. Run docker system prune -af --volumes every day overnight
3. Observe disk filling up over time

Expected behavior

After running docker system prune -af --volumes, all data from dangling images, completed containers etc should be removed

docker version

**Note:** Below o/p is collected _after_ a reinstall. The version running for the past months has been 24.0.6.

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-76-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.58GiB
 Name: ****
 ID: 2c71d842-867e-40fa-af3e-a6210ada7edd
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[EyuelMathewos]

Best Option to Solve this issue

1. if the we running as docker service -> docker service update --force
2. you can check /var/lib/docker/overlay2 in linux terminal command "du -h --max-depth=1 /var/lib/docker/overlay2" to check maximum disk usage & check every docker process "docker inspect docker CONTAINER ID" or you can use docker disk space usage of Docker containers and images using command "docker system df"
3. docker system prune --all --force, caution if you are cleaning completely by removing the service you will lose the network & volumes.

[timon-michel-scopevisio]

I think i have the same issue. I don't have any

• volumes
• containers
• images
• networks

right now. But if i list the contents of /var/lib/docker:

4.0K    /var/lib/docker/swarm
19M     /var/lib/docker/buildkit
16K     /var/lib/docker/plugins
24M     /var/lib/docker/image
9.6G    /var/lib/docker/overlay2
8.0K    /var/lib/docker/tmp
1.5M    /var/lib/docker/containers
76K     /var/lib/docker/network
4.0K    /var/lib/docker/runtimes
34G     /var/lib/docker/volumes
44G     /var/lib/docker

I have tried every command i could find on the internet, nothing seems to help, including your commands (except for nuking the whole folder, but i want to avoid that if possible)

Is there a safe way to manually delete the contents of the /var/lib/docker folder?

[cpuguy83]

Have you tried docker builder prune? I'm pretty sure this is not handled with docker system prune`

[emike922]

Thanks for the proposal @cpuguy83. Unfortunately, docker builder prune -af has no additional effect - frees 0B.
Any other suggestions? :-)

[cpuguy83]

@emike922 Did you, by chance, switch graphdrivers or even switch to the containerd backed image store?

[timon-michel-scopevisio]

I have changed the docker service to be rootless. After starting up the service as root again and cleaning up there, i have managed to clean my 44G down to 6G. After a quick sudo docker system prune --all --force (sudo or simply as root), it shows to have cleaned everything down to 18M (which is not completely clean, but that's alright)

So all in all - i understood that the rootless uses the data than the root-version, which doesn't seem to be the case. After thinking about it, it makes sense though. Maybe we can add a note to the documentation though, that you may want to clean up the installation before disabling the service? (here: https://docs.docker.com/engine/security/rootless/)

[emike922]

Did you, by chance, switch graphdrivers or even switch to the containerd backed image store?

No, I didn't change anything, at least not consciously. We essentially run what is described at https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository.

[EyuelMathewos]

@emike922 if Container are running actively it will not clean any of use by docker system i think that why. i also recommend to run the command as root user also "sudo docker system prune --all --force" this command clean only unused containers data( containers with state stop ).

[emike922]

@EyuelMathewos I have made sure that no containers are running before running docker system prune -af, and running it as root made no difference either.

As an aside, why should I have to run the prune command as root? Our dockerd is already running as root. Is prune not executed with dockerd's permissions?

[EyuelMathewos]

@emike922 for checking the issue which docker dir is consuming alot of your space run "du -h --max-depth=1 /var/lib/docker" share the result and also check if there is no docker service running with replication 0?

[emike922]

@EyuelMathewos it's overlay2 that doesn't get pruned properly, please refer to my original logs in the issue description for the command output you requested.

How do I check for docker services with replication 0? Does the below o/p already answer the question implicitly?

$ docker service ls
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

[EyuelMathewos]

@emike922
First Option check restarting docker service by "systemctl restart docker"
Second Option "cd /var/lib/docker/overlay2" and "rm -r ./*" which remove the all overlay2 after that reboot the linux machine "reboot" & systemctl restart docker.
this happend because all space is consumed and docker is not working properly i hope this helps.

[timon-michel-scopevisio]

As an aside, why should I have to run the prune command as root? Our dockerd is already running as root. Is prune not executed with dockerd's permissions?

I don't understand what you mean. In order to be able to communicate with the docker daemon, you should need to run it as the same user as the daemon. Or am i wrong?

[emike922]

First Option check restarting docker service by "systemctl restart docker"

Interesting, that did clean up some of the data leftover in overlay2, but not all of it:

# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
# docker system prune -af
Total reclaimed space: 12.77GB
# du -ch --max-depth=1 /var/lib/docker/
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
12G     /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
8.9M    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
18M     /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
12G     /var/lib/docker/
12G     total
# systemctl restart docker
# du -ch --max-depth=1 /var/lib/docker/
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
4.4G    /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
8.9M    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
18M     /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
4.4G    /var/lib/docker/
4.4G    total

Second Option "cd /var/lib/docker/overlay2" and "rm -r ./*" which remove the all overlay2 after that reboot the linux machine "reboot" & systemctl restart docker.

That may work, I can't try right now, but I can't really go around rebooting the host every couple of days. :(
Ideally, docker system prune -af should catch and remove all of the stale/dangling entries in overlay2, wouldn't you agree?

I don't understand what you mean. In order to be able to communicate with the docker daemon, you should need to run it as the same user as the daemon. Or am i wrong?

I am running the docker CLI as non-privileged user in the "docker" group. As such, my user can access /var/run/docker.sock and therefore is able to issue commands like prune, which I expect will be executed by dockerd (which runs with root privileges). That's why I am wondering if my user really needs to be root (or use sudo), because dockerd is the one actually doing the pruning.

[EyuelMathewos]

@emike922 Okay that is progress let's try running & share the results it helps

• docker volume ls
• docker container ls -a
• docker network ls
• docker plugin ls
• docker image ls

[emike922]

@EyuelMathewos I would say those all look unremarkable:

# docker volume ls
DRIVER    VOLUME NAME
# docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
fe15e008745d   bridge    bridge    local
4890ede3afd1   host      host      local
2965bd8b6905   none      null      local
# docker plugin ls
ID        NAME      DESCRIPTION   ENABLED
# docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

[EyuelMathewos]

@emike922 All are clean what remains in "/var/lib/docker/overlay2" try "ls -a /var/lib/docker/overlay2" share the results

[emike922]

@EyuelMathewos This is what's remaining. Can you infer anything from that?

# du -h --max-depth=1 /var/lib/docker/overlay2 | sort -h
28K     /var/lib/docker/overlay2/022076e4c2ea8a14ef639aada3bac355b796ea3364dc258cc82b5c7542ce16fa
28K     /var/lib/docker/overlay2/0fcd0396b0fb81670a13013a4bd423cee81dc283c1020d677f4593981eca5b4b
28K     /var/lib/docker/overlay2/108e5f2461abdce0c9f4859b7e612da9008589a1fe9bce8923cce5d4f522c5b4
28K     /var/lib/docker/overlay2/1904f568beb17ddc4111f6aafb97f0c05958c64553d9b1182d03a23758ba8dfa
28K     /var/lib/docker/overlay2/1972924ff48e6a6cd817f6a0646c4e2ee8f8f43f5e3767704ef81a47d707f535
28K     /var/lib/docker/overlay2/2be37bfa72549381a40e8a85668ddc518fc0a9d49ffce9896fa72595126fd06a
28K     /var/lib/docker/overlay2/2dca9ae8a227be68aea01e99833c6d2f83cc9fa077324fd9130c9a228c9e4752
28K     /var/lib/docker/overlay2/30d9ffe8f9d498881dc63f6aa19248dafce0ef6f2851ebfbb704f2a011df809c
28K     /var/lib/docker/overlay2/31e8024da2b4562eae3fd6a549e10e2115f2ed3bd79b16e23fed5bfbfab89c09
28K     /var/lib/docker/overlay2/338fbb0f1dd54f0a97e5be396c69b34e0cf8bffba9312b4357590d898283be6b
28K     /var/lib/docker/overlay2/3559251f7f5d8e9f1af7d20fd99a97eb3c3d10ed3064ce078d0f216dafb7ffe0
28K     /var/lib/docker/overlay2/37f5a25ae1cc76b33a9e3b0e3bebdf1deae64b4968047ad5e035ca6cb1efcf3a
28K     /var/lib/docker/overlay2/3fb9a6a43893ed3af74c4e8f452ffcd318c7e5826c88c08316059358da8216ff
28K     /var/lib/docker/overlay2/414bf51ab93ff4bb7e5670452e0aed11521f0d5e2d085933348b53790c8d5614
28K     /var/lib/docker/overlay2/43447517babe4472af38fa3d7819f20b4603731f7dd99e1c0a44476eb662a7be
28K     /var/lib/docker/overlay2/43be2a03435ec4a2afd535632bd77188f116e4066523e598230f0e692f6753de
28K     /var/lib/docker/overlay2/47045eb6f50f6e2c0c24bad3c306980be186fdcddd56a608c0d7304048be9c08
28K     /var/lib/docker/overlay2/4a0aebf5d367f5ca652e9322774ff0bb9ef9d90c512ef10b7871fe35f7de224c
28K     /var/lib/docker/overlay2/4aa6ba3ea0648a3ed73a12d28b9f9457244cecce98560f0dc4ef4bbdd812caa5
28K     /var/lib/docker/overlay2/4dc6b336af7b0b3cbca805bd484ea75f3ddc3b4e4282f60c570f3aba119551d9
28K     /var/lib/docker/overlay2/526fe45515f88fd9d3c3b4b7f8c9751a994cb0b612bdaa0465cf7edeb852d23d
28K     /var/lib/docker/overlay2/53569f132403fb412b4903da9df7ecc208b3ca6c9ab5ec5176c1dfba68be758f
28K     /var/lib/docker/overlay2/5922c49234bfa9f8eed1c6ad4ab18bdb5b7cf62e21984d2b792f7583fba12608
28K     /var/lib/docker/overlay2/5da93877906f89ee681d2a9bdd75486b84fed841a338cfcac592fb860e1ac165
28K     /var/lib/docker/overlay2/5e3697ef7973751d3e4a36c0c2ac8ee49f9db2cb0a5496279a3269595cbb06e1
28K     /var/lib/docker/overlay2/6267cc8c93157afe27b0664d44931a1e37b431c1bc12b4b2b73b26d172cbeaad
28K     /var/lib/docker/overlay2/72c4ea6eb9de00560135c525814260f9da34d5057ba7674e873725d0145e9d1f
28K     /var/lib/docker/overlay2/7556694aa0e4f86481a028f3ca87fc347d290ae013fd009780e4ef06007082b3
28K     /var/lib/docker/overlay2/75d336cf76495fb05be12d0a5cb198bf19e8c38cf872b8239796b8fe752e9b8f
28K     /var/lib/docker/overlay2/76789d56b87543c07948bd8b0e760402f4ce89027fb943c7753cd10db44d7076
28K     /var/lib/docker/overlay2/7ceba0d6be3b903574b012ae9e4122c8c26befc0a07bd6634197dfe39bb2e1ec
28K     /var/lib/docker/overlay2/7ec76091da6e3be065f2d0a41ce3cb6f321406292976b6b34f1ef8f00d2a7499
28K     /var/lib/docker/overlay2/7ef50662610cb28764440228696a6778e5bb17777f6864441fd2ecfd815aea97
28K     /var/lib/docker/overlay2/803b84975bf7b8a9e5e190360f227763a2e2aabbffe199f342773531200fd94c
28K     /var/lib/docker/overlay2/809a5cad8f0b658d035157b5736bf76195d9c06485a03a6f1fa962e82f0faa51
28K     /var/lib/docker/overlay2/8f221ee08f7e839a5210bad804f2de9df3e4da10642208afea1ac1c50d0e2953
28K     /var/lib/docker/overlay2/9147c8e51e9043768165cb9f65dfbbaeda4fee7f7f5606d7924ea34fe82823dc
28K     /var/lib/docker/overlay2/91490331cd32b72813ac3c5c7df2db6f7f1dee078592e5edb8a6fce1e3d0ad24
28K     /var/lib/docker/overlay2/918de3a940cd106c480d97aaff2504ecec62365b4838010f5e737cda5248eeb2
28K     /var/lib/docker/overlay2/9220c2bc4c265e1ca8a100a04ab56e21d821428839d840a95239dda2d9fb07f8
28K     /var/lib/docker/overlay2/9507a82ab080a827690bc1c6d764148bfe898bddc10372345eb8d5ad38454bb4
28K     /var/lib/docker/overlay2/97a57ca48d588a1e831a183a9323531410fc5d78ac94a1550d8a5e9ed8d248fb
28K     /var/lib/docker/overlay2/97b793f9325621eeb1a0c10cc18b9b63477edb7f46e8846af8124d1fe831bc7e
28K     /var/lib/docker/overlay2/9b380e9d350530581c7d01aa63e3bedeefed7682f6e7cfd5d92ad305c8826086
28K     /var/lib/docker/overlay2/a4eb0dfa3181c9429b5fc969f6e08e42ba133d0a340b40dad98f70341e7d710d
28K     /var/lib/docker/overlay2/a6471fdd4b709128a7db6ec3bf7ab74864f3ac346dbf31a2cbc74083a37100ba
28K     /var/lib/docker/overlay2/aac30f188a7b70dd8ceb9105f079bad9b6f7fb2e8d21473e72dfefce4144c093
28K     /var/lib/docker/overlay2/ab45b2a3a41e6df3ac90609cec4a5e3beacd5471c70e1c21a8307cd32c91b08c
28K     /var/lib/docker/overlay2/b2284c477db1d38995ecb55786bb7de10b7cab3c7e486accbfbb48be61cc8ac4
28K     /var/lib/docker/overlay2/b539972b90b41cd527d0cff6f78928a5029ee614c5a069c0019321d58c908de0
28K     /var/lib/docker/overlay2/b56fe94e35916b5ab6f1bb86b9c6466c2c18eadbcfafffd75e7b86eca4c2ab1f
28K     /var/lib/docker/overlay2/bc6d65968d77e5966785e85d3512b82bfc698d3dfe4e59b4ff70f78305289b07
28K     /var/lib/docker/overlay2/c136dbe94c420a1f2ce05add152dbb96d80a309513c43c87106016be9568573f
28K     /var/lib/docker/overlay2/c2a7d137110f29cc13953c1ef0094f25c4c161d3b449b63fb2beb6b5c89936b4
28K     /var/lib/docker/overlay2/c60f1691b727df2a249e37518b995df6a7ff0fd06e112391e3bd331dcc45028a
28K     /var/lib/docker/overlay2/ca82888ab4baa4e493dfe66f7f976892d597d538738b74ca076d76dc505f81cc
28K     /var/lib/docker/overlay2/cc1ea6b3cd19fcc8f6d1470135b45f2706056d18838ee7b2c78aeb269c2f4750
28K     /var/lib/docker/overlay2/ccdc55d46e440a23454c5b5e7878eebe46fd410e91fb83a54a516c32d6e9134c
28K     /var/lib/docker/overlay2/cf4783fec91839be29d9d4b4bda86d2d3535ca33067bc5a7e2891b5f32ec017a
28K     /var/lib/docker/overlay2/d1f3986e914b3bd28570665d4db2c9ca1747d1e0379d2f7a3ab25bf38a3fd861
28K     /var/lib/docker/overlay2/d2071e1c19daa0fc5683bd33bec80d278b238928fc11a8dd807d92bee92430c7
28K     /var/lib/docker/overlay2/e1143a7af864458600d0352f3b9be8f4d9a93b0435522abcfd00f6d08f4a4669
28K     /var/lib/docker/overlay2/e164d963bf77f593657c3c2a0821545ecdda2142d9ef08692b515695164354df
28K     /var/lib/docker/overlay2/e4f20f3382f7ddd6677fa6aae6979a0acdc6af27631f4980eecf23f7ca48782f
28K     /var/lib/docker/overlay2/ed7ed01076698aa79eadb2d08d09ebe4c0213e7e7298d4a8ffe1af05086160b3
28K     /var/lib/docker/overlay2/f2e53971f499eeb8bc26b9b6615ecd27a23bb2133b2c937eda811fb727dbbdf9
28K     /var/lib/docker/overlay2/f48de88276d01a858d7e77077950b9fc26f63bac545cf9e7b52dcffa4122f027
28K     /var/lib/docker/overlay2/f5ce3a39158839b60cf9a4a40d9c7c2d034df5492c5b633375a8c4428d44bb56
40K     /var/lib/docker/overlay2/022076e4c2ea8a14ef639aada3bac355b796ea3364dc258cc82b5c7542ce16fa-init
40K     /var/lib/docker/overlay2/0fcd0396b0fb81670a13013a4bd423cee81dc283c1020d677f4593981eca5b4b-init
40K     /var/lib/docker/overlay2/108e5f2461abdce0c9f4859b7e612da9008589a1fe9bce8923cce5d4f522c5b4-init
40K     /var/lib/docker/overlay2/1904f568beb17ddc4111f6aafb97f0c05958c64553d9b1182d03a23758ba8dfa-init
40K     /var/lib/docker/overlay2/1972924ff48e6a6cd817f6a0646c4e2ee8f8f43f5e3767704ef81a47d707f535-init
40K     /var/lib/docker/overlay2/2be37bfa72549381a40e8a85668ddc518fc0a9d49ffce9896fa72595126fd06a-init
40K     /var/lib/docker/overlay2/2dca9ae8a227be68aea01e99833c6d2f83cc9fa077324fd9130c9a228c9e4752-init
40K     /var/lib/docker/overlay2/30d9ffe8f9d498881dc63f6aa19248dafce0ef6f2851ebfbb704f2a011df809c-init
40K     /var/lib/docker/overlay2/31e8024da2b4562eae3fd6a549e10e2115f2ed3bd79b16e23fed5bfbfab89c09-init
40K     /var/lib/docker/overlay2/338fbb0f1dd54f0a97e5be396c69b34e0cf8bffba9312b4357590d898283be6b-init
40K     /var/lib/docker/overlay2/3559251f7f5d8e9f1af7d20fd99a97eb3c3d10ed3064ce078d0f216dafb7ffe0-init
40K     /var/lib/docker/overlay2/37f5a25ae1cc76b33a9e3b0e3bebdf1deae64b4968047ad5e035ca6cb1efcf3a-init
40K     /var/lib/docker/overlay2/3fb9a6a43893ed3af74c4e8f452ffcd318c7e5826c88c08316059358da8216ff-init
40K     /var/lib/docker/overlay2/414bf51ab93ff4bb7e5670452e0aed11521f0d5e2d085933348b53790c8d5614-init
40K     /var/lib/docker/overlay2/43447517babe4472af38fa3d7819f20b4603731f7dd99e1c0a44476eb662a7be-init
40K     /var/lib/docker/overlay2/43be2a03435ec4a2afd535632bd77188f116e4066523e598230f0e692f6753de-init
40K     /var/lib/docker/overlay2/47045eb6f50f6e2c0c24bad3c306980be186fdcddd56a608c0d7304048be9c08-init
40K     /var/lib/docker/overlay2/4a0aebf5d367f5ca652e9322774ff0bb9ef9d90c512ef10b7871fe35f7de224c-init
40K     /var/lib/docker/overlay2/4aa6ba3ea0648a3ed73a12d28b9f9457244cecce98560f0dc4ef4bbdd812caa5-init
40K     /var/lib/docker/overlay2/4dc6b336af7b0b3cbca805bd484ea75f3ddc3b4e4282f60c570f3aba119551d9-init
40K     /var/lib/docker/overlay2/526fe45515f88fd9d3c3b4b7f8c9751a994cb0b612bdaa0465cf7edeb852d23d-init
40K     /var/lib/docker/overlay2/53569f132403fb412b4903da9df7ecc208b3ca6c9ab5ec5176c1dfba68be758f-init
40K     /var/lib/docker/overlay2/5922c49234bfa9f8eed1c6ad4ab18bdb5b7cf62e21984d2b792f7583fba12608-init
40K     /var/lib/docker/overlay2/5da93877906f89ee681d2a9bdd75486b84fed841a338cfcac592fb860e1ac165-init
40K     /var/lib/docker/overlay2/5e3697ef7973751d3e4a36c0c2ac8ee49f9db2cb0a5496279a3269595cbb06e1-init
40K     /var/lib/docker/overlay2/6267cc8c93157afe27b0664d44931a1e37b431c1bc12b4b2b73b26d172cbeaad-init
40K     /var/lib/docker/overlay2/72c4ea6eb9de00560135c525814260f9da34d5057ba7674e873725d0145e9d1f-init
40K     /var/lib/docker/overlay2/7556694aa0e4f86481a028f3ca87fc347d290ae013fd009780e4ef06007082b3-init
40K     /var/lib/docker/overlay2/75d336cf76495fb05be12d0a5cb198bf19e8c38cf872b8239796b8fe752e9b8f-init
40K     /var/lib/docker/overlay2/76789d56b87543c07948bd8b0e760402f4ce89027fb943c7753cd10db44d7076-init
40K     /var/lib/docker/overlay2/7ceba0d6be3b903574b012ae9e4122c8c26befc0a07bd6634197dfe39bb2e1ec-init
40K     /var/lib/docker/overlay2/7ec76091da6e3be065f2d0a41ce3cb6f321406292976b6b34f1ef8f00d2a7499-init
40K     /var/lib/docker/overlay2/7ef50662610cb28764440228696a6778e5bb17777f6864441fd2ecfd815aea97-init
40K     /var/lib/docker/overlay2/803b84975bf7b8a9e5e190360f227763a2e2aabbffe199f342773531200fd94c-init
40K     /var/lib/docker/overlay2/809a5cad8f0b658d035157b5736bf76195d9c06485a03a6f1fa962e82f0faa51-init
40K     /var/lib/docker/overlay2/8f221ee08f7e839a5210bad804f2de9df3e4da10642208afea1ac1c50d0e2953-init
40K     /var/lib/docker/overlay2/9147c8e51e9043768165cb9f65dfbbaeda4fee7f7f5606d7924ea34fe82823dc-init
40K     /var/lib/docker/overlay2/91490331cd32b72813ac3c5c7df2db6f7f1dee078592e5edb8a6fce1e3d0ad24-init
40K     /var/lib/docker/overlay2/918de3a940cd106c480d97aaff2504ecec62365b4838010f5e737cda5248eeb2-init
40K     /var/lib/docker/overlay2/9220c2bc4c265e1ca8a100a04ab56e21d821428839d840a95239dda2d9fb07f8-init
40K     /var/lib/docker/overlay2/9507a82ab080a827690bc1c6d764148bfe898bddc10372345eb8d5ad38454bb4-init
40K     /var/lib/docker/overlay2/97a57ca48d588a1e831a183a9323531410fc5d78ac94a1550d8a5e9ed8d248fb-init
40K     /var/lib/docker/overlay2/97b793f9325621eeb1a0c10cc18b9b63477edb7f46e8846af8124d1fe831bc7e-init
40K     /var/lib/docker/overlay2/9b380e9d350530581c7d01aa63e3bedeefed7682f6e7cfd5d92ad305c8826086-init
40K     /var/lib/docker/overlay2/a4eb0dfa3181c9429b5fc969f6e08e42ba133d0a340b40dad98f70341e7d710d-init
40K     /var/lib/docker/overlay2/a6471fdd4b709128a7db6ec3bf7ab74864f3ac346dbf31a2cbc74083a37100ba-init
40K     /var/lib/docker/overlay2/aac30f188a7b70dd8ceb9105f079bad9b6f7fb2e8d21473e72dfefce4144c093-init
40K     /var/lib/docker/overlay2/ab45b2a3a41e6df3ac90609cec4a5e3beacd5471c70e1c21a8307cd32c91b08c-init
40K     /var/lib/docker/overlay2/b2284c477db1d38995ecb55786bb7de10b7cab3c7e486accbfbb48be61cc8ac4-init
40K     /var/lib/docker/overlay2/b539972b90b41cd527d0cff6f78928a5029ee614c5a069c0019321d58c908de0-init
40K     /var/lib/docker/overlay2/b56fe94e35916b5ab6f1bb86b9c6466c2c18eadbcfafffd75e7b86eca4c2ab1f-init
40K     /var/lib/docker/overlay2/bc6d65968d77e5966785e85d3512b82bfc698d3dfe4e59b4ff70f78305289b07-init
40K     /var/lib/docker/overlay2/c136dbe94c420a1f2ce05add152dbb96d80a309513c43c87106016be9568573f-init
40K     /var/lib/docker/overlay2/c2a7d137110f29cc13953c1ef0094f25c4c161d3b449b63fb2beb6b5c89936b4-init
40K     /var/lib/docker/overlay2/c60f1691b727df2a249e37518b995df6a7ff0fd06e112391e3bd331dcc45028a-init
40K     /var/lib/docker/overlay2/ca82888ab4baa4e493dfe66f7f976892d597d538738b74ca076d76dc505f81cc-init
40K     /var/lib/docker/overlay2/cc1ea6b3cd19fcc8f6d1470135b45f2706056d18838ee7b2c78aeb269c2f4750-init
40K     /var/lib/docker/overlay2/ccdc55d46e440a23454c5b5e7878eebe46fd410e91fb83a54a516c32d6e9134c-init
40K     /var/lib/docker/overlay2/cf4783fec91839be29d9d4b4bda86d2d3535ca33067bc5a7e2891b5f32ec017a-init
40K     /var/lib/docker/overlay2/d1f3986e914b3bd28570665d4db2c9ca1747d1e0379d2f7a3ab25bf38a3fd861-init
40K     /var/lib/docker/overlay2/d2071e1c19daa0fc5683bd33bec80d278b238928fc11a8dd807d92bee92430c7-init
40K     /var/lib/docker/overlay2/e1143a7af864458600d0352f3b9be8f4d9a93b0435522abcfd00f6d08f4a4669-init
40K     /var/lib/docker/overlay2/e164d963bf77f593657c3c2a0821545ecdda2142d9ef08692b515695164354df-init
40K     /var/lib/docker/overlay2/e4f20f3382f7ddd6677fa6aae6979a0acdc6af27631f4980eecf23f7ca48782f-init
40K     /var/lib/docker/overlay2/ed7ed01076698aa79eadb2d08d09ebe4c0213e7e7298d4a8ffe1af05086160b3-init
40K     /var/lib/docker/overlay2/f2e53971f499eeb8bc26b9b6615ecd27a23bb2133b2c937eda811fb727dbbdf9-init
40K     /var/lib/docker/overlay2/f48de88276d01a858d7e77077950b9fc26f63bac545cf9e7b52dcffa4122f027-init
40K     /var/lib/docker/overlay2/f5ce3a39158839b60cf9a4a40d9c7c2d034df5492c5b633375a8c4428d44bb56-init
848K    /var/lib/docker/overlay2/l
12M     /var/lib/docker/overlay2/0f6eec1f33ff745a875c6da25cc63192211ed7cde6f6e1fa88a144e5efb61258
12M     /var/lib/docker/overlay2/11bd070e718a4ad3a5fd8b95b2ad3dee8bd194f1fda9aa47a7b6a0ee13360996
12M     /var/lib/docker/overlay2/12e0c9266f57ab7d188939bd0caf22e20acc0ea95b114e7077b4c06e10546e2a
12M     /var/lib/docker/overlay2/14a3ec226cf924ae8d53c0dab26000782ca661d1f0c3213713d6976355e4a2a0
12M     /var/lib/docker/overlay2/18038f74bf029813c2d6a5d025efcff69c027d9dd2e0a686c052a6bcb4a14820
12M     /var/lib/docker/overlay2/1d5f2d45079dd5947b54c348fd79c612bf25b760853e309acd25f640a6624246
12M     /var/lib/docker/overlay2/22aafcfb818ebcc579f26473df50d878f81874015c598852224300d27806e11a
12M     /var/lib/docker/overlay2/32b886491d2a86ceec1997149be442ad1e83d0a5d9e475dcf1ecc95c314f3df9
12M     /var/lib/docker/overlay2/339ce9be6c0ecb84d6c4c470ae22107a1cfd993d205cd569f80d4317664698ce
12M     /var/lib/docker/overlay2/3507739e44bb87b5a0d5f3ef6a5fbd8faa760bd189949b529f1bc7e21ed94cdd
12M     /var/lib/docker/overlay2/3a3145821215e2aeabff60761f011a38ed79de590b23eb651797299b4c22f5f9
12M     /var/lib/docker/overlay2/447c1f1c9c2a162b7a12f4804ea7772acabd187fad6a93e741c4af43bb1527a0
12M     /var/lib/docker/overlay2/487fe220f27259b9ffc07d6bb86a1234c6d10682c7fcb18d14bc003717a39ab1
12M     /var/lib/docker/overlay2/4a2775b006d77a1a0b5861fff51007ed1ad4e043261aae82bac400e4fc3b71c4
12M     /var/lib/docker/overlay2/4d7820155509f96d95d1c2a4e328cb0c6d714de1575ea6b69dbacf387bcaa14b
12M     /var/lib/docker/overlay2/500707b675fc231c8a313ffb1afd1bbd9da73f20bce9395b18dbdcc675d204d2
12M     /var/lib/docker/overlay2/5e1b2c4bbbfd51c93fcf23baeb66be6023173e3eda06436581cbb0d156eef224
12M     /var/lib/docker/overlay2/703d8ba6155597f72fd1c5ce356634dca7099f38fa5ebf1b226b10f4b3b35a15
12M     /var/lib/docker/overlay2/71a0d1a35aa8dfabf9981f2354fbcfe5aed51b733a9918031b1ce7b2cc985571
12M     /var/lib/docker/overlay2/8364866b9f7539541ac2ce3532f98aa01c74610ab1d42d6ddc1dd782f01df777
12M     /var/lib/docker/overlay2/8ab0ab17cc229de1c6268c824deb17d10d697262e2c06a6d249f7e3a3c3ffad6
12M     /var/lib/docker/overlay2/9414481a8cc0dc7be44ae674627d0ccd0a6448230e0abcd927e9dce2cad2b117
12M     /var/lib/docker/overlay2/a98c079a4e95b8d73eb5090846863bc1f37b432cf808ee47e3948e175ac2669e
12M     /var/lib/docker/overlay2/b5f2c0d455769d041cce8c7c8f2f1aeeb177ee226644a9b5576a8497e29a8da1
12M     /var/lib/docker/overlay2/bcd58aab586d43bf42b2de42eb33e0d110e80b359c1aa0380d468f74e94d814f
12M     /var/lib/docker/overlay2/cc4e13d5d1e0a70d3b777925566825a3eccc4d338e659c06e222a14282b2a071
12M     /var/lib/docker/overlay2/d10459fbe32625f99251269cc1ee18ccddf246d4f3a1db5ab4c8fc81f6083df8
12M     /var/lib/docker/overlay2/d53d571be663ad78f04de744e78c23107aa9c647a2abdc312f3de1b52214f437
12M     /var/lib/docker/overlay2/d982a71269464169f40dd9b97165aa65f5d905b5b38ba142ba1a0ae2c2814f33
12M     /var/lib/docker/overlay2/dece2e0ebe3c4be2b977148311878680dfc2494419aa42405f4d92f3f3131514
12M     /var/lib/docker/overlay2/eb6ade431731747c02694951e965fd87c259afe626d9ef2ab1135978f0242f94
12M     /var/lib/docker/overlay2/eb774ed51705a1e2b9d0a05fb4c9e9cbf7fa28e65936f888e11efaff4238db92
12M     /var/lib/docker/overlay2/f3bee4679e4ba49ef1448dd4dc7f911239d93ef0bb4dac99fc30042cf8c1dc0d
12M     /var/lib/docker/overlay2/ff684ec445f579fabcc5f948d643efc22f7de71748d5841d2a5d230217865632
43M     /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792
118M    /var/lib/docker/overlay2/114f461ddc4a8dc93baebfaeae6b99df6ed2af6e1ce139d40aa2d6b3570bd972
118M    /var/lib/docker/overlay2/1bdaa1e424e599e988066acd1187b07bc199ed37f91c1179d4a01d200f37720d
118M    /var/lib/docker/overlay2/1e1a587688a9f9e3691f02851dda6196c04d0a3f8c999af5c191db4cfab76a9b
118M    /var/lib/docker/overlay2/24e5eb43bd4d7100b1956011acb5d35a6771f915a53069eebd8d62c8fa216ecc
118M    /var/lib/docker/overlay2/284c5bfaf3217d18abe21e596fef954a412e2b49c38b3fae3c1891fcc632636a
118M    /var/lib/docker/overlay2/2b31b050d250d6fc23f6c3bae706e871debba9c1ff86f6a1c2d0bdd39c8d42ab
118M    /var/lib/docker/overlay2/39364c897e59457c6ac627a4ed65fc0b6d7d5292fcc0ff70afe0c64baeda3254
118M    /var/lib/docker/overlay2/426bb0fddab6257b7fdf13fe764f4d01810779eb5f7464965c369b2222b823fc
118M    /var/lib/docker/overlay2/446e7eb52b903547fd3f94206a54c1fcc3ca6bcf55b68b53d89be46acd012692
118M    /var/lib/docker/overlay2/486162e92862c37bf6a9392a7af340b9c050a83990193abb06d6520fb5861406
118M    /var/lib/docker/overlay2/51bdc4f2a842b3c4b8fd9336deffa226de20696304251e3434aef8f98cb02d62
118M    /var/lib/docker/overlay2/5fc85c3e2f3e6abd178ce61768434f9d842fa94e946cf6d6f9030b0645e0aa2a
118M    /var/lib/docker/overlay2/64eab9f58bc9b4e5e88f3f3fc0351cbe3406f3b4cdc21bcd52c8ea4e22ddf26e
118M    /var/lib/docker/overlay2/67302d6cc7028e02964400c76e235380e78ac8a4ae01ece598f566b35f1e8f75
118M    /var/lib/docker/overlay2/7868200c332a8a61c020e152b2b7fc8ad9b4a6efb7fdf957d1d07405aebfd046
118M    /var/lib/docker/overlay2/9da93202f6c38ef718a63f92760590fa44ff92eb457eb6906a30ce344016993d
118M    /var/lib/docker/overlay2/9eace8be6aa95e9dea7f045e4599a0356431fa3db7ebbe06c48163db21b20789
118M    /var/lib/docker/overlay2/a1da59e1f0e8faeb11c3bf00b0242160a6542d9d3def648082cf6e6d7ef791e0
118M    /var/lib/docker/overlay2/a4250fee2a542a56f71f3c27b676893277d64bb9bfa2cb2dee9486b99b1aa86e
118M    /var/lib/docker/overlay2/af320e7e38464f3c30f9f0f1e2430a7811ad549fd893a3eb4521d7770398fd26
118M    /var/lib/docker/overlay2/af66e4862b415f6a1a7bfc82291c0d147214f4e79ec2f548db8f7dd8ef10f8b6
118M    /var/lib/docker/overlay2/b164dce7d62ef01efdccd8ba9d41aa673222882242414964efac532b3c826637
118M    /var/lib/docker/overlay2/be5adaaefd075d71492de48d216baab4aa7d55ce0212f8b6c96bbef59a5d9a6e
118M    /var/lib/docker/overlay2/c37b262d2c42cc54ade4ba8c49940b0b30dd685d0f8dec6409163fda52c1807e
118M    /var/lib/docker/overlay2/cacf2b3a775594495996b2b977c6fd15b65ed0d7ef4ae97cf5c8a6c8dad0db81
118M    /var/lib/docker/overlay2/d8d54950701480994d5a7674f8d9d13862bd744952c21d88372d415a915c61eb
118M    /var/lib/docker/overlay2/e1514fc1d13639311aa4d2243bd16de3488c4027e0db84333bdad906bf3d70c5
118M    /var/lib/docker/overlay2/e841a346a27e857f15730468911efd3dc42dcc1e8f48ad04298d54e3456ad1db
118M    /var/lib/docker/overlay2/e8fb3a906f66e781e97aa15136c9197bdc1c5fb4b86bb89178de33d77c0d08dd
118M    /var/lib/docker/overlay2/e9145e0bfcdc0b5ad0980819dfbcaf0a7e6f11b76db7b6ac4d7d6f57151f7cbf
118M    /var/lib/docker/overlay2/efc83e03194a0a189868cfe1c3d9b552d71aad5b5340aa85ce9e8e21f7226a0a
118M    /var/lib/docker/overlay2/f165555f036fa67e0ace61a0b30a47339bce1eb03e835fe5441fadfac06eb578
118M    /var/lib/docker/overlay2/f3d8e6bf9be751e98ecf62a3c99ef0708b66129f3f11741a19d75d3506c09a93
118M    /var/lib/docker/overlay2/fdd9df4cb04e56078881befc7ba4c2ceaec391adf8b725ef5743c7c32f41f7d1
4.4G    /var/lib/docker/overlay2

[EyuelMathewos]

@emike922 we should list out where is max disk usage "du -h --max-depth=1 /var/lib/docker/overlay2"

[emike922]

@EyuelMathewos I updated the output in the previous comment with the information in an attempt to limit the length of this thread somewhat.

[EyuelMathewos]

@emike922 Okay that is helpful, let try

• "systemctl stop docker"
• "systemctl stop docker.socket"
• check the status "systemctl status docker.socket"
• after that then -> "cd /var/lib/docker/overlay2" and "rm -r ./*"

[EyuelMathewos]

@emike922 Finally start

• "systemctl start docker"
• "systemctl start docker.socket"

[emike922]

@EyuelMathewos That certainly removed everything from overlay2, however, docker still seems to "know" about and attempt to use the deleted entries:

+ docker build --build-arg BASE_IMAGE_VERSION="1.2.3" --tag "target-image:1.0.0" .
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.87kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [auth] path/base-image:pull token for private-registry.example.com
#3 DONE 0.0s

#4 [internal] load metadata for private-registry.example.com/path/base-image:1.2.3
#4 DONE 0.5s

#5 [ 1/12] FROM private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc
#5 resolve private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc done
#5 sha256:5c13c8f19ac777c393e8916cd38d5a6b3b66e873e9d8692968452ba08dbe9159 1.70kB / 1.70kB done
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 4.19MB / 31.45MB 0.1s
#5 sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc 741B / 741B done
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 31.45MB / 31.45MB 0.3s
#5 sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3 31.45MB / 31.45MB 0.4s done
#5 extracting sha256:bab6eb336678c7c650680c940838514d2e58b5aab8b2a7f61bc889cd084d66c3
------
 > [ 1/12] FROM private-registry.example.com/path/base-image:1.2.3@sha256:7b6850578fdb5bc31dd59ac0b10b4b131d201603cf680212e0e5b918a22eafcc:
------
Dockerfile:4
--------------------
   2 |     ARG BASE_IMAGE_VERSION
   3 |     
   4 | >>> FROM ${BASE_IMAGE_NAME}:${BASE_IMAGE_VERSION}
   5 |     
   6 |     # Redacted comment
--------------------
ERROR: failed to solve: failed to register layer: open /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792/.tmp-committed3238700635: no such file or directory

[EyuelMathewos]

@emike922 did you stop all docker linux services before deleting the overlay2 dir files ? "systemctl start docker"

• "systemctl start docker.socket"
• Try cleaning "docker builder prune --all --force"

[emike922]

@EyuelMathewos Sorry, I forgot to include the logs for that.. But yes, I did stop all Linux services as you suggested:

# systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
# systemctl stop docker.socket
# systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2024-04-24 15:37:48 CEST; 8s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)

Apr 22 18:07:38 node-10-210-155-101 systemd[1]: Starting Docker Socket for the API.
Apr 22 18:07:38 node-10-210-155-101 systemd[1]: Listening on Docker Socket for the API.
Apr 24 15:37:48 node-10-210-155-101 systemd[1]: docker.socket: Succeeded.
Apr 24 15:37:48 node-10-210-155-101 systemd[1]: Closed Docker Socket for the API.
# cd /var/lib/docker/overlay2
# rm -r ./*
# ls -lh
total 0
# systemctl start docker
# systemctl start docker.socket
# systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-04-24 15:39:05 CEST; 32s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)
      Tasks: 0 (limit: 18991)
     Memory: 4.0K
     CGroup: /system.slice/docker.socket

Apr 24 15:39:05 node-10-210-155-101 systemd[1]: Starting Docker Socket for the API.
Apr 24 15:39:05 node-10-210-155-101 systemd[1]: Listening on Docker Socket for the API.

I also did a docker builder prune --all --force just now, but the build still produces the exact error shared before (complaining about /var/lib/docker/overlay2/c6b098a51cf9ecd3fadbdfe7e4595969a897d158ef50217adf9dd045ee244792). Do you have any other ideas how to make docker "forget" that it used to exist?

By the way, thanks a lot for your continued support on this! :)

[EyuelMathewos]

sudo systemctl stop docker
sudo rm -rf /var/lib/docker/buildkit
sudo systemctl start docker

[emike922]

I followed your steps, but the error is unfortunately completely unchanged..

[EyuelMathewos]

@emike922 remove all docker engine files begin with new by cleaning

• "systemctl stop docker"
• "systemctl stop docker.socket"
• sudo rm -rf /var/lib/docker
• "systemctl start docker"
• "systemctl start docker.socket"

[emike922]

@EyuelMathewos That seems to have done it! Docker recovered after the restart and is no longer generating the above error during builds 🥳

Does any of that help to narrow down why docker system prune -af is not able to detect and remove all of the unused data?
Ideally that should be the only command I need to run every now and then. For sure nuking the entire /var/lib/docker dir every time is a non-starter..