Docker push to registry gets 500 error for some layers

[junzhengca]

Description

When pushing Docker images to container registry, I get 500 errors for some layers after automated retries.

5f70bf18a086: Layer already exists 
adfb1d834c14: Layer already exists 
245b4b04eefc: Layer already exists 
075c1f9ddfb2: Pushing [==================================================>]  2.995MB/2.995MB
d4fa65af1494: Waiting 
284b272e4c5d: Waiting 
aa49f4877180: Waiting 
1ef96cc3773b: Waiting 
0fbf9681f063: Waiting 
4b778def1fd6: Waiting 
6d7bfc74a78d: Waiting 
df87cf2cbe5a: Waiting 
c03b855bf7ce: Waiting 
5a578fe88ffc: Waiting 
ecbda56a4ecd: Waiting 
2ca469df2d7a: Waiting 
72c690143394: Waiting 
received unexpected HTTP status: 500 Internal Server Error

What I have tried:

• Reducing concurrent pushes to 1 layer at a time: does nothing, still same error
• Increasing Nginx reverse proxy body size: does nothing, Nginx doesn't report any errors, I am using the official reverse proxy configuration https://docs.gitea.com/administration/reverse-proxies with certbot for SSL.
• Looking at Gitea logs doesn't raise any immidiate red flags

2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:43 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:c4f956cad6340013f20d5a3e6487b341ee703a1f2f6d6bccae93fffe7dd63854 for 173.35.35.243:0, 200 OK in 5.1ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/<REDACTED_REGISTRY_PATH>/blobs/sha256:f134d980ef08b059da2509fe2b1c86c755f951c5ecfedc76d46fb55a566779ae for 173.35.35.243:0, 200 OK in 10.6ms @ container/container.go:486(container.HeadBlob)
2024/06/19 21:26:44 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 21.0ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:26:49 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 20.0ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:27:00 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 16.1ms @ container/container.go:218(container.InitiateUploadBlob)
2024/06/19 21:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/<REDACTED_REGISTRY_PATH>/blobs/uploads/ for 173.35.35.243:0, 202 Accepted in 14.0ms @ container/container.go:218(container.InitiateUploadBlob)

Gitea Version

1.22.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

Assuming this doesn't matter as I am running the official Docker image

Operating System

Debian GNU/Linux 11 (bullseye) aarch64

How are you running Gitea?

I am running Gitea using a Docker compose file

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:1.22.0
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

Database

SQLite

[techknowlogick]

The logs you have provided are solely for the router, are you able to grab all the logs? If Gitea is providing a 500 then it will be logged if the logs for non-router are also enabled.
You can find details onto how to enable those logs here: https://docs.gitea.com/help/support

[GiteaBot]

We close issues that need feedback from the author if there were no new comments for a month. 🍵