Sony RAW(.arw) file thumbnail generation failed

[kchen0x]

The bug

upload SONY RAW (.arw) file as external library photos but can't not generate thumbnail. Details can be seen at related logs.

The OS that Immich Server is running on

Ubuntu 20.04.6 LTS

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.3

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /data02/Pictures:/mnt/media/pictures:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/data02/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. upload .arw file to `picture` as external library
2. Scan New Library Files
3. GENERATE THUMBNAILS of MISSING thumbnail

Relevant log output

[Nest] 7  - 06/15/2024, 12:51:52 PM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: Input file has corrupt header: magickload: Magick: Unsupported file format or not RAW file `/mnt/media/pictures/DSC00887.ARW' @ error/dng.c/ReadDNGImage/555 (null)
[Nest] 7  - 06/15/2024, 12:51:52 PM   ERROR [Microservices:JobService] Error: Input file has corrupt header: magickload: Magick: Unsupported file format or not RAW file `/mnt/media/pictures/DSC00887.ARW' @ error/dng.c/ReadDNGImage/555 (null)
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:158:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:135:29)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 06/15/2024, 12:51:52 PM   ERROR [Microservices:JobService] Object:
{
  "id": "94033a9f-371c-4adb-8e6c-b99c6648866c"
}

Additional information

No response

[Rubeast]

I have a similar problem since the last update. But it seems to be random. I can take 2 pictures. One is uploaded with thumbnail generation without problem. The other gives a broken thumbnail and this log:

[Nest] 6  - 06/16/2024, 7:38:33 PM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: Input file contains unsupported image format
[Nest] 6  - 06/16/2024, 7:38:33 PM   ERROR [Microservices:JobService] Error: Input file contains unsupported image format
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:158:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:135:29)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 6  - 06/16/2024, 7:38:33 PM   ERROR [Microservices:JobService] Object:
{
  "id": "4e65ef49-c5e3-4e03-b9a5-a3cea96c9b29",
  "source": "upload"

[mertalev]

Can you share a sample image so I can reproduce the issue?

[mertalev]

Possibly related issue in LibRaw. We should really update to their latest.

[kchen0x]

Can you share a sample image so I can reproduce the issue?

Sure: DSC00965.zip

This photo can be previewed by macOS well:

[hugohabicht01]

i've noticed something somewhat similar, raw files (.dng in my case, shot on google pixel) don't have a thumbnail (as shown in the images above), until i click on it. After that they have thumbnails and it shows in the server logs that webp versions were created
(Successfully generated WEBP image thumbnail for asset 923d0475-5ed6-41db-9561-fd1f5cc0a4f3)
So the issue might not be that thumbnails don't get created at all, but just that it takes some time or a specific request

[mertalev]

Fixed via #10457