Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker image cannot be used with SQLite in a mounted volume #546

Closed
schlamar opened this issue Dec 31, 2022 · 22 comments · Fixed by #907
Closed

Docker image cannot be used with SQLite in a mounted volume #546

schlamar opened this issue Dec 31, 2022 · 22 comments · Fixed by #907
Assignees
Labels
tag:more-info More information needed type:bug Something isn't working
Milestone

Comments

@schlamar
Copy link

Data

  • Shiori version: latest docker image (1.5.3)
  • Database Engine: SQLite
  • Operating system: N/A
  • CLI/Web interface/Web Extension: N/A

Describe the bug / actual behavior

If you run the Docker image with /shiori as mounted volume the shiori process cannot write the database. It crashes with "out of memory" (see stacktrace below).

Expected behavior

Shiori with SQLite can be used with a mounted volume (running without mounted volume doesn't make sense for production deployment).

To Reproduce

Create a compose.yaml

services:
  shiori:
    container_name: shiori
    image: "ghcr.io/go-shiori/shiori"
    restart: unless-stopped
    volumes:
      - ./data:/shiori  
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 8080:8080

Running docker compose up results in:

shiori  | panic: unable to open database file: out of memory (14)
shiori  |
shiori  | goroutine 1 [running]:
shiori  | github.com/jmoiron/sqlx.MustConnect(...)
shiori  |       /home/runner/go/pkg/mod/github.com/jmoiron/[email protected]/sqlx.go:654
shiori  | github.com/go-shiori/shiori/internal/database.OpenSQLiteDatabase({0xc0000ca438, 0x15fc350})
shiori  |       /home/runner/work/shiori/shiori/internal/database/sqlite.go:27 +0xaa
shiori  | github.com/go-shiori/shiori/internal/cmd.openSQLiteDatabase()
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:117 +0x6a
shiori  | github.com/go-shiori/shiori/internal/cmd.openDatabase()
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:111 +0x79
shiori  | github.com/go-shiori/shiori/internal/cmd.preRunRootHandler(0xc00021cc80, {0xbbd9a8, 0x0, 0x0})
shiori  |       /home/runner/work/shiori/shiori/internal/cmd/root.go:64 +0x116
shiori  | github.com/spf13/cobra.(*Command).execute(0xc00021cc80, {0x15fbd98, 0x0, 0x0})
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:840 +0x50a
shiori  | github.com/spf13/cobra.(*Command).ExecuteC(0xc0001a3400)
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
shiori  | github.com/spf13/cobra.(*Command).Execute(...)
shiori  |       /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
shiori  | main.main()
shiori  |       /home/runner/work/shiori/shiori/main.go:19 +0x1e

Notes

This is probably a permission issue. I don't think you should create an extra user in a Docker container to run an application in default use cases (disclaimer: no Docker expert here, I'm just guessing). Just run mkdir /shiori instead of addgroup/adduser probably fixes this issue.

@schlamar schlamar added the type:bug Something isn't working label Dec 31, 2022
@schlamar
Copy link
Author

Next issue on manual build, there is no alpine 3.16 on ghcr.io registry:

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: ghcr.io/ghcri/alpine:3.16: not found

Probably you should use the official Docker hub here.

@schlamar
Copy link
Author

This Dockerfile worked for me

# build stage
FROM golang:1.19-alpine3.16 AS builder
WORKDIR /src
COPY . .
RUN go build -ldflags '-s -w'

# server image

FROM alpine:3.16
LABEL org.opencontainers.image.source https://github.com/go-shiori/shiori
COPY --from=builder /src/shiori /usr/bin/
RUN mkdir /shiori
WORKDIR /shiori
EXPOSE 8080
ENV SHIORI_DIR /shiori/
ENTRYPOINT ["/usr/bin/shiori"]
CMD ["serve"]

@schlamar
Copy link
Author

A workaround is to create the "data" directory before running docker. It is only broken if the data directory is created by docker / the container.

The documented docker run from Usage.md works because there is no subdirectory involved.

@fmartingr
Copy link
Member

Hey @schlamar, have you tried this with the dev tag? It works correctly on my machine (though is a Mac), but it automatically migrates the database and creates the appropriate folders:

docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev

Though I tried with latest and it works as well in my case (and I have it setup on a server with a volume too, have been working for a while now). Which host operating system are you using?

@fmartingr fmartingr added the tag:more-info More information needed label Jan 10, 2023
@therysin
Copy link

therysin commented Jan 11, 2023

Hey @schlamar, have you tried this with the dev tag? It works correctly on my machine (though is a Mac), but it automatically migrates the database and creates the appropriate folders:

docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev

Though I tried with latest and it works as well in my case (and I have it setup on a server with a volume too, have been working for a while now). Which host operating system are you using?

Also having the same issue with both the latest and dev builds. I’m on Ubuntu 22.04.1 arm (Oracle Cloud).

Seemed like an awesome project but the container won’t start unfortunately (same out of memory error ).

I should also note that I’m using docker-compose(same file as OP to test), and it’s running behind traefik.

If the mounted volume lines are commented or removed from the compose file, shiori works fine (without data persistence).

@schlamar
Copy link
Author

schlamar commented Jan 11, 2023

Dev fails, too:

$ sudo docker run -v "$(pwd)/data:/shiori" -p 8080:8080 ghcr.io/go-shiori/shiori:dev
Unable to find image 'ghcr.io/go-shiori/shiori:dev' locally
dev: Pulling from go-shiori/shiori
...
Status: Downloaded newer image for ghcr.io/go-shiori/shiori:dev
Failed to open database: unable to open database file: out of memory (14)

Running Arch Linux with current docker release:

$ docker --version
Docker version 20.10.22, build 3a2c30b63a

@Denow
Copy link

Denow commented Jan 18, 2023

I have the same problem with Ubuntu 20.04.5 LTS (x86).

$ docker --version
Docker version 20.10.22, build 3a2c30b

@fmartingr
Copy link
Member

I have the same problem with Ubuntu 20.04.5 LTS (x86).

$ docker --version
Docker version 20.10.22, build 3a2c30b

Are you running ubuntu over oracle cloud as well? Do you have the exact same error as @schlamar ?

@Denow
Copy link

Denow commented Jan 18, 2023

No, I am running Ubuntu in a vm on an onprem esxi host. The error I am receiving is the same as @schlamar:

Failed to open database: unable to open database file: out of memory (14)

The container keeps restarting after this error message and the data dir is empty so it does look like a permission issue.

@Denow
Copy link

Denow commented Jan 18, 2023

Nevermind I was mounting the wrong volume...

@therysin
Copy link

Nevermind I was mounting the wrong volume...

Can you expand further on what was wrong/ what volume you mounted to fix?

@Denow
Copy link

Denow commented Jan 18, 2023

I was using the following tutorial for deploying shiori: https://noted.lol/tutorial-setting-up-shiori.
The docker-compose file in this tutorial changes the shiori data dir with an environment variable to SHIORI_DIR=/data and creates a volume for /data. Using the same docker-compose file gave me the Failed to open database: unable to open database file: out of memory (14) error.

version: "3"
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    environment:
      - PUID=1000
      - PGID=1000
      - SHIORI_DIR=/data
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /data/shiori:/data

I removed the environment variable so shiori would write the db to the default location /shiori and also changed the volume path to the same location, now everything is working great.

@therysin
Copy link

I was using the following tutorial for deploying shiori: https://noted.lol/tutorial-setting-up-shiori. The docker-compose file in this tutorial changes the shiori data dir with an environment variable to SHIORI_DIR=/data and creates a volume for /data. Using the same docker-compose file gave me the Failed to open database: unable to open database file: out of memory (14) error.

version: "3"
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    environment:
      - PUID=1000
      - PGID=1000
      - SHIORI_DIR=/data
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /data/shiori:/data

I removed the environment variable so shiori would write the db to the default location /shiori and also changed the volume path to the same location, now everything is working great.

Thanks. Was hoping it would be something I overlooked, but I’m already using the default shiori location 🥲. I’m using the exact same compose file as the OP.

@therysin
Copy link

I ended up using a regular docker volume for persistence and not a bind mount.

Probably a temp fix, but works out in the short term.

@fmartingr
Copy link
Member

I have been trying to run and connect to an Oracle Cloud ubuntu instance for 15 minutes now. I wanted to try over there just in case, but this works on my mac/linux machines. You should be able to create a mounted volume if you have enough permissions over the folder being mounted. I'll leave this open for now, in case someone else can chip in here.

@danielnbalasoiu
Copy link

I had the same error and I fixed it by creating the local directory and then set the correct permissions expected by shiori.

mkdir ./shiori-data
chown -R 1000:1000 ./shiori-data

@milosimpson
Copy link

milosimpson commented Oct 23, 2023

Also running into this issue trying to get shiori running in docker on a Synology NAS.

If I don't specify a data volume, I can get v.1.5.5 to boot because shiori will make a sqlite db in the/shiori dir inside the docker container, which works but isn't great as there is no upgrade path as the db is "trapped" inside the container.

When I try to give it an external volume/mount, it fails to be able to make the sqllite db file because user 1000/shiori doesn't have write permissions to the mounted directory.

I am guessing you want to use a non-root user for security reasons since this service exposes an http service.

I think the fix might be to: while booting, as root chown the SHIORI_DIR to the uid 1000/shiori user, then switch to the uid 1000/shiori user.

I tried to test this by pre-populating a shiori.db file in the mount volumn/directory, to get past the boot failure so that I could shell into the container and try to run sudo chown -R 1000:1000 on the mounted dir to see if that would temporarily make things work, but the image does not contain sudo.

Shelled into the container (v1.5.5) to look at who owns /shiori/shiori.db on a setup where there is no external volume mount.
User shiori owns the file.
Screenshot 2023-10-22 at 10 04 41 PM

Shelled into the container with an external volume mount with pre-existing shiori.db file (v1.5.3 that it doesn't try to migrate db on boot). User root owns that file/directory which means it isn't usable.
Screenshot 2023-10-22 at 10 21 53 PM

@GeekyNinja2003
Copy link

GeekyNinja2003 commented Oct 23, 2023 via email

@fmartingr
Copy link
Member

I need to unify both Dockerfiles, since we use one for the CI (production builds) and one for development, and that makes no sense right now. That said, Shiori should work using a mounted volume without much issue as I have been running it like that for some time now.

Maybe allowing to customize the user and group IDs on the image via environment variables would work well, but that something I need to investigate and make backwards compatible, so it will be after we release 1.6.0, since that already contains enough changes as it is.

@fmartingr fmartingr added this to the 1.6.1 milestone Oct 29, 2023
@milosimpson
Copy link

milosimpson commented Nov 12, 2023

Figured out my problem. It is a Synology NAS specific issue. The fix was to create a specific non-root user with access to the mount directory ala https://drfrankenstein.co.uk/step-2-setting-up-a-restricted-docker-user-and-obtaining-ids/ and then use that user with Docker Compose. The Synology "Container Manager" app is basically Synology's Portainer, and it can run docker compose yaml files.

version: '3.3'
services:
  shiori:
    image: ghcr.io/go-shiori/shiori
    container_name: shiori
    user: UID:GID   // this is the important part; set the user and group that have access to  /volume2/docker/shiori
    ports:
      - 8080:8080
    restart: unless-stopped
    volumes:
      - /volume2/docker/shiori:/shiori           
      // note you have to specify the the "real" path on host not the nice Synology share name

It seemed like most suggestions on the internet were to make user and group that the docker image wants on the host OS, which is easy to do on Linux systems, but not on the Synology.

The catch is that it seems like only compose and the yaml file approach can set the user and group id, not the "docker" command line or simple docker run UIs in the Synology.

The one nice thing that Shiroi could do is take UID and PID to use as environment variables, that way it could be run w/out compose.

@danielyrovas
Copy link

I ran into this issue using podman running in rootless mode. I manged to solve it by building a container as below:

FROM ghcr.io/go-shiori/shiori:latest AS source

FROM docker.io/alpine:3.19
LABEL org.opencontainers.image.source https://github.com/go-shiori/shiori
COPY --from=source /usr/bin/shiori /usr/bin/

EXPOSE 8080
WORKDIR /shiori
ENV SHIORI_DIR /shiori/
ENTRYPOINT ["/usr/bin/shiori"]
CMD ["server"]

@fmartingr
Copy link
Member

I just tested the image on PR #907 on a linux laptop and it required no further modifications in order to mount a local path:

$ cd $(mktemp -d)
$ docker run -v ./data:/shiori -p 8080:8080 ghcr.io/go-shiori/shiori:pr-907
... server running, add bookmarks
^C
$ ls data
shiori.db
$ sqlite3 data/shiori.db "SELECT COUNT(*) FROM bookmark;
1

@fmartingr fmartingr self-assigned this May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tag:more-info More information needed type:bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

8 participants