Docker image: how to config the volumes for data persistency

[LucaMaurelliC]

I'm learning the Docker technology and I would like to map a volume for data persistency. I understood the standard image of Mosquitto binds mount the config file directly and then maps also two subfolders for the log and data.
What I would like to do is to mount just a volume, e.g. "\mosquitto" into the image at "\mosquitto" and let the application creates its subfolders and files (and copy my config file later, if needed). At the moment when I run the image like suggested, instead, it creates three volumes. What can I do to use just one volume?

What I am using:
docker run -d --network mynetwork --network-alias mosquitto --name mosquitto -p 1883:1883 -v "C:\myWindowsPath\mosquitto.conf":/mosquitto/config/mosquitto.conf -v /mosquitto/data -v /mosquitto/log eclipse-mosquitto
What I tried:
docker run -d --network mynetwork --network-alias mosquitto --name mosquitto -p 1883:1883 -v mosquitto:/mosquitto

[Daedaluz]

Firstly, it looks like your configuration is at C:\myWindowsPath\mosquitto.conf, while you only mount the /mosquitto path.
This way your broker will be config-less, and probably only accept connections on localhost which is in most cases useless in a docker environment.

You could try create a folder somewhere named mosquitto and have a structure like:

mosquitto/config
mosquitto/config/mosquitto.conf
mosquitto/log
mosquitto/data

then run something like docker run -d --name mosquitto -p 1883:1883 -v %cd%:/mosquitto while standing in the mosquitto directory.

What exactly did happen when you run docker run -d --network mynetwork --network-alias mosquitto --name mosquitto -p 1883:1883 -v /mosquitto:/mosquitto?

[LucaMaurelliC]

EDIT: Similar to database images, when using the official Microsoft SQL Server image, for instance, one can use -v "sql1data:/var/opt/mssql" and its creates in Host machine a folder named sql1data where the containers creates its structure (subfolders and files).

If I do docker run -d --network mynetwork --network-alias mosquitto --name mosquitto -p 1883:1883 -v mosquitto:/mosquitto eclipse-mosquitto (note the omitted slash in the first argument of the -v command) it creates the mosquitto volume and the right structures but also other two volumes named randomly, for instance "0a7d9cd40fae0d1958c19f064dcb9b8c2ed45b53c4cd69f3210865cc75d0e006".

See figures below:

[Daedaluz]

I think this behavior comes from the VOLUME directive in Dockerfiles.
I couldn't find a flag to ignore them or avoid creating them if they are not specifically mounted, unfortunately.

• <volume_name>:/mosquitto will use <volume_name> as a source dir (created by docker) and mount it on /mosquitto in the container.
• C:\mosquitto:/mosquitto will mount the hosts path C:\mosquitto on /mosquitto in the container

The fact that the 2 random volumes are created anyway, makes me think that they are also automatically mounted on their given paths in the container.

Your data probably will end up in the random volumes too, even though you mount the parent path /mosquitto but i'm not sure.

[NorbertHeusser]

The two random volumes will be created due to the two VOLUMES, which are specified in the Mosquitto docker image (see README.md.
If you have created a volume mount for /mosquitto while starting your docker container the persistent data will end up in this folder independent of the random volume overlays created.
The more "clean" way would be to use three different mounts when starting your container. This way you may seperate configuration, data and log of your broker (e.g. use different disk spaces)
config-folder:/mosquitto/config
data-folder:/mosquitto/data
log-folder:/mosquitto/log

[Daedaluz]

I just tested this, and at least on linux, my guess were correct;

• create a mosquitto docker container
  $ docker run --rm -ti --name mosquitto -v mosquitto:/mosquitto eclipse-mosquitto
• create a test file in data folder $ docker exec -ti mosquitto touch /mosquitto/data/test
• list new random data volume
  $ sudo ls -la $(docker container inspect mosquitto | jq -r '.[0].Mounts[] | select(.Destination == "/mosquitto/data") | .Source')

  drwxr-xr-x 2 1883 1883 4096 Apr  4 22:56 .
  drwx-----x 3 root root 4096 Apr  4 22:56 ..
  -rw-r--r-- 1 root root    0 Apr  4 22:56 test
  

• list "mosquitto" volume
  $ sudo ls -la $(docker container inspect mosquitto | jq -r '.[0].Mounts[] | select(.Destination == "/mosquitto") | .Source')

  drwxr-xr-x 5 1883 1883 4096 Apr  4 22:50 .
  drwx-----x 3 root root 4096 Apr  4 22:50 ..
  drwxr-xr-x 2 1883 1883 4096 Apr  4 22:50 config
  drwxr-xr-x 2 1883 1883 4096 Nov 12 06:08 data
  drwxr-xr-x 2 1883 1883 4096 Nov 12 06:08 log
  

  It looks like that it indeed creates the folders, but then again, they are only placeholders for the other 2 random
  volume mounts
• listing the "mosquitto" volume data folder suggests it's empty:
  sudo ls -la $(docker container inspect mosquitto | jq -r '.[0].Mounts[] | select(.Destination == "/mosquitto") | .Source')/data

  drwxr-xr-x 2 1883 1883 4096 Nov 12 06:08 .
  drwxr-xr-x 5 1883 1883 4096 Apr  4 22:50 ..
  

as for a "clean" way, i usually avoid the volume directive as they tend to litter a lot for my use-cases.
eg, you are most likely to mount a reasonable configuration, that could specify where to store data and how to log.
If you configure mosquitto to log to stdout instead of file, you have no need for the log volume.
You could configure mosquitto to store data in /data instead of /mosquitto/data and just mount something there instead. (this would leave the data volume unused)

Obviously, the mosquitto container does a little bit of "fixing" before starting the mosquitto broker, like changing permissions on the database etc, and changing these paths or config parameters is probably going to cause you more issues than it solves in most cases with the standard eclipse-mosquitto container.

If you want to use the standard eclipse-mosquitto container you probably need to mount both volumes somewhere.
To get rid of this volume behavior, you could just create your own image without the volume directive.

[Daedaluz]

Feel free to close the issue if there is nothing else :)