Documentation should mention and suggest Docker installation rather than installing from sources

[perillamint]

While reading the documentation, I wonder why the documentation only suggests direct source install to the server.

In my opinion, excluding inevitable case like, rigging up development setup, it should not recommend source install, and I think the documentation should recommend Docker compose way instead.

Since mastodon/mastodon already have docker-compose file, the only thing we need is

• Document existence of official docker image, tootsuite/mastodon
• Document bring-up procedure using docker

This will

• Reduce confusion to new instance admins
• Reduce maintenance burden to new instance admins

And will help fediverse keeps its shape in good form.

Before working on documentation, I want to hear from the current maintainers.

[melroy89]

Indeed I would like to see your doc site being updated instead.

For now use the current guide:

https://gist.github.com/melroy89/6fe7d05bdc0cfd2153b77310abf62990

[nightpool]

We do not recommend admins use Docker Compose unless they have significant prior experience running docker in production. In our experience helping hundreds of admins troubleshoot and set up their server, there are just too many footguns and new areas of complexity compared to the standard systemd workflow that people are used to.

[melroy89]

We do not recommend admins use Docker Compose unless they have significant prior experience running docker in production. In our experience helping hundreds of admins troubleshoot and set up their server, there are just too many footguns and new areas of complexity compared to the standard systemd workflow that people are used to.

You said this earlier as well I noticed. I don't agree at all with this statement. Because your documentation doesn't say anything at all regarding how to setup/upgrade the Docker instance.
Of course people getting confused.

See my setup gist for now: https://gist.github.com/melroy89/6fe7d05bdc0cfd2153b77310abf62990

At least allow us to document it on your official site, rather than random gists on the GitHub!

Please, consider reopening the issue and work together with the community, we can help contributing.

I'm running a Mastodon instance for years using Docker without any issues, high-up time, auto-scaling and much more.

[Daniel15]

We do not recommend admins use Docker Compose unless they have significant prior experience running docker in production.

An inexperienced system admin is going to have a far better time using Docker compared to directly installing dependencies like Postgres. It also makes it significantly easier to move to a different server.

In any case, even if you don't recommend it, it should still be documented. Let the community maintain the documentation, with a giant "This is community supported, and not officially supported by the Mastodon developers" banner at the top.

the standard systemd workflow that people are used to.

The 'standard' workflow for systems with a moderate level of complexity is actually slowly moving towards containerisation, mostly using Docker for single-node systems and Kubernetes for large systems with multiple nodes. A lot of people are running common self-hosting stuff using https://www.linuxserver.io/, email hosting using https://mailcow.email/, etc. both of which are using Docker.

Looks like LinuxServer.io started working on a Mastodon Docker image: https://hub.docker.com/r/linuxserver/mastodon. Seems very new, and there's not much documentation about it yet.

[klogdog]

Docker Containers are WAY easier to properly maintain and configure then some esoteric ruby backend, js frontend, redis server, postgres server, potentially elastic search, etc.

Docker is the way.

Docker allows low repetition of configuration and way more automation. If you truly want decentralized you will support and move forward with docker as the standard

"docker-compose up"

[perillamint]

Adding some comment:

IMO, docker is way better and easy way to install/manage mastodon for newcommers.

Since Mastodon is not a single-binary deployable project but complex fleet of services (PostgreSQL, redis, ES/OS(optional), Puma, Sidekiq, NodeJS streaming svc...) and every service has its own dependencies (some programs refuse to run when they encounter external libpq (no pgdg for them), nodejs should stay in range to make the react happy, system must have a java version which ES/OS wants to run on it, sometimes change in ffmpeg can cause breakage, etc.)

It is literally pain in the a** to manage them and keep them healty. especially for newcomers.

The more complicating scenario? sometimes random program A wants some specific version of software B and in case of mastodon requires another version of B, there is no easy way to resolve this conflict for the sysadmins. The worst case is, sysadmin does not notice some random mastodon feature breaks which depends on program B and when they found the feature breakage, they have to bring out their sysadmin skill to diagnose / troubleshoot the problem.

I don't think this scenario is ideal, especially for hobbiest sysadmins / unexperienced amateur homelabber which are very new to managing the systems (not everyone running Mastodon does not have > 3y of sysadmin experience).

[NoaHimesaka1873]

We do not recommend admins use Docker Compose unless they have significant prior experience running docker in production. In our experience helping hundreds of admins troubleshoot and set up their server, there are just too many footguns and new areas of complexity compared to the standard systemd workflow that people are used to.

lmao. Setting things just right and scaling up later on is difficult without Docker/Podman etc. There's less complexity because user don't have to care about distro packages, etc. since Docker sets everything up right at the first place. Providing well written Docker Compose file will be much easier way than setting everything up manually bare metal. systemd workflow should be only for experienced sysadmins with enough resources to scale everything to expected demand and have no need to scale up.

[NoaHimesaka1873]

@nightpool Also, please provide what issue people faced using Docker instead of systemd.

[josejfernandez]

We do not recommend admins use Docker Compose unless they have significant prior experience running docker in production. In our experience helping hundreds of admins troubleshoot and set up their server, there are just too many footguns and new areas of complexity compared to the standard systemd workflow that people are used to.

Hello. I think the issue could be reopened as Documentation should mention Docker installation, maybe not with the "recommend" part. I understand your point with the complexity, as a Docker newbie myself, however it is not up to you to decide which way suits me better, I think. Therefore, the Docker install should be documented at least.

[Lastorder-DC]

In our experience helping hundreds of admins troubleshoot and set up their server, there are just too many footguns and new areas of complexity compared to the standard systemd workflow that people are used to.

It's complex since there are no official documention for docker installation.

[melroy89]

Could we at least have a second opinion on this issue, maybe @Gargron?

Reconsider opening the ticket again.

[Lastorder-DC]

There are similar issue - #770.
But that was already rejected 2 years ago.

Things change fast. back then, docker installation was not standard, nor easy way to install software. Now? Some software are only provide docker as install method, leaving source installation for devs who want to contribute to project.

Please, we should at lease mention about docker.

It's easy to "reset" docker installation since All I need to do is backup docker's volume - and recreate new one.
But It will be hard to "reset" source installation. What If I do something wrong? something prevent me to login into my server?

[Znuff]

@nightpool Also, please provide what issue people faced using Docker instead of systemd.

I'm currently setting this up with docker-compose, and so far it's a nightmare.

I'm experienced with docker-compose by running other projects without big issues -- and most projects are providing common pitfalls and common fixes to issues users might encounter.

First of all -- the .env.production is too specific to running stand-alone, a .env.docker with sane defaults could probably fix quite a few issues that people run into.

I've had to figure out that I actually have to specify the proper hostname from the internal-network of Docker, specifically:

• for PostgreSQL - db
• for Redis - redis
• for ElasticSearch - es

Next up, how to actually generate the secret keys. Before you go all nuts on docker compose up, you need to run this twice:

docker run --rm tootsuite/mastodon bundle exec rake secret

Once for SECRET_KEY_BASE and once for OTP_SECRET, or at least that's what I was able to figure out. Still not sure this was the correct move, but at least Ruby doesn't throw up in my face about it.

You'll also have to generate the VAPID keys, with:

docker run --rm tootsuite/mastodon bundle exec rake mastodon:webpush:generate_vapid_key

Edit your .env.production file with the correct values.

The next part I had issues was with the database provisioning.

I admit I don't have much experience with PostgreSQL myself, so I had to read a bit about the specifics of the Docker image.

My suggestion here to avoid all the possible issues that people may encounter -- to edit the docker-compose.yaml file and initiate the DB properly. The default compose file DOES NOT match the values that .env.production.sample has.

Specifically, the env file states:

# PostgreSQL
# ----------
DB_HOST=/var/run/postgresql
DB_USER=mastodon
DB_NAME=mastodon_production
DB_PASS=
DB_PORT=5432

But by default, the docker image of PostgreSQL will initiate a database called postgres with a username of postgres.

The quickest fix for me was to edit the db section of the docker-compose file like this:

version: '3'
services:
  db:
    restart: always
    image: postgres:14-alpine
    shm_size: 256mb
    networks:
      - internal_network
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'mastodon', '-d', 'mastodon_production']
    volumes:
      - ./postgres14:/var/lib/postgresql/data
    environment:
      - 'POSTGRES_HOST_AUTH_METHOD=trust'
      - 'POSTGRES_USER=mastodon'
      - 'POSTGRES_PASSWORD=ChangeMePlease'
      - 'POSTGRES_DB=mastodon_production'

Notice I changed the test line, to use the correct user and database (so you don't get FATAL messages in your logs). Also notice how I added the POSTGRES_ environment variables to create a mastodon user with a mastodon_production database.

You can also use an environment file (.env.db is suggested) instead, but I gave up and just slapped it in the compose file at this point.

Now, no, you still can not run docker-compose up -d and call it a day. While postgresql initiated the database, Mastodon itself hasn't created the tables and anything else.

For this purpose, you're supposed to run db:setup: (notice I'm running docker compose run and not just docker run)

docker compose run --rm web bundle exec rake db:setup

You will get a message about the database mastodon_production already existing. You can safely ignore it.

The thing that db:setup doesn't do is setting up an account for you, so you probably want:

docker compose run --rm web bin/tootctl accounts create YourUsername [email protected] --confirmed --role=Owner

The password will be printed to your console.

Note: apparently you CAN NOT use admin as a user here, because it's reserved -- which is strange, because mastodon:setup will happily allow you to make an admin user.

The next step would technically be to set up nginx (or whatever your choice of a web server/ssl terminator) is.

This is where I still haven't exactly figure out how it's supposed to go.

Usually when you opt to use pre-built Docker images, a docker-compose.yml and maybe an .env file is all you need.

The nginx config sample in the dist/ directory of the repository assumes you have the static files located in /home/mastodon/live/public, but in a Docker setup that directory will end up empty, so you have no static assets to serve to clients... unless you clone it.

I feel that git clone-ing the repository/version you are running from Docker kind of defeats the purpose of Docker itself.

The image itself should provide all possible files that are needed, and if the static assets aren't present, then it should

I have attempted to run:

docker compose run --rm sidekiq bundle exec rails assets:precompile

But this fails and bails out:
t

Compiling...
Compilation failed:
/opt/mastodon/node_modules/webpack-cli/bin/cli.js:93
                                throw err;
                                ^

Error: ENOENT: no such file or directory, open '/opt/mastodon/public/emoji/1f602.svg'
    at Object.openSync (node:fs:594:3)
    at readFileSync (node:fs:462:35)
    at /opt/mastodon/config/webpack/production.js:45:22
    at Array.map (<anonymous>)
    at Object.<anonymous> (/opt/mastodon/config/webpack/production.js:43:64)
    at Module._compile (/opt/mastodon/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Module.load (node:internal/modules/cjs/loader:1004:32)
    at Function.Module._load (node:internal/modules/cjs/loader:839:12)
    at Module.require (node:internal/modules/cjs/loader:1028:19)
    at require (/opt/mastodon/node_modules/v8-compile-cache/v8-compile-cache.js:159:20)
    at WEBPACK_OPTIONS (/opt/mastodon/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
    at requireConfig (/opt/mastodon/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
    at /opt/mastodon/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
    at Array.forEach (<anonymous>)
    at module.exports (/opt/mastodon/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
    at /opt/mastodon/node_modules/webpack-cli/bin/cli.js:71:45
    at Object.parse (/opt/mastodon/node_modules/webpack-cli/node_modules/yargs/yargs.js:576:18)
    at /opt/mastodon/node_modules/webpack-cli/bin/cli.js:49:8
    at Object.<anonymous> (/opt/mastodon/node_modules/webpack-cli/bin/cli.js:366:3)
    at Module._compile (node:internal/modules/cjs/loader:1126:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Module.load (node:internal/modules/cjs/loader:1004:32)
    at Function.Module._load (node:internal/modules/cjs/loader:839:12)
    at Module.require (node:internal/modules/cjs/loader:1028:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/opt/mastodon/node_modules/webpack/bin/webpack.js:156:2)
    at Module._compile (node:internal/modules/cjs/loader:1126:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1180:10)
    at Module.load (node:internal/modules/cjs/loader:1004:32) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/opt/mastodon/public/emoji/1f602.svg'
}

At this point, I'm lost.

All the webpacks that my browser tries to fetch are non-existant, all end up in 404: https://i.imgur.com/SDndVGj.png

[ineffyble]

@shleeable you might be able to help with the above

[ineffyble]

Relevant link: https://blog.plein.org/2022/11/19/work-in-progress-easy-docker-compose-for-mastodon/

[ineffyble]

See also: https://gist.github.com/TrillCyborg/84939cd4013ace9960031b803a0590c4

[perillamint]

@Znuff There is no need to precompile assets in the docker. If you are using official image, tootsuite/mastodon, it is already baked into it.

Do not use the dist/nginx.conf as-is, because the container serves the files, there is no need to serve the file through nginx separately.

Please follow the configuration guide https://gist.github.com/danger89/6fe7d05bdc0cfd2153b77310abf62990, especially the nginx one.

The only thing missing from the gist is, properly setting up the PostgreSQL tho. (use proper password authentication)

[Znuff]

Thank you very much for that. Finally got it up and working.

I have read your gist (and others) during my searches. Yours does not mention anything about the keys and the actual DB setup, which is very nonintuitive for most people.

This is another big issue with not having an official docker-compose instruction list for how to get it running.

Lots of people will actually write lots of conflicting documentation about it. I think I've read about 10 different documents on how to do it, and in neither one of them I have read that you need a different nginx config to serve the static assets from rails directly.

Documentation fragmentation is a thing.

[melroy89]

The only thing missing from the gist is, properly setting up the PostgreSQL tho. (use proper password authentication)
I have read your gist (and others) during my searches. Yours does not mention anything about the keys and the actual DB setup, which is very non-intuitive for most people.

Thanks for the feedback. I updated the gist with all the extra info you all really want and need.

This is still a good reason why we want to document this on the official site. Reopen this ticket.

[Lastorder-DC]

Looks like only SW is open, not maintainers. What a shameful behavior.

[Daniel15]

@Znuff For what it's worth, I didn't have to do a lot of things you mentioned in your comment. Running docker-compose run --rm web bundle exec rake mastodon:setup generated the contents of a valid .env.production file including all the secret keys. It asks if you're using Docker, and if so, it configures all the hostnames correctly.

[melroy89]

@Znuff For what it's worth, I didn't have to do a lot of things you mentioned in your comment. Running docker-compose run --rm web bundle exec rake mastodon:setup generated the contents of a valid .env.production file including all the secret keys. It asks if you're using Docker, and if so, it configures all the hostnames correctly.

That is indeed correct. I updated my gist once more. But keeping still the reference to how the config should look like. In case people fail to complete the setup or select the wrong values during the setup.

[magnus919]

footguns and new areas of complexity

That's less of a Docker problem

[Znuff]

@Znuff For what it's worth, I didn't have to do a lot of things you mentioned in your comment. Running docker-compose run --rm web bundle exec rake mastodon:setup generated the contents of a valid .env.production file including all the secret keys. It asks if you're using Docker, and if so, it configures all the hostnames correctly.

Well, that'd be great if that was in the official documentation.

99% of the docker stuff I run is usually with docker compose up and most will just start without issues.

Obviously once I saw that doesn't work right out of the gate, I started looking for answers elsewhere, and due to the lack of official instructions, the community has written their own way in how they understand it works.

[perillamint]

In my opinion, pouring the output of mastodon:setup into .env.production is... kind of ugly.

Ofc, it will work, but it removes useful information from base .env.production, especially the comment sections.

[arjenpdevries]

Let me second that improved "Run with Docker" instructions would be great!

Instructions how to run with docker were there when I started running an instance in 2018, and it was not that difficult. But, it's not really been documented ever - I ended up guessing a lot. E.g., I have always build my own image, but realized this week that you could also pull images for upgrades from the docker-hub, instead of creating your own; but I never saw instructions about what you have to do after pulling the container, just copy env.production into the pulled one? So, I'll have to experiment - but would have been nice if I could just read about it ;-)

I glanced over the gist, will dive into it more deeply next week @Danger89 - it looks very useful!

[otrapersona]

I also agree, docker instructions are necessary.

[magnus919]

In my opinion, pouring the output of mastodon:setup into .env.production is... kind of ugly.

It's also not the Docker way. It's like some weird old Ruby on Rails convention I think.

Configuration for Docker apps works best with externalized environment variables read on entry. The same image should be pushed to prod and non prod. If any state is to be kept at all, ideally it's externalized to the database.

Having to run a setup execution first is also... weird.

[lnlyssg]

Yup, the current set-up is extremely odd - you have to create the .env.production file first but nothing gets read or written to it by the setup so why does it need to exist? Ideally nginx should also be part of the docker-compose.

I'm not sure why there's such reluctance from the maintainers to address Docker, it's by far the easiest was to get Mastodon up and running once you get past the lack of official instructions!

[Znuff]

Ideally nginx should also be part of the docker-compose.

Please no.

[melroy89]

you have to create the .env.production file first but nothing gets read or written to it by the setup so why does it need to exist?

Thanks for your feedback. I improved my gist - Mastodon setup guide even more now.

[BirdInFire]

Fun fact how they use the excuse of "normal" workflow, when they don't seem to know that the normal workflow in 2022/2023 is kubernetes and docker.
Forcing their "native from source" installation is somthing that was seen 10 year ago.

[Lastorder-DC]

#1090

There are even opened PR for addressing docker installation(but not recommend it yet), but looks like devs are never care of documentation. What a shame...

[arjenpdevries]

#1090

There are even opened PR for addressing docker installation(but not recommend it yet), but looks like devs are never care of documentation. What a shame...

At this moment, I think @Danger89 provides best write-up: see #1035 (comment)

[pexus]

svg

path: '/opt/mastodon/public/emoji/1f602.svg'

@Znuff - were you able to fix this issue ? I see the public directory is empty and i was trying to precompile the assets but ran into this same issue. The above thread does not mention about this. Wondering if setting by the proxy server is enough and we don't need this?

[Znuff]

svg

path: '/opt/mastodon/public/emoji/1f602.svg'

@Znuff - were you able to fix this issue ? I see the public directory is empty and i was trying to precompile the assets but ran into this same issue. The above thread does not mention about this. Wondering if setting by the proxy server is enough and we don't need this?

Yes, please check the nginx.conf sample, specifically:

  # If Docker is used for deployment and Rails serves static files,
  # then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.

Just do that on all location entries and you're good.

[danielkrajnik]

While reading the documentation, I wonder why the documentation only suggests direct source install to the server.

My initial impression is that mastodon is an openwashed scheme to create another centralized service. Or am I jumping to conclusions too early?

[Sieboldianus]

I added a blog post that describes how I set up Mastodon in docker (rootless) with the official nginx.conf:

https://du.nkel.dev/blog/2023-12-12_mastodon-docker-rootless/

I had to remove the upstream server definitions because of Bad Gateway Errors, afterwards everything worked fine. Find my final configuration here directly:
https://du.nkel.dev/blog/files/mastodon.example.com.conf

Things changed:

• upstream backend and upstream streaming removed and replaced with direct forwarding to 127.0.0.1:4000 and 127.0.0.1:3000
• commented # root /home/mastodon/live/public; - all files are served with rails
• try_files $uri =404; replaced with try_files $uri @proxy;, as suggested by the ominous comment