This GitHub repository provides a simple script that can pull missing posts into Mastodon using the Mastodon API. FediFetcher has no further dependencies, and can be run as either a GitHub Action, as a scheduled cron job, or a pre-packaged container. Here is what FediFetcher can do:

1. It can pull missing remote replies to posts that are already on your server into your server. Specifically, it can
   1. fetch missing replies to posts that users on your instance have already replied to,
   2. fetch missing replies to the most recent posts in your home timeline,
   3. fetch missing replies to your bookmarks.
   4. fetch missing replies to your favourites.
2. It can also backfill profiles on your instance. In particular it can
   1. fetch missing posts from users that have recently appeared in your notifications,
   2. fetch missing posts from users that you have recently followed,
   3. fetch missing posts form users that have recently followed you,
   4. fetch missing posts form users that have recently sent you a follow request.

Each part of this script is fully configurable, and you can completely disable parts that you are not interested in.

FediFetcher will store posts and profiles it has already pulled in on disk, to prevent re-fetching the same info in subsequent executions.

Be aware, that this script may run for a very long time. This is particularly true, the first time this script runs, and/or if you enable all parts of this script. You should ensure that you take steps to prevent multiple overlapping executions of this script, as that will lead to unpleasant results. There are detailed instructions for this below.

For detailed information on the how and why, please read the FediFetcher for Mastodon page.

FediFetcher makes use of the Mastodon API. It'll run against any instance implementing this API, and whilst it was built for Mastodon, it's been confirmed working against Pleroma as well.

FediFetcher will pull in posts and profiles from any server that implements the Mastodon API, including Mastodon, Pleroma, Akkoma, Pixelfed, and probably others.

You can run FediFetcher either as a GitHub Action, as a scheduled cron job on your local machine/server, or from a pre-packed container.

Regardless of how you want to run FediFetcher, you must first get an access token:

1. In Mastodon go to Preferences > Development > New Application
   1. Give it a nice name
   2. Enable the required scopes for your options. You could tick read and admin:read:accounts, or see below for a list of which scopes are required for which options.
   3. Save
   4. Copy the value of Your access token

If you are not a server admin, you do not have access to Preferences > Development. You can use GetAuth for Mastodon to generate an Access Token instead.

To run FediFetcher as a GitHub Action:

1. Fork this repository
2. Add your access token:
   1. Go to Settings > Secrets and Variables > Actions
   2. Click New Repository Secret
   3. Supply the Name ACCESS_TOKEN and provide the Token generated above as Secret
3. Provide the required environment variables, to configure your Action:
   1. Go to Settings > Environments
   2. Click New Environment
   3. Provide the name Mastodon
   4. Add environment variables to configure your action as described below.
4. Finally go to the Actions tab and enable the action. The action should now automatically run approximately once every 10 min.

Keep in mind that the schedule event can be delayed during periods of high loads of GitHub Actions workflow runs.

If you want to, you can of course also run FediFetcher locally as a cron job:

1. To get started, clone this repository.
2. Install requirements: pip install -r requirements.txt
3. Then simply run this script like so: python find_posts.py --access-token=<TOKEN> --server=<SERVER> etc. (Read below, or run python find_posts.py -h to get a list of all options.)

An example script can be found in the examples folder.

When using a cronjob, we are using file based locking to avoid multiple overlapping executions of the script. The timeout period for the lock can be configured using --lock-hours.

If you are running FediFetcher locally, my recommendation is to run it manually once, before turning on the cron job: The first run will be significantly slower than subsequent runs, and that will help you prevent overlapping during that first run.

Note: if you wish to run FediFetcher using Windows Task Scheduler, you can rename the script to the .pyw extension instead of .py, and it will run silently, without opening a console window.

FediFetcher is also available in a pre-packaged container, FediFetcher - Thank you @nikdoof.

1. Pull the container from ghcr.io, using Docker or your container tool of choice: docker pull ghcr.io/nanos/fedifetcher:latest
2. Run the container, passing the command line arguments like running the script directly: docker run -it ghcr.io/nanos/fedifetcher:latest --access-token=<TOKEN> --server=<SERVER>

The same rules for running this as a cron job apply to running the container: don't overlap any executions.

Persistent files are stored in /app/artifacts within the container, so you may want to map this to a local folder on your system.

An example Kubernetes CronJob for running the container is included in the examples folder.

FediFetcher has quite a few configuration options, so here is my quick configuration advice, that should probably work for most people (use the Environment Variable Name if you are running FediFetcher has a GitHub Action, otherwise use the Command line flag):

If you configure FediFetcher this way, it'll fetch missing remote replies to the last 200 posts in your home timeline. It'll additionally backfill profiles of the last 80 people you followed, and of every account who appeared in your notifications during the past hour.

Please find the list of all configuration options, including descriptions, below:

If you wish to run FediFetcher for multiple users on your instance, you can supply the --access-token argument multiple times, with different access tokens for different users. That will allow you to fetch replies and/or backfill profiles for multiple users on your account. Have a look at the sample script provided.

This is only supported when running FediFetcher as cron job, or container. Multi-user support is not available when running FediFetcher as GitHub Action.

• For all actions, your access token must include these scopes:
  • read:search
  • read:statuses
  • read:accounts
• If you are supplying REPLY_INTERVAL_IN_HOURS / --reply-interval-in-hours you must additionally enable this scope:
  • admin:read:accounts
• If you are supplying MAX_FOLLOW_REQUESTS / --max-follow-requests you must additionally enable this scope:
  • read:follows
• If you are supplying MAX_BOOKMARKS / --max-bookmarks you must additionally enable this scope:
  • read:bookmarks
• If you are supplying MAX_FAVOURITES / --max-favourites you must additionally enable this scope:
  • read:favourites
• If you are supplying FROM_NOTIFICATIONS / --from-notifications you must additionally enable this scope:
  • read:notifications

The original inspiration of this script, as well as parts of its implementation are taken from Abhinav Sarkar. Thank you Abhinav!