Add remote borg backup support

[timdiels]

This adds support for backing up directly to a remote borg repo. (I tried reaching out earlier)

Why?

I'd like to backup/restore via the UI, resist ransomware and store the backups remotely without storing a local (compressed) copy of the nextcloud data.

Alternatives (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#are-remote-borg-backups-supported):

• Mount a network file system: Ransomware can encrypt the backups on the network storage.
• rclone: Ransomware and stores a local copy of the borg repo
• borg backup the local repo to borgbase: Stores a local compressed copy of the nextcloud data as at least 1 borg backup
• create your own solution: backups cannot be managed through the UI and others will have to reinvent my work

How to use it

Instead of entering a local repo path (which remains supported), you can choose to instead enter a remote borg repo url. (These screenshots are from a disaster recovery but it's analogous, just ignore the passphrase):

The first you try to initialise the repo, the backup container will create an ssh key and foolishly try to init the borg repo resulting in error because you have to authorise the ssh key it generated first. So, e.g. I copy paste the public key shown here to my borgbase repo:

Then I can try again as instructed and it should work. The following backup info is shown:

Changes

• Support for storing backups directly at a remote borg repo (Replace commands to not assume the repo is local, e.g. use BORG_REPO env var, ...)
• Exclude files from backups if we don't plan on restoring them anyway, such as the audit log
• Restore with borg extract because, at least with a remote repo, borg mount is very slow (20 seconds vs 24 minutes restore for pretty much an empty nextcloud install).

FYI Oddly the original code never restores host-mounts.

Testing

I hacked the code til I could run local deployments of nextcloud-aio without any builds and manually tested:

• Entering invalid stuff into forms I changed
• Remote borg repo init
• Creating and restoring a backup. I created some files in the master container volume; some in an ignored directory to make sure those are not deleted, some in a directory that is part of the backup, after the backup made some changes and made sure that once restored, the ignored files were left alone, new (non-ignored) files were deleted and changed files restored. I can't really open the user UI with my local test setup.
• Checking a backup
• Setting up a new nextcloud from backup

TODO

• I did at one point test local backups, but I should redo that.
• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great
• Update the readme, but I first want to hear your thoughts before putting more time in it

[szaimen]

Hi, first of all thank you for your contribution!

I've thought a bit about this the last week and came to the conclusion that I do not want to maintain this since it requires an additional server for me for testing (with a not so easy to reproduce setup) and has the potential of many users needing help getting this to work (and thus only a limited usecase). So I would still only test local backup also in the future.

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

FYI Oddly the original code never restores host-mounts.

Yes, this is expected and documented.

TODO

• I did at one point test local backups, but I should redo that.

Sounds good!

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Yeah, we could create a dev instance from this if we proceed...

• Update the readme, but I first want to hear your thoughts before putting more time in it

Yes

[szaimen]

I would personally like to keep rsync at least for local archives since as you found out borg extract does not allow for deleting existing files and rsync is much faster at least for local archives... but we could split the logic and use rsync for local archives and borg extract for remote archives...

[timdiels]

Sure, I'll bring back the borg mount and rsync for local repos.

[timdiels]

Done

[timdiels]

Hi,

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Note to self:

• Update readme
• Local repo should use borg mount and rsync
• Test local and remote on dev build

[szaimen]

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

Cool, then we can go ahead with this PR. I've invited you to the repo for easier collaboration :)

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

Great :)

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Usually it is indeed always via push to github, wait for build and deploy. So probably it is easier for you to keep your debug branch 👍

[gregjohnsonsaltaire]

Very much appreciate the PR guys!
I'm happy to help test informally ... I have 3 nextcloud-aio sites of 20GB, 40GB & 350GB doing borg backups to Hetzner Storageboxes that I'm prepared to convert to the PR ... Regards

[asavageiv]

Should this be "Likely borg init will error on a partially initialized repo"?

[timdiels]

Probably most borg commands would fail but it's referring to the borg info on line 120. If we ever end up with a partially initialized repo (sounds very rare), I think manual intervention is required anyway? The important thing is I believe borg info will fail on such a corrupt repo and will end up at line 124 instead of trying to initialize it again. The config file was used as a sentinel to detect whether the repo has been initialized already (see the deleted line 107), which why it is removed here and why it's probably no longer necessary to do this.

[timdiels]

I tried out local backups on my machine and made all the requested changes I believe. Could we get a a dev build going?

[szaimen]

TODO: revert this later on

[szaimen]

I tried out local backups on my machine and made all the requested changes I believe.

Nice!

Could we get a a dev build going?

Yes, it should be available via nextcloud/all-in-one:develop-4804

[timdiels]

I'm trying it out on my own server (with a VM snapshot before switching to dev). Will I be able to later update this to :latest after it's included in a release or should I revert my snapshot after testing instead?

[timdiels]

Created a backup on my server, I'll enable autobackup/update and try out restoring after some changes later (might be next weekend):

Duration: 1 hours 2 minutes 27.77 seconds
Number of files: 34348
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                9.93 GB              8.89 GB              8.65 GB

(the duration is pretty much optimal considering my 19 Mbps ISP upload speed)

[timdiels]

I rolled back to an old snapshot because apparently the dev build was older than the latest released nextcloud, so it wouldn't start anymore because e.g. elastic can't roll back. I'll try again on the weekend with a tmp vm instead.

[msullivan-lc]

Appreciate all the work you are doing on this @timdiels. Very much looking forward to this addition!