Borg Backup Troubleshooting

[jaume-rsl]

Hi!
I've been running Nextcloud AIO for some weeks with no issues. But, when trying to backup through Borg, it gets stucked. I have AIO set up on an old laptop repurposed as a home server with Ubuntu, with two external HDDs, one for the Nextcloud Data, the other as a Backup drive, both of them used previously with the same role on a NextcloudPi instance.

While doing the first backup I did dome changes in the drive (my intention was not to touch any of the files on NC datadir, but I'm starting to thing that maybe I did). After a loooooong time I killed and removed the borg container and I restarted everything.

What I've tried so far is to remove borg directory, remove borg container, and cleaning configuration in mastercontainer with

sudo docker exec nextcloud-aio-mastercontainer rm /mnt/docker-aio-config/data/borg.config

output is still similar:

2023-06-06T21:50:58.758755870Z initializing repository...
2023-06-06T21:50:59.222210536Z using builtin fallback logging configuration
2023-06-06T21:50:59.456055330Z 33 self tests completed in 0.23 seconds
2023-06-06T21:50:59.625338037Z Initializing repository at "/mnt/borgbackup/borg"
2023-06-06T21:51:00.096217308Z Key in "<Repository /mnt/borgbackup/borg>" created.
2023-06-06T21:51:00.096291200Z Keep this key safe. Your data will be inaccessible without it.
2023-06-06T21:51:00.316232077Z check_free_space: few segments, not requiring a full free segment
2023-06-06T21:51:00.316865387Z check_free_space: calculated working space for compact as 0 bytes
2023-06-06T21:51:00.317046703Z check_free_space: required bytes 169138, free bytes 3314785595392
2023-06-06T21:51:00.929581775Z security: previous location file /root/.config/borg/security/7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f/location not found
2023-06-06T21:51:00.929640004Z security: manifest timestamp file /root/.config/borg/security/7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f/manifest-timestamp not found
2023-06-06T21:51:00.929687787Z security: determined newest manifest timestamp as
2023-06-06T21:51:00.929753989Z security: remembering previously unknown repository
2023-06-06T21:51:00.929828949Z security: saving state for 7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f to /root/.config/borg/security/7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f
2023-06-06T21:51:00.929863822Z security: current location /mnt/borgbackup/borg
2023-06-06T21:51:00.929930692Z security: key type 5
2023-06-06T21:51:00.929973809Z security: manifest timestamp 2023-06-06T21:51:00.096262
2023-06-06T21:51:00.932990214Z security: repository checks ok, allowing access
2023-06-06T21:51:00.934391503Z Synchronizing chunks cache...
2023-06-06T21:51:00.934648575Z Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
2023-06-06T21:51:00.935305712Z Verified integrity of /mnt/borgbackup/borg/index.1
2023-06-06T21:51:00.935356305Z Done.
2023-06-06T21:51:00.935452450Z security: saving state for 7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f to /root/.config/borg/security/7bae7175e3fda459d230196bf1e18bfa7fa7a880006842f85c5d8f5fdfabbd7f
2023-06-06T21:51:00.935494046Z security: current location /mnt/borgbackup/borg
2023-06-06T21:51:00.935554079Z security: key type 5
2023-06-06T21:51:00.935629489Z security: manifest timestamp 2023-06-06T21:51:00.096262
2023-06-06T21:51:00.942132769Z
2023-06-06T21:51:00.942159612Z By default repositories initialized with this version will produce security
2023-06-06T21:51:00.942169332Z errors if written to with an older version (up to and including Borg 1.0.8).
2023-06-06T21:51:00.942177112Z
2023-06-06T21:51:00.942183860Z If you want to use these older versions, you can disable the check by running:
2023-06-06T21:51:00.942267583Z borg upgrade --disable-tam /mnt/borgbackup/borg
2023-06-06T21:51:00.942280217Z
2023-06-06T21:51:00.942288102Z See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.
2023-06-06T21:51:00.942296169Z
2023-06-06T21:51:00.942303143Z IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
2023-06-06T21:51:00.942310841Z If you used a repokey mode, the key is stored in the repo, but you should back it up separately.
2023-06-06T21:51:00.942318892Z Use "borg key export" to export the key, optionally in printable format.
2023-06-06T21:51:00.942326697Z Write down the passphrase. Store both at safe place(s).
2023-06-06T21:51:00.942334243Z
2023-06-06T21:51:02.321214184Z Repository successfully initialized.
2023-06-06T21:51:02.321243642Z Performing backup...
2023-06-06T21:51:02.321489767Z Starting the backup...
2023-06-06T21:51:02.903008918Z Creating archive at "/mnt/borgbackup/borg::20230606_215102-nextcloud-aio"

After 10 hours, no more log entries.

As I monitor the hardware with Grafana, I see the following data that might be relevant:

• CPU usage is about 25% from the start of the process, and keeps like so.
• During 1h30m, there have been many writing operations on backup drive (and reading operations on data drive)
• After that, minor writing operations on backup drive and constantly reading in data drive

Any idea how should I procede or what status this behaviour responds to? I'm totally lost.

Many thanks!

[szaimen]

Hi I think that the initial backup just actually takes so long.

We will make this better debugable with #2697 by adding the progress option to the initial backup.

[jaume-rsl]

Yep! BTRFS. Will borg backup the whole drive??? Snapshots are from the previus "backup solution", but were not meant to be included in the backup now, the same as additional folders unrelated to NextCloug (e.g. multimedia for my Plex server)

[szaimen]

Yep! BTRFS. Will borg backup the whole drive??? Snapshots are from the previus "backup solution", but were not meant to be included in the backup now, the same as additional folders unrelated to NextCloug (e.g. multimedia for my Plex server)

You would need to put the datadir into a subfolder on the drive then in order to fix this. The top-level folder of the drive is not a good place to store the datadir as you can see.

[jaume-rsl]

Poor system management on my side... Thank you for the advice!

[szaimen]

I've added a hint via 3396c83

[szaimen]

The progress on initial backup is now released with v6.2.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel