restic prune fatal not found in the new index - "data blobs seem to be missing" #3123
Comments
This is what the In your example, did check print out no warning? If yes, can you retry the |
|
There are also beta build available at https://beta.restic.net/?sort=time&order=desc . The problem is probably caused by the two pack files which failed listing. Please check whether the files in |
|
I will get the latest beta build and retry both files have the right
I meant a command to backup everything again and not trust previous data, not just check the data. Ideally that would be run for critical sections that I want to make double sure is included in the data backup. |
|
Hello, I've retried I would like to backup again but making sure that both packs are re-written again (e.g. akin to deleting those two files and backing up again). Is this doable? |
You can manually delete or move them, then run However, your situation is a bit weird. If I would be interested in the output of |
|
I had an identical error message to OP on TL;DR - using the latest beta build and running prune, then check again seemed Original error message using 0.11.0: Immediately afterwards, using the restic beta version (dry run prune, then That seemed to fix the issue so I think I'll continue using this beta version |
|
Well...I ran a Apparently there are still issues :/ Some other reading indicates perhaps an issue with data corruption and B2 backend? How to recover? Feel free to split my comments to a new issue if it's not related to the OP. Thanks! |
Sorry @aawsome due to timing constraints I had to delete the corrupted files, re-index and re-run the whole backup process, so I can't provide that Will look out for it in the future. |
|
@firecat53 Thanks for your pruning report. In fact, I wanted to write yesterday that you should run To recover, you can try
@ldhieu No worries. Hope your repo keeps staying in good shape from now 👍 |
|
I really wonder what causes these damaged pack files on B2. B2 is right now the only backend for which restic sends along a SHA1 hash of the pack files during upload (which should rule out file corruption during upload). So either we are seeing lots of failed downloads without any further error indications or something breaks before / during upload. @firecat53 Can you take a look at the damaged (?) pack files in your B2 bucket? |
|
Another remark: as |
|
After my initial post (after the initial
To answer the specific questions you asked after I ran For file
For file
I hope I answered all of your questions...where to go from here? Thanks! Edit: I did notice that all of the |
That's normal and as it should be. The problem here is, that your file seems to be all correct, but for some reason restic Sorry, but I did not think that this could be possible and therefore my advice was totally misleading. Is there a way that allows you to restore the files you deletet from B2 in the
step? Might be that they were if fact all fine.. Before you proceed with steps to repair your repo (or do anything to modify this repo), you should find the reason for this SHA256 mismatch. Honestly, most likely this is due to a hardware error on your PC. Can you
|
No, there's no versioning enabled.
Lenovo TS140 Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz, 8GB ECC RAM. Archlinux on BTRFS RAID 1.
I was able to do this on one of the files multiple times on both server and
Tried I've been trying to avoid running check --read-data-subset multiple times just cause
Ran Lenovo memory checkers (about 3 hours) and memtest86+ (for about 20 min) I had a suspicion that these errors were all related to the large directory After this, I was able to run a I'm afraid my fix may have prevented further troubleshooting to narrow down the issue and for that I'm sorry!! TL;DR - no obvious hardware issues found. Suspect something happened during a |
|
@firecat53 The backup format doesn't really have the concept of deleted files. I'd rather expect that adding lots of new data could lead to some damaged files (simply because there are more files which makes it more likely that something goes wrong). Which linux kernel version do you use? We recently had a issue with pack files which randomly failed the integrity check, which seems to have been caused by a buggy kernel version: https://forum.restic.net/t/troubleshooting-unreproducible-consistency-errors/3372/16 [Edit]As you're using archlinux you're kernel version is probably already much newer than 5.5?[/Edit] |
|
I'm running restic 0.11.0 on Windows 10 20H2 with a REST backend and got a similar error message when trying to prune after removing a number of old snapshots on my 12TiB backup: I'm running check now and it also reports problems with missing data blobs. |
|
@JPPhoto Please open a separate issue and add a (redacted) snippet of the check output there. I'm closing this issue as the two originally reported repository corruptions have been resolved and there's nothing left to investigate. |
Output of
restic versionrestic 0.11.0 compiled with go1.15.3 on windows/amd64How did you run restic exactly?
What backend/server/service did you use to store the repository?
I'm using a samba server, mounted on Windows, both sides are connected via OpenVPN.
The server is running alpine linux
5.4.72-0-virt x86_64andsamba-4.12.9-r0Expected behavior
Prune should run without error.
Actual behavior
Steps to reproduce the behavior
restic -r $remote-place backup --verbose=2 --tag $tag $local-placerestic -r $remote-place check --read-data --verbose=2restic -r $remote-place prune --verbose=2Do you have any idea what may have caused this?
I believe this was due to backing up the directory via SFTP first, which had some errors (it seemed like it was related to
ServerAliveInterval, however I tried different things after searching online, but it did not fix it, SFTP always failed after a while).This means I'm pretty sure the server had some bad files (in the middle of writing but got cut off).
So I had to switch to samba + mounting it via OpenVPN instead. I re-ran the backup from start, with the same tag. Then ran it again with
backup --force --no-cache.After that, I ran
check, thencheck --read-data, thenpruneand got the error above.Do you have an idea how to solve the issue?
I'm not sure. I'd like an option to do a backup that goes through all files again to make sure no data is corrupted. Is there any way to do this?
Did restic help you today? Did it make you happy in any way?
I like
restic. I tried backing up with:rdiff-backup,borg, andduplicity; restic was not only the fastest one but the one that did not choke on backing up in my particular case (except for that sftp error which I still have not figured out why).The text was updated successfully, but these errors were encountered: