CouchDB should provide advice on safe use with BTRFS storage #4227
Comments
|
This looks like something that could go in our documentation. Would you be willing to write a PR for this? |
|
I would expect CouchDB to make (or at least ask) that setting during the installation, but if you think that this is user's responsibility, then it's related to the documentation. I would love to create a PR for this. |
|
I read the links a bit more carefully, so I want to ask: are you sure this is a problem for btrfs? Have you seen a significant CouchDB performance change when altering the file setting? Alternatively, what does Unlike traditional SQL databases, CouchDB doesn't do random writes to its database files. It only ever appends to a file. (It is, in fact, a copy-on-write B-tree database itself, just like btrfs.) The link you gave suggests that this is only an issue for files that have lots of random writes. That never happens with CouchDB: it will only ever write to a disk block once. Also, compaction will rewrite the database and all of its views/indexes to new files when necessary. CouchDB 3.x has a fairly aggressive compaction setting, meaning databases will get compacted if they see a lot of writes anyway. In this scenario, given the entire database & index files are rewritten as part of compaction, I wonder how bad the situation really is. If this is really an issue (and I'm not saying it's not, I'd just like to be sure that it is), it is a pretty specific, narrow use case. In general, we don't detect or address this sort of thing automatically at installation or setup time. Given how long CouchDB's been around, and this is the first we're hearing of this, I'm guessing there's either not a problem (see above) or CouchDB+btrfs use in production is limited. (Most deployments I know of use ext4, xfs, or zfs.) I would add this as a callout at this line, just before the "Installation from source" section. You can use a warning box, like this one. I would not include the entire script, especially not with all of the moves/etc. We like to treat our users as capable system administrators, it should be enough to provide the basics. So I would say something like: That's only suggested text, please rewrite however you see fit. |
|
And - thank you so much for helping us build and maintain CouchDB! 🥰👨🚀🏋️♀️ |
This needs to be measured. Luckily I have a production setup on BTRFS where I forgot to change that CoW flag, so system was running without CoW flag disabled for a year. I was experiencing lots of interesting issues with database microservice, but that might not be related directly with this topic. Let's see. CouchDB 3.x on my Laptop in LXC container (Debian Buster, freshly installed), with ~20K documents synced after CouchDB 3.x on my server in LXC container (Debian Buster, freshly installed), with ~20K documents synced after CouchDB 2.x on my server in LXC container (Debian Stretch), with ~20K documents. This is my production server (where CoW is not disabled): How should we interpret those results? |
|
Hm, that's more extents, but it doesn't look terrible. And 3.x will be doing more aggressive compaction than 2.x, so we're also seeing version vs. version bias here (and 2.x is end-of-life). I'm reading that just adding Or it might be acceptable to simply run This is definitely a finer point of tuning, and while I can see that this would be useful info for CouchDB users, it's not something I'd call out as critical in the installation section. Maybe this would be better to put in the best practices under the "if it ain't broke, don't fix it" approach? I'll definitely merge a PR if it shows up, I'm just not sure we have enough data here to define this as a best practice. |
|
I'm having an interesting trouble here, as the script I provided in my first post worked just well on my laptop (CouchDB-3.x on Debian-Buster on LXC on my-laptop which uses BTRFS), however, CouchDB-2.x refuses to build views on my production server when I applied those changes. I'm on it and I'll provide the tested directions ASAP.
Well, it's hard for me to clearly state anything, but I've a feeling that this is a critical matter. I experienced a disk failure where my server got unresponsive (naturally). When I tried to recover any data I could, I realized that the corrupted files were especially belonged to CouchDB (this might be a coincidence, I'm aware). See this and this:
I'll be get in touch as soon as I solve the puzzle. |
|
Looking forward to it, let us know. I'm going to be away for a few days, so I can get back to this in January. Your post on linux-btrfs and your gist suggest your root cause is disk failure to me. CouchDB insists on fsync() after every write (unconditionally in 3.x, and in 2.x by default unless you disable a poorly documented setting). Happy Holidays! |
|
@ceremcem Any progress on this, especially with The easy thing to do would be recommending the use of |
wohali
changed the title
|
Hi @wohali. I just didn't spare time to debug any further. I got the schedule, I'll do my best. |
Summary
BTRFS filesystem does not play well with database files due to its Copy-on-Write feature. This kind of files should be created inside a folder whose CoW feature is turned off.
See https://btrfs.wiki.kernel.org/index.php/Gotchas#Fragmentation
Disabling CoW for "data" folder on a BTRFS filesystem would be a wise move.
Possible Solution
Following script fixes the issue:
The text was updated successfully, but these errors were encountered: