Getting 'entered unreachable code' panic from btree_mutator after interrupting casey/ord

[hantuzun]

I interrupted @casey's ord (10f866) because it was stuck while indexing. I cannot start ord anymore due to a panic from redb-0.12.1:

$ RUST_BACKTRACE=full ord server
Listening on http:https://0.0.0.0:80
[indexing blocks] ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 494301/774770thread '<unnamed>' panicked at 'internal error: entered unreachable code', /Users/han/.cargo/registry/src/github.com-1ecc6299db9ec823/redb-0.12.1/src/tree_store/btree_mutator.rs:971:18
stack backtrace:
   0:        0x10eee7216 - std::backtrace_rs::backtrace::libunwind::trace::h310cbd77a7d2ae59
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10eee7216 - std::backtrace_rs::backtrace::trace_unsynchronized::h5768bae568840507
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10eee7216 - std::sys_common::backtrace::_print_fmt::hd104a205649a2ffb
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x10eee7216 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h521420ec33f3769d
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10ef084ba - core::fmt::write::h694a0d7c23f57ada
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/fmt/mod.rs:1208:17
   5:        0x10eee064c - std::io::Write::write_fmt::h1920a3973ad439e5
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/io/mod.rs:1682:15
   6:        0x10eee6ffa - std::sys_common::backtrace::_print::h75582c4ed1a04abb
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys_common/backtrace.rs:47:5
   7:        0x10eee6ffa - std::sys_common::backtrace::print::hef1aa4dbdc07ee06
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys_common/backtrace.rs:34:9
   8:        0x10eee8c23 - std::panicking::default_hook::{{closure}}::h529701a1070b4ce0
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:267:22
   9:        0x10eee8978 - std::panicking::default_hook::hfeeab2c667b2d7c2
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:286:9
  10:        0x10eee9353 - std::panicking::rust_panic_with_hook::h1b5245192f90251d
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:688:13
  11:        0x10eee90e3 - std::panicking::begin_panic_handler::{{closure}}::h3658f3a9566379d4
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:577:13
  12:        0x10eee76b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h9e01645d962f8882
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys_common/backtrace.rs:137:18
  13:        0x10eee8ded - rust_begin_unwind
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:575:5
  14:        0x10ef39ad3 - core::panicking::panic_fmt::h0097ad8ec0b07517
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:64:14
  15:        0x10ef39ba7 - core::panicking::panic::ha47a4f67bad0ba2d
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:111:5
  16:        0x10e7bf712 - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete_helper::h0f30e752f74cd4bf
  17:        0x10e7bbc46 - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete_helper::h0f30e752f74cd4bf
  18:        0x10e7bbc46 - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete_helper::h0f30e752f74cd4bf
  19:        0x10e7bbc46 - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete_helper::h0f30e752f74cd4bf
  20:        0x10e7bbc46 - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete_helper::h0f30e752f74cd4bf
  21:        0x10e7edfff - redb::tree_store::btree_mutator::MutateHelper<K,V>::delete::h29ae2cb93920f9e8
  22:        0x10e8ffe3f - redb::tree_store::btree::BtreeMut<K,V>::remove::hb3e22d5a358d036a
  23:        0x10e9150ce - ord::index::updater::inscription_updater::InscriptionUpdater::index_transaction_inscriptions::hb10f2eb470695894
  24:        0x10e8094db - ord::index::updater::Updater::update::hedc89b041ebaa722
  25:        0x10e8adfdc - std::sys_common::backtrace::__rust_begin_short_backtrace::h1260a421b607b62b
  26:        0x10e74e9f3 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h055c324cb991daa8
  27:        0x10eeede57 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h21c45e2812e9405d
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/alloc/src/boxed.rs:2000:9
  28:        0x10eeede57 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h88e23a22c920f493
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/alloc/src/boxed.rs:2000:9
  29:        0x10eeede57 - std::sys::unix::thread::Thread::new::thread_start::h7200ef980f8e52c3
                               at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/sys/unix/thread.rs:108:17
  30:     0x7ff80e87a259 - __pthread_start

redb is supposed to be "crash-safe by default", so I'm opening the issue here.

I'm running macOS Ventura on a Xeon processor with ECC RAM.

[cberner]

Well that's definitely a bug! :/

I'll try to reproduce this. What command line options did you pass to ord when it hung?

[hantuzun]

I did not pass any command line arguments. I have a collection of large ord index.redb files. I documented some of them in https://github.com/casey/ord/issues/1471. I can upload them to a service you point to.

[cberner]

I think this is very likely fixed by #516 . That PR fixes a very bad data corruption bug in databases greater than ~4GB is size. Can you try running with the latest master?

[hantuzun]

Thanks @cberner, I'll try! I've read https://github.com/cberner/redb/pull/516/files. It seems like this fix can work with "corrupted" databases, right?

[cberner]

Not really. There's no way to fix the corruption unfortunately, so while it might allow you to open the database (which I'm going to disallow in #518) the database will still be corrupted. You'll need to delete the ord database and rebuild it

[hantuzun]

The fix works fine for now.

[hantuzun]

Ord had indexed 300k blocks with this for now. Thanks a lot @cberner!