[BUG]cnosdb restart panic

[Benxiaohai001]

Describe the bug

cnosdb restart panic
version:fc729e3242b7c98099b443c045a61d530f03cecd
deploy:singleton

To Reproduce

1. Kill process with leaked handle
2. restart the cnosdb

thread 'main' panicked at /home/runner/work/cnosdb/cnosdb/tskv/src/kvcore.rs:119:18:
called `Result::unwrap()` on an `Err` value: IndexErr { source: IndexStorage { msg: "Condition failed: `rest.is_empty()`", location: Location { file: "/home/runner/work/cnosdb/cnosdb/tskv/src/index/engine.rs", line: 35, column: 67 }, backtrace: Backtrace(   0: tskv::version_set::VersionSet::new::{{closure}}
   1: tskv::kvcore::TsKv::open::{{closure}}
   2: cnosdb::server::ServiceBuilder::create_tskv::{{closure}}
   3: cnosdb::server::ServiceBuilder::build_query_storage::{{closure}}
   4: cnosdb::main::{{closure}}
   5: cnosdb::main
   6: std::sys_common::backtrace::__rust_begin_short_backtrace
   7: std::rt::lang_start::{{closure}}
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
      std::panicking::try::do_call
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
      std::panicking::try
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal::{{closure}}
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
      std::panicking::try
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal
             at rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
   9: main
  10: <unknown>
  11: __libc_start_main
  12: _start
             at opt/zig-linux-x86_64-0.11.0/lib/libc/glibc/sysdeps/x86_64/start-2.33.S:120
) } }
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at ./rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   3: tskv::kvcore::TsKv::open::{{closure}}
   4: cnosdb::server::ServiceBuilder::create_tskv::{{closure}}
   5: cnosdb::server::ServiceBuilder::build_query_storage::{{closure}}
   6: cnosdb::main::{{closure}}
   7: cnosdb::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[root@localhost ~]#

Expected behavior

No response

Additional context

No response

[Benxiaohai001]

I tried it and found that several vnodes were broken. It is estimated that all of them are damaged. Could it be caused by restarting docker and synchronizing data with the host machine?(from @bartliu827 )