opendb module provides observability and configurability for rocksdb databases.

When you open RocksDB with provided API:

• opendb registers prometheus metrics
• opendb launches a reportMetrics goroutine which every rocksdb.report-metrics-interval-secs seconds reports metrics to prometheus, reportMetrics goroutine performs following steps:
  • uses grocksdb API to gather rocksdb properties/stats in text format
    • more info can be found here: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-statistics
  • parses rocksdb properties/stats from text and reports them as prometheus metrics

List of reported metrics and their documentation can be found in:

• source code: registerMetrics() function in metrics.go
• corresponding grafana dashboard

opendb allows you to override default rocksdb settings, more info can be found here:

• https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning
• https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

There are 2 main scenarios:

• create a new rocksdb database
• open already existing database

• opendb starts with default cometbft-db rocksdb configuration, see for the details: https://github.com/Kava-Labs/cometbft-db/blob/main/rocksdb.go
• opendb overrides options which explicitly specified in appOpts (app.toml)

• opendb loads stored rocksdb configuration and starts with it
• opendb overrides options which explicitly specified in appOpts (app.toml)

app.toml example:

[rocksdb]
enable-metrics = true
report-metrics-interval-secs = 15
max-open-files = 16384
...

[rocksdb.blockstore]
max-open-files = -1
block_size = 4096

[rocksdb.tx_index]
max-open-files = -1
block_size = 4096

[rocksdb] section contains default/fallback database configuration, but you can override it with database-specific configuration

it's very convenient approach, because rocksdb configuration is pretty extensive, contains around 30 parameters, but usually we override only few of them

so we can define standard configuration in [rocksdb] section, and then override only few params in database-specific configurations

#####################
# RocksDB Tuning
#####################
rocksdb_enable_metrics: true
rocksdb_report_metrics_interval_secs: 15

rocksdb_max_open_files: 16384 # increase default max # of open files from 4096
rocksdb_max_file_opening_threads: 16
rocksdb_table_cache_numshardbits: 6
rocksdb_allow_mmap_writes: false
rocksdb_allow_mmap_reads: false
rocksdb_use_fsync: false
rocksdb_use_adaptive_mutex: false
rocksdb_bytes_per_sync: 0
rocksdb_max_background_jobs: 16

rocksdb_write_buffer_size: 134217728
rocksdb_num_levels: 7
rocksdb_max_write_buffer_number: 6
rocksdb_min_write_buffer_number_to_merge: 2
rocksdb_max_bytes_for_level_base: 536870912
rocksdb_max_bytes_for_level_multiplier: 10.000000
rocksdb_target_file_size_base: 67108864
rocksdb_target_file_size_multiplier: 1
rocksdb_level0_file_num_compaction_trigger: 2
rocksdb_level0_slowdown_writes_trigger: 20

rocksdb_block_cache_size: 1073741824
rocksdb_bits_per_key: 10
rocksdb_block_size: 16384 # 16K to match default zfs. Decreases block index memory usage by 4x from the default 4K
rocksdb_cache_index_and_filter_blocks: false
rocksdb_pin_l0_filter_and_index_blocks_in_cache: false
rocksdb_format_version: 5

# https://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html
rocksdb_read_async_io: true # help speed up iterations