Skip to content
/ dariadb Public

dariadb - is a numeric time-series database storage engine.

License

Notifications You must be signed in to change notification settings

lysevi/dariadb

Repository files navigation

dariadb - numeric time-series database.

Continuous Integration

version build & tests test coverage
master Build Status codecov
develop Build Status codecov

Features

  • True columnar storage
  • Can be used as a server application or an embedded library.
  • Full featured http api.
  • Golang client (see folders "go" and "examples/go")
  • Accept unordered data.
  • Each measurement contains:
    • Id - x32 unsigned integer value.
    • Time - x64 timestamp.
    • Value - x64 float.
    • Flag - x32 unsigned integer.
  • Write strategies:
    • wal - little cache and all values storing to disk in write ahead log. optimised for big write load(but slower than 'memory' strategy).
    • compressed - all values compressed for good disk usage without writing to sorted layer.
    • memory - all values stored in memory and dropped to disk when memory limit is ended.
    • cache - all values stored in memory with writes to disk.
    • memory-only - all valeus stored only in memory.
  • LSM-like storage struct with three layers:
    • Memory cache or Append-only files layer, for fast write speed and crash-safety(if strategy is 'wal').
    • Old values stored in compressed block for better disk space usage.
  • High write speed:
    • as embedded engine - to disk - 1.5 - 3.5 millions values per second to disk
    • as memory storage(when strategy is 'memory') - 7-9 millions.
    • across the network - 700k - 800k values per second
  • Shard-engine: you can split values per shard in disk, for better compaction and read speed up.
  • Crash recovery.
  • CRC32 for all values.
  • Two variants of API:
    • Functor API (async) - engine apply given function to each measurement in the incoming request.
    • Standard API - You can Query interval as list or values in time point as dictionary.
  • Compaction old data with filtration support;
  • Statistic:
    • time min/max
    • value min/max
    • measurement count
    • values sum
  • Statistical functions:
    • minimum
    • maximum
    • count
    • average
    • median
    • sigma(standard deviation)
    • percentile90
    • percentile99
  • Interval aggregation support. Available intervals: raw,minute, half hour, hour, day, week, month, year.

Usage example

  • See folder "examples"
  • How to use dariadb as a embedded storage engine: dariadb-example

Dependencies

  • Boost 1.54.0 or higher: system, filesystem, date_time,regex, program_options, asio.
  • cmake 3.1 or higher
  • c++ 14/17 compiler (MSVC 2015, gcc 6.0, clang 3.8)

Build


Install dependencies

$ sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
$ sudo apt-get update
$ sudo apt-get install -y libboost-dev  libboost-filesystem-dev libboost-program-options-dev libasio-dev libboost-date-time-dev cmake  g++-6  gcc-6 cpp-6 
$ export CC="gcc-6"
$ export CXX="g++-6"

Jemalloc

Optionaly you can install jemalloc for better memory usage.

$ sudo apt-get install libjemalloc-dev

Or you may use builtin jemalloc source in dariadb - just add build option -DSYSTEM_JEMALLOC=OFF

Git submodules

$ cd dariadb
$ git submodules init 
$ git submodules update

Available build options

  • DARIADB_ENABLE_TESTS - Enable testing of the dariadb. - ON
  • DARIADB_ENABLE_INTEGRATION_TESTS - Enable integration test. - ON
  • DARIADB_ENABLE_SERVER - Enable build dariadb server. - ON
  • DARIADB_ENABLE_BENCHMARKS - Enable build dariadb benchmarks. - ON
  • DARIADB_ENABLE_SAMPLES - Build dariadb sample programs. - ON
  • DARIADB_ASAN_UBSAN - Enable address & undefined behavior sanitizer for binary. - OFF
  • DARIADB_MSAN - Enable memory sanitizer for binary. - OFF
  • DARIADB_SYSTEM_JEMALLOC - Use jemalloc installed in the system. - ON

Configure to build with all benchmarks, but without tests and server.


$ cmake  -DCMAKE_BUILD_TYPE=Release -DDARIADB_ENABLE_TESTS=OFF -DDARIADB_ENABLE_INTEGRATION_TESTS=OFF -DDARIADB_ENABLE_BENCHMARKS=ON -DDARIADB_ENABLE_SERVER=OFF . 

clang


Clang currently does not supported.

$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="${CMAKE_CXX_FLAGS_RELEASE} -stdlib=libc++" -DCMAKE_EXE_LINKER_FLAGS="${CMAKE_EXE_LINKER_FLAGS} -lstdc++" .
$ make

gcc


$ cmake -DCMAKE_BUILD_TYPE=Release .
$ make

Microsoft Visual Studio


$ cmake -G "Visual Studio 14 2015 Win64" .
$ cmake --build .

if you want to build benchmarks and tests

$ cmake -G "Visual Studio 14 2015 Win64" -DBUILD_SHARED_LIBS=FALSE  .
$ cmake --build .

build with non system installed boost


$ cmake  -DCMAKE_BUILD_TYPE=Release -DBOOST_ROOT="path/to/boost/" .
$ make

About

dariadb - is a numeric time-series database storage engine.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages