…ytes

Summary:
Cockroachdb exposed this bug in #1778. The bug happens when a compaction's output files are ended due to exceeding max_compaction_bytes. In that case we weren't taking into account the next file's start key when deciding how far to extend the current file's max_key. This caused the non-overlapping key-range invariant to be violated.

Note this was correctly handled for the usual case of cutting compaction output, which is file size exceeding max_output_file_size. I am not sure why these are two separate code paths, but we can consider refactoring it to prevent such errors in the future.
Closes #1784

Differential Revision: D4430235

Pulled By: ajkr

fbshipit-source-id: 80af748

Summary:
Remove the logic since we don't use buffer cache with direct IO. Resolve
read regression we currently have.
Closes #1782

Differential Revision: D4430408

Pulled By: yiwu-arbug

fbshipit-source-id: 5557bba

Summary:
Consider the following single column family scenario:
prepare in log A
commit in log B
*WAL is too large, flush all CFs to releast log A*
*CFA is on log B so we do not see CFA is depending on log A so no flush is requested*

To fix this we must also consider the log containing the prepare section when determining what log a CF is dependent on.
Closes #1768

Differential Revision: D4403265

Pulled By: reidHoruff

fbshipit-source-id: ce800ff

Summary:
Fix the bug when sync log fail, FlushJob::Run() will not be execute and
reference to cfd->current() will not be release.
Closes #1792

Differential Revision: D4441316

Pulled By: yiwu-arbug

fbshipit-source-id: 5523e28

Summary:
If users directly call OptimizeForPointLookup(), it is broken as the option isn't compatible with parallel memtable insert. Fix it by using memtable bloomo filter instead.
Closes #1791

Differential Revision: D4442836

Pulled By: siying

fbshipit-source-id: bf6c9cd

Summary:
If concurrent memtable insert is enabled, and one prepare command and a normal command are grouped into a commit group, the sequence ID will be calculated incorrectly.
Closes #1730

Differential Revision: D4371081

Pulled By: siying

fbshipit-source-id: cd40c6d

Summary:
when writing RangeDelAggregator::AddToBuilder, I forgot that there are sentinel tombstones in the middle of the interval map since gaps between real tombstones are represented with sentinels.

blame: #1614
Closes #1804

Differential Revision: D4460426

Pulled By: ajkr

fbshipit-source-id: 69444b5

…sion

Summary:
GetAndRefSuperVersion() should not be called again in the same thread before ReturnAndCleanupSuperVersion() is called.

If we have a compaction filter that is using DB::Get, This will happen
```
CompactFiles() {
  GetAndRefSuperVersion() // -- first call
    ..
    CompactionFilter() {
      GetAndRefSuperVersion() // -- second call
      ReturnAndCleanupSuperVersion()
    }
    ..
  ReturnAndCleanupSuperVersion()
}
```

We solve this issue in the same way Iterator is solving it, but using GetReferencedSuperVersion()

This was discovered in facebook/mysql-5.6#427 by alxyang
Closes #1803

Differential Revision: D4460155

Pulled By: IslamAbdelRahman

fbshipit-source-id: 5e54322

This patches adds support for using the SPDK BlobFS as the
storage engine for a RocksDB database.

The Makefile changes assume that SPDK and DPDK have been
built according to the SPDK README (https://github/spdk/spdk)
and share the same parent directory as the RocksDB repository.

Flags to configure the SPDK storage engine:

spdk -
  Defines the name of the SPDK configuration file.  If
  omitted, will use PosixEnv instead of SpdkEnv.  This
  configuration file defines the block devices that may
  contain the BlobFS for the database.

spdk_bdev -
  Defines the name of the SPDK block device which
  contains the BlobFS for this database.

spdk_cache_size -
  Defines the amount of userspace cache memory used by
  BlobFS.  Specified in terms of megabytes (MB).  Default
  is 4096 (4GB).

Signed-off-by: Jim Harris <[email protected]>
Change-Id: I01d42d43fc1112571eb262799b669cfba572d654

Change-Id: I9d2c16716f6428e8a0a4ec34f75c0314bbcd630d
Signed-off-by: Daniel Verkamp <[email protected]>

Rather than using bdev->name or the spdk_bdev_get_name() accessor
function, maintain compatibility with SPDK 17.03 as well as newer SPDK
that will remove the definition of struct spdk_bdev from the public API.

Change-Id: I91f3cc0717ccc391d3db7e8e84d2d274522e0c92
Signed-off-by: Daniel Verkamp <[email protected]>

Change-Id: I73293bbad220dbc3f9fa809b860630c2484033dd
Signed-off-by: Ben Walker <[email protected]>
Reviewed-on: https://review.gerrithub.io/362086
Reviewed-by: Daniel Verkamp <[email protected]>

Change-Id: I6671bf829a008a1f1269a0d5234fab2a53173b1d
Signed-off-by: Ben Walker <[email protected]>
Reviewed-on: https://review.gerrithub.io/362263
Reviewed-by: Daniel Verkamp <[email protected]>
Reviewed-by: Jim Harris <[email protected]>
Tested-by: Jim Harris <[email protected]>

This solves the chicken and egg problem with updating
SPDK APIs.

Change-Id: I3865d90ab41e5cac9186e193b5cd21f112ac98d9
Signed-off-by: Ben Walker <[email protected]>
Reviewed-on: https://review.gerrithub.io/364532
Reviewed-by: Ziye Yang <[email protected]>
Reviewed-by: Jim Harris <[email protected]>

This allows SPDK patches to easily update the
SPDK-specific parts of the RocksDB build.

Change-Id: I9c0381c84414229c5bca50b513a198cda90a166b
Signed-off-by: Ben Walker <[email protected]>
Reviewed-on: https://review.gerrithub.io/365064
Reviewed-by: Jim Harris <[email protected]>