New Feature: Database Partitions

[davisp]

Overview

This PR introduces a new feature, user-defined partitioned databases.

A new kind of database can be created with the ?partitioned=true option. All documents within the database must have document ids of the following format;

partition_name:doc_id

both partition_name and doc_id must follow the couchdb id format (can't begin with _, etc).

All documents with the same partition_name are guaranteed to be mapped to the same shard range. When querying an index, the new /db/_partition/$partition/_view endpoint can query the view more efficiently, by only consulting the single shard range holding the partition. This is much more efficient and scales the same way that primary key lookup (GET /dbname/docid) does (approximately linearly).

Testing recommendations

The PR contains multiple tests for basic functionality and all existing tests still pass. When testing the PR, it is important to try the feature yourself, interactively, with docs and views of your choosing, to give us confidence in this new feature.

Related Issues or Pull Requests

This supersedes PR #1605.

Checklist

• Code is written and works correctly;
• Changes are covered by tests;
• Documentation reflects the changes;

[iilyak]

I didn't finish my review yet. But this is looking great so far.
There is one aspect of the codebase which I don't like. However it is very easy to fix.
There are 3 places where we leak implementation detail that the partition separator is ":".
Would you mind adding following two functions into couch_partition.erl:

-compile({inline, [
    start_key/1,
    end_key/1
]}).
start_key(Partition) -> 
    <<Partition/binary, ":">>.

end_key(Partition) -> 
    <<Partition/binary, ";">>.

Then we can use these functions:

• in couch_bt_engine:get_partition_info/2
• in couch_mrview_util:apply_all_docs_partition/2 instead of PARTITION_START and PARTITION_END macro.

The third place I mentioned is in couch_mrview_updater:partition/1. I already have a comment about it.

[iilyak]

Some tests need styling updates:

** (Mix) mix format failed due to --check-formatted.
The following files were not formatted:

  * test/partition_view_test.exs
  * test/partition_mango_test.exs
  * test/partition_crud_test.exs
  * test/all_docs_test.exs

[iilyak]

After updating Makefile to skip elixir-check-formatted I am getting multiple test failures in Elixir test suite. Some of them might not be related:

ReplicationTest
  * test unauthorized replication cancellation (1038.8ms)

  1) test unauthorized replication cancellation (ReplicationTest)
     test/replication_test.exs:156
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: anonymous fn/1 in ReplicationTest.set_user_context/1
       (ex_unit) lib/ex_unit/on_exit_handler.ex:140: ExUnit.OnExitHandler.exec_callback/1
       (ex_unit) lib/ex_unit/on_exit_handler.ex:126: ExUnit.OnExitHandler.on_exit_runner_loop/0

  * test non-admin or reader user on source - remote-to-remote (90.7ms)

  2) test non-admin or reader user on source - remote-to-remote (ReplicationTest)
     test/replication_test.exs:248
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin user on target - remote-to-local (51.4ms)

  3) test non-admin user on target - remote-to-local (ReplicationTest)
     test/replication_test.exs:243
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin user on target - local-to-local (64.0ms)

  4) test non-admin user on target - local-to-local (ReplicationTest)
     test/replication_test.exs:243
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin user on target - local-to-remote (36.3ms)

  5) test non-admin user on target - local-to-remote (ReplicationTest)
     test/replication_test.exs:243
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin or reader user on source - local-to-remote (71.0ms)

  6) test non-admin or reader user on source - local-to-remote (ReplicationTest)
     test/replication_test.exs:248
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin or reader user on source - local-to-local (49.4ms)

  7) test non-admin or reader user on source - local-to-local (ReplicationTest)
     test/replication_test.exs:248
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin or reader user on source - remote-to-local (192.6ms)

  8) test non-admin or reader user on source - remote-to-local (ReplicationTest)
     test/replication_test.exs:248
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

  * test non-admin user on target - remote-to-remote (54.9ms)

  9) test non-admin user on target - remote-to-remote (ReplicationTest)
     test/replication_test.exs:243
     Expected truthy, got false
     code: assert HTTPotion.Response.success?(resp)
     stacktrace:
       test/replication_test.exs:2: ReplicationTest.create_user/1
       test/replication_test.exs:2: ReplicationTest.set_user_context/1
       (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
       test/replication_test.exs:2: ReplicationTest.__ex_unit_setup_0/1
       test/replication_test.exs:1: ReplicationTest.__ex_unit__/2

PartitionCrudTest
  ...
  * test saves attachment with partitioned doc (171.8ms)

 10) test saves attachment with partitioned doc (PartitionCrudTest)
     test/partition_crud_test.exs:246
     Assertion with == failed
     code:  assert body["_attachments"] == %{"foo.txt" => %{"content_type" => "text/plain", "digest" => "md5-OW2BoZAtMqs1E+fAnLpNBw==", "length" => 31, "revpos" => 1, "stub" => true}}
     left:  %{"foo.txt" => %{"content_type" => "text/plain", "length" => 31, "revpos" => 1, "stub" => true, "digest" => "md5-wJORG+sY7HTKqCWOnk5tIQ=="}}
     right: %{"foo.txt" => %{"content_type" => "text/plain", "length" => 31, "revpos" => 1, "stub" => true, "digest" => "md5-OW2BoZAtMqs1E+fAnLpNBw=="}}
     stacktrace:
       test/partition_crud_test.exs:269: (test)

BasicsTest
  ...
  * test PUT error when body not an object (147.4ms)

 11) test PUT error when body not an object (BasicsTest)
     test/basics_test.exs:243
     Assertion with == failed
     code:  assert resp.status_code() == 400
     left:  500
     right: 400
     stacktrace:
       test/basics_test.exs:246: (test)

[davisp]

@iilyak Those inline calls I believe only apply to the module, I don't think that'll affect anywhere else they're used as external functions? However we could at least hoist the macros into a couch_partitions.hrl for re-use.

For the mix format, those should be fixed after one of my recent-ish force pushes. I rebased for something else and forgot to check that after we added Credo.

I'm looking into the other test failures to see what's going on with those.

[jaydoane]

After make clean && git clean -xffd && ./configure --dev && make, I'm seeing numerous setup failures and a couple test failures running make eunit apps=chttpd:

=======================================================
  Failed: 2.  Skipped: 0.  Passed: 184.

compared to a successful result when run against the master commit this is based against:

=======================================================
  All 233 tests passed.

The two failures are

chttpd_view:123: t_check_include_docs_throw_validation_error...*failed*
in function chttpd_view:'-t_check_include_docs_throw_validation_error/0-fun-0-'/0 (src/chttpd_view.erl, line 127)
**error:{assertException,
    [{module,chttpd_view},
     {line,127},
     {expression,
         "multi_query_view ( Req , db , ddoc , << \"v\" >> , [ Query ] )"},
     {pattern,"{ throw , Throw , [...] }"},
     {unexpected_exception,
         {error,function_clause,
             [{couch_db,is_clustered,[db],[{file,...},{...}]},
              {fabric_util,validate_args,3,[{...}|...]},
              {lists,map,2,[...]},
              {chttpd_view,multi_query_view,5,...},
              {chttpd_view,
                  '-t_check_include_docs_throw_validation_error/0-fun-0-',...},
              {eunit_test,...},
              {...}|...]}}]}
  output:<<"">>

chttpd_view:132: t_check_user_can_override_individual_query_type...*failed*
in function couch_db:is_clustered/1 (src/couch_db.erl, line 217)
  called as is_clustered(db)

There's also a failure in mem3:

module 'mem3_util_test'
  mem3_util_test: hash_test...*failed*
in function mem3_util:hash/1
  called as hash(0)
in call from mem3_util_test:'-hash_test/0-fun-0-'/0 (test/mem3_util_test.erl, line 19)
in call from mem3_util_test:hash_test/0 (test/mem3_util_test.erl, line 19)
in call from mem3_util_test:hash_test/0
**error:undef
  output:<<"">>

since mem3_util:hash has been removed.

[davisp]

@jaydoane chttpd tests should be fixed now. Am running through checking that everything else works as well.

Fix for the ones you noted was to use test_util:fake_db/1 instead of the atom db.

[davisp]

@jaydoane @iilyak All tests both eunit and elixir should be passing now. Apologies for the delay!

[AlexanderKaraberov]

Hello guys!
I'm trying to sync with this new cool feature as it totally passed by me. Please correct me if I'm wrong. As I understand this adds something akin to list partitioning in relational databases to CouchDB (on top of current composite hash-range partitioning) where each partition is defined and selected based on the membership of a document id?
Could you please point me to other discussions, sources (mailing list perhaps?) where I can read more about this new feature and ponder beforehand about the changes in our backend in order to leverage it. Thanks!

[janl]

@AlexanderKaraberov https://lists.apache.org/thread.html/4d09ee0fcc4afec8f25dbd540f9c4f0af67d3fdcc41f26f36ff7a32d@%3Cdev.couchdb.apache.org%3E

[AlexanderKaraberov]

@janl Thank you! Looks exactly like what I was looking for.