Mango Indexes on FoundationDB #2585

garrensmith · 2020-02-20T11:39:50Z

Overview

Updates Mango view indexes to work on top of FoundationDB.
Mango view indexes are replaced by Mango json indexes.

Mango json indexes are updated in the same transaction that a document
is updated. A background indexer will build a new index and get it up
to date.

Thanks to @jaydoane for contributing code to this.

Testing recommendations

All mango tests should pass. Create mango indexes and make sure the results returned are expected

Related Issues or Pull Requests

Checklist

Code is written and works correctly
Changes are covered by tests
Any new configurable parameters are documented in rel/overlay/etc/default.ini
A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation

davisp · 2020-02-25T22:34:46Z

Overall, this looks to be a really good first step getting mango updated for fdb. I've only had a few hours to spend reviewing this today and it'll take me a day or two to let it all soak in but there's a few things I'd like jot down while they're paged in.

My first major impression is that there's an awkward cross over between couch_views and the new mango fdb modules. AFAICT, this is mostly due to the way that mango jobs stop indexing once they reach the version stamp where they were created along with being able to just update a single document. It would seem to me that if we just added these two bits as optional features of couch_views it would slim down this entire PR considerably. It might also be worth getting the basic async-index-update mango working before jumping to having the single transaction updates. It feels as there's some criss cross between the changes for a new fdb combined with changes for the incremental updates that aren't quite gelled together.

The changes to fabric2_fdb need to be redone. There shouldn't be a single mention of mango in that module. There are a number of features that will want to know about db changes that should all share a single interface.

For the PR itself, this should definitely be split up into smaller chunks of logic that are described individually. The last one or two might be a fairly large commit but there's a lot of churn that could be processed more efficiently stepping through individual commits. If you need help on how to break down a large commit I'd happy to pair with you to show you the basic git commands for handling that.

I really like your build status concept. I'd like to extend that for views as well to show initial build vs incremental update that would be useful for operators. Moving that status under a prefix would also give us the necessary structure for efficient view cleanups as well which I'm looking at working on next.

garrensmith · 2020-02-26T08:58:14Z

@davisp thanks for reviewing. Sorry I made a real mess with my commits and cheated a little and squashed into one. I've broken them up into a few commits. Its not perfect but breaks the context up a bit more and should make it easier to review.

src/couch/src/couch_proc_manager.erl

tonysun83 · 2020-03-03T17:09:10Z

Just started going over this PR. I noticed we're going through the EPI route for each doc update. Looks like Ken may be utilizing the EPI route as well.. Will it make more sense to plug in the hook into Ken later on as part of Ken's consolidated auto indexing scheme?

I see work currently being done by @nickva in this regard: b16c4a1

davisp · 2020-03-03T20:49:40Z

@tonysun83 Yeah, both of these are definitely touching on related bits triggering view work from the update loop. Whichever lands second will fixup things so there's a unified interface there.

iilyak · 2020-03-04T16:41:12Z

Just started going over this PR. I noticed we're going through the EPI route for each doc update. Looks like Ken may be utilizing the EPI route as well.. Will it make more sense to plug in the hook into Ken later on as part of Ken's consolidated auto indexing scheme?

From EPI point of view it has a mode where it calls plugins one by one. So it is possible to have multiple plugins on the same extension hook.

garrensmith · 2020-03-11T11:22:02Z

Quick update on this. I've cleaned up the commits and all python and eunit tests are passing. Also all pouchdb-find tests are passing.

davisp · 2020-03-11T15:00:20Z

Can you update make check-fdb to include the commadn to run the mango python tests?

davisp

Looks pretty good. Most of my comments were style related. I think changing the build_to_vs field is probably the biggest change in behavior and that shouldn't be a major change. We also need to get the values passed to couch_eval:map_docs hammered out. It looks as though there's a change to using ErlJSON docs when its expecting #doc{} records in some places still.

src/chttpd/src/chttpd_db.erl

src/couch/src/couch_proc_manager.erl

src/couch_views/src/couch_views.erl

src/couch_views/src/couch_views_fdb.erl

src/mango/test/eunit/mango_indexer_test.erl

src/mango/test/mango.py

src/mango/test/user_docs.py

garrensmith

Thanks for the review Paul. I've left some comments to answer some of your questions.

src/chttpd/src/chttpd_db.erl

src/couch_views/src/couch_views.erl

src/couch_views/src/couch_views_fdb.erl

src/fabric/src/fabric2_fdb.erl

src/mango/test/01-index-crud-test.py

src/mango/test/19-find-conflicts.py

src/mango/test/20-no-timeout-test.py

src/mango/test/user_docs.py

src/mango/test/12-use-correct-index-test.py

src/mango/src/mango_indexer.erl

davisp

I've got mixed up in my review so flushing the couple notes I wrote so I can refresh the conversation tab.

src/couch_views/src/couch_views_indexer.erl

src/couch_views/test/couch_views_trace_index_test.erl

src/mango/test/12-use-correct-index-test.py

davisp

Looking really good! Just a few things left to fix up.

src/couch_views/src/couch_views_jobs.erl

This creates a versionstamp for when an indexed was created and build status for indexes. if the index has a creation_vs, then couch_views_indexer will built the index to this creation versionstamp.

This adds the ability for couch_views to index an index in the docs update transaction. This only happens if a design doc has the field <<"interactive">> = true.

Adds a max value to use for encoding. This is useful when getting the max range when encoding startkey/endkeys.

Removes the view callback that was performed on the nodes before sending the results back to the co-ordinator.

Removing quorum stats since they are not relevant with FDB.

This uses couch_views_updater to create mango indexes in the doc update along with the couch_views_indexer to update the indexes in the background up to the creation versionstamp.

garrensmith

thanks @davisp I've rebased with the latest couch_rate work and addressed all your comments.

src/couch_views/src/couch_views_indexer.erl

src/couch_views/src/couch_views_jobs.erl

src/couch_views/test/couch_views_trace_index_test.erl

davisp

One very simple last change:

diff --git a/Makefile b/Makefile
index 1326d40f7..cc33c61d8 100644
--- a/Makefile
+++ b/Makefile
@@ -350,7 +350,7 @@ mango-test: devclean all
 	@cd src/mango && \
 		python3 -m venv .venv && \
 		.venv/bin/python3 -m pip install -r requirements.txt
-	@cd src/mango && ../../dev/run "$(TEST_OPTS)" -n 1 --admin=adm:pass '.venv/bin/python3 -m nose --with-xunit'
+	@cd src/mango && ../../dev/run "$(TEST_OPTS)" -n 1 --admin=adm:pass --erlang-config=rel/files/eunit.config '.venv/bin/python3 -m nose -v --with-xunit'
 
 ################################################################################
 # Developing

The --erlang-config flag is the important change there. I added the -v for nose to list tests so we can see what's skipped. Its like 98% text indexing plus the two or so that we're leaving for the _users database. Up to you whether you keep the verbosity or not.

Clean up unused mango_utils functions.

garrensmith · 2020-04-06T10:10:36Z

@davisp I like the -v I've added the change in.

davisp

+1

wohali · 2020-04-06T15:29:52Z

Makefile

@@ -160,9 +160,10 @@ endif

 .PHONY: check-fdb
 check-fdb:
- make eunit apps=couch_eval,couch_expiring_cache,ctrace,couch_jobs,couch_views,fabric
+ make eunit apps=couch_eval,couch_expiring_cache,ctrace,couch_jobs,couch_views,fabric,mango


Anyone planning to copy these changes into Makefile.win?

garrensmith requested review from davisp, jaydoane and tonysun83 February 20, 2020 11:39

garrensmith force-pushed the fdb-mango-indexes branch from 0448466 to c7906ef Compare February 26, 2020 08:57

garrensmith force-pushed the fdb-mango-indexes branch 2 times, most recently from 20af02b to a3c6379 Compare March 2, 2020 15:06

davisp force-pushed the prototype/fdb-layer branch from b3bd36b to bdd0578 Compare March 2, 2020 22:53

tonysun83 reviewed Mar 3, 2020

View reviewed changes

src/couch/src/couch_proc_manager.erl Outdated Show resolved Hide resolved

garrensmith force-pushed the fdb-mango-indexes branch 4 times, most recently from 1acee3a to 7f295f3 Compare March 9, 2020 17:18

This was referenced Mar 10, 2020

couch_jobs resubmit updates job data #2649

Merged

Fix bug in reverse folding with startkey_docid #2650

Merged

garrensmith force-pushed the fdb-mango-indexes branch 4 times, most recently from 0f6837d to 1bfd0db Compare March 11, 2020 11:21

davisp requested changes Mar 17, 2020

View reviewed changes

garrensmith commented Mar 17, 2020

View reviewed changes

davisp reviewed Mar 17, 2020

View reviewed changes

src/mango/src/mango_indexer.erl Outdated Show resolved Hide resolved

garrensmith force-pushed the fdb-mango-indexes branch 2 times, most recently from 4d5b4eb to e072b88 Compare March 19, 2020 15:56

garrensmith force-pushed the fdb-mango-indexes branch from 8288107 to e0aff0e Compare March 31, 2020 15:20

davisp reviewed Apr 1, 2020

View reviewed changes

src/couch_views/src/couch_views_indexer.erl Outdated Show resolved Hide resolved

src/couch_views/test/couch_views_trace_index_test.erl Outdated Show resolved Hide resolved

src/mango/test/12-use-correct-index-test.py Show resolved Hide resolved

davisp requested changes Apr 1, 2020

View reviewed changes

src/couch_views/src/couch_views_jobs.erl Outdated Show resolved Hide resolved

garrensmith added 16 commits April 2, 2020 12:50

fix all_docs call to return row

a553bc2

move all_doc view options to fabric2_util

3179147

add include_docs option to fold_docs

ec50197

add fabric2 after_doc_write plugin

bd192ea

Add couch_views_indexer build to creation versionstamp

9dfe9e5

This creates a versionstamp for when an indexed was created and build status for indexes. if the index has a creation_vs, then couch_views_indexer will built the index to this creation versionstamp.

All couch_view queries to run across transactions

5000260

Add couch_views_updater interactive indexer

3c86190

This adds the ability for couch_views to index an index in the docs update transaction. This only happens if a design doc has the field <<"interactive">> = true.

Add couch_views_encoding max value

fd323c1

Adds a max value to use for encoding. This is useful when getting the max range when encoding startkey/endkeys.

remove mango native proc

daecc5d

remove partition opts from mango

0df103b

remove unneeded r/w parameter

6170372

Remove view_cb predicate push down

825aafc

Removes the view callback that was performed on the nodes before sending the results back to the co-ordinator.

Remove quorum stats

a2fc004

Removing quorum stats since they are not relevant with FDB.

Add mango indexing

e438a36

This uses couch_views_updater to create mango indexes in the doc update along with the couch_views_indexer to update the indexes in the background up to the creation versionstamp.

Update Mango query to work with couch_views

5fc1900

Update mango test creds to same as elixir tests

370e490

garrensmith force-pushed the fdb-mango-indexes branch from e0aff0e to 95477cc Compare April 2, 2020 12:39

garrensmith commented Apr 2, 2020

View reviewed changes

src/couch_views/src/couch_views_indexer.erl Outdated Show resolved Hide resolved

src/couch_views/src/couch_views_jobs.erl Outdated Show resolved Hide resolved

src/couch_views/test/couch_views_trace_index_test.erl Outdated Show resolved Hide resolved

davisp requested changes Apr 5, 2020

View reviewed changes

garrensmith added 2 commits April 6, 2020 12:10

update mango tests to work with Mango on FDB

b06a713

remove defer and load_ddocs from mango_utils

a154b75

Clean up unused mango_utils functions.

garrensmith force-pushed the fdb-mango-indexes branch from 95477cc to a154b75 Compare April 6, 2020 10:10

davisp approved these changes Apr 6, 2020

View reviewed changes

wohali reviewed Apr 6, 2020

View reviewed changes

garrensmith merged commit 5652e72 into prototype/fdb-layer Apr 6, 2020

wohali deleted the fdb-mango-indexes branch October 21, 2020 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mango Indexes on FoundationDB #2585

Mango Indexes on FoundationDB #2585

garrensmith commented Feb 20, 2020

davisp commented Feb 25, 2020

garrensmith commented Feb 26, 2020

tonysun83 commented Mar 3, 2020 •

edited

Loading

davisp commented Mar 3, 2020

iilyak commented Mar 4, 2020

garrensmith commented Mar 11, 2020

davisp commented Mar 11, 2020

davisp left a comment

garrensmith left a comment

davisp left a comment

davisp left a comment

garrensmith left a comment

davisp left a comment

garrensmith commented Apr 6, 2020

davisp left a comment

wohali Apr 6, 2020

Mango Indexes on FoundationDB #2585

Mango Indexes on FoundationDB #2585

Conversation

garrensmith commented Feb 20, 2020

Overview

Testing recommendations

Related Issues or Pull Requests

Checklist

davisp commented Feb 25, 2020

garrensmith commented Feb 26, 2020

tonysun83 commented Mar 3, 2020 • edited Loading

davisp commented Mar 3, 2020

iilyak commented Mar 4, 2020

garrensmith commented Mar 11, 2020

davisp commented Mar 11, 2020

davisp left a comment

Choose a reason for hiding this comment

garrensmith left a comment

Choose a reason for hiding this comment

davisp left a comment

Choose a reason for hiding this comment

davisp left a comment

Choose a reason for hiding this comment

garrensmith left a comment

Choose a reason for hiding this comment

davisp left a comment

Choose a reason for hiding this comment

garrensmith commented Apr 6, 2020

davisp left a comment

Choose a reason for hiding this comment

wohali Apr 6, 2020

Choose a reason for hiding this comment

tonysun83 commented Mar 3, 2020 •

edited

Loading