feat(dev): add Clouseau to the developer setup

[pgj]

Provide a way to the developers to deploy and launch the Clouseau search module more easily, therefore making it more accessible. This can help with running the Search-based Elixir tests as well as the Mango text search tests. This could allow us to catch more bugs ahead of time and might even inspire more improvements in the area.

The extension is designed in a way to make it simple to integrate with the CI — only the Java environment of the proper version needs to be deployed, everything else could be managed from this repository. Such as bumping the version of Clouseau or fine-tuning the configuration parameters.

This is an opt-in feature on two levels. First, one has to tell ./configure --enable-clouseau to instantiate Clouseau locally. This will create the clouseau sub-directory that holds the contents of the Clouseau distribution of the specified version, which currently defaults to 2.22.0 (the latest). If an older version is needed, use the --clouseau-version flag.

Then the mango-test and elixir-search targets will try to use Clouseau automatically just to make it easy to use in the old way. However, the dev/run script will not do the same. It requires the --with-clouseau flag to be passed for that. That is because the developer may not necessarily want to launch Clouseau ad hoc, even if it is available.

With this, note that elixir-search is added to the check target. In lack of a configured Clouseau instance, it will become a no-op and a warning is emitted.

Testing recommendations

There are many ways to try this change. The conventional approach is to have some flavor of JDK 8 installed, via asdf for example, configure CouchDB to deploy Clouseau and run the corresponding test suites.

asdf plugin add java
asdf install java zulu-8.74.0.17
export CLOUSEAU_JAVA_HOME=$(asdf where java zulu-8.74.0.17)
./configure --enable-clouseau <other-configure-flag-of-yours>
make elixir-search
make mango-test

The elixir-search target should not emit a warning (but run the tests) and mango-test should have only a single skipped test.

The dev/run script could be run individually where Search could be talked to, e.g.:

dev/run --with-clouseau -n 1 -a adm:pass
curl -sS http:https://127.0.0.1:15984/ | jq -rc '.features'

The last command should show something similar to this:

["search","access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"]

Conversely, if no Clouseau is configured, there shall be no clouseau sub-directory, no Elixir Search tests or Mango text tests are run (more skipped cases), and no search is reported as a feature. When trying to use dev/run --with-clouseau without Clouseau being added, there shall be errors.

Related Issues or Pull Requests

• apache/couchdb-ci #62

Checklist

• Code is written and works correctly
• Add Windows support
• Jenkinsfile changes (maybe as a follow-up PR once the Docker images are updated)
• devcontainer changes (needs the updated base image)

[jiahuili430]

The code is very clean and makes Clouseau easier to use. Thank you, Gábor.

[jaydoane]

This is great work! It's about time Clouseau became a first class developer component.

I followed the well-written configuration and runtime documentation, and all went well.

I ran the relevant tests:

❯ make elixir-search

SearchTest [test/elixir/test/search_test.exs]
  * test drilldown single key multiple values for POST (776.2ms) [L#159]
  * test drilldown multiple keys single values for POST (51.9ms) [L#172]
  * test drilldown single key single value for POST (51.2ms) [L#146]
  * test drilldown single key single value for GET (46.4ms) [L#80]
  * test facet ranges, non-empty (74.5ms) [L#265]
  * test drilldown single key multiple values for GET (45.2ms) [L#93]
  * test drilldown multiple query definitions for GET (45.2ms) [L#119]
  * test search returns all items for GET (45.2ms) [L#67]
  * test clean up search index with invalid design document (17.3ms) [L#224]
  * test facet counts, non-empty (54.4ms) [L#235]
  * test facet counts, empty (46.6ms) [L#250]
  * test facet ranges, empty (50.9ms) [L#280]
  * test drilldown multiple keys multiple values for POST (44.8ms) [L#198]
  * test search returns all items for POST (46.5ms) [L#133]
  * test drilldown multiple query definitions for POST (41.8ms) [L#211]
  * test drilldown three keys single values for POST (51.6ms) [L#185]
  * test drilldown multiple keys single values for GET (49.4ms) [L#106]

PartitionSearchTest [test/elixir/test/partition_search_test.exs]
  * test Works with limit using POST for on non-partitioned db (45.7ms) [L#112]
  * test Works with limit using POST for partitioned db (41.8ms) [L#123]
  * test Works with bookmarks and limit (70.3ms) [L#84]
  * test rejects conflicting partition values (10.7ms) [L#210]
  * test Simple query returns partitioned search results (51.4ms) [L#52]
  * test normal search on non-partitioned dbs without limit (41.0ms) [L#173]
  * test normal search on non-partitioned dbs with limit (38.2ms) [L#186]
  * test normal search on non-partitioned dbs with over limit (10.8ms) [L#199]
  * test restricted parameters are not allowed in query or body (15.3ms) [L#221]
  * test Cannot do partition query with global search ddoc (11.1ms) [L#147]
  * test Only returns docs in partition not those in shard (42.3ms) [L#71]
  * test normal search on non-partitioned dbs still work (38.3ms) [L#160]
  * test Cannot do global query with partition view (13.4ms) [L#134]

Finished in 3.2 seconds (0.00s async, 3.2s sync)
30 tests, 0 failures

and

❯ make mango-test

[ * ] Setup environment ... ok
[ * ] Ensure CouchDB is built ... ok
[ * ] Ensure Erlang boot script exists ... ok
[ * ] Prepare configuration files ... ok
[ * ] Start node node1 ... ok
[ * ] Start Clouseau node clouseau1 ... ok
[ * ] Check node at http:https://127.0.0.1:15984/ ... ok
[ * ] Check Clouseau node clouseau1 ... ok
[ * ] Running cluster setup ... ok
[ * ] Exec command COUCH_USER=adm COUCH_PASS=pass src/mango/.venv/bin/nose2 -s src/mango/test -c src/mango/unittest.cfg  ... .....................................................................................................................................................................................................................................s..........................................................................................................................................................
----------------------------------------------------------------------
Ran 384 tests in 38.257s

And started a remsh (never seen those clouseau nodes boot so fast) to play around in:

❯ dev/run -a adm:pass --with-haproxy --with-clouseau --no-eval REMSHID=1 dev/remsh
[ * ] Setup environment ... ok
[ * ] Ensure CouchDB is built ... ok
[ * ] Ensure Erlang boot script exists ... ok
[ * ] Prepare configuration files ... ok
[ * ] Start node node1 ... ok
[ * ] Start node node2 ... ok
[ * ] Start node node3 ... ok
[ * ] Start Clouseau node clouseau1 ... ok
[ * ] Start Clouseau node clouseau2 ... ok
[ * ] Start Clouseau node clouseau3 ... ok
[ * ] Check node at http:https://127.0.0.1:15984/ ... ok
[ * ] Check Clouseau node clouseau1 ... ok
[ * ] Check node at http:https://127.0.0.1:25984/ ... ok
[ * ] Check Clouseau node clouseau2 ... ok
[ * ] Check node at http:https://127.0.0.1:35984/ ... ok
[ * ] Check Clouseau node clouseau3 ... ok
[ * ] Running cluster setup ... ok
[ * ] Exec command REMSHID=1 dev/remsh ... Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Enabled docsh 0.7.0 from: /Users/jay/repos/docsh/_build/default/lib/docsh
Call h(docsh) for interactive help.

Eshell V13.2.2  (abort with ^G)
([email protected])1> net_adm:ping('[email protected]').
pong
([email protected])2> net_adm:ping('[email protected]').
pong
([email protected])3> net_adm:ping('[email protected]').
pong

Confirmed clouseau logs appeared in dev/logs/.

Thank you for this huge improvement in the developer experience!