Optimise fabric:all_dbs()

[nickva]

Previous version was awfully inefficient: we only need the dbname, but we opened each document body, parsed the shards out of it, generated Q*N shards, then put them in a list and usorted them back down to a list of dbnames.

We don't use this in many places (mem3 security sync, fabric bench) but it might be nice to have an efficient version of it anyway.

[jaydoane]

I tested this out on a cluster with a large shards db. The current implementation takes over 12 seconds:

(dbcore@db10.bigblue.cloudant.net)7> timer:tc(fun() -> fabric:all_dbs(<<"jdoane">>) end).
{12599252,
 {ok,[<<"jdoane/cats">>,<<"jdoane/d/">>,<<"jdoane/db+plus">>,
      <<"jdoane/db_$()f">>,<<"jdoane/devops-dev-products">>,
      <<"jdoane/foo/bar">>,<<"jdoane/hogs">>,<<"jdoane/large">>,
      <<"jdoane/n0hjj4$x$a">>,<<"jdoane/replicated-cats">>,
      <<"jdoane/sec-test">>,<<"jdoane/testy_db_jydrjarchp">>,
      <<"jdoane/testy_db_target">>,
      <<"jdoane/testy_db_uxlqexkazh">>,<<"jdoane/tmp">>,
      <<"jdoane/wgb4g">>]}}

I copied a slightly modified mem3_shards:fold_dbs/3 into a remsh and it only takes about a couple milliseconds:

(dbcore@db10.bigblue.cloudant.net)10> timer:tc(fun() -> Dbs = _fold_dbs(<<"jdoane">>, FoldFun, []), {ok, lists:reverse(Dbs)} end).
{1755,
 {ok,[<<"jdoane/cats">>,<<"jdoane/d/">>,<<"jdoane/db+plus">>,
      <<"jdoane/db_$()f">>,<<"jdoane/devops-dev-products">>,
      <<"jdoane/foo/bar">>,<<"jdoane/hogs">>,<<"jdoane/large">>,
      <<"jdoane/n0hjj4$x$a">>,<<"jdoane/replicated-cats">>,
      <<"jdoane/sec-test">>,<<"jdoane/testy_db_jydrjarchp">>,
      <<"jdoane/testy_db_target">>,
      <<"jdoane/testy_db_uxlqexkazh">>,<<"jdoane/tmp">>,
      <<"jdoane/wgb4g">>]}}

which is several orders of magnitude improvement.

Fantastic work!

[jaydoane]

This seems like a clever way of not having to use an end key like e.g. <<Prefix/binary, ?HIGH_VALUE_UNICODE/binary>>