Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(mango): keys-only covering indexes #4482

Closed
wants to merge 0 commits into from
Closed

feat(mango): keys-only covering indexes #4482

wants to merge 0 commits into from

Conversation

pgj
Copy link
Contributor

@pgj pgj commented Mar 20, 2023 •
edited
Loading

This pull request aims to implement the first phase of the covering indexes work ("keys-only covering indexes") as it was described in the corresponding RFC document by @mikerhodes, see #4410 and #4413 for the details.

Ideally, end users should not notice any change in the behavior but speed-ups for certain kind of queries. Performance results obtained by k6 on my MacBook Pro 2021 M1 Max, with 10,000 documents, compared against a9bce2f598edc8ef843baa9412c60d22157eeabf with 3 nodes:

  • 1-field index covers the query:
    • before: min=0.60ms max=3223.65ms p(95)=2113.95ms p(98)=2354.79ms p(99)=2505.30ms
    • after: min=0.50ms max=615.87ms p(95)=95.89ms p(98)=140.80ms p(99)=171.90ms
  • 2-field index covers the query:
    • before: min=73.09ms max=6245.72ms p(95)=3835.45ms p(98)=4376.79ms p(99)=5022.26ms
    • after: min=2.65ms max=579.62ms p(95)=211.18ms p(98)=264.66ms p(99)=305.37ms
  • generic case:
    • before: min=0.80ms max=14991.06ms p(95)=13171.23ms p(98)=13975.51ms p(99)=14456.11ms
    • after: min=0.87ms max=15379.69ms p(95)=10748.65ms p(98)=11998.88ms p(99)=12171.89ms

Please find the details of the performance evaluation in pgj/couchdb-k6/mango/couchdb_4482.

Unit test coverage increased significantly, see before:

$ make eunit apps=mango
==> mango (compile)
==> rel (compile)
==> couchdb (compile)
==> couchdb (setup_eunit)
==> mango (eunit)
======================== EUnit ========================
[..]
module 'mango_idx_view'
module 'mango_cursor_view'
  mango_cursor_view: does_not_refetch_doc_with_value_test...ok
  mango_cursor_view: doc_member_and_extract_fields_test...ok
  mango_cursor_view: match_and_extract_doc_match_test...ok
  mango_cursor_view: match_and_extract_doc_matchextract_test...ok
  mango_cursor_view: match_and_extract_doc_nomatch_test...ok
  mango_cursor_view: match_and_extract_doc_nomatch_fields_test...ok
  mango_cursor_view: choose_best_index_with_singleton_test...ok
  mango_cursor_view: choose_best_index_lowest_difference_test...ok
  mango_cursor_view: choose_best_index_least_number_of_fields_test...ok
  mango_cursor_view: choose_best_index_lowest_index_triple_test...ok
  [done in 0.030 s]
[..]
=======================================================
  All 81 tests passed.
[..]

Code Coverage:
mango_app             :   0%
mango_crud            :   0%
mango_cursor          :   0%
mango_cursor_special  :   0%
mango_cursor_text     :   0%
mango_cursor_view     :  33%
mango_doc             :   3%
mango_epi             : 100%
mango_error           :   0%
mango_execution_stats :   0%
mango_fields          :  50%
mango_httpd           :   0%
mango_httpd_handlers  :   0%
mango_idx             :  18%
mango_idx_special     :   0%
mango_idx_text        :  63%
mango_idx_view        :   1%
mango_json            :  10%
mango_json_bookmark   :   0%
mango_native_proc     :   5%
mango_opts            :  25%
mango_selector        :  62%
mango_selector_text   :  86%
mango_sort            :   0%
mango_sup             :   0%
mango_util            :  38%

Total                 : 31%
==> rel (eunit)
==> couchdb (eunit)

and then after:

==> mango (compile)
==> rel (compile)
==> couchdb (compile)
==> couchdb (setup_eunit)
==> mango (eunit)
======================== EUnit ========================
[..]
module 'mango_idx_view'
  mango_idx_view: indexable_fields_empty_test...ok
  mango_idx_view: indexable_fields_and_test...ok
  mango_idx_view: indexable_fields_or_test...ok
  mango_idx_view: indexable_fields_nor_test...ok
  mango_idx_view: indexable_fields_all_test...ok
  mango_idx_view: indexable_fields_elemMatch_test...ok
  mango_idx_view: indexable_fields_allMatch_test...ok
  mango_idx_view: indexable_fields_keyMapMatch_test...ok
  mango_idx_view: indexable_fields_in_test...ok
  mango_idx_view: indexable_fields_nin_test...ok
  mango_idx_view: indexable_fields_not_test...ok
  mango_idx_view: indexable_fields_lt_test...ok
  mango_idx_view: indexable_fields_lte_test...ok
  mango_idx_view: indexable_fields_eq_test...ok
  mango_idx_view: indexable_fields_ne_test...ok
  mango_idx_view: indexable_fields_gte_test...ok
  mango_idx_view: indexable_fields_gt_test...ok
  mango_idx_view: indexable_fields_mod_test...ok
  mango_idx_view: indexable_fields_regex_test...ok
  mango_idx_view: indexable_fields_exists_test...ok
  mango_idx_view: indexable_fields_type_test...ok
  mango_idx_view: indexable_fields_size_test...ok
  mango_idx_view: covers_all_fields_test...ok
  mango_idx_view: covers_all_docs_test...ok
  mango_idx_view: covers_empty_index_test...ok
  mango_idx_view: covers_regular_index_test...ok
  [done in 0.081 s]
module 'mango_cursor_view'
  mango_cursor_view: viewcbargs_test...ok
  mango_cursor_view: maybe_replace_max_json_test...ok
  mango_cursor_view: base_opts_test...ok
  mango_cursor_view: apply_opts_empty_test...ok
  mango_cursor_view: apply_opts_r_test...ok
  mango_cursor_view: apply_opts_conflicts_test...ok
  mango_cursor_view: apply_opts_sort_test...ok
  mango_cursor_view: apply_opts_stale_test...ok
  mango_cursor_view: apply_opts_stable_test...ok
  mango_cursor_view: apply_opts_update_test...ok
  mango_cursor_view: apply_opts_partition_test...[0.020 s] ok
  mango_cursor_view: consider_index_coverage_positive_test...ok
  mango_cursor_view: consider_index_coverage_negative_test...ok
  mango_cursor_view: derive_doc_from_index_test...ok
  mango_cursor_view: composite_indexes_test...ok
  mango_cursor_view: create_test...ok
  mango_cursor_view: explain_test...ok
  mango_cursor_view: execute_empty_test...[0.122 s] ok
  mango_cursor_view: execute_ok_all_docs_test...[0.316 s] ok
  mango_cursor_view: execute_ok_query_view_test...[0.297 s] ok
  mango_cursor_view: execute_error_test...[0.152 s] ok
  mango_cursor_view: view_cb_meta_test...[0.046 s] ok
  mango_cursor_view: view_cb_row_matching_regular_doc_test...[0.055 s] ok
  mango_cursor_view: view_cb_row_non_matching_regular_doc_test...[0.065 s] ok
  mango_cursor_view: view_cb_row_null_doc_test...[0.053 s] ok
  mango_cursor_view: view_cb_row_missing_doc_triggers_quorum_fetch_test...[0.056 s] ok
  mango_cursor_view: view_cb_row_matching_covered_doc_test...[0.145 s] ok
  mango_cursor_view: view_cb_row_non_matching_covered_doc_test...[0.052 s] ok
  mango_cursor_view: view_cb_row_backwards_compatible_test...[0.060 s] ok
  mango_cursor_view: view_cb_complete_test...[0.054 s] ok
  mango_cursor_view: view_cb_ok_test...[0.047 s] ok
  mango_cursor_view: maybe_send_mango_ping_nop_test...[0.054 s] ok
  mango_cursor_view: maybe_send_mango_ping_happens_test...[0.048 s] ok
  mango_cursor_view: ddocid_test...ok
  mango_cursor_view: is_design_doc_test...ok
  mango_cursor_view: handle_message_meta_test...ok
  mango_cursor_view: handle_message_row_ok_above_limit_test...[0.223 s] ok
  mango_cursor_view: handle_message_row_ok_at_limit_test...ok
  mango_cursor_view: handle_message_row_ok_skip_test...ok
  mango_cursor_view: handle_message_row_ok_triggers_quorum_fetch_match_test...[1.424 s] ok
  mango_cursor_view: handle_message_row_ok_triggers_quorum_fetch_no_match_test...ok
  mango_cursor_view: handle_message_row_no_match_test...ok
  mango_cursor_view: handle_message_row_error_test...[0.047 s] ok
  mango_cursor_view: handle_message_execution_stats_test...ok
  mango_cursor_view: handle_message_complete_test...ok
  mango_cursor_view: handle_message_error_test...ok
  mango_cursor_view: handle_all_docs_message_ddoc_test...ok
  mango_cursor_view: handle_all_docs_message_row_test...ok
  mango_cursor_view: handle_all_docs_message_regular_test...ok
  mango_cursor_view: does_not_refetch_doc_with_value_test...ok
  mango_cursor_view: doc_member_and_extract_fields_test...ok
  mango_cursor_view: match_and_extract_doc_match_test...ok
  mango_cursor_view: match_and_extract_doc_matchextract_test...ok
  mango_cursor_view: match_and_extract_doc_nomatch_test...ok
  mango_cursor_view: match_and_extract_doc_nomatch_fields_test...ok
  mango_cursor_view: choose_best_index_with_singleton_test...ok
  mango_cursor_view: choose_best_index_lowest_difference_test...ok
  mango_cursor_view: choose_best_index_least_number_of_fields_test...ok
  mango_cursor_view: choose_best_index_lowest_index_triple_test...ok
  mango_cursor_view: update_bookmark_keys_test...ok
  [done in 3.526 s]
[..]
=======================================================
  All 157 tests passed.
[..]

Code Coverage:
mango_app             :   0%
mango_crud            :   0%
mango_cursor          :  20%
mango_cursor_special  :   0%
mango_cursor_text     :   0%
mango_cursor_view     : 100%
mango_doc             :   4%
mango_epi             : 100%
mango_error           :   0%
mango_execution_stats :  63%
mango_fields          :  57%
mango_httpd           :   0%
mango_httpd_handlers  :   0%
mango_idx             :  21%
mango_idx_special     :   0%
mango_idx_text        :  63%
mango_idx_view        :  39%
mango_json            :  12%
mango_json_bookmark   :  25%
mango_native_proc     :   5%
mango_opts            :  27%
mango_selector        :  63%
mango_selector_text   :  86%
mango_sort            :   5%
mango_sup             :   0%

Total                 : 48%
==> rel (eunit)
==> couchdb (eunit)

Tasks:

  • Code is written and works correctly
  • Changes are covered by tests
  • Documentation changes were made in the src/docs folder

@pgj pgj force-pushed the feat/covering-indexes-phase-1 branch 5 times, most recently from 4e33d87 to 618832d Compare March 27, 2023 17:46
@pgj pgj closed this Apr 4, 2023
@pgj pgj force-pushed the feat/covering-indexes-phase-1 branch from ace6a5b to b976247 Compare April 4, 2023 16:57
@pgj
Copy link
Contributor Author

pgj commented Apr 4, 2023

This pull request was closed by accident.

@pgj
Copy link
Contributor Author

pgj commented Apr 4, 2023

The PR was re-submitted as #4512.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pgj