Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDEV-33408 Alter HNSW graph storage and fix memory leak #3197

Open
wants to merge 1 commit into
base: bb-11.4-vec-vicentiu
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
MDEV-33408 Alter HNSW graph storage and fix memory leak
This commit changes the way HNSW graph information is stored in the
second table. Instead of storing connections as separate records, it now
stores neighbors for each node, leading to significant performance
improvements and storage savings.

Comparing with the previous approach, the insert speed is 5 times faster,
search speed improves by 23%, and storage usage is reduced by 73%, based
on ann-benchmark tests with random-xs-20-euclidean and
random-s-100-euclidean datasets.

Additionally, in previous code, vector objects were not released after
use, resulting in excessive memory consumption (over 20GB for building
the index with 90,000 records), preventing tests with large datasets.
Now ensure that vectors are released appropriately during the insert and
search functions. Note there are still some vectors that need to be
cleaned up after search query completion. Needs to be addressed in a
future commit.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
  • Loading branch information
HugoWenTD committed Apr 12, 2024
commit 437e21444ceec6992cd35f4449632a5dec07a9bf
2 changes: 1 addition & 1 deletion sql/sql_base.cc
Original file line number Diff line number Diff line change
Expand Up @@ -9842,7 +9842,7 @@ int TABLE::hlindex_open(uint nr)
}
TABLE *table= (TABLE*)alloc_root(&mem_root, sizeof(*table));
if (!table ||
open_table_from_share(in_use, s->hlindex, &empty_clex_str, db_stat, 0,
open_table_from_share(in_use, s->hlindex, &empty_clex_str, db_stat, EXTRA_RECORD,
in_use->open_options, table, 0))
return 1;
hlindex= table;
Expand Down