Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify vector_dim and embedding_dim parameter in Document Store #1922

Merged

Conversation

mathew55
Copy link
Contributor

@mathew55 mathew55 commented Dec 23, 2021

Closes Issue #1740.

The issue is about having consistency in the 'embedding dimension' parameter used across the different document stores.

Milvus1.x, Milvus2.x & Faiss were using vector_dim to capture this while all other document stores were using embedding_dim.

The embedding_dim parameter is introduced to Milvus & Faiss Document Stores. Deprecation warnings will be given to users using vector_dim.

Status (please check what you already did):

  • First draft (up for discussions & feedback)
  • Final code
  • Added tests
  • Updated documentation

@mathew55 mathew55 marked this pull request as ready for review December 23, 2021 20:41
@bogdankostic
Copy link
Contributor

Hi @mathew55, thank you so much for your very first contribution to Haystack! This looks already really good! There are just two more smaller points that need to be addressed with regard to the naming of embedding_dim parameter:

  • In some of the tests, we still use vector_dim. We should switch there to embedding_dim as well.
  • In Tutorial 12 we still call FAISSDocumentStore with vector_dim parameter. We should switch there to embedding_dim as well.

Do you want to add these changes to your PR? :)

@bogdankostic bogdankostic linked an issue Dec 27, 2021 that may be closed by this pull request
@mathew55
Copy link
Contributor Author

@bogdankostic , Thank you for reviewing, I will make the above changes and add it to the PR.

@bogdankostic bogdankostic added topic:document_store type:refactor Not necessarily visible to the users labels Dec 30, 2021
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@mathew55
Copy link
Contributor Author

@bogdankostic , I have updated the changes you have mentioned. Will you be able to check again?

I had trouble running milvus locally, so was not able to run the unit test cases on my local.

Copy link
Contributor

@bogdankostic bogdankostic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This look really good to me, we just need to change this line to use embedding_dim instead of vector_dim.

Then all the tests should pass and we should be able to merge this PR! :)

@mathew55
Copy link
Contributor Author

mathew55 commented Jan 9, 2022

@bogdankostic , Can you check now, I have made the change mentioned

@mathew55 mathew55 closed this Jan 9, 2022
@mathew55 mathew55 reopened this Jan 9, 2022
@bogdankostic bogdankostic changed the title WIP: Refactored code to unify vector_dim and embedding_dim parameter in Document Store Unify vector_dim and embedding_dim parameter in Document Store Jan 10, 2022
@bogdankostic
Copy link
Contributor

FYI, I moved the usage of the vector_dim parameter to the if-block where we print out the deprecation warning to make it easier in the future when we'll remove this parameter.

Copy link
Contributor

@bogdankostic bogdankostic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good to me! Thank you very much for your first contribution to Haystack 😊

@bogdankostic bogdankostic merged commit a44b6c1 into deepset-ai:master Jan 10, 2022
@mathew55 mathew55 deleted the 1740_unify_vector_dim_embedding_dim branch January 10, 2022 22:23
@mathew55
Copy link
Contributor Author

@bogdankostic , Thank you for the helping me with the reviews! Hoping to contribute more in the future!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:document_store type:refactor Not necessarily visible to the users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unify vector_dim/embedding_dim parameter in DocumentStores
2 participants