Use separate analyzer on the search side for ngram-based search. #1711
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The n-gram search on texts were commented out, most likely because it negatively impacts the search relevance. Upon a closer look, there seem to be two main issues:
(1) The same n-gram analyzer was used for both indexing and search. Typically the search side needs a different analyzer that does not use the n-gram based token filters.
(2) The general n-gram token filter was used, which would include n-grams inside a word and thus may negatively impact the relevance.
This pull request is meant to fix the above two issues and re-enable the n-gram based search for texts at the word boundary (as opposed to at the beginning of the whole text only).
The changes in this pull request have been deployed/used for months at the Lister Hill Center of the National Library of Medicine and the results seem to be positive so far.