Enable stemming and choosing tokenizer, when doing full text search in tantivy

[josca42]

SDK

Python

Description

Enabling stemming and using a language specific tokenizer tend to improve recall quite a bit, when doing full text search.

Tantivy has support for this through the tokenizer_name argument in add_text_field.

As far as I can tell the change needed is to add tokenizer_name argument to the following line

And then add the tokenizer_name argument to the create_fts_index method.

I would personally really prefer if the argument could be exposed instead of just enabling the usage of the english stemmer. Tantivy supports a few different language tokenizers, which I think a lot of people would like to use instead of english

I can create a pull request with the suggested changes if you think it is a good idea :-).

[wjones127]

This all sounds good to me. Feel free to make a PR :)