[BUG] IDF change to 0 after writing the same text three times

[freecw]

[meiravgri]

@freecw Hi.
Indeed looks like a bug! Good catch!
Thanks.

[freecw]

@meiravgri it seems the "IDF" will change to 0 after we deleting another key

[meiravgri]

I've found the root cause for this bug and opened an internal ticket to our team.
We'll address this asap. Thanks for reporting.

[freecw]

@meiravgri Thanks, could you please share the root cause? And any expected time for addressing this?

[meiravgri]

sure
IDF score depends on sp->docs.size and idx->numDocs like so:

logb(1.0F + sp->docs.size / (idx->numDocs ? idx->numDocs : (double)1));

Since idx->numDocs reflects the number of valid docs + updated docs that weren’t yet cleaned by the gc, AND the default gc threshold is 100, the score is calculated based on wrong index statistics.

As a temporary workaround you can decrease the the gc threshold:
FT.CONFIG set FORK_GC_CLEAN_THRESHOLD 0

[freecw]

Thanks @meiravgri . Any update on addressing this issue is highly appreciated.