This is a fix that is intended to stamp out a lingering edge case in _encode_pair() (#1053 #1297 ) where the target continuation ends up with no tokens assigned to it, in the case that the target gets folded entirely into the last token of the context.

The reason we originally introduced _encode_pair() was so that we could avoid the behavior exhibited by Llama's tokenizer where if you pass " <word>" into it, it returns <BOS if added> 29871 <actual token for word, where word is beginning of a word in sentencepiece> rather than <BOS if added> <actual token for word, where word is beginning of a word in sentencepiece>.

29871 in the Llama tokenizer is the SPIECE_UNDERLINE character, which we don't want to be tokenized standalone in the middle of a context.
See huggingface/transformers#26273
oobabooga/text-generation-webui#2606
ggerganov/llama.cpp#3664

for further reference of this issue.

Opening a PR so that I do not forget this needs merging, but it's probably blocked on getting huggingface/transformers#28010 merged--at which point it may be possible to handle this cleanly rather than resorting to hacks.

In an ideal world we may also be able to decide to remove _encode_pair() (or leave it, but only use when a user passes a flag to use the legacy behavior.) and avoid more complicated hacks from compounding. However, we should only do so if it seems like this will not affect the context + continuation behavior in the vast majority of cases.

closes #1297 #1053 .

Will also rebase as to not be dependent on #1287 .