Add splitting/chunking utilities in order to do in-memory RAG.

[homanp]

Since cosine search is implemented in the package wouldn't it make sense to also add utilities for splitting/chunking text? This would allow users to do "in memory RAG" without having to store vectors in a db.

We use this approach for our web research agents and would be up for contributing.

[lgrammel]

Splitting / chunking can quickly become a very large topic with tokenizers, different splitters for different content types, etc.

Do you have any splitters that are most important to you?

[homanp]

We primarily use a recursive text splitter which perform best on regular text which I believe is the primary use case for cosine search anyways.

[lgrammel]

what do you use as max size for the leaf chunks? token counts or character length? i implemented this with token counting before, which requires tokenization. a simpler version that does not require the whole tokenizer part would be text length

[homanp]

what do you use as max size for the leaf chunks? token counts or character length? i implemented this with token counting before, which requires tokenization. a simpler version that does not require the whole tokenizer part would be text length

We don't use tokenizers. It's a combo of length and different separators.