What's new in 0.25.0

[nicolay-r]

Support Batching for effecting imputing LLM into text processing pipelines

Previosly, the whole text processing pipeline was relying on the sentence / text part.
Now we overcome that liimitation and therefore we can consider multiple sentences, formed in list i.e. batch.
This step is so important for LLM, LM, neural networks, for which batching accelerates the performance.
As the result, overall pipeline launching is expected to perform faster.

• Pipelines -- refactoring core concept, source customization selection for ppl items #539 🔥
• Pipelines -- Batching sentences in document parser [ARElight backlog] #535
• Pipeline.run might be just a concept of launchers, there is no need to combine storage of items with run operation #540
• SentenceObjectsParserPipelineItem -- rename to the ObjectsParserPipelineItem concept #541

Sources collections are no longer going to be a part of AREkit ✨
Tha allow us to lightweight 🪶 the overall framework and so that purely focus on data processing techniques

• Sources Movement in AREkit-ss [including the related dependencies] #537
• Remove requests library dependency 🪶
• Move all the tutorials 📚 to the AREkit-ss project. 🪶

Flexibility and Performance Enhancements

• Pipelines -- refactoring core concept, source customization selection for ppl items #539 🔥
• Pipelines -- Batching sentences in document parser [ARElight backlog] #535
• Sources Movement in AREkit-ss [including the related dependencies] #537
• Source Registration Challenges arekit-ss#80 (Issue has been moved to arekit-ss project)
  • Dynamic creation of labels provider #511
• ➕ SQlite-based readers and storage providers #538
• Partitioning -- fancy last operations of the SentenceObjectsParserPipelineItem which has no longer application [ARElight backlog] #542 🔧

Fixed bugs

• 🔧 RowCacheStorageProvider fixed bug with mismatching size of type list and columns list in case of other force collected columns (ad4312c)

Minor Updates

• ❌ Removed OpinionsIO (76b4c1f)
• ❌ Removed suffix -0 in filenames for samples. (76b4c1f)
• ❌ TermsSplitterParser -- is no longer required [ARElight backlog] #543
• ❌ _get_text is no longer needed #544
• ❌ SamplesIO.create_target -- provide this parameter as function [ARElight backlog] #547

[nicolay-r]

Published 🥳