ZMongoRetriever
is a Python library designed to facilitate the retrieval, processing, and encoding of documents from MongoDB collections. It's especially suited for handling large datasets that require chunking and embedding for advanced machine learning applications. Through an elegant interface, it supports document splitting, custom encoding with OpenAI models, and direct integration with MongoDB databases.
- Document Retrieval: Seamlessly fetch documents from MongoDB collections.
- Dynamic Chunking: Split documents into manageable chunks based on character count or embedding size.
- Embedding Support: Encode document chunks using OpenAI's embedding models for deep learning tasks.
- Flexible Configuration: Customize chunk sizes, token overlaps, and database connections to fit your project needs.
- Metadata Conversion: Convert JSON to structured metadata for enhanced document handling.
Before you begin, ensure you have MongoDB and Python 3.6+ installed on your system. Clone this repository or download the ZMongoRetriever
module directly. Dependencies can be installed via pip:
pip install -r requirements.txt
You must have a file named '.env' with the appropriate values for the following:
OPENAI_API_KEY=___
To get started with ZMongoRetriever
, follow these steps:
- Initialize MongoDB Connection:
from pymongo import MongoClient
from zconstants import MONGO_URI
client = MongoClient(MONGO_URI)
- Create an Instance of ZMongoRetriever:
from zmongo_retriever import ZMongoRetriever
retriever = ZMongoRetriever(mongo_uri=MONGO_URI, db_name='your_database', collection_name='your_collection')
- Retrieve and Process Documents:
object_ids = ["65f28c8103fc21342e2dc04d", "65f28c8403fc21342e2dc064"]
documents = retriever.invoke(object_ids=object_ids, page_content_key='report.details.content')
Enable encoding to process document chunks with OpenAI's embeddings:
retriever.use_encoding = True
encoded_chunks = retriever.invoke(object_ids=object_ids, page_content_key='report.details.content')
Customize the chunk size and token overlap for nuanced control over document processing:
retriever.chunk_size = 1024 # Characters
retriever.overlap_prior_chunks = 2 # Number of chunks repeated in a subsequent Document list
Distributed under the MIT License. See LICENSE
for more information.