Limiting Memory Usage When Loading Large Collections #33031
Unanswered
legion15q
asked this question in
Q&A and General discussion
Replies: 1 comment 4 replies
-
In my opinion, to limit memory, you should use k8s resources.limit.memory. |
Beta Was this translation helpful? Give feedback.
4 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello Milvus Community!
I am encountering an issue with high memory consumption when using the collection.load function for large datasets. Despite applying approaches like mmap and using diskANN and HSNW indexes, which indeed reduce memory usage, they do not allow for a strict limitation on maximum memory usage.
Is there a way in Milvus to configure data loading such that the search process can be batched or not entirely in-memory? I am willing to sacrifice search speed to reduce memory usage.
I would appreciate any suggestions or recommendations!
Beta Was this translation helpful? Give feedback.
All reactions