A high-throughput and memory-efficient inference and serving engine for LLMs

Get up and running with Llama 3.1, Mistral, Gemma 2, and other large language models.

Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.