A high-throughput and memory-efficient inference and serving engine for LLMs

Production-Grade Container Scheduling and Management

Kubernetes website and documentation repo:

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

The release repo for "Vicuna: An Open Chatbot Impressing GPT-4"

A library developed by Volcano Engine for high-performance reading and writing of PyTorch model files.

Stateless cluster local OCI registry mirror.

Enhancements tracking repo for Kubernetes

Serving LLMs on heterogeneous decentralized clusters.

Serve 100s of Fine-Tuned LLMs in Production for the Cost of 1

OpenAgents: An Open Platform for Language Agents in the Wild

A efficient and effective few-shot NL2SQL method on GPT-4.

Generate comic panels using a LLM + SDXL. Powered by Hugging Face 🤗

The TikTok OpenSDK features Login Kit and Share Kit which allow your users to log in using their TikTok account and share content from your app to TikTok.

Contributed modules to ray

Easy, Fast, Secure and Cost-Efficient LLM Pipelines to generate GhatGPT-like private domain models and knowledgeable agents for your organization.

⚡ Building applications with LLMs through composability ⚡

Instruct-tune LLaMA on consumer hardware

A framework for few-shot evaluation of autoregressive language models.

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110).

Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.

An open and reliable container runtime

CLI tool for spawning and running containers according to the OCI specification

runc bindings for Go