A high-throughput and memory-efficient inference and serving engine for LLMs

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

LLM Frontend for Power Users.

A fast inference library for running LLMs locally on modern consumer-class GPUs

Fast and memory-efficient exact attention

An OAI compatible exllamav2 API that's both lightweight and fast

Efficient Triton Kernels for LLM Training

Utilities to use the Hugging Face Hub API

A general fine-tuning kit geared toward Stable Diffusion 2.1, DeepFloyd, and SDXL.

Development repository for the Triton language and compiler

Auto convert transformers models to QuaRot.

Grok open release

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Port of Facebook's LLaMA model in C/C++

Easy to use LLM quantization via the QuIP# technique

Preprint: Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

LLM Powered discord bot, Character Card enabled Chat page, Stable Diffusion discord bot, and overall AI tool. All from one app, TalOS: Reborn.

SillyTavern MultiPlayer is an LLM chat interface, created by RossAscends, that allows multiple users to chat together with each other and an AI.

⚡ Building applications with LLMs through composability ⚡

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

Tools for merging pretrained large language models.