S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Microsoft Automatic Mixed Precision Library

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Fast and memory-efficient exact attention

A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)

🐙 Guides, papers, lecture, notebooks and resources for prompt engineering

The simplest, fastest repository for training/finetuning medium-sized GPTs.

Pipeline Parallelism for PyTorch

Intel® Performance Counter Monitor (Intel® PCM)

A multi-party collaborative machine learning framework

Ongoing research training transformer models at scale

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A PyTorch implementation of the Transformer model in "Attention is All You Need".

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

A lightweight parameter server interface

A high performance and generic framework for distributed DNN training

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.