EAGLE: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

An Efficient LLM Fine-Tuning Factory Optimized for MoE PEFT

MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.

LLaMA 2 implemented from scratch in PyTorch

CoMamba: Real-time Cooperative Perception Unlocked with State Space Models

GRadient-INformed MoE

A high-throughput and memory-efficient inference and serving engine for LLMs

A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 and reasoning techniques.

A curated list of resources for using LLMs to develop more competitive grant applications.

Implementation of PhysioMTL (CHIL 2022)

The official implementation of Self-Play Preference Optimization (SPPO)

OLMoE: Open Mixture-of-Experts Language Models

A library for mechanistic interpretability of GPT-style language models

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

MetaDrive-0.2.6.0 that is compatible with newest gym version.

Gin provides a lightweight configuration framework for Python

Plots from "Can AI Help Reduce Disparities in General Medical and Mental Health Care?"

Efficient Triton Kernels for LLM Training

Codebase for Merging Language Models (ICML 2024)

System prompts from Apple's new Apple Intelligence on MacOS Sequoia

[ICCV 2017] Torch code for Grad-CAM

LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models

Building modular LMs with parameter-efficient fine-tuning.

Pytorch implementation of the PEER block from the paper, Mixture of A Million Experts, by Xu Owen He at Deepmind

A collection of AWESOME things about mixture-of-experts

Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"