DiWA: Diverse Weight Averaging for Out-of-Distribution Generalization

Official Implementation of SWAD (NeurIPS 2021)

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Toolkit for creating, sharing and using natural language prompts.

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

A family of open-sourced Mixture-of-Experts (MoE) Large Language Models

A JAX research toolkit for building, editing, and visualizing neural networks.

The official Meta Llama 3 GitHub site

[ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"

Model Stock: All we need is just a few fine-tuned models

official code for "Large Language Models as Optimizers"

Code examples and resources for DBRX, a large language model developed by Databricks

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Official repository of Evolutionary Optimization of Model Merging Recipes

Sphinx theme from Read the Docs

Grok open release

Official implementation for Sparse MetA-Tuning (SMAT)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

[NAACL 2024] Embodied Executable Policy Learning with Language-based Scene Summarization

Official PyTorch Implementation of "Learning to Learn with Generative Models of Neural Network Checkpoints"

Load multiple LoRA modules simultaneously and automatically switch the appropriate combination of LoRA modules to generate the best answer based on user queries.