GoldFinch and other hybrid transformer components

[ICML 2024] When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models

GraphRAG using Ollama with Gradio UI and Extra Features

A modular graph-based Retrieval-Augmented Generation (RAG) system

Hackathon winner at AI Engineer World Fair Hackathon: Transforming code, one function at a time, to reduce digital carbon footprints and create a more sustainable digital world.

Offcial Repo of Paper "Eliminating Position Bias of Language Models: A Mechanistic Approach""

Official code for "Block Transformer: Global-to-Local Language Modeling for Fast Inference"

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

The modern replacement for Jupyter Notebooks

convert files / GitHub repos into LLM-ready markdown.md files

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023

Reference implementation for Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model

Research project on the capabilities of RNNs.

Continual Resilient (CoRe) Optimizer for PyTorch

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

MaskLID: Code-Switching Language Identification through Iterative Masking -- ACL 2024

Official implementation of Goldfish Loss: Mitigating Memorization in Generative LLMs

MINT-1T: A one trillion token multimodal interleaved dataset.

ACL2024 Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation

Source code for our paper: "SelfGoal: Your Language Agents Already Know How to Achieve High-level Goals".

This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"

LLM Analytics

TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data

Code for Husky, an open-source language agent that solves complex, multi-step reasoning tasks. Husky v1 addresses numerical, tabular and knowledge-based reasoning tasks.

Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".

Autoregressive Model Beats Diffusion: 🦙 Llama for Scalable Image Generation