A MAD laboratory to improve AI architecture designs 🧪

Stick-breaking attention

📦 Repomix (formerly Repopack) is a powerful tool that packs your entire repository into a single, AI-friendly file. Perfect for when you need to feed your codebase to Large Language Models (LLMs) o…

High performance AI inference stack. Built for production. @ziglang / @openxla / MLIR / @bazelbuild

Qwen2.5 is the large language model series developed by Qwen team, Alibaba Cloud.

Official implementation for Yuan & Liu & Zhong et al., KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches. EMNLP Findings 2024

Simple and fast low-bit matmul kernels in CUDA / Triton

Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.

Using FlexAttention to compute attention with different masking patterns

Experimental paper writing linter.

`dattri` is a PyTorch library for developing, benchmarking, and deploying efficient data attribution algorithms.

An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.

Efficient Triton Kernels for LLM Training

Inference RWKV with multiple supported backends.

Influence Functions with (Eigenvalue-corrected) Kronecker-Factored Approximate Curvature

Scalable neural net training via automatic normalization in the modular norm.

使用 cutlass 仓库在 ada 架构上实现 fp8 的 flash attention

RWKV6 in native pytorch and triton:)

WIP

lina-speech : linear attention based text-to-speech

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

Interaction Fingerprints for protein-ligand complexes and more

Solve puzzles. Improve your pytorch.

Run PyTorch LLMs locally on servers, desktop and mobile

Triton implementation of FlashAttention2 that adds Custom Masks.

GoldFinch and other hybrid transformer components