#
Starred repositories
Lightning-fast serving engine for AI models. Flexible. Easy. Enterprise-scale.
Dynamic Memory Management for Serving LLMs without PagedAttention
"The gift of mental power comes from God, Divine Being, and if we concentrate our minds on that truth, we become in tune with this great power. My Mother had taught me to seek all truth in the Bibl…
A fast communication-overlapping library for tensor parallelism on GPUs.
AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and versatility of software and hardware.
Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
List of papers related to neural network quantization in recent AI conferences and journals.
đź“–A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.
A framework for few-shot evaluation of language models.
A tool for generating information about the matrix multiplication instructions in AMD Radeon™ and AMD Instinct™ accelerators
A one stop repository for generative AI research updates, interview resources, notebooks and much more!
RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing …
Python bindings for the Transformer models implemented in C/C++ using GGML library.
Assembler and Decompiler for NVIDIA (Maxwell Pascal Volta Turing Ampere) GPUs.
Home of the JELOS Linux distribution.
A Easy-to-understand TensorOp Matmul Tutorial
WebUI for Fine-Tuning and Self-hosting of Open-Source Large Language Models for Coding
