PyTorch extensions for high performance and large scale training.

YaFSDP: Yet another Fully Sharded Data Parallel

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

fabric is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.

Go ahead and axolotl questions

Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.

Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI

🤗 LeRobot: End-to-end Learning for Real-World Robotics in Pytorch

Manage ML configuration with pydantic

An LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.

Build powerful CLIs with simple idiomatic Python, driven by type hints. Not all arguments are bad.

Intuitive, easy CLIs based on python type hints.

Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

🛰️ An approximate nearest-neighbor search library for Python and Java with a focus on ease of use, simplicity, and deployability.

SimPO: Simple Preference Optimization with a Reference-Free Reward

ONNX Script enables developers to naturally author ONNX functions and models using a subset of Python.

Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.

Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust

awesome synthetic (text) datasets

Dataset Crafting and Efficient Fine-Tuning Using Only Free Open-Source Tools

Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.

Python implementation of TextRank algorithms ("textgraphs") for phrase extraction

TTS Generation Web UI (Bark, MusicGen + AudioGen, Tortoise, RVC, Vocos, Demucs, SeamlessM4T, MAGNet, StyleTTS2, MMS)

Blazingly fast LLM inference.

A minimal GPU design in Verilog to learn how GPUs work from the ground up

GitHub Action for Deploying Lambda code to an existing function

Cohere Toolkit is a collection of prebuilt components enabling users to quickly build and deploy RAG applications.

Typed interactions with the GitHub API v3

CuVS integration for Lucene