Simple, fast unsupervised word aligner

Conversion between Traditional and Simplified Chinese

Open language modeling toolkit based on PyTorch

Tools for evaluating the performance of MT metrics on data from recent WMT metrics shared tasks.

a curated list of speech datasets (110+ datasets, 75+ easy to download)

Brand new TTS solution

SONAR, a new multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders.

A fast inference library for running LLMs locally on modern consumer-class GPUs

vits2 backbone with multilingual-bert

Powerful Free DeepL API, No Token Required

KenLM: Faster and Smaller Language Model Queries

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

Bicleaner fork that uses neural networks

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)

Repo for ACL2023 paper "BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training"

Train transformer language models with reinforcement learning.

NTREX -- News Test References for MT Evaluation

Fast numerical array expression evaluator for Python, NumPy, Pandas, PyTables and more

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Minimal keyword extraction with BERT

Fast Open-Source Search & Clustering engine × for Vectors & 🔜 Strings × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍

Easy-to-use word-to-word translations for 3,564 language pairs.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Free English to Chinese Dictionary Database

An Efficient Lexical Analyzer for Chinese

A repository for experiments in quality-aware decoding