Implementation of OpenAI paper with Simple Noise Scale on Fastai V2

An approximate implementation of the OpenAI paper - An Empirical Model of Large-Batch Training for MNIST

[ICSE 2024 Industry Challenge Track] Official implementation of "ReposVul: A Repository-Level High-Quality Vulnerability Dataset".

Longitudinal Evaluation of LLMs via Data Compression

Codebase for Merging Language Models (ICML 2024)

RewardBench: the first evaluation tool for reward models.

Robust recipes to align language models with human and AI preferences

Comprehensive toolkit for Reinforcement Learning from Human Feedback (RLHF) training, featuring instruction fine-tuning, reward model training, and support for PPO and DPO algorithms with various c…

ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Languag…

Reaching LLaMA2 Performance with 0.1M Dollars

Retrieval and Retrieval-augmented LLMs

Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip

Imitate OpenAI with Local Models

[EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs

An application providing a RESTful API similar to OpenAI Embedding, supporting BERT, SBERT, and CoSENT models for generating text embedding vectors.

Code for paper 'Are We Falling in a Middle-Intelligence Trap? An Analysis and Mitigation of the Reversal Curse'

Datasets, tools, and benchmarks for representation learning of code.

Enhacing Code Pre-trained Models by Contrastive Learning

The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.

[ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"

Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Ongoing research training transformer models at scale

CodeXGLUE