[AISTATS 2023] Error Estimation for Random Fourier Features

Implementation of a Transformer using ReLA (Rectified Linear Attention) from https://arxiv.org/abs/2104.07012

An implementation of masked language modeling for Pytorch, made as concise and simple as possible

Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"

Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch

Reformer, the efficient Transformer, in Pytorch

Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts

Implementation of an Attention layer where each head can attend to more than just one token, using coordinate descent to pick topk

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Implementation of Linformer for Pytorch

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

An implementation of local windowed attention for language modeling

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of the convolutional module from the Conformer paper, for use in Transformers

A simple but complete full-attention transformer with a set of promising experimental features from various papers

The Electronic World Atlas of Varieties of English

Chaospy - Toolbox for performing uncertainty quantification.

A Python wrapper around the Tasmanian sparse grid library

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)

Long Range Arena for Benchmarking Efficient Transformers

[DEPRECATED] Repo for exploring multi-task learning approaches to learning sentence representations