We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
add sub-groupnorm in multihead ema, for use in audio modeling
fix laplace activation function thanks to @boweny-cerebras
fix residual within mega layer, thanks to @VHellendoorn
prenorm requires a final layernorm
fix residual for prenorm
expose multi-headed learned EMA, for use outside of repo
offer prenorm architecture
handle bidirectional better
setup enwik8 autoregressive training
improvise on bidirectional for multi-head learned ema