Commits

Commits on Jul 5, 2023

fix states being overridden in generate
lucidrains
committedJul 5, 2023

Commits on Apr 24, 2023

change of plans
lucidrains
committedApr 24, 2023

Commits on Apr 20, 2023

handle an edge case
lucidrains
committedApr 20, 2023
update to einops 0.6.1 or greater
lucidrains
committedApr 20, 2023
add compressed memories
lucidrains
committedApr 20, 2023

Commits on Apr 18, 2023

Commits on Apr 17, 2023

remove enhanced recurrence, does not seem to do all that much here
lucidrains
committedApr 17, 2023
refactor to process block by block, to ready for decoupling layers at which state is read or written
lucidrains
committedApr 17, 2023

Commits on Apr 4, 2023

Commits on Mar 31, 2023

actually use the recurrent states instead of always initial state, thanks to @wj210 for identifying the bug
lucidrains
committedMar 31, 2023

Commits on Mar 26, 2023

cache causal mask and rotary positional embeddings
lucidrains
committedMar 26, 2023

Commits on Mar 24, 2023

bug with xpos for rotary embeddings
lucidrains
committedMar 24, 2023
product management
lucidrains
committedMar 24, 2023
intent
lucidrains
committedMar 24, 2023
it turns out flash attention in pytorch 2.0 is not handling causal correctly when key and query lengths differ. get around with it by manually constructing mask
lucidrains
committedMar 24, 2023
switch to rotary positional embedding with xpos, to prepare for flash attention
lucidrains
committedMar 24, 2023
product management
lucidrains
committedMar 24, 2023
shoot for the moon
lucidrains
committedMar 24, 2023

Commits on Feb 27, 2023

address #1
lucidrains
committedFeb 27, 2023

Commits on Feb 26, 2023

allow for returning memories and states during training, for potentially training on RL
lucidrains
committedFeb 26, 2023

Commits on Feb 23, 2023

negative result on experiment
lucidrains
committedFeb 23, 2023
product management
lucidrains
committedFeb 23, 2023

Commits on Feb 22, 2023

project management
lucidrains
committedFeb 22, 2023
leave comment in readme
lucidrains
committedFeb 22, 2023
0.0.17
lucidrains
committedFeb 22, 2023
default to qk norm for only the xl and recurrent layers (with state vectors), but allow for overriding to use qk norm throughout network
lucidrains
committedFeb 22, 2023
add single headed kv
lucidrains
committedFeb 22, 2023

Commits on Feb 21, 2023

seems to work
lucidrains
committedFeb 21, 2023