Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Draft] RWKV LM #207

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

[Draft] RWKV LM #207

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
3f33328
Begin scaffolfing rwkv hf interface
Kyle1668 Apr 17, 2023
fa55ee3
RWKV works maybe?
Kyle1668 Apr 18, 2023
b7b8aec
prog
Kyle1668 Apr 18, 2023
d94132d
Pivot to getting the token embeddings
Kyle1668 Apr 22, 2023
d61375c
Additional notebook tests
Kyle1668 Apr 22, 2023
69194ee
Reset extraction
Kyle1668 Apr 22, 2023
c84a74d
Add instantiate_config(cfg.model)
Kyle1668 Apr 22, 2023
15baa83
Refactored RWKV and merged from main
Kyle1668 Apr 22, 2023
f4c88d5
Try exposing hiddens
Kyle1668 Apr 24, 2023
1a0dcae
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 24, 2023
53e3f04
Add HF hub download links within the model code
Apr 25, 2023
55ad3b4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 25, 2023
2fa92d6
Got better than random results
Kyle1668 Apr 27, 2023
aeb1f20
Merge from main
Kyle1668 May 6, 2023
6c6a324
Got good results with post-ln
Kyle1668 May 6, 2023
2536e5a
Stop tracking notebooks in git
Kyle1668 May 6, 2023
dc173fb
Stop tracking VS code debugging config
Kyle1668 May 6, 2023
a970761
Remove custom approach and begin generalizing
Kyle1668 May 6, 2023
7ae8637
Further refactoring
Kyle1668 May 6, 2023
086ecc0
Furhter cleanup
Kyle1668 May 6, 2023
14f66e1
Undo settings
Kyle1668 May 6, 2023
2102b4b
Oly get one hidden state per layer
Kyle1668 May 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
prog
  • Loading branch information
@Kyle1668
Kyle1668 committed Apr 18, 2023
commit b7b8aec817b6f7360be041e499ea0a0e8db821a2
2 changes: 1 addition & 1 deletion elk/rwkv_lm/rwkv_hf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ class RWKVConfig(PretrainedConfig):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.hidden_size = 2048
self.num_hidden_layers = 1
self.num_hidden_layers = 120
self.is_encoder_decoder = False
self.architectures = ["RWKV-LM"]

Expand Down