Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Python 3.11 to CI tests #91

Merged
merged 18 commits into from
Feb 19, 2023
Merged

Add Python 3.11 to CI tests #91

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
608bbc7
load dset
AlexWan0 Feb 16, 2023
0464c22
use Datasets.from_generator for multiprocessing
AlexWan0 Feb 18, 2023
f66ef4c
revert some debugging stuff
AlexWan0 Feb 18, 2023
f2b7abf
remove accidentally included files
AlexWan0 Feb 18, 2023
5717d19
switch to correct multiprocessing lib + switch to spawn
AlexWan0 Feb 18, 2023
189bb30
split dataset before multiprocessing
AlexWan0 Feb 18, 2023
5af52d7
nicer prints
AlexWan0 Feb 18, 2023
3862e92
Replace custom method with np.array_split
norabelrose Feb 18, 2023
03b782d
Merge branch 'main' into multiprocessing
norabelrose Feb 19, 2023
9b3bbe3
Move multiprocessing inside generator
norabelrose Feb 19, 2023
f5b8a4a
change yield to put
AlexWan0 Feb 19, 2023
54f7f9a
change to use model_str
AlexWan0 Feb 19, 2023
bfb7955
fix connectionreseterror
AlexWan0 Feb 19, 2023
c6e6dd1
switch extract to return dataset instead of saving to disk
AlexWan0 Feb 19, 2023
5e9662e
Switch to torch.multiprocessing.spawn
norabelrose Feb 19, 2023
a0a7c81
Merge branch 'main' into multiprocessing
norabelrose Feb 19, 2023
5a950a1
Merge remote-tracking branch 'origin/main' into multiprocessing
norabelrose Feb 19, 2023
27e3759
Start testing on Python 3.11
norabelrose Feb 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix connectionreseterror
  • Loading branch information
@AlexWan0
AlexWan0 committed Feb 19, 2023
commit bfb795527e59c4867891d58d8df748347a575597
9 changes: 7 additions & 2 deletions elk/extraction/extraction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,9 +215,11 @@ def reduce_seqs(
output_hidden_states=True,
)
# [batch_size, num_layers, num_choices, hidden_size]
# need to convert hidden states to numpy array first or
# you get a ConnectionResetErrror
queue.put(
{
"hiddens": torch.stack(outputs.decoder_hidden_states, dim=2),
"hiddens": torch.stack(outputs.decoder_hidden_states, dim=2).cpu().numpy(),
"labels": labels,
}
)
Expand All @@ -228,9 +230,12 @@ def reduce_seqs(

# Skip the input embeddings which are unlikely to be interesting
h = model(**choices, output_hidden_states=True).hidden_states[1:]

# need to convert hidden states to numpy array first or
# you get a ConnectionResetErrror
queue.put(
{
"hiddens": reduce_seqs(h, choices["attention_mask"]),
"hiddens": reduce_seqs(h, choices["attention_mask"]).cpu().numpy(),
"labels": labels,
}
)