Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi datasets #123

Merged
merged 46 commits into from
Mar 28, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
681698d
add multiple datasets support
Benw8888 Mar 9, 2023
ac1b9f1
Merge branch 'main' of github.com:EleutherAI/elk into multi-datasets
Benw8888 Mar 9, 2023
b864c77
train_reporter works on a list of layers now
Benw8888 Mar 10, 2023
7d7d97c
changing printed layer names
Benw8888 Mar 10, 2023
4fe61e9
fixed concatenation bug
Benw8888 Mar 11, 2023
fe61d67
minor edits
Benw8888 Mar 13, 2023
74da878
fixed pyright issues
Benw8888 Mar 13, 2023
569ef05
Merge branch 'main' of github.com:EleutherAI/elk into multi-datasets
Benw8888 Mar 14, 2023
b62b679
Merge branch 'main' into multi-datasets
norabelrose Mar 20, 2023
fe94c22
Fix tests
norabelrose Mar 20, 2023
bba24d8
Now working sorta
norabelrose Mar 22, 2023
03ba6e0
Skip slow BalancedBatchSampler test
norabelrose Mar 22, 2023
15ab351
Slightly relax test_output_is_roughly_balanced
norabelrose Mar 22, 2023
a80369e
Make BalancedSampler deterministic
norabelrose Mar 22, 2023
d304ab3
InitVar
norabelrose Mar 22, 2023
761c82d
Support multi class again
norabelrose Mar 22, 2023
f29743b
Fix naming issue
norabelrose Mar 22, 2023
b7b7e23
Support few shot prompts
norabelrose Mar 23, 2023
1afb563
Merge branch 'main' into multi-datasets
norabelrose Mar 23, 2023
225d4c7
fix multiclass labels
AlexTMallen Mar 23, 2023
9368dc8
Merge branch 'multi-datasets' of github.com:EleutherAI/elk into multi…
AlexTMallen Mar 23, 2023
a858b65
Merge branch 'main' into multi-datasets
norabelrose Mar 24, 2023
5dc2ec6
Merge branch 'multi-datasets' of github.com:EleutherAI/elk into multi…
norabelrose Mar 24, 2023
b1b95e5
Fix dumb part of test failures
norabelrose Mar 25, 2023
ee3911e
Fix assert_allclose warning
norabelrose Mar 25, 2023
a55b3de
Switch to torch.testing.assert_close in EigenReporter test
norabelrose Mar 25, 2023
44dc25c
Shuffle load_prompts output by default
norabelrose Mar 25, 2023
93d8d87
Fix smoke test failure
norabelrose Mar 25, 2023
fad4d74
Remove debug prints
AlexTMallen Mar 25, 2023
0a054f4
Remove more debug print statements
AlexTMallen Mar 25, 2023
177eec2
make min_memory usable; broadcast mmax_examples in __post_init__
AlexTMallen Mar 26, 2023
3a762b0
prompt loading refactor to enable better streaming
AlexTMallen Mar 26, 2023
f66c054
remove shuffle arg
AlexTMallen Mar 26, 2023
d3d87fc
remove unused @dataclass
lauritowal Mar 26, 2023
3d08147
merge
lauritowal Mar 27, 2023
c9a43e1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 27, 2023
94290aa
add concatenated_layer_offset to eval
lauritowal Mar 27, 2023
f9298e4
Merge branch 'multi-datasets' of https://github.com/EleutherAI/elk in…
lauritowal Mar 27, 2023
3765c4f
add self.
lauritowal Mar 27, 2023
2b05193
replace target with data
lauritowal Mar 27, 2023
83731bb
add self.
lauritowal Mar 27, 2023
764fda9
remove second arg
lauritowal Mar 27, 2023
d2c66b0
fix passing the wrong params for world size / rank
thejaminator Mar 28, 2023
9186326
Update prompt_loading.py
lauritowal Mar 28, 2023
3f99a4d
fix pre-commit errors
lauritowal Mar 28, 2023
148130d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix tests
  • Loading branch information
norabelrose committed Mar 20, 2023
commit fe94c22c83579e093b610c317ede48524403b211
3 changes: 2 additions & 1 deletion tests/distilgpt2_copa_cfg.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ layers: []
model: distilgpt2
prompts:
balance: true
dataset: super_glue copa
datasets:
- "super_glue copa"
label_column: null
max_examples:
- 2
Expand Down
3 changes: 2 additions & 1 deletion tests/distilgpt2_dbpedia_cfg.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ layers: []
model: distilgpt2
prompts:
balance: true
dataset: dbpedia_14
datasets:
- "dbpedia_14"
label_column: null
max_examples:
- 3
Expand Down
2 changes: 1 addition & 1 deletion tests/test_prompt_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
def test_prompt_dataset_getitem_boolq():
def test_prompt_dataset_getitem(cfg: ExtractionConfig, split: str):
prompt_ds = PromptDataset(cfg.prompts, rank=0, world_size=1, split=split)
ds_name, _, config_name = cfg.prompts.dataset.partition(" ")
ds_name, _, config_name = cfg.prompts.datasets[0].partition(" ")

prompter = DatasetTemplates(ds_name, config_name or None)
assert len(prompt_ds) == cfg.prompts.max_examples[-1]
Expand Down