Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spar mt prompt invar #184

Merged
merged 55 commits into from
Apr 20, 2023
Merged

spar mt prompt invar #184

Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
7b68f5f
add boolq_pt template and christykoh as included user
ChristyKoh Mar 9, 2023
edfdd6d
add templates.yaml for boolqpt
ChristyKoh Mar 14, 2023
d285798
pt yaml
reaganjlee Mar 26, 2023
6c91be2
Merge remote-tracking branch 'origin/main' into spar_boolq_pt
reaganjlee Apr 5, 2023
0ff1609
add ag_news template, translated to pt
ChristyKoh Apr 5, 2023
7180a64
add ag_news template, translated to pt
ChristyKoh Apr 5, 2023
b8f5e8b
save eval runs to separate subfolders by target dataset
ChristyKoh Apr 7, 2023
51fed5d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 7, 2023
9cebbdd
Merge branch 'main' into spar_pt
ChristyKoh Apr 7, 2023
6be825a
Merge branch 'spar_boolq_pt' into spar_pt
ChristyKoh Apr 7, 2023
2660844
eval multiple datasets
ChristyKoh Apr 7, 2023
f35a6a8
Merge branch 'eval_dirs' of github.com:EleutherAI/elk into eval_dirs
ChristyKoh Apr 7, 2023
b230209
change prompt answer chouces to portuguese
ChristyKoh Apr 7, 2023
74c9915
add imdb_pt template
ChristyKoh Apr 11, 2023
85fd9e4
implement prompt sharing, generate combined templates.yaml
ChristyKoh Apr 11, 2023
8383f26
fix num templates logic
ChristyKoh Apr 12, 2023
df41ab4
fix pt answer choice
ChristyKoh Apr 12, 2023
c03742d
Merge branch 'spar_pt' of github.com:EleutherAI/elk into spar_mt_prom…
ChristyKoh Apr 12, 2023
e536aa6
Merge branch 'main' of github.com:EleutherAI/elk into main
ChristyKoh Apr 12, 2023
208200d
Merge branch 'main' into spar_pt
ChristyKoh Apr 12, 2023
a132f40
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 12, 2023
9f3f218
Merge branch 'spar_pt' of github.com:EleutherAI/elk into spar_mt_prom…
ChristyKoh Apr 12, 2023
c71bf1c
remove empty prompt_dataset file
ChristyKoh Apr 12, 2023
1ec5787
fix empty prompters bug
ChristyKoh Apr 12, 2023
66c7a6b
fix multiclass label bug
ChristyKoh Apr 12, 2023
d91acac
move prompt combination to PromptConfig post_init logic
ChristyKoh Apr 12, 2023
89b2346
Merge branch 'main' of github.com:EleutherAI/elk into main
ChristyKoh Apr 12, 2023
715bba8
Merge branch 'main' into spar_mt_prompt_invar
ChristyKoh Apr 12, 2023
0c2f5c4
fix refactor bugs, runnable state
ChristyKoh Apr 12, 2023
066cd44
rewrite template merging, regenerate prompter every run
ChristyKoh Apr 12, 2023
846b78c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 12, 2023
1aecd7e
line len fixes
ChristyKoh Apr 12, 2023
b7bbee0
Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…
ChristyKoh Apr 12, 2023
b0c0f63
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 12, 2023
34bc364
Merge remote-tracking branch 'origin/main' into spar_mt_prompt_invar
ChristyKoh Apr 12, 2023
2da069a
update README with prompt invariance argument
ChristyKoh Apr 12, 2023
4c6d344
fix bugs, add dataset col checks
ChristyKoh Apr 12, 2023
53d186b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 12, 2023
b78355f
fix prompter init typing
ChristyKoh Apr 12, 2023
3486ded
Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…
ChristyKoh Apr 12, 2023
1975410
Update README.md
lauritowal Apr 12, 2023
8a5fb0d
try to fix typing again
ChristyKoh Apr 12, 2023
6d46f7e
Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…
ChristyKoh Apr 12, 2023
a7f5a8b
assert datasettemplates type
ChristyKoh Apr 12, 2023
ba090b7
Merge remote-tracking branch 'origin/main' into eval_dirs
ChristyKoh Apr 13, 2023
7af1a1b
bugfix to run eval separately on each dataset
ChristyKoh Apr 13, 2023
74551fa
add combine_evals flag to differentiate a multi dataset eval from a b…
ChristyKoh Apr 13, 2023
0b3d3c9
Merge branch 'main' of github.com:EleutherAI/elk into main
ChristyKoh Apr 19, 2023
d8cee8b
Merge branch 'main' into eval_dirs
ChristyKoh Apr 19, 2023
54aa710
Merge branch 'eval_dirs' into spar_mt_prompt_invar, separate combinin…
ChristyKoh Apr 19, 2023
f7a4713
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 19, 2023
6e2f54c
define ds_name
ChristyKoh Apr 19, 2023
3b0765d
Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…
ChristyKoh Apr 19, 2023
5f0f32a
fix ds_name bug
ChristyKoh Apr 20, 2023
8fa07b4
Update README.md
lauritowal Apr 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
  • Loading branch information
@pre-commit-ci
pre-commit-ci[bot] committed Apr 12, 2023
commit a132f4011b095f219bc50700540b3064658ea786
4 changes: 2 additions & 2 deletions elk/promptsource/templates/christykoh/boolq_pt/templates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ templates:
{% if label != -1 %}

{{answer_choices[label]}}

{% endif %}'
metadata: !TemplateMetadata
choices_in_prompt: true
Expand Down Expand Up @@ -112,7 +112,7 @@ templates:
{% if label != -1 %}

{{ answer_choices[label] }}

{% endif %}'
metadata: !TemplateMetadata
choices_in_prompt: false
Expand Down