spar mt prompt invar #184

ChristyKoh · 2023-04-12T12:13:58Z

Fixes #172

add merge_templates_from method to DatasetTemplates
generate combined templates.yaml when PromptConfig is initialized

currently, checks for alignment of col number and ClassLabel classes

for more information, see https://pre-commit.ci

…pt_invar

for more information, see https://pre-commit.ci

…pt_invar

elk/extraction/extraction.py

elk/extraction/prompt_loading.py

for more information, see https://pre-commit.ci

… spar_mt_prompt_invar

…atch of single-dataset evals

lauritowal

Thanks for the pull-request! Just tried it out and it seems to work. You might want to split up your code into multiple functions to make it a bit more readable. I've added some comments.

elk/extraction/prompt_loading.py

…g logic into functions

for more information, see https://pre-commit.ci

… spar_mt_prompt_invar

ChristyKoh and others added 30 commits March 14, 2023 05:16

add boolq_pt template and christykoh as included user

7b68f5f

add templates.yaml for boolqpt

edfdd6d

pt yaml

d285798

Merge remote-tracking branch 'origin/main' into spar_boolq_pt

6c91be2

add ag_news template, translated to pt

0ff1609

add ag_news template, translated to pt

7180a64

save eval runs to separate subfolders by target dataset

b8f5e8b

[pre-commit.ci] auto fixes from pre-commit.com hooks

51fed5d

for more information, see https://pre-commit.ci

Merge branch 'main' into spar_pt

9cebbdd

Merge branch 'spar_boolq_pt' into spar_pt

6be825a

eval multiple datasets

2660844

Merge branch 'eval_dirs' of github.com:EleutherAI/elk into eval_dirs

f35a6a8

change prompt answer chouces to portuguese

b230209

add imdb_pt template

74c9915

implement prompt sharing, generate combined templates.yaml

85fd9e4

fix num templates logic

8383f26

fix pt answer choice

df41ab4

Merge branch 'spar_pt' of github.com:EleutherAI/elk into spar_mt_prom…

c03742d

…pt_invar

Merge branch 'main' of github.com:EleutherAI/elk into main

e536aa6

Merge branch 'main' into spar_pt

208200d

[pre-commit.ci] auto fixes from pre-commit.com hooks

a132f40

for more information, see https://pre-commit.ci

Merge branch 'spar_pt' of github.com:EleutherAI/elk into spar_mt_prom…

9f3f218

…pt_invar

remove empty prompt_dataset file

c71bf1c

fix empty prompters bug

1ec5787

fix multiclass label bug

66c7a6b

move prompt combination to PromptConfig post_init logic

d91acac

Merge branch 'main' of github.com:EleutherAI/elk into main

89b2346

Merge branch 'main' into spar_mt_prompt_invar

715bba8

fix refactor bugs, runnable state

0c2f5c4

rewrite template merging, regenerate prompter every run

066cd44

ChristyKoh requested a review from norabelrose April 12, 2023 12:27

lauritowal requested changes Apr 12, 2023

View reviewed changes

elk/extraction/extraction.py Outdated Show resolved Hide resolved

elk/extraction/prompt_loading.py Outdated Show resolved Hide resolved

elk/extraction/prompt_loading.py Outdated Show resolved Hide resolved

ChristyKoh and others added 9 commits April 12, 2023 17:20

Merge remote-tracking branch 'origin/main' into spar_mt_prompt_invar

34bc364

update README with prompt invariance argument

2da069a

fix bugs, add dataset col checks

4c6d344

[pre-commit.ci] auto fixes from pre-commit.com hooks

53d186b

for more information, see https://pre-commit.ci

fix prompter init typing

b78355f

Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…

3486ded

… spar_mt_prompt_invar

Update README.md

1975410

try to fix typing again

8a5fb0d

Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…

6d46f7e

… spar_mt_prompt_invar

ChristyKoh marked this pull request as ready for review April 12, 2023 20:24

assert datasettemplates type

a7f5a8b

ChristyKoh self-assigned this Apr 12, 2023

ChristyKoh added 3 commits April 13, 2023 09:11

Merge remote-tracking branch 'origin/main' into eval_dirs

ba090b7

bugfix to run eval separately on each dataset

7af1a1b

add combine_evals flag to differentiate a multi dataset eval from a b…

74551fa

…atch of single-dataset evals

lauritowal requested changes Apr 16, 2023

View reviewed changes

ChristyKoh and others added 8 commits April 19, 2023 06:11

Merge branch 'main' of github.com:EleutherAI/elk into main

0b3d3c9

Merge branch 'main' into eval_dirs

d8cee8b

Merge branch 'eval_dirs' into spar_mt_prompt_invar, separate combinin…

54aa710

…g logic into functions

[pre-commit.ci] auto fixes from pre-commit.com hooks

f7a4713

for more information, see https://pre-commit.ci

define ds_name

6e2f54c

Merge branch 'spar_mt_prompt_invar' of github.com:EleutherAI/elk into…

3b0765d

… spar_mt_prompt_invar

fix ds_name bug

5f0f32a

Update README.md

8fa07b4

lauritowal approved these changes Apr 20, 2023

View reviewed changes

lauritowal merged commit 89a2122 into main Apr 20, 2023
4 checks passed

lauritowal deleted the spar_mt_prompt_invar branch April 20, 2023 11:46

norabelrose mentioned this pull request Apr 20, 2023

Revert "spar mt prompt invar" #205

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spar mt prompt invar #184

spar mt prompt invar #184

ChristyKoh commented Apr 12, 2023 •

edited

lauritowal left a comment

spar mt prompt invar #184

spar mt prompt invar #184

Conversation

ChristyKoh commented Apr 12, 2023 • edited

lauritowal left a comment

Choose a reason for hiding this comment

ChristyKoh commented Apr 12, 2023 •

edited