Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow for arbitrary hyperparameter selection for sweep #198

Closed
wants to merge 6 commits into from
Closed

Allow for arbitrary hyperparameter selection for sweep #198

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6eb18a4
Refactor metrics into evaluate_preds
norabelrose Apr 17, 2023
9f5759b
Fix stupid CCS bug
norabelrose Apr 17, 2023
14d1323
Cluster bootstrap for AUROC; boost default sample size
norabelrose Apr 17, 2023
bc3f29c
Cluster bootstrap for accuracy
norabelrose Apr 17, 2023
b829538
Allow for arbitrary hparam selection in sweep
norabelrose Apr 17, 2023
9c59751
Merge remote-tracking branch 'origin/main' into sweep-extra
norabelrose Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Allow for arbitrary hparam selection in sweep
  • Loading branch information
@norabelrose
norabelrose committed Apr 17, 2023
commit b829538c78dd7aa84280bd8a1134b30b2962d034
24 changes: 15 additions & 9 deletions elk/training/sweep.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from copy import deepcopy
from dataclasses import InitVar, dataclass

from ..extraction import Extract, PromptConfig
Expand All @@ -18,6 +19,14 @@ class Sweep:

name: str | None = None

# A bit of a hack to add all the command line arguments from Elicit
run_template: Elicit = Elicit(
data=Extract(
model="<placeholder>",
prompts=PromptConfig(datasets=["<placeholder>"]),
)
)

def __post_init__(self, add_pooled: bool):
if not self.datasets:
raise ValueError("No datasets specified")
Expand Down Expand Up @@ -49,12 +58,9 @@ def execute(self):
# plus signs. This means we can pool datasets together inside of a
# single sweep.
datasets = [ds.strip() for ds in dataset_str.split("+")]
Elicit(
data=Extract(
model=model_str,
prompts=PromptConfig(
datasets=datasets,
),
),
out_dir=out_dir,
).execute()

run = deepcopy(self.run_template)
run.data.model = model_str
run.data.prompts.datasets = datasets
run.out_dir = out_dir
run.execute()