Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add custom_ds_n CLI arg #187

Closed
wants to merge 5 commits into from
Closed

add custom_ds_n CLI arg #187

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
03e463c
add custom_ds_n CLI arg
reaganjlee Apr 13, 2023
c7d6ba3
cleanup comments + typechecking
reaganjlee Apr 13, 2023
f26bac1
cleanup comments + typechecking
reaganjlee Apr 13, 2023
62cb109
Merge branch 'renaming' of https://github.com/EleutherAI/elk into ren…
reaganjlee Apr 13, 2023
b0f6d17
Change to Path types
reaganjlee Apr 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add custom_ds_n CLI arg
  • Loading branch information
@reaganjlee
reaganjlee committed Apr 13, 2023
commit 03e463cec1f4a84f19c0ef79fd937e7717414ac1
5 changes: 4 additions & 1 deletion elk/evaluation/evaluate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,17 @@ class Eval(Serializable):
num_gpus: int = -1
skip_baseline: bool = False
concatenated_layer_offset: int = 0
custom_ds_n: str = None

def execute(self):
datasets = self.data.prompts.datasets

transfer_dir = elk_reporter_dir() / self.source / "transfer_eval"

for dataset in datasets:
run = Evaluate(cfg=self, out_dir=transfer_dir / dataset)
run = Evaluate(
cfg=self, out_dir=transfer_dir / dataset, custom_ds_n=self.custom_ds_n
)
run.evaluate()


Expand Down
4 changes: 4 additions & 0 deletions elk/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ class Run(ABC):
cfg: Union["Elicit", "Eval"]
out_dir: Optional[Path] = None
dataset: DatasetDict = field(init=False)
custom_ds_n: str = None

def __post_init__(self):
# Extract the hidden states first if necessary
Expand All @@ -44,6 +45,9 @@ def __post_init__(self):
# Save in a memorably-named directory inside of
# ELK_REPORTER_DIR/<model_name>/<dataset_name>
ds_name = ", ".join(self.cfg.data.prompts.datasets)
print(f"self.custom_root_name: {self.custom_ds_n}")
if self.custom_ds_n:
ds_name = self.custom_ds_n
root = elk_reporter_dir() / self.cfg.data.model / ds_name
Comment on lines +48 to 50
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we would need to do the same for models too.
However, I feel like, what we want instead maybe, is to be able to set a custom path for the whole root ... or at least what follows after elk_reporter_dirt()


self.out_dir = memorably_named_dir(root)
Expand Down
5 changes: 4 additions & 1 deletion elk/training/train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,12 @@ class Elicit(Serializable):
# if nonzero, appends the hidden states of layer concatenated_layer_offset before
debug: bool = False
out_dir: Optional[Path] = None
custom_ds_n: str = None

def execute(self):
train_run = Train(cfg=self, out_dir=self.out_dir)
print(f"at train.py the out_dir is currently: {self.out_dir}")
print(f"at train.py the custom_root_name is currently: {self.custom_ds_n}")
train_run = Train(cfg=self, out_dir=self.out_dir, custom_ds_n=self.custom_ds_n)
train_run.train()


Expand Down