Add option to save probabilities #289
Conversation
| @@ -31,7 +31,7 @@ def execute(self, highlight_color: Color = "cyan"): | |||
| @torch.inference_mode() | |||
| def apply_to_layer( | |||
| self, layer: int, devices: list[str], world_size: int | |||
| ) -> dict[str, pd.DataFrame]: | |||
| ) -> tuple[dict[str, pd.DataFrame], dict]: | |||
There was a problem hiding this comment.
Maybe you could do something similar to here: https://github.com/EleutherAI/elk/pull/259/files#diff-d13b83b80dc8fe2ae73e22669dd7a1a3167a1ae731d341fa96f03a766d877933R37
🟢
Instead of having tuple[dict[str, pd.DataFrame], dict]
But we can also leave it for now, and once we merge our pull-request it will be changed anyway
| { | ||
| **meta, | ||
| "ensembling": mode, | ||
| "inlp_iter": i, |
There was a problem hiding this comment.
Iterated nullspace projection iteration for the logistic regression model.
| @@ -11,7 +11,7 @@ | |||
| from simple_parsing import subgroups | |||
There was a problem hiding this comment.
The function apply_to_layer was already a bit long. I think we should refactor it a bit in a second pull-request
| get_logprobs(val_lr_credences, mode).detach().cpu() | ||
| ) | ||
|
|
||
| row_bufs["train_lr_eval"].append( |
There was a problem hiding this comment.
the names are getting a bit confusing. Maybe we should have a subfolder
"evals" containing
- trainset_lr.csv
- validationset_lr.csv
- trainset_ccs.csv
- etc.
There was a problem hiding this comment.
(Could be also included in a second pull-request) 🟢
|
included in #292 |
prepare_datato return aLayerDataobjecttrain_lr_eval.csvto results dir