Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

train probe per prompt #271

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

train probe per prompt #271

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
47bcfb2
added cli arg
derpyplops Jul 13, 2023
a50fe57
refactor reporter training
derpyplops Jul 13, 2023
52b1394
WIP add multiprobe training
derpyplops Jul 14, 2023
2420ae0
multiprobe elicit works
derpyplops Jul 14, 2023
f35626a
fix pyright
derpyplops Jul 14, 2023
898c3f1
implemented multi probe for elicit
derpyplops Jul 18, 2023
01d5baa
undo list
derpyplops Jul 18, 2023
7701c29
add more types and sorting
derpyplops Jul 18, 2023
4310def
weird duplicate arg
derpyplops Jul 18, 2023
96a3dab
resolved circular import
derpyplops Jul 19, 2023
9c2def0
fixed index passing
derpyplops Jul 19, 2023
29eeb7f
fixed index passing again
derpyplops Jul 20, 2023
f533418
add assert
derpyplops Jul 20, 2023
0f5ce0b
fix prompt index in loading
derpyplops Jul 20, 2023
327d1eb
remove redundant method
derpyplops Jul 20, 2023
51b7d3c
correctly eval with multiple probes and some renaming
derpyplops Jul 20, 2023
75fe560
remove wrong function
derpyplops Jul 20, 2023
1b6757a
pyright
derpyplops Jul 20, 2023
0d6c8b9
pytest
derpyplops Jul 20, 2023
785537b
pyright
derpyplops Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
pyright
  • Loading branch information
@derpyplops
derpyplops committed Jul 20, 2023
commit 1b6757a78518dcb035f3107b1486e1df0a43e993
4 changes: 2 additions & 2 deletions elk/evaluation/evaluate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def eval_all(
prompt_index: int | Literal["multi"] | None = None,
i: int = 0,
):
prompt_index = (
prompt_index_dict = (
{"prompt_index": prompt_index} if prompt_index is not None else {}
)
for ds_name, (val_h, val_gt, _) in val_output.items():
Expand All @@ -73,7 +73,7 @@ def eval_all(
**meta,
"ensembling": mode,
**evaluate_preds(val_gt, val_credences, mode).to_dict(),
**prompt_index,
**prompt_index_dict,
}
)

Expand Down
5 changes: 4 additions & 1 deletion elk/training/multi_reporter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,11 @@
if reporter[0].train_loss is not None
else None
)

self.train_loss = (
sum(train_losses) / len(train_losses) if train_losses is not None else None
sum(train_losses) / len(train_losses)

Check failure on line 32 in elk/training/multi_reporter.py

View workflow job for this annotation

GitHub Actions / run-tests (3.10, ubuntu-latest) 

Argument of type "list[float | None]" cannot be assigned to parameter "__iterable" of type "Iterable[_SupportsSumNoDefaultT@sum]" in function "sum"
  "list[float | None]" is incompatible with "Iterable[_SupportsSumNoDefaultT@sum]"
    TypeVar "_T_co@Iterable" is covariant
      Type "float | None" cannot be assigned to type "_SupportsSumWithNoDefaultGiven"
        Type "float | None" cannot be assigned to type "_SupportsSumWithNoDefaultGiven"
          "__add__" is not present
          "__radd__" is not present (reportGeneralTypeIssues)
if train_losses is not None # type: ignore
else None
)

def __call__(self, h):
Expand Down
1 change: 0 additions & 1 deletion elk/training/train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,6 @@ def apply_to_layer(
str_i = str(prompt_index).zfill(2)
base = self.out_dir / "reporters" / f"prompt_{str_i}"
reporters_path = base / "reporters"
base / "lr_models"

reporter_train_result = self.train_and_save_reporter(
device, layer, reporters_path, prompt_train_dict, prompt_index
Expand Down