Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

train probe per prompt #271

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

train probe per prompt #271

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
47bcfb2
added cli arg
derpyplops Jul 13, 2023
a50fe57
refactor reporter training
derpyplops Jul 13, 2023
52b1394
WIP add multiprobe training
derpyplops Jul 14, 2023
2420ae0
multiprobe elicit works
derpyplops Jul 14, 2023
f35626a
fix pyright
derpyplops Jul 14, 2023
898c3f1
implemented multi probe for elicit
derpyplops Jul 18, 2023
01d5baa
undo list
derpyplops Jul 18, 2023
7701c29
add more types and sorting
derpyplops Jul 18, 2023
4310def
weird duplicate arg
derpyplops Jul 18, 2023
96a3dab
resolved circular import
derpyplops Jul 19, 2023
9c2def0
fixed index passing
derpyplops Jul 19, 2023
29eeb7f
fixed index passing again
derpyplops Jul 20, 2023
f533418
add assert
derpyplops Jul 20, 2023
0f5ce0b
fix prompt index in loading
derpyplops Jul 20, 2023
327d1eb
remove redundant method
derpyplops Jul 20, 2023
51b7d3c
correctly eval with multiple probes and some renaming
derpyplops Jul 20, 2023
75fe560
remove wrong function
derpyplops Jul 20, 2023
1b6757a
pyright
derpyplops Jul 20, 2023
0d6c8b9
pytest
derpyplops Jul 20, 2023
785537b
pyright
derpyplops Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove wrong function
  • Loading branch information
@derpyplops
derpyplops committed Jul 20, 2023
commit 75fe56099306368c81db40fd99409fd4607b8432
4 changes: 1 addition & 3 deletions elk/evaluation/evaluate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

from ..files import elk_reporter_dir
from ..metrics import evaluate_preds
from ..run import Run, select_data
from ..run import Run
from ..training.multi_reporter import MultiReporter, SingleReporter
from ..utils import Color

Expand Down Expand Up @@ -38,8 +38,6 @@
device = self.get_device(devices, world_size)
val_output = self.prepare_data(device, layer, "val")

val_output = select_data(val_output, self.prompt_indices)

experiment_dir = elk_reporter_dir() / self.source

def load_reporter() -> SingleReporter | MultiReporter:
Expand All @@ -62,7 +60,7 @@
prompt_index: int | Literal["multi"] | None = None,
i: int = 0,
):
prompt_index = (

Check failure on line 63 in elk/evaluation/evaluate.py

View workflow job for this annotation

GitHub Actions / run-tests (3.10, macos-latest) 

Expression of type "dict[str, int | str]" cannot be assigned to declared type "int | Literal['multi'] | None"
  Type "dict[str, int | str]" cannot be assigned to type "int | Literal['multi'] | None"
    "dict[str, int | str]" is incompatible with "int"
    Type cannot be assigned to type "None"
    "dict[str, int | str]" cannot be assigned to type "Literal['multi']" (reportGeneralTypeIssues)

Check failure on line 63 in elk/evaluation/evaluate.py

View workflow job for this annotation

GitHub Actions / run-tests (3.11, macos-latest) 

Expression of type "dict[str, int | str]" cannot be assigned to declared type "int | Literal['multi'] | None"
  Type "dict[str, int | str]" cannot be assigned to type "int | Literal['multi'] | None"
    "dict[str, int | str]" is incompatible with "int"
    Type cannot be assigned to type "None"
    "dict[str, int | str]" cannot be assigned to type "Literal['multi']" (reportGeneralTypeIssues)
{"prompt_index": prompt_index} if prompt_index is not None else {}
)
for ds_name, (val_h, val_gt, _) in val_output.items():
Expand All @@ -75,7 +73,7 @@
**meta,
"ensembling": mode,
**evaluate_preds(val_gt, val_credences, mode).to_dict(),
**prompt_index,

Check failure on line 76 in elk/evaluation/evaluate.py

View workflow job for this annotation

GitHub Actions / run-tests (3.10, macos-latest) 

Expected mapping for dictionary unpack operator (reportGeneralTypeIssues)

Check failure on line 76 in elk/evaluation/evaluate.py

View workflow job for this annotation

GitHub Actions / run-tests (3.11, macos-latest) 

Expected mapping for dictionary unpack operator (reportGeneralTypeIssues)
}
)

Expand Down
11 changes: 0 additions & 11 deletions elk/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,6 @@
PreparedData = dict[str, tuple[Tensor, Tensor, Tensor | None]]


def select_data(prepared_data: PreparedData, prompt_indices: list[int]):
return {
ds_name: (
train_h[:, prompt_indices, ...],
train_gt,
lm_preds[:, prompt_indices, ...] if lm_preds is not None else None,
)
for ds_name, (train_h, train_gt, lm_preds) in prepared_data.items()
}


@dataclass
class Run(ABC, Serializable):
data: Extract
Expand Down
Loading