Make dspy.Evaluate Typed and DSPy Only (No dsp) #889
Conversation
|
|
||
|
|
||
| class Evaluate(BaseModel): | ||
| # Example is not yet Pydantic valid |
There was a problem hiding this comment.
can remove this comment now :)
| ncorrect = 0 | ||
| ntotal = 0 | ||
| reordered_devset = [] | ||
|
|
||
| pbar = tqdm.tqdm(total=len(devset), dynamic_ncols=True, disable=not display_progress, file=sys.stdout) | ||
| for idx, arg in devset: | ||
| with logging_redirect_tqdm(): | ||
| print(f"IDX: {idx}, {arg}, {type(idx)}, {type(arg)}") |
There was a problem hiding this comment.
should this be printed or logged and outputted in a verbose/debug setting?
|
|
||
|
|
||
| def answer_exact_match(example: Example, pred: Prediction, frac: float = 0.90, *_args, **_kwargs) -> bool: | ||
| if not isinstance(example.answer, (str, list)): |
There was a problem hiding this comment.
fine to leave as original assert statement?
|
|
||
|
|
||
| def answer_passage_match(example: Example, pred: Prediction, *_args, **_kwargs): | ||
| if not isinstance(example.answer, (str, list)): |
There was a problem hiding this comment.
same comment as above
| return dsp.answer_match(pred.answer, [example.answer], frac=frac) | ||
| else: # type(example.answer) is list | ||
| return dsp.answer_match(pred.answer, example.answer, frac=frac) | ||
| def em(prediction: str, answers_list: list[str]) -> bool: |
| return dsp.passage_match(pred.context, [example.answer]) | ||
| else: # type(example.answer) is list | ||
| return dsp.passage_match(pred.context, example.answer) | ||
| def f1(prediction: str, answers_list: list[str]) -> float: |
|
Thanks for this PR @KCaverly ! As we're moving the metrics from dsp to dspy, it seems like we are missing some metrics in there (novel_f1_score, precision_score, hotpot_f1_score, HotPotF1, nF1). Granted, these are not used in the repo but maybe we can keep them for now until a greater refactor (similar to the case with datasets). We can probably revisit this later, but with this change, should be good to remove them from the dsp/ folder now. Also feel free to check PR #935 since there may be some overlap/changes. |
|
LGTM |
In an effort to harden the Evaluate function, I have moved all metrics from dsp to dspy, added type hints and made Evaluate Pydantic validated. This should not affect any fundamental functionality, and is simply an effort to increase code quality and coverage.