Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[alpha] Improvements to ModelWrapper and better QA/Classification implementation #8

Merged
merged 16 commits into from
Mar 10, 2023
Merged

[alpha] Improvements to ModelWrapper and better QA/Classification implementation #8

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e44baa1
Add barebone unit test for jury format conversion for references
NISH1001 Feb 21, 2023
93b66d6
Add more tests for jury format conversion
NISH1001 Feb 21, 2023
619f6f8
Fix test function name
NISH1001 Feb 22, 2023
0b3657a
Add misc datasets loader module
NISH1001 Feb 22, 2023
13f9c56
Add device parameter to DefaultQAWrapper
NISH1001 Feb 22, 2023
3ab89a2
Update __init__ files with respective module imports
NISH1001 Feb 22, 2023
edbf0d1
Add metrics for computing semantics similarity
NISH1001 Feb 25, 2023
09239f3
Merge branch 'main' of github.com:NASA-IMPACT/evalem into feature/met…
NISH1001 Feb 27, 2023
55574fa
Add preprocessing and post-processing mechanism to ModelWrapper
NISH1001 Feb 27, 2023
7aee9f7
Update code documentation for HFPipelineWrapper docstring
NISH1001 Feb 27, 2023
2ba14a9
Implement HF pipeline wrapper/evaluator for text classification
NISH1001 Feb 28, 2023
a379446
Add per_instance_score flag to BertScore metric
NISH1001 Mar 1, 2023
a34165a
Merge branch 'main' of github.com:NASA-IMPACT/evalem into feature/met…
NISH1001 Mar 2, 2023
00e0150
Default to bert-base-uncased for BertScore
NISH1001 Mar 2, 2023
c0d92f3
Bugfix tokenizer parameter in QuestionAnsweringHFPipelineWrapper and
NISH1001 Mar 2, 2023
8e62cc7
Change BertScore's per_instance_score behaviour to compute mean
NISH1001 Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update code documentation for HFPipelineWrapper docstring
  • Loading branch information
@NISH1001
NISH1001 committed Feb 27, 2023
commit 7aee9f7e1daa9151fd6c3c322c897938cf35e01c
8 changes: 8 additions & 0 deletions evalem/models/_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,14 @@ class HFPipelineWrapper(HFWrapper):
pipe = hf_pipeline("question-answering")
wrapped_model = HFPipelineWrapper(pipe)

# Or: if you want to specify how to post-process predictions,
# provide the processor explicitly.
wrapped_model = HFPipelineWrapper(
pipeline("question-answering", model="deepset/roberta-base-squad2"),
predictions_postprocessor=lambda xs: list(map(lambda x: x["answer"], xs))
)


# compute predictions
# (format?) and pass to evaluator along with references
predictions = wrapped_model.predict(<inputs>)
Expand Down