This repository has been archived by the owner on Mar 21, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 142
Minor changes to CovidModel config parameters and updated report #554
Merged
Merged
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
66896c0
Update report and minor changes to config parameters
Shruthi42 18c4aa9
Fix or skip tests and update CHANGELOG.md
Shruthi42 57f0fe6
Address PR comments and pin cryptography
Shruthi42 44760ab
Remove cryptography from list of invalid packages
Shruthi42 9c68f17
Address PR comments
Shruthi42 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
# ------------------------------------------------------------------------------------------ | ||
# Copyright (c) Microsoft Corporation. All rights reserved. | ||
# Licensed under the MIT License (MIT). See LICENSE in the repo root for license information. | ||
# ------------------------------------------------------------------------------------------ | ||
|
||
from InnerEye.Common.common_util import ModelProcessing, get_best_epoch_results_path | ||
from InnerEye.Common.output_directories import OutputFolderForTests | ||
from InnerEye.Common.metrics_constants import LoggingColumns | ||
from InnerEye.ML.common import ModelExecutionMode | ||
from InnerEye.ML.configs.classification.CovidModel import CovidModel | ||
from InnerEye.ML.model_testing import MODEL_OUTPUT_CSV | ||
|
||
|
||
def test_generate_custom_report(test_output_dirs: OutputFolderForTests) -> None: | ||
""" | ||
Test that the Covid model report is generated correctly | ||
(especially when there are NaN values in the hierarchical task). | ||
""" | ||
|
||
model = CovidModel() | ||
model.set_output_to(test_output_dirs.root_dir) | ||
report_dir = test_output_dirs.root_dir / "reports" | ||
report_dir.mkdir() | ||
|
||
train_csv_path = model.outputs_folder / get_best_epoch_results_path(mode=ModelExecutionMode.TRAIN, | ||
model_proc=ModelProcessing.DEFAULT) \ | ||
/ MODEL_OUTPUT_CSV | ||
train_csv_path.parent.mkdir(parents=True) | ||
train_csv_path.write_text(f"""{LoggingColumns.Patient.value},{LoggingColumns.Hue.value},{LoggingColumns.Label.value},{LoggingColumns.ModelOutput.value},{LoggingColumns.CrossValidationSplitIndex.value} | ||
1,CVX0,1,0.7,-1 | ||
1,CVX1,0,0.1,-1 | ||
1,CVX2,0,0.1,-1 | ||
1,CVX3,0,0.1,-1 | ||
2,CVX0,0,0.1,-1 | ||
2,CVX1,1,0.7,-1 | ||
2,CVX2,0,0.1,-1 | ||
2,CVX3,0,0.1,-1 | ||
3,CVX0,0,0.7,-1 | ||
3,CVX1,0,0.1,-1 | ||
3,CVX2,1,0.1,-1 | ||
3,CVX3,0,0.1,-1 | ||
4,CVX0,0,0.0,-1 | ||
4,CVX1,0,1.0,-1 | ||
4,CVX2,0,0.0,-1 | ||
4,CVX3,1,0.0,-1 | ||
5,CVX0,0,0.0,-1 | ||
5,CVX1,0,0.0,-1 | ||
5,CVX2,1,1.0,-1 | ||
5,CVX3,0,0.0,-1 | ||
6,CVX0,0,0.0,-1 | ||
6,CVX1,1,1.0,-1 | ||
6,CVX2,0,0.0,-1 | ||
6,CVX3,0,0.0,-1 | ||
""") | ||
|
||
report_path = model.generate_custom_report(report_dir=report_dir, model_proc=ModelProcessing.DEFAULT) | ||
report_text = report_path.read_text() | ||
|
||
assert report_text == f"""{ModelExecutionMode.TRAIN.value} | ||
CVX03vs12 Accuracy: 0.6667 | ||
CVX0vs3 Accuracy: 1.0000 | ||
Warning: CVX0vs3 accuracy was computed skipping 1 NaN model outputs. | ||
CVX1vs2 Accuracy: 0.7500 | ||
Multiclass Accuracy: 0.6667 | ||
|
||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor: for this kind of statement I believe the intent would be clearer with
int(labels[1] or labels[2])
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, I meant
label_cvx03vs12 = int(labels[1] or labels[2])
. on second thought it may be better to revert to how it was initially, for consistency with the rest 🙂