Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Enable building an ensemble model from the cross validation checkpoints of a BYO Lightning model #529

Closed
wants to merge 101 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
1250de0
Use registered model for inference
Shruthi42 Jun 23, 2021
efc34fb
Merge branch 'main' into shbannur/load_registered_models
Shruthi42 Jun 23, 2021
f94b2e9
Bug fix
Shruthi42 Jun 23, 2021
7430785
Fix tests
Shruthi42 Jun 23, 2021
229b7ea
Fix tests
Shruthi42 Jun 23, 2021
16a09c3
mypy
Shruthi42 Jun 23, 2021
ee957d8
Fix tests
Shruthi42 Jun 23, 2021
596efee
Add tests
Shruthi42 Jun 24, 2021
2c2160f
Fix tests
Shruthi42 Jun 24, 2021
fe6fa93
Fix tests
Shruthi42 Jun 24, 2021
45895e7
Add test
Shruthi42 Jun 24, 2021
41f1b48
Fix tests
Shruthi42 Jun 24, 2021
f6bb7a2
Fix test
Shruthi42 Jul 5, 2021
3e3b069
Remove unnecessary function
Shruthi42 Jul 5, 2021
c0de1e6
Update tests
Shruthi42 Jul 5, 2021
0889a88
Flake8
Shruthi42 Jul 5, 2021
f98b22e
Fix tests
Shruthi42 Jul 5, 2021
d73f769
mypy
Shruthi42 Jul 5, 2021
f4dfbe7
Merge branch 'main' into shbannur/load_registered_models
Shruthi42 Jul 5, 2021
2767e18
Loosening multiple checkpoint check in run_inference_for_lightning_mo…
Jul 6, 2021
8a63e0a
WiP very scrappy!
Jul 7, 2021
116e566
WiP more mess
Jul 8, 2021
7107e27
WiP: bones of test class
Jul 8, 2021
d0c7724
Refactoring run_inference_for_lightning_models
Jul 9, 2021
6735d2c
mypy fixes
Jul 9, 2021
585511b
Merge branch 'main' into timregan/527-ensembles-for-BYOL-xval
Jul 9, 2021
4df8c09
WiP annotations and test
Jul 10, 2021
05a67dc
WiP: saving mid task for lunch
dumbledad Jul 10, 2021
8d93907
Correcting GPU -> CPU typo in comment
dumbledad Jul 11, 2021
e7ba7f1
Method can be static
dumbledad Jul 11, 2021
e4ebfcb
WiP fiddling
dumbledad Jul 11, 2021
7625a48
Example ensemble from InnerEyeInference
dumbledad Jul 11, 2021
f5b288c
flake8 and mypy fixes
dumbledad Jul 11, 2021
a79673f
mypy fixes
dumbledad Jul 11, 2021
3e9ec7e
tidying unused parameters
Jul 11, 2021
559a717
WiP simple temp test for train/test ensemble
Jul 11, 2021
f7be221
Renaming InnerEyeInference methods
Jul 12, 2021
a3179a8
tidy up
Jul 12, 2021
f7bdc7f
Matching new naming
Jul 12, 2021
939ec5a
renaming params
Jul 12, 2021
f206057
naming
Jul 12, 2021
d71f498
Unit test WiP
Jul 12, 2021
6987f99
don't be strict with state_dict
dumbledad Jul 12, 2021
f6e025f
first unit test takes shape
dumbledad Jul 12, 2021
500bf11
Unit test works, but doesn't check much
dumbledad Jul 12, 2021
0d97fe9
renaming unit test
Jul 13, 2021
b204f08
Merge branch 'main' into shbannur/load_registered_models
Shruthi42 Jul 13, 2021
dd17d78
Change docstring
Shruthi42 Jul 13, 2021
388e0a8
Update CHANGELOG.md
Shruthi42 Jul 13, 2021
d727fe1
Rename
Shruthi42 Jul 13, 2021
fa7a6e4
Fix test
Shruthi42 Jul 13, 2021
581c6a9
WiP ensemble unit test with value check
Jul 13, 2021
1134c0f
Merge branch 'main' into timregan/527-ensembles-for-BYOL-xval
Jul 14, 2021
e35db5b
Address PR comments
Shruthi42 Jul 14, 2021
9093b7e
Use list of pytest markers
Shruthi42 Jul 14, 2021
bf072c0
Move model_id to WorkflowParams
Shruthi42 Jul 14, 2021
5a76cc1
missed some name changes
Jul 14, 2021
2d75d24
WiP swapping back to checkpoints not accruing child runs
Jul 14, 2021
e064483
Refactor extra_downloaded_run_id
Shruthi42 Jul 14, 2021
168eb29
unit test working
Jul 14, 2021
838bb48
Update documentation and argparser
Shruthi42 Jul 14, 2021
0f3690c
flake & mypy
Jul 14, 2021
9c7f6b4
Revert changes to generic_parsing
Shruthi42 Jul 14, 2021
d612c6e
Update documentation
Shruthi42 Jul 14, 2021
6861178
Flake8 and mypy
Shruthi42 Jul 14, 2021
98c7683
Shruthi's changes to run_ml
Jul 14, 2021
54c845e
Merge branch 'shbannur/load_registered_models' into timregan/527-ense…
Jul 14, 2021
48aca37
WiP abstracting ensemble inference
Jul 14, 2021
d3d8477
WiP
dumbledad Jul 15, 2021
a4ea25a
Ensemble inference base
dumbledad Jul 15, 2021
5e7f0b1
Merge branch 'main' into timregan/527-ensembles-for-BYOL-xval
dumbledad Jul 15, 2021
a71fa90
Ended up with changes to 2 files I did not touch!
dumbledad Jul 15, 2021
0874a74
Restoring (and fixing) run_ml changes
dumbledad Jul 16, 2021
88bb96d
mypy
dumbledad Jul 16, 2021
424b2e7
WiP
dumbledad Jul 16, 2021
4903b96
run_ml unit test v1
Jul 16, 2021
547ed7d
WiP pre pruning ensemble stuff
Jul 16, 2021
538a7a4
refactored to avoid recursion blow-up
Jul 16, 2021
002886c
Merge branch 'main' into timregan/527-ensembles-for-BYOL-xval
dumbledad Jul 17, 2021
c6ca034
additional comments and remove inheritance
dumbledad Jul 17, 2021
484fe4a
removing duplicated unit test
dumbledad Jul 17, 2021
b771137
more comments
dumbledad Jul 17, 2021
6d57d08
test tidy
dumbledad Jul 17, 2021
6a9fd75
flake fixes
dumbledad Jul 17, 2021
cea51e4
WiP
dumbledad Jul 17, 2021
9fc696a
on_ensemble_inference_start needn't call down
dumbledad Jul 17, 2021
b947452
Old WiP changes
dumbledad Jul 17, 2021
f7d22f4
Adding HelloEnsembleInference
dumbledad Jul 17, 2021
af079ae
run_ml changes with parameter
dumbledad Jul 17, 2021
38b12b7
Wi{ on mypy and tidy pre unit test fix
dumbledad Jul 18, 2021
c255774
mypy fixes
dumbledad Jul 18, 2021
2db1857
import fix so test discovery works
dumbledad Jul 18, 2021
21044e6
Fixing test discovery
dumbledad Jul 18, 2021
4f10169
WiP fixing unit test
dumbledad Jul 18, 2021
e0e31e6
Unit test works
dumbledad Jul 18, 2021
1d23735
file system test
dumbledad Jul 18, 2021
1486bff
Adding register and actually building ensemble
Jul 19, 2021
ff8fef1
Removed call to innnereye_config
Jul 19, 2021
ddb3c9f
unit test fix
dumbledad Jul 22, 2021
e71d49b
fix for the reference error on AzureML
dumbledad Jul 23, 2021
db18fdd
Merge branch 'main' into timregan/527-ensembles-for-BYOL-xval
dumbledad Jul 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
unit test fix
  • Loading branch information
dumbledad committed Jul 22, 2021
commit ddb3c9f698e0c2cea17a2f706070de462325c516
12 changes: 7 additions & 5 deletions InnerEye/ML/run_ml.py
Original file line number Diff line number Diff line change
Expand Up @@ -439,11 +439,13 @@ def run(self) -> None:

# As above for InnerEyeContainers, if this is a cross validation run, and the present run is child run
# 0, then wait for the sibling runs, build the ensemble model, and write a report for that.
sibling_runs_checkpoint_handler = self.wait_and_collect_sibling_runs_if_required()
if sibling_runs_checkpoint_handler:
self.create_ensemble_model_and_run_inference_from_lightningmodule_checkpoints(
self.container.model,
sibling_runs_checkpoint_handler.get_best_checkpoints())
if not self.is_offline_run and PARENT_RUN_CONTEXT is not None:
sibling_runs_checkpoint_handler = self.wait_and_collect_sibling_runs_if_required()
logging.info("DEBUGGING: about to create_ensemble_model_and_run_inference_from_lightningmodule_checkpoints")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use logging.debug instead?

if sibling_runs_checkpoint_handler:
self.create_ensemble_model_and_run_inference_from_lightningmodule_checkpoints(
self.container.model,
sibling_runs_checkpoint_handler.get_best_checkpoints())

if self.container.regression_test_folder:
# Comparison with stored results for cross-validation runs only operates on child run 0. This run
Expand Down