Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Allow inference on ensemble models and disentangle run recovery from inference only runs. #377

Closed
Shruthi42 opened this issue Jan 28, 2021 · 0 comments
Closed

Allow inference on ensemble models and disentangle run recovery from inference only runs. #377

Shruthi42 opened this issue Jan 28, 2021 · 0 comments
Labels
architecture Anything around possible extensions or re-structuring of code

Comments

@Shruthi42
Copy link
Contributor

Shruthi42 commented Jan 28, 2021 •
edited by azure-boards bot

Currently, we use run recovery to both recover training on failed runs and re-run inference using models trained in previous runs. We should:

  • Support inference runs on ensemble models.
  • Restrict the use of run recovery to training run recovery.
  • Use the registered model to perform inference in inference only runs.

AB#3916
@ant0nsc ant0nsc added the architecture Anything around possible extensions or re-structuring of code label Feb 2, 2021
@Shruthi42 Shruthi42 mentioned this issue Feb 26, 2021
Closed
@ant0nsc ant0nsc added no changelog needed CHANGELOG.md does not need to be updated in this PR and removed no changelog needed CHANGELOG.md does not need to be updated in this PR labels Apr 9, 2021
@Shruthi42 Shruthi42 mentioned this issue May 12, 2021
Merged
Shruthi42 added a commit that referenced this issue May 12, 2021
@Shruthi42
Register all models after training, not only Segmentation models. (#455)
aa09b9d 
This PR changes the codepath so all models trained on AzureML are registered. The codepath previously allowed only segmentation models (subclasses of `SegmentationModelBase`) to be registered. Models are registered after a training run or if the `only_register_model` flag is set. Models may be legacy InnerEye config-based models or may be defined using the LightningContainer class.

The PR also removes the AzureRunner conda environment. The full InnerEye conda environment is needed to submit a training job to AzureML.

It splits the `TrainHelloWorldAndHelloContainer` job in the PR build into two jobs, `TrainHelloWorld` and `TrainHelloContainer`. It adds a pytest marker `after_training_hello_container` for tests that can be run after training is finished in the `TrainHelloContainer` job.

This will solve the issue of model registration in #377 and #398.
@dumbledad dumbledad mentioned this issue Jul 6, 2021
Open
@ant0nsc ant0nsc closed this as completed Jul 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
architecture Anything around possible extensions or re-structuring of code
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Register all models and allow inference on registered models microsoft/InnerEye-DeepLearning
2 participants
@ant0nsc @Shruthi42