Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Enable model registration and scoring for classification models #398

Closed
JacopoTeneggi opened this issue Feb 13, 2021 · 2 comments
Closed

Enable model registration and scoring for classification models #398

JacopoTeneggi opened this issue Feb 13, 2021 · 2 comments
Labels
triaged An item on Azure Boards has been created and prioritized

Comments

@JacopoTeneggi
Copy link
Contributor

JacopoTeneggi commented Feb 13, 2021 •
edited by azure-boards bot
Loading

After the run is successfully completed, I see a .tar archive named 50_checkpoint.pth.tar in outputs/checkpoints under the "Outputs & Logs" tab, but I do not see the model under the "Models" tab.

As mentioned in the docs no clear pipeline is defined for classification models.

How can I run a trained model on AML without a score.py file?

AB#3920
@ant0nsc
Copy link
Contributor

ant0nsc commented Feb 15, 2021 •
edited
Loading

One of the conceptual difficulties for classification models is their (potentially) very complex input. We have designed the classification models such that you can feed in a combination of image and non-image features - to use that at inference time in all its generality, you need to feed in a folder with a mini mock dataset for the test data.

We have not yet had a use case for that, hence it is not in the toolbox.

If you'd like to help extend the toolbox here, we'd be very grateful! The necessary steps are:

  • Extend the model registration code here to also register classification models
  • Model registration for classification models is almost identical to that for segmentation models. The only piece of code that is in the way is copy_child_paths, where we create an object that is specific to segmentation models. Change inference to take DICOM input and produce DICOM-RT output for segmentation models #366 is already removing those offending fields, and I would expect that we can use the same model registration code for segmentation and classification models after.
  • Extend score.py to deal with models that are not segmentation models. This code would require quite a bit of cleanup, because it contains quite a bit of logic that, in my view, should actually live inside the models, and not in a scoring script.

The first 2 steps should be relatively simple. Once you have that, you can download a trained model from AzureML, read out the checkpoint, and do pretty much anything with it, without having to rely on fancy score.py

@ant0nsc ant0nsc changed the title [Sample classification task] How to run trained model Enable model registration and scoring for classification models Feb 17, 2021
@ant0nsc ant0nsc added the triaged An item on Azure Boards has been created and prioritized label Mar 9, 2021
@ant0nsc ant0nsc added no changelog needed CHANGELOG.md does not need to be updated in this PR and removed no changelog needed CHANGELOG.md does not need to be updated in this PR labels Apr 9, 2021
@Shruthi42 Shruthi42 mentioned this issue May 12, 2021
Merged
Shruthi42 added a commit that referenced this issue May 12, 2021
@Shruthi42
Register all models after training, not only Segmentation models. (#455)
aa09b9d 
This PR changes the codepath so all models trained on AzureML are registered. The codepath previously allowed only segmentation models (subclasses of `SegmentationModelBase`) to be registered. Models are registered after a training run or if the `only_register_model` flag is set. Models may be legacy InnerEye config-based models or may be defined using the LightningContainer class.

The PR also removes the AzureRunner conda environment. The full InnerEye conda environment is needed to submit a training job to AzureML.

It splits the `TrainHelloWorldAndHelloContainer` job in the PR build into two jobs, `TrainHelloWorld` and `TrainHelloContainer`. It adds a pytest marker `after_training_hello_container` for tests that can be run after training is finished in the `TrainHelloContainer` job.

This will solve the issue of model registration in #377 and #398.
@ant0nsc
Copy link
Contributor

ant0nsc commented Jul 27, 2021

Classification models are registered now, see #455. We do not yet have a clear story for running those with score.py.

@ant0nsc ant0nsc closed this as completed Jul 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
triaged An item on Azure Boards has been created and prioritized
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ant0nsc @JacopoTeneggi