Ensemble aggregation is using storage_account instead of Run context

[javier-alvarez]

2020-09-12T03:23:17Z INFO
Starting the daemon thread to refresh tokens in background for process with pid = 134

The experiment failed. Finalizing run...
2020-09-12T03:23:17Z INFO Exiting context: TrackUserError
2020-09-12T03:23:17Z INFO Exiting context: RunHistory
[2020-09-12T03:23:17.966397] TimeoutHandler init
[2020-09-12T03:23:17.966479] TimeoutHandler enter
Cleaning up all outstanding Run operations, waiting 300.0 seconds
9 items cleaning up...
Cleanup took 0.9337708950042725 seconds
[2020-09-12T03:23:19.147607] TimeoutHandler exit
2020-09-12T03:23:19Z INFO Exiting context: Dataset
Enter exit of DatasetContextManager
Exit exit of DatasetContextManager
2020-09-12T03:23:19Z INFO Exiting context: ProjectPythonPath
Traceback (most recent call last):
File "InnerEye/ML/runner.py", line 391, in
main()
File "InnerEye/ML/runner.py", line 387, in main
post_cross_validation_hook=default_post_cross_validation_hook)
File "InnerEye/ML/runner.py", line 381, in run
return runner.run()
File "InnerEye/ML/runner.py", line 249, in run
self.run_in_situ()
File "InnerEye/ML/runner.py", line 329, in run_in_situ
self.wait_for_cross_val_runs_to_finish_and_aggregate()
File "/azureml-envs/azureml_25e24d323549738d2dc10e4c7fc4746d/lib/python3.7/site-packages/stopit/utils.py", line 148, in wrapper
return func(*args, **kwargs)
File "InnerEye/ML/runner.py", line 124, in wait_for_cross_val_runs_to_finish_and_aggregate
self.create_ensemble_model()
File "InnerEye/ML/runner.py", line 158, in create_ensemble_model
self.azure_config, self.model_config, PARENT_RUN_CONTEXT, output_subdir_name=OTHER_RUNS_SUBDIR_NAME)
File "/mnt/batch/tasks/shared/LS_root/jobs/radnetinnereyedev/azureml/hd_3d2f42e2-5087-42f2-91d7-eaa50c2d656a_0/mounts/workspaceblobstore/azureml/HD_3d2f42e2-5087-42f2-91d7-eaa50c2d656a_0/InnerEye/ML/utils/run_recovery.py", line 92, in download_checkpoints_from_run
run=run
File "/mnt/batch/tasks/shared/LS_root/jobs/radnetinnereyedev/azureml/hd_3d2f42e2-5087-42f2-91d7-eaa50c2d656a_0/mounts/workspaceblobstore/azureml/HD_3d2f42e2-5087-42f2-91d7-eaa50c2d656a_0/InnerEye/Azure/azure_config.py", line 273, in download_outputs_from_run
raise ValueError("self.storage_account cannot be None")
ValueError: self.storage_account cannot be None

2020/09/12 03:23:31 logger.go:293: Failed to run the wrapper cmd with err: exit status 1
2020/09/12 03:23:31 sysutils_linux.go:221: mpirun version string: {
mpirun (Open MPI) 3.1.2