You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.
When training models in InnerEye the error below is encountered due to recent changes in AML. This has been fixed in the latest hi-ml version (v0.2.5), so IE-DL needs to be updated to this.
Code for reproduction
pythonInnerEye/ML/runner.py--model=Lung--azureml
Actual outcome
Training run fails
Error messages
File "InnerEye/ML/runner.py", line 466, in <module>
main()
File "InnerEye/ML/runner.py", line 460, in main
run(project_root=fixed_paths.repository_root_directory(),
File "InnerEye/ML/runner.py", line 456, in runreturn runner.run()
File "InnerEye/ML/runner.py", line 220, in runself.run_in_situ(azure_run_info)
File "InnerEye/ML/runner.py", line 408, in run_in_situ
set_environment_variables_for_multi_node()
File "/mnt/azureml/cr/j/bc3f99f19bb745519fd9272cfd730249/exe/wd/InnerEye/Azure/azure_runner.py", line 313, in set_environment_variables_for_multi_node
env_vars =", ".join(f"{var} = {os.environ[var]}"for var in [ENV_MASTER_ADDR, ENV_MASTER_PORT, ENV_NODE_RANK])
File "/mnt/azureml/cr/j/bc3f99f19bb745519fd9272cfd730249/exe/wd/InnerEye/Azure/azure_runner.py", line 313, in <genexpr>
env_vars =", ".join(f"{var} = {os.environ[var]}"for var in [ENV_MASTER_ADDR, ENV_MASTER_PORT, ENV_NODE_RANK])
File "/azureml-envs/azureml_e12c14b51edf42f47eec39c741162949/lib/python3.8/os.py", line 675, in __getitem__raiseKeyError(key) fromNoneKeyError: 'NODE_RANK'
Is there an existing issue for this?
Bug summary
When training models in InnerEye the error below is encountered due to recent changes in AML. This has been fixed in the latest
hi-ml
version (v0.2.5), so IE-DL needs to be updated to this.Code for reproduction
Actual outcome
Training run fails
Error messages
Expected outcome
Successful training run
System info
No response
AB#7305
The text was updated successfully, but these errors were encountered: