Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

FastMRI dataset onboarding script and detailed examples #444

Merged
merged 64 commits into from
May 19, 2021
Merged

Conversation

ant0nsc
Copy link
Contributor

@ant0nsc ant0nsc commented Apr 22, 2021

This PR adds necessary tooling and examples for running fastMRI reconstruction models.

  • Script to create and run an Azure Data Factory to download the raw data, and place them into a storage account
  • Detailed examples to run the VarNet model from the fastMRI github repo
  • Ability to work with fixed mounting points for datasets

melanibe
melanibe previously approved these changes May 13, 2021
Copy link
Contributor

@melanibe melanibe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great new capabilities added 🚀 And really liked the documentation!
Left some minor comments, feel free to ignore if they don't make sense.

melanibe
melanibe previously approved these changes May 14, 2021
melanibe
melanibe previously approved these changes May 18, 2021
@ant0nsc ant0nsc enabled auto-merge (squash) May 18, 2021 11:59
@hansenms
Copy link

At the moment, the prepare_fastmri.py script does not work. At first glance, there are problems with binary datasets having space in their names, e.g. " On Azure`. I tried a quick hack to remove all the spaces in various names, but I am now getting this error (when running the pipelines):

{"stack":"Error: Error: Error: TypeError: Cannot set property 'enablePartitionDiscovery' of undefined\n    at t.<anonymous> (https://ms-adf.azure.com/main.01ccec2db0cb1598b89b.js:2:1661940)\n    at https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:112098\n    at Object.throw (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:112203)\n    at s (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:110993)\n    at e.invoke (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:14216)\n    at Object.onInvoke (https://ms-adf.azure.com/main.01ccec2db0cb1598b89b.js:2:3754478)\n    at e.invoke (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:14156)\n    at t.run (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:9419)\n    at https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:21175\n    at e.invokeTask (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:14900)\n    at Object.onInvokeTask (https://ms-adf.azure.com/main.01ccec2db0cb1598b89b.js:2:3754390)\n    at e.invokeTask (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:14821)\n    at t.runTask (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:10082)\n    at g (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:17167)\n    at t.invokeTask (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:16012)\n    at invoke (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:15874)\n    at n.args.<computed> (https://ms-adf.azure.com/polyfills.7135c37b2f707063692f.js:2:45244)","message":"Error: Error: TypeError: Cannot set property 'enablePartitionDiscovery' of undefined"}

@ant0nsc ant0nsc disabled auto-merge May 18, 2021 21:23
@hansenms
Copy link

correction on my comment above, appears there was an exit(1) in there for debugging purposes that caused the scrip to exit prematurely. Continuing testing with that fixed.

@ant0nsc ant0nsc enabled auto-merge (squash) May 19, 2021 08:18
progress_bar_refresh_rate = container.pl_progress_bar_refresh_rate
if progress_bar_refresh_rate is None and is_azureml_run:
# When running in AzureML, the default progress bar clutters the output files with thousands of lines.
progress_bar_refresh_rate = 50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be set here? I imagine most containers would override this if necessary (the FastMri container does), and this introduces a small hidden difference between an offline and AzureML run.

@ant0nsc ant0nsc merged commit 8bae42e into main May 19, 2021
@ant0nsc ant0nsc deleted the antonsc/fastmri branch May 19, 2021 15:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants