Add postprocessing step and tests for PanNuke #49
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… load one at a time instead of loading entire fold into single np array. Also add some tests.
ryanccarelli
approved these changes
Jan 4, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When PanNuke dataset is downloaded, it comes in 3 folds and each fold is a single .npy file for the images and a single .npy file for the maks and a third .npy file for the tissue type labels. There are ~2500 images per fold, so these .npy files for the images and masks are pretty big.
The current implementation calls np.load() on the big .npy files, and then slices them to get the right images/masks. Loading these arrays can be somewhat slow though, and they also take up memory.
I added a postprocessing step to write each individual image to .png and each individual mask to .npy. I also modified the PanNukeDataset as needed. Tissue type and fold labels are in the filenames.
This setup is more intuitive and hopefully will be faster since we avoid loading the big .npy arrays during training.
I also added tests