Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tarfiles #27

Open
wants to merge 54 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
9f3eb7b
Open MIMIC from tarfile
bganglia Aug 3, 2020
0414355
Merge branch 'master' of https://github.com/ieee8023/torchxrayvision …
bganglia Aug 6, 2020
289747d
Merge branch 'master' of https://github.com/ieee8023/torchxrayvision …
bganglia Aug 6, 2020
303832c
revert whitespace
bganglia Aug 8, 2020
da2490b
don't use get_image() in NIH_Dataset
bganglia Aug 8, 2020
8fb3f03
NIH_Dataset extends TarDataset
bganglia Aug 8, 2020
395e5e4
Store tarfiles in dictionary
bganglia Aug 8, 2020
fa69973
use getnames intead of getmembers
bganglia Aug 8, 2020
abbbfec
use O(n) method for determining imgid from tar_path
bganglia Aug 9, 2020
2ba6f5d
random data in MIMIC format
bganglia Aug 9, 2020
cacc3ad
script for generating random MIMIC data
bganglia Aug 9, 2020
ecbf302
track random MIMIC data
bganglia Aug 9, 2020
04f1a32
tarfile test using random MIMIC data
bganglia Aug 9, 2020
90129ab
fix test directory
bganglia Aug 9, 2020
0aa52a7
use .close() on tarfile and regenerate test directory
bganglia Aug 9, 2020
349babb
support for tarfiles in NIH dataset
bganglia Aug 9, 2020
6999bd3
Inherit from TarDataset in PC_Dataset
bganglia Aug 10, 2020
842ddf8
Storage-agnostic dataset
bganglia Aug 10, 2020
37afa4e
Inherit from storage agnostic loader
bganglia Aug 10, 2020
bbd4007
tidy up tarfile code
bganglia Aug 10, 2020
34daddb
remove previous TarDataset, ZipDataset classes
bganglia Aug 10, 2020
727d9ff
Scripts for generating test data
bganglia Aug 13, 2020
d2ae7c0
Test data
bganglia Aug 13, 2020
41b50c4
Tests for zip, tar in MIMIC, NIH, and PC
bganglia Aug 13, 2020
48d8170
clean up storage classes
bganglia Aug 13, 2020
5c4117e
save progress
bganglia Aug 26, 2020
2773c69
inherit from Dataset in NIH_Dataset
bganglia Aug 26, 2020
7ffc252
Add code for automated tests with script-generated data
bganglia Aug 26, 2020
68a71ae
script for writing random data
bganglia Aug 26, 2020
ec9777b
fall back on .index() instead of trying to load a cached version in .…
bganglia Aug 26, 2020
29498a6
support multiprocessing
bganglia Aug 27, 2020
3674357
Clean up new code for tests and format interfaces
bganglia Aug 27, 2020
ccec9ae
write partial metadata files with subset of columns
bganglia Aug 27, 2020
c091734
Improve caching
bganglia Aug 27, 2020
e56a565
fix tests
bganglia Aug 28, 2020
1dde4b7
fix error in data-generation script
bganglia Aug 28, 2020
1628db4
create .torchxrayvision if it does not already exist
bganglia Aug 28, 2020
124467c
fix line adding .torchxrayvision
bganglia Aug 28, 2020
28816e5
Commit sample data for testing NLM_TB datasets, instead of auto-gener…
bganglia Aug 28, 2020
ce38e57
Commit covid test cases
bganglia Aug 28, 2020
281935c
Include parallel tests again
bganglia Aug 28, 2020
9c2c9d2
trycatch on reading/writing stored_mappings, with disk_unwriteable_ou…
bganglia Aug 28, 2020
7c6aebb
work when .torchxrayvision is not writeable
bganglia Aug 28, 2020
cb97e70
remove some print statements
bganglia Aug 28, 2020
950ae96
add test simulating an unwriteable disk
bganglia Aug 28, 2020
300c9d7
use filesystem instead of dictionary
bganglia Aug 28, 2020
218fa75
rewrite data generation scripts as python, not bash scripts; add para…
bganglia Aug 30, 2020
b22cead
cleanup: better variable names and use blake2b instead of hash (works…
bganglia Aug 31, 2020
ae09bc9
Add test for asserting a dataset loads faster the second time
bganglia Aug 31, 2020
30c043b
Don't invoke duration test, to avoid spurious errors
bganglia Aug 31, 2020
bfdebf2
Call on new data generation script
bganglia Aug 31, 2020
0f7ea51
simplify and improve documentation
bganglia Sep 5, 2020
71c7a50
reorganize
bganglia Sep 19, 2020
1715b9d
Fix path length in CheX_Dataset
bganglia Sep 19, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Test data
  • Loading branch information
bganglia committed Aug 13, 2020
commit d2ae7c0d3e0ef1cd3581a87d25a10cdc61b4f383
Binary file added tests/NIH_test_data/folder/00000001_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000002_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000003_001.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000005_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000006_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000007_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000008_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000009_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000010_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/folder/00000011_000.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/NIH_test_data/tar.tar
Binary file not shown.
Binary file added tests/NIH_test_data/zip.zip
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/PC_test_data/tar.tar
Binary file not shown.
Binary file added tests/PC_test_data/zip.zip
Binary file not shown.
Binary file modified tests/gen_mimic/images-224.tar
Binary file not shown.
Binary file added tests/gen_mimic/images-224.zip
Binary file not shown.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
20 changes: 10 additions & 10 deletions tests/gen_mimic/mimic-cxr-2.0.0-metadata.csv
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
dicom_id,subject_id,study_id,PerformedProcedureStepDescription,ViewPosition,Rows,Columns,StudyDate,StudyTime,ProcedureCodeSequence_CodeMeaning,ViewCodeSequence_CodeMeaning,PatientOrientationCodeSequence_CodeMeaning
c5402f0b-2ecf6365-bd88b424-252ad568,085cff35,a433a81c,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Recumbent
63036743-d14a06a9-2784f480-257e30e1,edc6b925,d03e6b53,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Recumbent
9160eb4a-257a3140-8f138287-8981fc11,0639398e,ab8e71f2,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Recumbent
746857e8-930e2b92-830ba925-90a97e7e,67111dd3,97d40f2a,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Recumbent
1d5cc6d7-8c6bc168-7b694c53-4df47582,182aeefb,5239d871,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Recumbent
7d4c4448-a12969b6-e4f1fce9-e42b5a6c,88530ff8,732de0c4,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
c94bbfae-e39038df-5b7efb8e-31f9ee55,32b572e6,9ba81d90,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
da54bf19-4bfcbe16-b364a677-b61c0743,636c6ae0,c8cca22c,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
d8d6c36e-f6d5be79-6d8d0627-4d147195,709a3d20,04003d64,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Erect
1c9bdf33-12762299-6e5a617d-b2b0ef88,700bf17d,da301b01,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Recumbent
dc7a8b5e-4ad1afd8-8f994433-643c39f8,346b3f01,14b48a44,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Recumbent
9622b5f2-3a3e307e-f5ca9164-375a6f70,11a8f57f,bf2ecf26,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Erect
8cf7e7b2-46bdc39f-31564b20-71ee6b6d,4eedf9fe,f8521632,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Erect
6a301286-456d8528-c59bd118-195ce98c,bc0ee611,89cfed09,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
67dc7d13-32da76c2-ff22a175-5d4c3ced,54e08d2a,1c41417d,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Erect
b69fd4aa-55745241-f15af2bb-b979004a,3b3b7d36,f076c36f,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Erect
7e393cbe-6eac27c9-469708b4-dc5f22ef,310345ea,274bcf57,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
a0082cb3-93689ac6-4fbbad4e-ddd68866,bfb2d1b6,2c5d98ee,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Recumbent
4197fbbc-1ed2f09e-8a82308b-971009a3,3d404d22,1e752e16,CHEST (PA AND LAT),LATERAL,224,224,0,0,CHEST (PA AND LAT),lateral,Erect
baf27490-fe6678a0-fe0b7379-665d78ba,6145fd64,7968e508,CHEST (PA AND LAT),PA,224,224,0,0,CHEST (PA AND LAT),postero-anterior,Recumbent
Binary file modified tests/gen_mimic/mimic-cxr-2.0.0-metadata.csv.gz
Binary file not shown.
20 changes: 10 additions & 10 deletions tests/gen_mimic/mimic-cxr-2.0.0-negbio.csv
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
subject_id,study_id,Atelectasis,Cardiomegaly,Consolidation,Edema,Enlarged Cardiomediastinum,Fracture,Lung Lesion,Lung Opacity,No Finding,Pleural Effusion,Pleural Other,Pneumonia,Pneumothorax,Support Devices
085cff35,a433a81c,1.0,0.0,1.0,,,,,1.0,,0.0,0.0,0.0,,1.0
edc6b925,d03e6b53,0.0,1.0,1.0,-1.0,,,-1.0,-1.0,0.0,-1.0,1.0,0.0,0.0,-1.0
0639398e,ab8e71f2,,1.0,,1.0,,1.0,,1.0,1.0,0.0,-1.0,1.0,-1.0,
67111dd3,97d40f2a,0.0,0.0,0.0,,0.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,0.0,0.0,0.0
182aeefb,5239d871,1.0,,0.0,-1.0,0.0,1.0,,0.0,0.0,,,1.0,1.0,
88530ff8,732de0c4,,1.0,-1.0,-1.0,1.0,-1.0,1.0,,1.0,1.0,0.0,-1.0,-1.0,1.0
32b572e6,9ba81d90,,0.0,,,1.0,,,1.0,0.0,1.0,,-1.0,0.0,0.0
636c6ae0,c8cca22c,1.0,-1.0,,-1.0,0.0,0.0,-1.0,-1.0,1.0,,,1.0,,-1.0
709a3d20,04003d64,0.0,1.0,1.0,-1.0,,1.0,,0.0,-1.0,1.0,-1.0,0.0,,-1.0
700bf17d,da301b01,0.0,0.0,1.0,-1.0,-1.0,,0.0,,-1.0,0.0,1.0,1.0,0.0,1.0
346b3f01,14b48a44,-1.0,-1.0,0.0,0.0,-1.0,-1.0,,-1.0,0.0,,,,,
11a8f57f,bf2ecf26,1.0,1.0,0.0,0.0,-1.0,0.0,-1.0,-1.0,0.0,1.0,1.0,-1.0,,1.0
4eedf9fe,f8521632,,,1.0,0.0,1.0,1.0,,,,1.0,0.0,-1.0,-1.0,
bc0ee611,89cfed09,0.0,,0.0,-1.0,-1.0,1.0,1.0,1.0,,0.0,-1.0,0.0,-1.0,0.0
54e08d2a,1c41417d,1.0,-1.0,0.0,,,-1.0,-1.0,1.0,-1.0,-1.0,,,1.0,0.0
3b3b7d36,f076c36f,,0.0,0.0,0.0,,1.0,0.0,0.0,0.0,-1.0,,,1.0,0.0
310345ea,274bcf57,,0.0,0.0,-1.0,1.0,1.0,0.0,0.0,1.0,0.0,0.0,1.0,0.0,-1.0
bfb2d1b6,2c5d98ee,-1.0,,,,,,1.0,,1.0,0.0,0.0,0.0,-1.0,-1.0
3d404d22,1e752e16,0.0,0.0,1.0,-1.0,0.0,-1.0,1.0,1.0,0.0,-1.0,-1.0,1.0,-1.0,1.0
6145fd64,7968e508,,-1.0,0.0,,-1.0,1.0,1.0,0.0,1.0,1.0,,1.0,,
Binary file modified tests/gen_mimic/mimic-cxr-2.0.0-negbio.csv.gz
Binary file not shown.