Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tarfiles #27

Open
wants to merge 54 commits into
base: master
Choose a base branch
from
Open

Tarfiles #27

wants to merge 54 commits into from

Commits on Aug 6, 2020

  1. Open MIMIC from tarfile

    @bganglia
    bganglia committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    9f3eb7b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ieee8023/torchxrayvision … 

    …into tarfiles
    @bganglia
    bganglia committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    0414355 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/ieee8023/torchxrayvision … 

    …into tarfiles
    @bganglia
    bganglia committed Aug 6, 2020
    Configuration menu
    Copy the full SHA
    289747d View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2020

  1. revert whitespace

    @bganglia
    bganglia committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    303832c View commit details
    Browse the repository at this point in the history

  2. don't use get_image() in NIH_Dataset

    @bganglia
    bganglia committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    da2490b View commit details
    Browse the repository at this point in the history

  3. NIH_Dataset extends TarDataset

    @bganglia
    bganglia committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    8fb3f03 View commit details
    Browse the repository at this point in the history

  4. Store tarfiles in dictionary

    @bganglia
    bganglia committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    395e5e4 View commit details
    Browse the repository at this point in the history

  5. use getnames intead of getmembers

    @bganglia
    bganglia committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    fa69973 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2020

  1. use O(n) method for determining imgid from tar_path

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    abbbfec View commit details
    Browse the repository at this point in the history

  2. random data in MIMIC format

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    2ba6f5d View commit details
    Browse the repository at this point in the history

  3. script for generating random MIMIC data

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    cacc3ad View commit details
    Browse the repository at this point in the history

  4. track random MIMIC data

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    ecbf302 View commit details
    Browse the repository at this point in the history

  5. tarfile test using random MIMIC data

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    04f1a32 View commit details
    Browse the repository at this point in the history

  6. fix test directory

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    90129ab View commit details
    Browse the repository at this point in the history

  7. use .close() on tarfile and regenerate test directory

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    0aa52a7 View commit details
    Browse the repository at this point in the history

  8. support for tarfiles in NIH dataset

    @bganglia
    bganglia committed Aug 9, 2020
    Configuration menu
    Copy the full SHA
    349babb View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2020

  1. Inherit from TarDataset in PC_Dataset

    @bganglia
    bganglia committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    6999bd3 View commit details
    Browse the repository at this point in the history

  2. Storage-agnostic dataset

    @bganglia
    bganglia committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    842ddf8 View commit details
    Browse the repository at this point in the history

  3. Inherit from storage agnostic loader

    @bganglia
    bganglia committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    37afa4e View commit details
    Browse the repository at this point in the history

  4. tidy up tarfile code

    @bganglia
    bganglia committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    bbd4007 View commit details
    Browse the repository at this point in the history

  5. remove previous TarDataset, ZipDataset classes

    @bganglia
    bganglia committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    34daddb View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2020

  1. Scripts for generating test data

    @bganglia
    bganglia committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    727d9ff View commit details
    Browse the repository at this point in the history

  2. Test data

    @bganglia
    bganglia committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    d2ae7c0 View commit details
    Browse the repository at this point in the history

  3. Tests for zip, tar in MIMIC, NIH, and PC

    @bganglia
    bganglia committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    41b50c4 View commit details
    Browse the repository at this point in the history

  4. clean up storage classes

    @bganglia
    bganglia committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    48d8170 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2020

  1. save progress

    @bganglia
    bganglia committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    5c4117e View commit details
    Browse the repository at this point in the history

  2. inherit from Dataset in NIH_Dataset

    @bganglia
    bganglia committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    2773c69 View commit details
    Browse the repository at this point in the history

  3. Add code for automated tests with script-generated data

    @bganglia
    bganglia committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    7ffc252 View commit details
    Browse the repository at this point in the history

  4. script for writing random data

    @bganglia
    bganglia committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    68a71ae View commit details
    Browse the repository at this point in the history

  5. fall back on .index() instead of trying to load a cached version in .… 

    …load_dataset()
    @bganglia
    bganglia committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    ec9777b View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2020

  1. support multiprocessing

    @bganglia
    bganglia committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    29498a6 View commit details
    Browse the repository at this point in the history

  2. Clean up new code for tests and format interfaces

    @bganglia
    bganglia committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    3674357 View commit details
    Browse the repository at this point in the history

  3. write partial metadata files with subset of columns

    @bganglia
    bganglia committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    ccec9ae View commit details
    Browse the repository at this point in the history

  4. Improve caching

    @bganglia
    bganglia committed Aug 27, 2020
    Configuration menu
    Copy the full SHA
    c091734 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2020

  1. fix tests

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    e56a565 View commit details
    Browse the repository at this point in the history

  2. fix error in data-generation script

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    1dde4b7 View commit details
    Browse the repository at this point in the history

  3. create .torchxrayvision if it does not already exist

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    1628db4 View commit details
    Browse the repository at this point in the history

  4. fix line adding .torchxrayvision

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    124467c View commit details
    Browse the repository at this point in the history

  5. Commit sample data for testing NLM_TB datasets, instead of auto-gener… 

    …ating it
    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    28816e5 View commit details
    Browse the repository at this point in the history

  6. Commit covid test cases

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    ce38e57 View commit details
    Browse the repository at this point in the history

  7. Include parallel tests again

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    281935c View commit details
    Browse the repository at this point in the history

  8. trycatch on reading/writing stored_mappings, with disk_unwriteable_ou… 

    …t_of_date variable to prevent reading from an out-of-date cache
    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    9c2c9d2 View commit details
    Browse the repository at this point in the history

  9. work when .torchxrayvision is not writeable

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    7c6aebb View commit details
    Browse the repository at this point in the history

  10. remove some print statements

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    cb97e70 View commit details
    Browse the repository at this point in the history

  11. add test simulating an unwriteable disk

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    950ae96 View commit details
    Browse the repository at this point in the history

  12. use filesystem instead of dictionary

    @bganglia
    bganglia committed Aug 28, 2020
    Configuration menu
    Copy the full SHA
    300c9d7 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2020

  1. rewrite data generation scripts as python, not bash scripts; add para… 

    …meter for length of all datasets
    @bganglia
    bganglia committed Aug 30, 2020
    Configuration menu
    Copy the full SHA
    218fa75 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2020

  1. cleanup: better variable names and use blake2b instead of hash (works… 

    … between sessions)
    @bganglia
    bganglia committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    b22cead View commit details
    Browse the repository at this point in the history

  2. Add test for asserting a dataset loads faster the second time

    @bganglia
    bganglia committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    ae09bc9 View commit details
    Browse the repository at this point in the history

  3. Don't invoke duration test, to avoid spurious errors

    @bganglia
    bganglia committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    30c043b View commit details
    Browse the repository at this point in the history

  4. Call on new data generation script

    @bganglia
    bganglia committed Aug 31, 2020
    Configuration menu
    Copy the full SHA
    bfdebf2 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2020

  1. simplify and improve documentation

    @bganglia
    bganglia committed Sep 5, 2020
    Configuration menu
    Copy the full SHA
    0f7ea51 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2020

  1. reorganize

    @bganglia
    bganglia committed Sep 19, 2020
    Configuration menu
    Copy the full SHA
    71c7a50 View commit details
    Browse the repository at this point in the history

  2. Fix path length in CheX_Dataset

    @bganglia
    bganglia committed Sep 19, 2020
    Configuration menu
    Copy the full SHA
    1715b9d View commit details
    Browse the repository at this point in the history