HelloWord Test error

[sahand68]

After activating InnerEye env, trying to run python InnerEye/ML/runner.py --model=HelloWorld i get this error.
Ps. I am following documentation step by step

(**InnerEye) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning$ export PYTHONPATH=`pwd`
(InnerEye) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning$ python InnerEye/ML/runner.py --model=HelloWorld
Setting up logging to stdout.
Setting logging level to 20
2020-09-29T16:33:30Z INFO     rpdb is handling traps. To debug: identify the main runner.py process, then as root: kill -TRAP <process_id>; nc 127.0.0.1 4444
2020-09-29T16:33:30Z INFO     Found class HelloWorld in file /home/sahand/InnerEye-DeepLearning/InnerEye/ML/configs/segmentation/HelloWorld.py
2020-09-29T16:33:30Z INFO     Creating the default output folder structure.
2020-09-29T16:33:30Z INFO     Running outside of AzureML.
2020-09-29T16:33:30Z INFO     All results will be written to a subfolder of the project root folder.
2020-09-29T16:33:30Z INFO     Run outputs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld
2020-09-29T16:33:30Z INFO     Logs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld/logs
2020-09-29T16:33:30Z INFO     Creating the adjusted output folder structure.
2020-09-29T16:33:30Z INFO     Running outside of AzureML.
2020-09-29T16:33:30Z INFO     All results will be written to a subfolder of the project root folder.
2020-09-29T16:33:30Z INFO     Run outputs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld
2020-09-29T16:33:30Z INFO     Logs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld/logs
2020-09-29T16:33:30Z INFO     extra_code_directory is unset
Setting logging level to 20
Setting up logging with level 20 to file /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld/logs/stdout.txt
2020-09-29T16:33:33Z INFO     Setting multiprocessing start method to 'forkserver'
2020-09-29T16:33:33Z INFO     Model training will use the local dataset provided in /home/sahand/InnerEye-DeepLearning/Tests/ML/test_data
2020-09-29T16:33:33Z INFO     
Arguments:
	__center_size_param_value: None
	__dataset_data_frame_param_value: None
	__inference_stride_size_param_value: None
	__largest_connected_component_foreground_classes_param_value: None
	__min_l_rate_param_value: 0
	__model_category_param_value: ModelCategory.Segmentation
	__model_name_param_value: HelloWorld
	__overrides_param_value: None
	__use_gpu_param_value: False
	_architecture_param_value: UNet3D
	_class_weights_param_value: [0.02, 0.49, 0.49]
	_colours_param_value: [(130, 183, 14), (238, 127, 26)]
	_comparison_blob_storage_paths_param_value: None
	_crop_size_param_value: (64, 64, 64)
	_datasets_for_inference: None
	_datasets_for_training: None
	_feature_channels_param_value: [4]
	_file_system_config_param_value: <DeepLearningFileSystemConfig DeepLearningFileSystemConfig00010>
	_fill_holes_param_value: [True, True]
	_ground_truth_ids_display_names_param_value: ['region', 'region_1']
	_ground_truth_ids_param_value: ['region', 'region_1']
	_image_channels_param_value: ['channel1', 'channel2']
	_instance__params : {}
	_l_rate_multi_step_milestones_param_value: None
	_level_param_value: 50
	_local_dataset_param_value: /home/sahand/InnerEye-DeepLearning/Tests/ML/test_data
	_mask_id_param_value: mask
	_multiprocessing_start_method_param_value: MultiprocessingStartMethod.forkserver
	_name_param_value : HelloWorld00008
	_norm_method_param_value: PhotometricNormalizationMethod.CtWindow
	_num_dataload_workers_param_value: 0
	_num_epochs_param_value: 2
	_param_watchers   : {}
	_save_start_epoch_param_value: 1
	_save_step_epochs_param_value: 1
	_slice_exclusion_rules_param_value: []
	_start_epoch_param_value: 0
	_summed_probability_rules_param_value: []
	_tail_param_value : None
	_test_crop_size_param_value: (64, 64, 64)
	_test_diff_epochs_param_value: 1
	_test_start_epoch_param_value: 2
	_test_step_epochs_param_value: 1
	_train_batch_size_param_value: 2
	_use_mixed_precision_param_value: True
	_window_param_value: 200
	initialized       : True
	param             : <param.parameterized.Parameters object at 0x7f3a5dc57b00>

2020-09-29T16:33:33Z INFO     
2020-09-29T16:33:33Z INFO     **** STARTING: Model training **********************************************************************
2020-09-29T16:33:33Z INFO     
2020-09-29T16:33:33Z INFO     Train: 3, Test: 1, and Val: 2. Total subjects: 6
2020-09-29T16:33:33Z INFO     Model Training: Random seed set to: 42
2020-09-29T16:33:33Z INFO     Starting to read and parse the datasets.
2020-09-29T16:33:33Z INFO     Processing dataset (name=None)
2020-09-29T16:33:33Z INFO     Processing dataset (name=None)
2020-09-29T16:33:33Z INFO     Creating the data loader for the training set.
2020-09-29T16:33:33Z INFO     Creating the data loader for the validation set.
2020-09-29T16:33:33Z INFO     Finished creating the data loaders.
2020-09-29T16:33:33Z INFO     Models are saved at /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T163330Z_HelloWorld/checkpoints
2020-09-29T16:33:33Z INFO     Writing model summary to: logs/model_summaries/model_log001.txt
Attempted to log scalar metric LoggingColumns.NumTrainableParameters:
506139
2020-09-29T16:33:33Z INFO     Making no adjustments to the model because no GPU was found.
2020-09-29T16:33:33Z INFO     Starting training
2020-09-29T16:33:33Z INFO     Starting epoch 1
2020-09-29T16:33:33Z ERROR    Model training/testing failed. Exception: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:107:
sitk::ERROR: Unable to determine ImageIO reader for "/home/sahand/InnerEye-DeepLearning/Tests/ML/test_data/train_and_test_data/id1_channel1.nii.gz"
Traceback (most recent call last):
  File "InnerEye/ML/runner.py", line 364, in run_in_situ
    self.create_ml_runner().run()
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 308, in run
    model_train(self.model_config, run_recovery)
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/model_training.py", line 150, in model_train
    train_epoch_results = train_or_validate_epoch(training_steps)
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/model_training.py", line 267, in train_or_validate_epoch
    for batch_index, sample in enumerate(train_val_params.data_loader):
  File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 403, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/dataset/cropping_dataset.py", line 38, in __getitem__
    sample=super().get_samples_at_index(index=i)[0],
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/dataset/full_image_dataset.py", line 289, in get_samples_at_index
    samples = [io_util.load_images_from_dataset_source(dataset_source=ds)]  # type: ignore
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 357, in load_images_from_dataset_source
    images = [load_nifti_image(channel, ImageDataType.IMAGE.value) for channel in dataset_source.image_channels]
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 357, in <listcomp>
    images = [load_nifti_image(channel, ImageDataType.IMAGE.value) for channel in dataset_source.image_channels]
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 182, in load_nifti_image
    img, header = read_image_as_array_with_header(path)
  File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 146, in read_image_as_array_with_header
    image: sitk.Image = sitk.ReadImage(str(file_path))
  File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/SimpleITK/SimpleITK.py", line 8876, in ReadImage
    return _SimpleITK.ReadImage(*args)
RuntimeError: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:107:
sitk::ERROR: Unable to determine ImageIO reader for "/home/sahand/InnerEye-DeepLearning/Tests/ML/test_data/train_and_test_data/id1_channel1.nii.gz"

```**

[MaherJendoubi]

Hi @sahand68 I am trying to reproduce your issue. Would you please describe your environment (OS version, Anaconda version, python version, etc)?

[sahand68]

@MaherJendoubi Thanks for the fast reply.
OS version: Ubuntu 20.04

(base) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning$ conda list anaconda
packages in environment at /home/sahand/anaconda3:
Name                    Version                   Build  Channel
anaconda                  2020.07                  py38_0  
anaconda-client           1.7.2                    py38_0  
anaconda-navigator        1.9.12                   py38_0  
anaconda-project          0.8.4                      py_0  

InnerEye) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning$ conda list
# packages in environment at /home/sahand/anaconda3/envs/InnerEye:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
absl-py                   0.10.0                   pypi_0    pypi
adal                      1.2.4                    pypi_0    pypi
aiofiles                  0.5.0                    pypi_0    pypi
aiohttp                   3.6.2                    pypi_0    pypi
alembic                   1.4.3                    pypi_0    pypi
ansiwrap                  0.8.4                    pypi_0    pypi
apipkg                    1.5                      pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
applicationinsights       0.11.9                   pypi_0    pypi
argon2-cffi               20.1.0                   pypi_0    pypi
async-generator           1.10                     pypi_0    pypi
async-timeout             3.0.1                    pypi_0    pypi
attrs                     20.2.0                   pypi_0    pypi
azure-common              1.1.25                   pypi_0    pypi
azure-core                1.8.1                    pypi_0    pypi
azure-graphrbac           0.61.1                   pypi_0    pypi
azure-identity            1.2.0                    pypi_0    pypi
azure-mgmt-authorization  0.61.0                   pypi_0    pypi
azure-mgmt-containerregistry 2.8.0                    pypi_0    pypi
azure-mgmt-core           1.2.0                    pypi_0    pypi
azure-mgmt-keyvault       7.0.0                    pypi_0    pypi
azure-mgmt-resource       10.2.0                   pypi_0    pypi
azure-mgmt-storage        16.0.0                   pypi_0    pypi
azure-storage-blob        2.1.0                    pypi_0    pypi
azure-storage-common      2.1.0                    pypi_0    pypi
azure-storage-file        2.1.0                    pypi_0    pypi
azureml-automl-core       1.12.0                   pypi_0    pypi
azureml-core              1.12.0.post2             pypi_0    pypi
azureml-dataprep          2.0.10                   pypi_0    pypi
azureml-dataprep-native   20.1.1                   pypi_0    pypi
azureml-dataset-runtime   1.12.0                   pypi_0    pypi
azureml-mlflow            1.12.0                   pypi_0    pypi
azureml-pipeline          1.12.0                   pypi_0    pypi
azureml-pipeline-core     1.12.0                   pypi_0    pypi
azureml-pipeline-steps    1.12.0                   pypi_0    pypi
azureml-sdk               1.12.0                   pypi_0    pypi
azureml-telemetry         1.12.0                   pypi_0    pypi
azureml-tensorboard       1.12.0                   pypi_0    pypi
azureml-train             1.12.0                   pypi_0    pypi
azureml-train-automl-client 1.12.0.post2             pypi_0    pypi
azureml-train-core        1.12.0                   pypi_0    pypi
azureml-train-restclients-hyperdrive 1.12.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
backports-tempfile        1.0                      pypi_0    pypi
backports-weakref         1.0.post1                pypi_0    pypi
bitstring                 3.1.7                    pypi_0    pypi
black                     20.8b1                   pypi_0    pypi
blas                      1.0                         mkl  
bleach                    3.2.1                    pypi_0    pypi
blobxfer                  1.9.4                    pypi_0    pypi
blosc                     1.20.0               hd408876_0  
ca-certificates           2020.7.22                     0  
cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.6.20                py37_0  
cffi                      1.14.3                   pypi_0    pypi
chardet                   3.0.4                    pypi_0    pypi
click                     7.1.2                    pypi_0    pypi
cloudpickle               1.6.0                    pypi_0    pypi
colorama                  0.4.3                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
contextlib2               0.6.0.post1              pypi_0    pypi
coverage                  5.3                      pypi_0    pypi
cryptography              2.9.2                    pypi_0    pypi
cudatoolkit               10.2.89              hfd86e86_1  
cycler                    0.10.0                   pypi_0    pypi
databricks-cli            0.12.0                   pypi_0    pypi
dataclasses-json          0.5.2                    pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
defusedxml                0.6.0                    pypi_0    pypi
dill                      0.3.2                    pypi_0    pypi
distro                    1.5.0                    pypi_0    pypi
docker                    4.3.1                    pypi_0    pypi
dotnetcore2               2.1.16                   pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
execnet                   1.7.1                    pypi_0    pypi
flake8                    3.8.3                    pypi_0    pypi
flask                     1.1.2                    pypi_0    pypi
freetype                  2.10.2               h5ab3b9f_0  
fusepy                    3.0.1                    pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
gitdb                     4.0.5                    pypi_0    pypi
gitpython                 3.1.7                    pypi_0    pypi
google-auth               1.22.0                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
gorilla                   0.3.0                    pypi_0    pypi
gputil                    1.4.0                    pypi_0    pypi
grpcio                    1.32.0                   pypi_0    pypi
gunicorn                  20.0.4                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
idna                      2.8                      pypi_0    pypi
imageio                   2.9.0                    pypi_0    pypi
importlib-metadata        2.0.0                    pypi_0    pypi
iniconfig                 1.0.1                    pypi_0    pypi
intel-openmp              2020.2                      254  
ipykernel                 5.3.4                    pypi_0    pypi
ipython                   7.18.1                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.5.1                    pypi_0    pypi
isodate                   0.6.0                    pypi_0    pypi
itsdangerous              1.1.0                    pypi_0    pypi
jedi                      0.17.2                   pypi_0    pypi
jeepney                   0.4.3                    pypi_0    pypi
jinja2                    2.11.2                   pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    0.16.0                   pypi_0    pypi
jpeg                      9b                   h024ee3a_2  
jsonpickle                1.4.1                    pypi_0    pypi
jsonschema                3.2.0                    pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            6.1.5                    pypi_0    pypi
jupyter-console           6.2.0                    pypi_0    pypi
jupyter-core              4.6.3                    pypi_0    pypi
jupyterlab-pygments       0.1.1                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
lcms2                     2.11                 h396b838_0  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_1  
llvmlite                  0.34.0                   pypi_0    pypi
lz4-c                     1.9.2                he6710b0_1  
mako                      1.1.3                    pypi_0    pypi
markdown                  3.2.2                    pypi_0    pypi
markupsafe                1.1.1                    pypi_0    pypi
marshmallow               3.8.0                    pypi_0    pypi
marshmallow-enum          1.5.1                    pypi_0    pypi
matplotlib                3.3.0                    pypi_0    pypi
mccabe                    0.6.1                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
mkl                       2020.2                      256  
mkl-service               2.3.0            py37he904b0f_0  
mkl_fft                   1.2.0            py37h23d657b_0  
mkl_random                1.1.1            py37h0573a6f_0  
mlflow                    1.8.0                    pypi_0    pypi
more-itertools            8.5.0                    pypi_0    pypi
msal                      1.5.0                    pypi_0    pypi
msal-extensions           0.1.3                    pypi_0    pypi
msrest                    0.6.19                   pypi_0    pypi
msrestazure               0.6.4                    pypi_0    pypi
multidict                 4.7.6                    pypi_0    pypi
mypy                      0.770                    pypi_0    pypi
mypy-extensions           0.4.3                    pypi_0    pypi
nbclient                  0.5.0                    pypi_0    pypi
nbconvert                 6.0.6                    pypi_0    pypi
nbformat                  5.0.7                    pypi_0    pypi
ncurses                   6.2                  he6710b0_1  
ndg-httpsclient           0.5.1                    pypi_0    pypi
nest-asyncio              1.4.1                    pypi_0    pypi
networkx                  2.5                      pypi_0    pypi
ninja                     1.10.1           py37hfd86e86_0  
notebook                  6.1.4                    pypi_0    pypi
numba                     0.51.2                   pypi_0    pypi
numpy                     1.19.1           py37hbc911f0_0  
numpy-base                1.19.1           py37hfa32c7d_0  
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                     py37_0  
openssl                   1.1.1h               h7b6447c_0  
packaging                 20.4                     pypi_0    pypi
pandas                    1.1.0                    pypi_0    pypi
pandocfilters             1.4.2                    pypi_0    pypi
papermill                 2.1.2                    pypi_0    pypi
param                     1.9.3                    pypi_0    pypi
parso                     0.7.1                    pypi_0    pypi
pathspec                  0.8.0                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    7.2.0            py37hb39fc2d_0  
pip                       20.1.1                   py37_1  
pluggy                    0.13.1                   pypi_0    pypi
portalocker               1.7.1                    pypi_0    pypi
prometheus-client         0.8.0                    pypi_0    pypi
prometheus-flask-exporter 0.18.0                   pypi_0    pypi
prompt-toolkit            3.0.7                    pypi_0    pypi
protobuf                  3.13.0                   pypi_0    pypi
psutil                    5.7.2                    pypi_0    pypi
ptyprocess                0.6.0                    pypi_0    pypi
py                        1.9.0                    pypi_0    pypi
pyarrow                   1.0.1                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.6.0                    pypi_0    pypi
pycparser                 2.20                     pypi_0    pypi
pydicom                   2.0.0                    pypi_0    pypi
pyflakes                  2.2.0                    pypi_0    pypi
pygments                  2.7.1                    pypi_0    pypi
pyjwt                     1.7.1                    pypi_0    pypi
pyopenssl                 19.1.0                   pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
pyrsistent                0.17.3                   pypi_0    pypi
pytest                    6.0.1                    pypi_0    pypi
pytest-cov                2.10.1                   pypi_0    pypi
pytest-forked             1.3.0                    pypi_0    pypi
pytest-xdist              1.34.0                   pypi_0    pypi
python                    3.7.3                h0371630_0  
python-blosc              1.7.0            py37h7b6447c_0  
python-dateutil           2.8.1                    pypi_0    pypi
python-editor             1.0.4                    pypi_0    pypi
pytorch                   1.6.0           py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch
pytz                      2020.1                   pypi_0    pypi
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.3.1                    pypi_0    pypi
pyzmq                     19.0.2                   pypi_0    pypi
qtconsole                 4.7.7                    pypi_0    pypi
qtpy                      1.9.0                    pypi_0    pypi
querystring-parser        1.2.4                    pypi_0    pypi
radio                     0.1.0                    pypi_0    pypi
readline                  7.0                  h7b6447c_5  
regex                     2020.9.27                pypi_0    pypi
requests                  2.22.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rich                      5.1.1                    pypi_0    pypi
rpdb                      0.1.6                    pypi_0    pypi
rsa                       4.6                      pypi_0    pypi
ruamel-yaml               0.16.12                  pypi_0    pypi
ruamel-yaml-clib          0.2.2                    pypi_0    pypi
scikit-image              0.17.2                   pypi_0    pypi
scikit-learn              0.23.2                   pypi_0    pypi
scipy                     1.5.2                    pypi_0    pypi
seaborn                   0.10.1                   pypi_0    pypi
secretstorage             3.1.2                    pypi_0    pypi
send2trash                1.5.0                    pypi_0    pypi
setuptools                49.6.0                   py37_1  
simpleitk                 1.2.4                    pypi_0    pypi
simplejson                3.17.2                   pypi_0    pypi
six                       1.15.0                     py_0  
smmap                     3.0.4                    pypi_0    pypi
snappy                    1.1.8                he6710b0_0  
sqlalchemy                1.3.13                   pypi_0    pypi
sqlite                    3.33.0               h62c20be_0  
sqlparse                  0.3.1                    pypi_0    pypi
stopit                    1.1.2                    pypi_0    pypi
stringcase                1.2.0                    pypi_0    pypi
tabulate                  0.8.7                    pypi_0    pypi
tenacity                  6.2.0                    pypi_0    pypi
tensorboard               2.3.0                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorboardx              2.1                      pypi_0    pypi
terminado                 0.9.1                    pypi_0    pypi
testpath                  0.4.4                    pypi_0    pypi
textwrap3                 0.9.2                    pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
tifffile                  2020.9.28                pypi_0    pypi
tk                        8.6.10               hbc83047_0  
toml                      0.10.1                   pypi_0    pypi
torchprof                 1.1.1                    pypi_0    pypi
torchvision               0.7.0                py37_cu102    pytorch
tornado                   6.0.4                    pypi_0    pypi
tqdm                      4.50.0                   pypi_0    pypi
traitlets                 5.0.4                    pypi_0    pypi
typed-ast                 1.4.1                    pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
typing-inspect            0.6.0                    pypi_0    pypi
urllib3                   1.25.10                  pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          0.57.0                   pypi_0    pypi
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.35.1                     py_0  
widgetsnbextension        3.5.1                    pypi_0    pypi
xz                        5.2.5                h7b6447c_0  
yarl                      1.6.0                    pypi_0    pypi
zipp                      3.2.0                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0  

`

[MaherJendoubi]

@sahand68 Thanks for the detailed environment. I still doesn't able to reproduce it.
I am suscpecting a corrupted file during the git clone operation.
Please cd /InnerEye-DeepLearning/Tests/ML/test_data and run ls -altrh in order to check the size at least.
Then paste the result here to compare it with mine.
After that, let's run conda deactivate then run conda env remove --name InnerEye
I suggest to delete InnerEye-DeepLearning folder with rm -r command then retry again the steps.
If the problem persists, please share the detailed steps to investigate the issue.

[sahand68]

@MaherJendoubi
There you go:
'base) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning/Tests/ML/test_data$ ls -altrh
total 340K
-rw-rw-r-- 1 sahand sahand 20K Sep 28 20:11 042_slice_001.png
-rw-rw-r-- 1 sahand sahand 5.7K Sep 28 20:11 042_slice_001_contour.png
-rw-rw-r-- 1 sahand sahand 9.1K Sep 28 20:11 Val_outliers.txt
-rw-rw-r-- 1 sahand sahand 7.5K Sep 28 20:11 Val_agg_splits.csv
-rw-rw-r-- 1 sahand sahand 7.6K Sep 28 20:11 Test_agg_splits.csv
-rw-rw-r-- 1 sahand sahand 121 Sep 28 20:11 ResultsByMode.csv
-rw-rw-r-- 1 sahand sahand 794 Sep 28 20:11 ResultsByModeAndStructure.csv
-rw-rw-r-- 1 sahand sahand 11K Sep 28 20:11 prefix042_class1_slice_001.png
-rw-rw-r-- 1 sahand sahand 131 Sep 28 20:11 posterior_bladder.nii.gz
drwxrwxr-x 5 sahand sahand 4.0K Sep 28 20:11 plot_cross_validation
-rw-rw-r-- 1 sahand sahand 432 Sep 28 20:11 metrics_aggregates.csv
-rw-rw-r-- 1 sahand sahand 16K Sep 28 20:11 MetricsAcrossAllRuns.csv
-rw-rw-r-- 1 sahand sahand 5.6K Sep 28 20:11 image_scaled_and_contour.png
-rw-rw-r-- 1 sahand sahand 9.0K Sep 28 20:11 image_and_multiple_contours.png
-rw-rw-r-- 1 sahand sahand 5.6K Sep 28 20:11 image_and_contour.png
drwxrwxr-x 3 sahand sahand 4.0K Sep 28 20:11 hdf5_data
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 full_header_csv
-rw-rw-r-- 1 sahand sahand 30K Sep 28 20:11 dice_per_epoch_3classes.png
-rw-rw-r-- 1 sahand sahand 92K Sep 28 20:11 dice_per_epoch_15classes.png
-rw-rw-r-- 1 sahand sahand 1.2K Sep 28 20:11 dataset_with_full_header.csv
-rw-rw-r-- 1 sahand sahand 1.6K Sep 28 20:11 dataset.csv
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 classification_data_sub_fold_cv
drwxrwxr-x 3 sahand sahand 4.0K Sep 28 20:11 classification_data_generated_random
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 classification_data_2d
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 classification_data
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 checkpoints
drwxrwxr-x 3 sahand sahand 4.0K Sep 28 20:11 train_and_test_data
-rw-rw-r-- 1 sahand sahand 127 Sep 28 20:11 test_img.nii.gz
-rw-rw-r-- 1 sahand sahand 129 Sep 28 20:11 test_good.nii.gz
-rw-rw-r-- 1 sahand sahand 94 Sep 28 20:11 test_dataset_parameters.csv
-rw-rw-r-- 1 sahand sahand 131 Sep 28 20:11 smoothed_posterior_bladder.nii.gz
drwxrwxr-x 2 sahand sahand 4.0K Sep 28 20:11 sequence_data_for_classification
-rw-rw-r-- 1 sahand sahand 128 Sep 28 20:11 scale_and_unscale_image.nii.gz
-rw-rw-r-- 1 sahand sahand 11K Sep 28 20:11 prefix042_class2_slice_002.png
drwxrwxr-x 11 sahand sahand 4.0K Sep 28 20:11 ..
drwxrwxr-x 12 sahand sahand 4.0K Sep 28 20:11 .
'

[MaherJendoubi]

Here is mine :

[sahand68]

I just removed all the repo with rm -rf and the environment as well.
I cloned the repo again. and I still get the same error.

(InnerEye) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning/Tests/ML/test_data$ ls -altrh total 340K -rw-rw-r-- 1 sahand sahand 9.1K Sep 29 13:09 Val_outliers.txt -rw-rw-r-- 1 sahand sahand 7.5K Sep 29 13:09 Val_agg_splits.csv -rw-rw-r-- 1 sahand sahand 7.6K Sep 29 13:09 Test_agg_splits.csv -rw-rw-r-- 1 sahand sahand 121 Sep 29 13:09 ResultsByMode.csv -rw-rw-r-- 1 sahand sahand 794 Sep 29 13:09 ResultsByModeAndStructure.csv -rw-rw-r-- 1 sahand sahand 432 Sep 29 13:09 metrics_aggregates.csv -rw-rw-r-- 1 sahand sahand 16K Sep 29 13:09 MetricsAcrossAllRuns.csv -rw-rw-r-- 1 sahand sahand 5.6K Sep 29 13:09 image_scaled_and_contour.png -rw-rw-r-- 1 sahand sahand 9.0K Sep 29 13:09 image_and_multiple_contours.png -rw-rw-r-- 1 sahand sahand 5.6K Sep 29 13:09 image_and_contour.png drwxrwxr-x 3 sahand sahand 4.0K Sep 29 13:09 hdf5_data drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 full_header_csv -rw-rw-r-- 1 sahand sahand 30K Sep 29 13:09 dice_per_epoch_3classes.png -rw-rw-r-- 1 sahand sahand 92K Sep 29 13:09 dice_per_epoch_15classes.png -rw-rw-r-- 1 sahand sahand 1.2K Sep 29 13:09 dataset_with_full_header.csv -rw-rw-r-- 1 sahand sahand 1.6K Sep 29 13:09 dataset.csv drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 classification_data_sub_fold_cv drwxrwxr-x 3 sahand sahand 4.0K Sep 29 13:09 classification_data_generated_random drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 classification_data_2d drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 classification_data drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 checkpoints -rw-rw-r-- 1 sahand sahand 20K Sep 29 13:09 042_slice_001.png -rw-rw-r-- 1 sahand sahand 5.7K Sep 29 13:09 042_slice_001_contour.png drwxrwxr-x 3 sahand sahand 4.0K Sep 29 13:09 train_and_test_data -rw-rw-r-- 1 sahand sahand 127 Sep 29 13:09 test_img.nii.gz -rw-rw-r-- 1 sahand sahand 129 Sep 29 13:09 test_good.nii.gz -rw-rw-r-- 1 sahand sahand 94 Sep 29 13:09 test_dataset_parameters.csv -rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 smoothed_posterior_bladder.nii.gz drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 sequence_data_for_classification -rw-rw-r-- 1 sahand sahand 128 Sep 29 13:09 scale_and_unscale_image.nii.gz -rw-rw-r-- 1 sahand sahand 11K Sep 29 13:09 prefix042_class2_slice_002.png -rw-rw-r-- 1 sahand sahand 11K Sep 29 13:09 prefix042_class1_slice_001.png -rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 posterior_bladder.nii.gz drwxrwxr-x 5 sahand sahand 4.0K Sep 29 13:09 plot_cross_validation drwxrwxr-x 11 sahand sahand 4.0K Sep 29 13:09 ..

[sahand68]

the error stack:
`(InnerEye) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning$ python InnerEye/ML/runner.py --model=HelloWorld

Setting up logging to stdout.
Setting logging level to 20
2020-09-29T20:12:41Z INFO rpdb is handling traps. To debug: identify the main runner.py process, then as root: kill -TRAP <process_id>; nc 127.0.0.1 4444
2020-09-29T20:12:41Z INFO Found class HelloWorld in file /home/sahand/InnerEye-DeepLearning/InnerEye/ML/configs/segmentation/HelloWorld.py
2020-09-29T20:12:41Z INFO Creating the default output folder structure.
2020-09-29T20:12:41Z INFO Running outside of AzureML.
2020-09-29T20:12:41Z INFO All results will be written to a subfolder of the project root folder.
2020-09-29T20:12:41Z INFO Run outputs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld
2020-09-29T20:12:41Z INFO Logs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld/logs
2020-09-29T20:12:41Z INFO Creating the adjusted output folder structure.
2020-09-29T20:12:41Z INFO Running outside of AzureML.
2020-09-29T20:12:41Z INFO All results will be written to a subfolder of the project root folder.
2020-09-29T20:12:41Z INFO Run outputs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld
2020-09-29T20:12:41Z INFO Logs folder: /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld/logs
2020-09-29T20:12:41Z INFO extra_code_directory is unset
Setting logging level to 20
Setting up logging with level 20 to file /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld/logs/stdout.txt
2020-09-29T20:12:43Z INFO Setting multiprocessing start method to 'forkserver'
2020-09-29T20:12:43Z INFO Model training will use the local dataset provided in /home/sahand/InnerEye-DeepLearning/Tests/ML/test_data
2020-09-29T20:12:43Z INFO
Arguments:
__center_size_param_value: None
__dataset_data_frame_param_value: None
__inference_stride_size_param_value: None
__largest_connected_component_foreground_classes_param_value: None
__min_l_rate_param_value: 0
__model_category_param_value: ModelCategory.Segmentation
__model_name_param_value: HelloWorld
__overrides_param_value: None
__use_gpu_param_value: False
_architecture_param_value: UNet3D
_class_weights_param_value: [0.02, 0.49, 0.49]
_colours_param_value: [(130, 183, 14), (238, 127, 26)]
_comparison_blob_storage_paths_param_value: None
_crop_size_param_value: (64, 64, 64)
_datasets_for_inference: None
_datasets_for_training: None
_feature_channels_param_value: [4]
_file_system_config_param_value:
_fill_holes_param_value: [True, True]
_ground_truth_ids_display_names_param_value: ['region', 'region_1']
_ground_truth_ids_param_value: ['region', 'region_1']
_image_channels_param_value: ['channel1', 'channel2']
_instance__params : {}
_l_rate_multi_step_milestones_param_value: None
_level_param_value: 50
_local_dataset_param_value: /home/sahand/InnerEye-DeepLearning/Tests/ML/test_data
_mask_id_param_value: mask
_multiprocessing_start_method_param_value: MultiprocessingStartMethod.forkserver
_name_param_value : HelloWorld00008
_norm_method_param_value: PhotometricNormalizationMethod.CtWindow
_num_dataload_workers_param_value: 0
_num_epochs_param_value: 2
_param_watchers : {}
_save_start_epoch_param_value: 1
_save_step_epochs_param_value: 1
_slice_exclusion_rules_param_value: []
_start_epoch_param_value: 0
_summed_probability_rules_param_value: []
_tail_param_value : None
_test_crop_size_param_value: (64, 64, 64)
_test_diff_epochs_param_value: 1
_test_start_epoch_param_value: 2
_test_step_epochs_param_value: 1
_train_batch_size_param_value: 2
_use_mixed_precision_param_value: True
_window_param_value: 200
initialized : True
param : <param.parameterized.Parameters object at 0x7f3947841828>

2020-09-29T20:12:43Z INFO
2020-09-29T20:12:43Z INFO **** STARTING: Model training **********************************************************************
2020-09-29T20:12:43Z INFO
2020-09-29T20:12:43Z INFO Train: 3, Test: 1, and Val: 2. Total subjects: 6
2020-09-29T20:12:43Z INFO Model Training: Random seed set to: 42
2020-09-29T20:12:43Z INFO Starting to read and parse the datasets.
2020-09-29T20:12:43Z INFO Processing dataset (name=None)
2020-09-29T20:12:43Z INFO Processing dataset (name=None)
2020-09-29T20:12:43Z INFO Creating the data loader for the training set.
2020-09-29T20:12:43Z INFO Creating the data loader for the validation set.
2020-09-29T20:12:43Z INFO Finished creating the data loaders.
2020-09-29T20:12:43Z INFO Models are saved at /home/sahand/InnerEye-DeepLearning/outputs/2020-09-29T201241Z_HelloWorld/checkpoints
2020-09-29T20:12:43Z INFO Writing model summary to: logs/model_summaries/model_log001.txt
Attempted to log scalar metric LoggingColumns.NumTrainableParameters:
506139
2020-09-29T20:12:43Z INFO Making no adjustments to the model because no GPU was found.
2020-09-29T20:12:43Z INFO Starting training
2020-09-29T20:12:43Z INFO Starting epoch 1
2020-09-29T20:12:43Z ERROR Model training/testing failed. Exception: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:107:
sitk::ERROR: Unable to determine ImageIO reader for "/home/sahand/InnerEye-DeepLearning/Tests/ML/test_data/train_and_test_data/id1_channel1.nii.gz"
Traceback (most recent call last):
File "InnerEye/ML/runner.py", line 364, in run_in_situ
self.create_ml_runner().run()
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 308, in run
model_train(self.model_config, run_recovery)
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/model_training.py", line 150, in model_train
train_epoch_results = train_or_validate_epoch(training_steps)
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/model_training.py", line 267, in train_or_validate_epoch
for batch_index, sample in enumerate(train_val_params.data_loader):
File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in next
data = self._next_data()
File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 403, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/dataset/cropping_dataset.py", line 38, in getitem
sample=super().get_samples_at_index(index=i)[0],
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/dataset/full_image_dataset.py", line 289, in get_samples_at_index
samples = [io_util.load_images_from_dataset_source(dataset_source=ds)] # type: ignore
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 357, in load_images_from_dataset_source
images = [load_nifti_image(channel, ImageDataType.IMAGE.value) for channel in dataset_source.image_channels]
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 357, in
images = [load_nifti_image(channel, ImageDataType.IMAGE.value) for channel in dataset_source.image_channels]
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 182, in load_nifti_image
img, header = read_image_as_array_with_header(path)
File "/home/sahand/InnerEye-DeepLearning/InnerEye/ML/utils/io_util.py", line 146, in read_image_as_array_with_header
image: sitk.Image = sitk.ReadImage(str(file_path))
File "/home/sahand/anaconda3/envs/InnerEye/lib/python3.7/site-packages/SimpleITK/SimpleITK.py", line 8876, in ReadImage
return _SimpleITK.ReadImage(*args)
RuntimeError: Exception thrown in SimpleITK ReadImage: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:107:
sitk::ERROR: Unable to determine ImageIO reader for "/home/sahand/InnerEye-DeepLearning/Tests/ML/test_data/train_and_test_data/id1_channel1.nii.gz"
`

[MaherJendoubi]

How about this command?

[sahand68]

(base) sahand@sahand-System-Product-Name:~/InnerEye-DeepLearning/Tests/ML/test_data/train_and_test_data$ ls -altrh
total 60K
-rw-rw-r-- 1 sahand sahand 1.2K Sep 29 13:09 scalar_prediction_target_metrics.csv
-rw-rw-r-- 1 sahand sahand 1006 Sep 29 13:09 scalar_epoch_metrics.csv
-rw-rw-r-- 1 sahand sahand 152 Sep 29 13:09 metrics.csv
-rw-rw-r-- 1 sahand sahand 405 Sep 29 13:09 metrics_aggregates.csv
-rw-rw-r-- 1 sahand sahand 130 Sep 29 13:09 id2_region.nii.gz
-rw-rw-r-- 1 sahand sahand 128 Sep 29 13:09 id2_mask.nii.gz
-rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 id2_channel2.nii.gz
-rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 id2_channel1.nii.gz
-rw-rw-r-- 1 sahand sahand 130 Sep 29 13:09 id1_region.nii.gz
-rw-rw-r-- 1 sahand sahand 128 Sep 29 13:09 id1_mask.nii.gz
-rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 id1_channel2.nii.gz
-rw-rw-r-- 1 sahand sahand 131 Sep 29 13:09 id1_channel1.nii.gz
drwxrwxr-x 2 sahand sahand 4.0K Sep 29 13:09 checkpoints
drwxrwxr-x 12 sahand sahand 4.0K Sep 29 13:09 ..
drwxrwxr-x 3 sahand sahand 4.0K Sep 29 13:09 .

[MaherJendoubi]

OK. Did you run git lfs install before cloning the git repo?

[sahand68]

Hey! you got it man! thank you so much, somehow I had missed that! my bad!

[MaherJendoubi]

@sahand68 You're welcome!

[ant0nsc]

Thanks a lot @MaherJendoubi for helping @sahand68 resolve this :-) Closing this issue now.