RuntimeError: More than one spatial_shape found in subject images

[ruby0101]

Hi, I am trying to train U-Net model for tumor segmentation with my MRI T1 image dataset. I just followed the tutorial
But when I call a train function, it gave me the following error:

RuntimeError: More than one spatial_shape found in subject images: {'brain': (76, 76, 48), 'mri': (75, 75, 48)}

I have already applied training_transform which is in the tutorial tio.CropOrPad((48, 60, 48)) to the training and validation dataset. But the error message, it seems like transformation didn't work properly. The processed sample size is {'brain': (76, 76, 48), 'mri': (75, 75, 48)}. Could you help me to solve it, please? What is a problem with it?

[fepegar]

Hi, @ruby0101. It's hard to diagnose without any code or data, but my guess is your brain and mri images have different associated affine matrices. Do you think you can provide a minimally working example as asked in the bug report template?

[fepegar]

Could you check my code please?

I would appreciate it if you could paste syntax-highlighted code and upload images here, on GitHub, so I don't have to go to all those links.

I am just wondered how do I feed original image data for training dataset after I updated it with label data. Could you clarify me on this please?

I am not sure what you mean, sorry. Could you elaborate?

[ruby0101]

Thanks a lot, finally I have solved the affine issues.

[ruby0101]

Model training part, another issue is raised. When training the model, it gave the below error.

I checked the tutorial notebook file, torch sizes are as same as mine.

Could you check it please? why it gives me the torch size error in my code?

[fepegar]

How do you instantiate the data loader?

[ruby0101]

Finally, it only worked with training_batch_size = 1 and num_workers=0 as shown below, otherwise it gives me the above mentioned error:

A learning curve as follows, the model is overfitted:

Do you know what preprocessing is recommended for preventing overfitting in this case? How can I check which part leads to overfitting in this MRI image dataset? Sorry, I am new to this area, that's why I might miss some important preprocessing.
I applied a transformation as same as the tutorial.

[mattwarkentin]

Hi @fepegar,

I just wanted to jump into this thread instead of creating an entirely new one. Do you think there should be some sort of way for the user to configure the tolerance for comparing the consistency of some of the attributes, such as the spacing/affine? I have found that rounding errors occur when some files are converted between types (DICOM <-> NRRD, for example), but they should have almost no practical effect.

Of course, in these cases I can simply resample the mask to the image space before, let's say, resampling them both to isotropy or doing some other transform. But this means I am having to resample twice in my preprocessing pipeline rather than just once (to isotropy), which I would prefer to avoid.

What are your thoughts? PyRadiomics uses 1e-16 by default, but I have had Slicer/pyrad devs tell me this tolerance may be too strict so I have often had to override this. Maybe the user should be able to pass a tolerance to torchio.Subject() and, if present, this will get used instead of the internal tolerance levels. Thoughts?

torchio/torchio/data/subject.py

Lines 237 to 255 in 6533cae

	def check_consistent_affine(self):
	# https://github.com/fepegar/torchio/issues/354
	affine = None
	first_image = None
	iterable = self.get_images_dict(intensity_only=False).items()
	for image_name, image in iterable:
	if affine is None:
	affine = image.affine
	first_image = image_name
	elif not np.allclose(affine, image.affine, rtol=1e-6, atol=1e-6):
	message = (
	f'Images "{first_image}" and "{image_name}" do not occupy'
	' the same physical space.'
	f'\nAffine of "{first_image}":'
	f'\n{pprint.pformat(affine)}'
	f'\nAffine of "{image_name}":'
	f'\n{pprint.pformat(image.affine)}'
	)
	raise RuntimeError(message)

[mattwarkentin]

I too would be interested in a simpler solution to very slight differences in the image affine (which in my case are due to differences in precision between DICOM and NRRD, and not real differences) instead of having to resample.

Either adding a transform to copy affine between subject volumes
I like this option I think. A transform where select images/masks are coerced to use the image affine of a chosen image might be simplest.

[romainVala]

i think a preprocessing transform should be easy to write,
let see what @fepegar say

[fepegar]

Which errors are you guys getting? Could you please paste the full traceback here?

[mattwarkentin]

I won't speak for @romainVala, but for me, my images (ScalarImage()) are usually in DICOM directories, and my masks (LabelMap()) are usually NRRD format. The spacings ARE the exact same, but due to differences in how they are stored, they differ by some small amount of rounding error.

Trying to perform transforms on these will throw errors due to different spacings. Currently my options are to resample the mask to the image space (or vice versa). But since I know they should have the exact same spacing, it would be less costly to just force the mask to adopt the affine matrix of the image.

[fepegar]

Thanks, @mattwarkentin.

Trying to perform transforms on these will throw errors due to different spacings

Could you please provide a pair of images and some code to reproduce the error?

[romainVala]

Hi
I can send you images if really needed, but it does not matter, as soon as you have 2 images with different qform
you can create on as you pointed out here #567 (comment)

About the code, Simple transform does not care about different affine :
for instance

i1 = tio.LabelMap('mask.nii')
i2  = tio.ScalarImage('T1.nii')
suj = tio.Subject({'t1':i2, 'mask':i1})
t = tio.CropOrPad([32,32,32])

in this case

suj_crop = t(suj)

just works fine for me. (any other tio transform too)
Actually only the call to check_consistent_affine is made only in tio.WeightedSampler

it must then be that @mattwarkentin also use a WeightedSampler

tio.data.WeightedSampler(64,'mask')
for path in tw(suj):
    print(patch['index_ini'])

does give me an error

/data/romain/toolbox_python/torchio/torchio/data/sampler/weighted.py in __call__(self, subject, num_patches)
     62             num_patches: Optional[int] = None,
     63             ) -> Generator[Subject, None, None]:
---> 64         subject.check_consistent_space()
     65         if np.any(self.patch_size > subject.spatial_shape):
     66             message = (

/data/romain/toolbox_python/torchio/torchio/data/subject.py in check_consistent_space(self)
    264     def check_consistent_space(self):
    265         self.check_consistent_spatial_shape()
--> 266         self.check_consistent_affine()
    267 
    268     def get_images_dict(

/data/romain/toolbox_python/torchio/torchio/data/subject.py in check_consistent_affine(self)
    260                     f'\n{pprint.pformat(image.affine)}'
    261                 )
--> 262                 raise RuntimeError(message)
    263 
    264     def check_consistent_space(self):

RuntimeError: Images "t1" and "mask" do not occupy the same physical space.

[mattwarkentin]

I was mistaken, the resampling works even with different spacings for the image and mask. However, if there is a very small difference in the affine spacing (due to, let's say, rounding error from different storage formats), this may cause an "off-by-one" issue with the new spatial shape after resampling. This will cause errors in downstream transforms, for example, tio.CropOrPad() or tio.Mask().

import numpy as np
import torch
import torchio as tio

img = torch.randn(1, 512, 512, 100)
mask = torch.tensor(img > 0).type(torch.int16)

af1 = np.array([[0.8, 0, 0, 0],
                [0, 0.8, 0, 0],
                [0, 0, 2.50000000000001, 0],
                [0, 0, 0, 1]])

af2 = np.array([[0.8, 0, 0, 0],
                [0, 0.8, 0, 0],
                [0, 0, 2.5, 0],
                [0, 0, 0, 1]])

sub = tio.Subject(
  image = tio.ScalarImage(tensor = img, affine = af1),
  mask = tio.LabelMap(tensor = mask, affine = af2)
)

sub.spacing # throws error

rs = tio.Resample() # Resample to isotropy
sub_rs = rs(sub)

sub_rs.spacing # no longer an error

sub_rs.spatial_shape # throws error

tio.CropOrPad(50)(sub_rs) # throws error

tio.Mask(masking_method = "mask")(sub_rs)  # throws error

[romainVala]

ok, thanks for the detail example,
indeed, small difference (in voxel resolution) does have some consequence
at least the fernando's proposition to replace your
tio.Resample()
with

resample = tio.Compose([
    tio.Resample(include=['image']),
    tio.Resample(target='image', exclude=['image']),
])

make the job

Thanks, Romain, that's helpful. An option is to remove the constraint that both images need the same affine. That will leave that responsibility to the user but will avoid getting errors due to tiny differences. The change in the code would be replacing subject.check_consistent_space() with subject.check_consistent_spatial_shape(). What do you guys think?

given this example, this will not be enough

Either we go to
1_ allowing overight of affin in the image init
2_ or adding a transform to copy affine, spacing ect ...

1_ would be simpler no ?

[romainVala]

Actually a transform CopyAffine, is also easy to do, and easy to insert in existing code,
we have a local version,
@fepegar should we propose a PR ?

[romainVala]

@fepegar any preference between the 2 options ?

[fepegar]

Option 1 is not practical because if you try to read the affine property of an image, the data will be loaded to RAM and that might not be desired. We could modify the behavior to read the affine using NiBabel without loading the data, but that's a different topic. I suppose a CopyAffine transform would be useful!

[romainVala]

I am surprise this check is not needed elsewhere, but ok, one solution would be to remove it,
But when different affine are in place this is often a sign that the user must double check what is going on.
So I thing to give an error here is a good thing, though it is annoying for tiny differences, but coding is often anoying ...

So I would prefer, to add an argument to WeigtedSampler, to explicitly ask to not do the check ...

tio.data.WeightedSampler(64,'mask', check_affine_consitency=False)

May be an other solution (instead of creating a new transform, that make an affine copy) would be to specify it at the creation
something like that

i2  = tio.ScalarImage('T1.nii')
i1 = tio.LabelMap('mask.nii', affine = i2.affine)
suj = tio.Subject({'t1':i2, 'mask':i1})

but the affine argument seems not to be taken into account

In [73]: np.max(np.abs(suj.t1.affine - suj.mask.affine))
Out[73]: 0.00064480513012474

I thing this argument was only though for tensor input ...

[ryancinsight]

@fepegar i’m getting this error trying to implement a cGan on T1 and T2 image sets. However, Croporpad does not seem to be working when subject dataset has multiple scaled image types. Is this possible or can you only have one scalar image and one mask?

[mattwarkentin]

Hmm, I tried looking into this, I wonder if this is a bug @fepegar. Should tio.CropOrPad() work on a subject with scalar images that possibly have different spacings and spatial shapes? It seems like it should, unless I am missing something.

Here is some example code to play with:

import torchio as tio

# These `ScalarImage`s have different spacings and orientations
brain = tio.datasets.Colin27().t1
ct = tio.datasets.Slicer('CTChest').CT_chest

sub = tio.Subject(ct=ct, brain=brain)

# This raises an error
tio.CropOrPad(100)(sub)

RuntimeError: More than one spatial_shape found in subject images:
{'brain': (181, 217, 181), 'ct': (512, 512, 139)}

# Transform images in same orientation and spacing
transforms = tio.Compose([
    tio.ToCanonical(),
    tio.Resample(1)
])

sub_t = transforms(sub)

# Still errors
tio.CropOrPad(100)(sub_t)

RuntimeError: More than one spatial_shape found in subject images:
{'brain': (181, 217, 181), 'ct': (391, 391, 348)}

[ryancinsight]

I get the same error with EnsureShapeMultiple as well. I'm not sure I understand why the check is at the beginning. I can set T1=T1 and T2=T1 so that they are the same image and these seem to work. Thus, the only way I've been able to bypass is create subjects zip style and apply the croporpad individually.

`transform = tio.Compose([
tio.ToCanonical(),
tio.Resample((2,2,2)),
tio.CropOrPad((100,100,100)),
])

subjects = [tio.Subject(
T1=transform(tio.ScalarImage(T1_path)),
T2=transform(tio.ScalarImage(T2_path)),
PD=transform(tio.ScalarImage(PD_path)),
) for (T2_path, T1_path, PD_path) in tqdm(zip(T2_paths, T1_paths, PD_paths))]
dataset = tio.SubjectsDataset(subjects)`

[romainVala]

Well the way it is implemented, you need to have the same spatial shape if you have multiple images in your subject
(because it compute one pad and/or one crop from the images shape compare to the request end shape.)

but I see your point it make sense to have different shape (that is why you add this croporpad transform, to have the same shape at the end)

It is possible to change the code : adding a loop over each subject image, to perform a specific crop and / or pad. (like it is done in the resample transform)

[fepegar]

Hmm I think this is a bit delicate. If the images have different shapes, chances are they are not in the same physical space, so their orientation and spacing might be different. Using CropOrPad would probably produce undesirable results. For those cases, Resample should be used instead.

@ryancinsight, could you please share a pair of your images so we understand the issue a bit better?

[mattwarkentin]

@fepegar I think it would be nice to have an FAQ or "common issues" section to provide small explanations and code snippets for these sorts of examples. It might not be immediately obvious (it wasn't to me) that tio.Resample(target='name') doesn't just resample all the images to the spacing of name, but also to have the same spatial shape.

[fepegar]

We agree then :)

[romainVala]

one should also emphasis that this Resample transform add an extra interpolation. then the solution with 2 CropOrPad, is still worth to mention ... I think ...

[ryancinsight]

I didn't realize this was possible, so if already registered I can resample all to the same image space with target='name' ?

[mattwarkentin]

@ryancinsight Yes, exactly!