OHIF deepgrow model gives Nonetype foreground interactions

[Douwe-Spaanderman]

Describe the bug
When using monailabel with OHIF I receive Nonetypes in my interactions on the z-axis. This happens specifically after correctly segmenting a sequence and moving to the next sequence in the same patient. Note, clearing points/interactions, making sure to remove previous segmentation, etc. did not seem to fix the issue. However, was able to fix the issue by refreshing the patient. Bug occurs independent of sample, i.e. if I refresh the patient I am able to segment the specific sequence.

Server logs
'foreground': [[114.18579234972675, 212.28415300546447, 10], [154.40437158469945, 174.1639344262295, 10], [171.54098360655735, 221.37704918032787, 10], [150.55737704918033, 225.224043715847, 13], [120.48087431693988, 162.62295081967213, 13], [178.88524590163934, 159.82513661202185, 13], [109.57377049180327, 179.0601092896175, None], [154.68852459016392, 178.36065573770492, None], [137.55191256830602, 202.49180327868854, None], [120.76502732240435, 176.2622950819672, None], [154.68852459016392, 187.80327868852459, None], [129.1584699453552, 207.3879781420765, None]], 'background': [], 'result_extension': '.nrrd', 'result_dtype': 'uint16', 'result_compress': False, 'description': 'Volumetric Interactivenet'}
[2024-03-15 13:24:41,491] [1] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2024-03-15 13:24:41,491] [1] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'foreground', 'background', 'result_extension', 'result_dtype', 'result_compress', 'description', 'image_path']
[2024-03-15 13:24:41,520] [1] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 0.0278; image: torch.Size([320, 320, 15])(torch.float32)
[2024-03-15 13:24:41,521] [1] [MainThread] [ERROR] (uvicorn.error:434) - Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 276, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 92, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 147, in simple_response
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "/usr/local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 180, in api_run_inference
return run_inference(background_tasks, model, image, session_id, params, file, label, output)
File "/usr/local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 162, in run_inference
result = instance.infer(request)
File "/usr/local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 307, in infer
result_file_name, result_json = task(request)
File "/usr/local/lib/python3.9/site-packages/monailabel/tasks/infer/basic_infer.py", line 295, in call
data = self.run_pre_transforms(data, pre_transforms)
File "/usr/local/lib/python3.9/site-packages/monailabel/tasks/infer/basic_infer.py", line 386, in run_pre_transforms
return run_transforms(data, transforms, log_prefix="PRE", use_compose=False)
File "/usr/local/lib/python3.9/site-packages/monailabel/interfaces/utils/transform.py", line 106, in run_transforms
data = t(data)
File "/usr/local/lib/python3.9/site-packages/monai/apps/deepgrow/transforms.py", line 571, in call
clicks = list(np.array(clicks, dtype=int))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

To Reproduce

1. Foreground interactions on sequence X of patient Z
2. Run deepgrow model
3. Submit label
4. Continue to sequence Y of patient Z (Note, in same patient)
   5a. (Clear points / previous segmentation map)
   5b. Create new segmentation map
5. Foreground interactions on sequence Y of patient Z
6. Run deepgrow model
7. BUG -> Nonetype in foreground interactions

Expected behavior
Should be able to work on multiple sequences in the same patient.

Environment
pytorch=1.12.1+cu113
monai==1.2.0
monailabel==0.7.0

[diazandr3s]

Hi @Douwe-Spaanderman,

Thanks for reporting this.
Which OHIF version are you using?
Unfortunately, if you're using OHIFv3, DeepGrow is not supported on this version: https://github.com/Project-MONAI/MONAILabel/releases/tag/0.8.0
If you want to use DeepGrow on OHIF, I'd suggest using the V2.

Hope this helps,

[Douwe-Spaanderman]

We are using OHIF V2 as we were aware that OHIFv3 for DeepGrow was not yet supported, so that can't be the issue here unfortunately...

[diazandr3s]

Hi @Douwe-Spaanderman,

Sorry, I missed a part of your post. I see that you can run inference when refreshing the patient.

Also, checking the logs in more detail I see the None values in the foreground array:

'foreground': [[114.18579234972675, 212.28415300546447, 10], [154.40437158469945, 174.1639344262295, 10], [171.54098360655735, 221.37704918032787, 10], [150.55737704918033, 225.224043715847, 13], [120.48087431693988, 162.62295081967213, 13], [178.88524590163934, 159.82513661202185, 13], [109.57377049180327, 179.0601092896175, None], [154.68852459016392, 178.36065573770492, None], [137.55191256830602, 202.49180327868854, None], [120.76502732240435, 176.2622950819672, None], [154.68852459016392, 187.80327868852459, None], [129.1584699453552, 207.3879781420765, None]]

I'm not sure why this is happening. Are you using a modified version of the Radiology app?

'description': 'Volumetric Interactivenet'}