Implementation of image size agnostic model in FastPathology #173

MarkusDrange · 2023-05-03T13:07:20Z

MarkusDrange
May 3, 2023

We are attempting to integrate a model in FastPathology where the image size can be arbitrary.

We export the ONNX model with the following dynamic axis (the pytorch lightning to_onnx() is used) :

out_dict = {'input' : [0, 2, 3]}
for i in range(9):
    out_dict[f'output{i}'] = [0]

model.to_onnx(
[...]
   dynamic_axes=out_dict
)

And then using the following attributes in the pipeline:

Attribute patch-size 256 256
Attribute input-nodes input:256,256,3
Attribute dimension-ordering "channel-first"

We however get the following error message:

Error parsing pipeline! OpenVINO exception caught: Check 'input_shape[1].compatible(filters_shape[1])' failed at src/core/shape_inference/include/convolution_shape_inference.hpp:324:
While validating node 'v1::Convolution /net/net/mobilevit/conv_stem/convolution/Conv/WithoutBiases (/net/Div[0]:f32[?,256,256,3], onnx::Conv_1819[0]:f32[16,3,3,3]) -> (f32[?,16,1..,1..])' with friendly_name '/net/net/mobilevit/conv_stem/convolution/Conv/WithoutBiases':
Data batch channel count (256) does not match filter input channel count (3).

If we change the order of the input-nodes attribute values to 3,256,256 we get:

FAST exception caught: Input image sent to executeNetwork has incorrect nr of channels: 3. Expected: 256.

Is this a bug or is there a way to solve this?

Full pipeline:

PipelineName "Tissue multiclass"
PipelineDescription "Multi task patch-wise image classification trained from ADP dataset"
PipelineInputData WSI "Whole-slide image"
PipelineOutputData E stitcher 0
PipelineOutputData C stitcher1 0
PipelineOutputData H stitcher2 0
PipelineOutputData S stitcher3 0
PipelineOutputData A stitcher4 0
PipelineOutputData M stitcher5 0
PipelineOutputData N stitcher6 0
PipelineOutputData G stitcher7 0
PipelineOutputData T stitcher8 0

Attribute classes "Tissue"

### Processing chain

ProcessObject tissueSeg TissueSegmentation
Attribute threshold 85
Input 0 WSI

ProcessObject patch PatchGenerator
Attribute patch-size 256 256
Attribute patch-magnification 20
Attribute patch-overlap 0.0
Attribute mask-threshold 0.05
Input 0 WSI
Input 1 tissueSeg 0

ProcessObject network NeuralNetwork
Attribute inference-engine "OpenVINO" 
#"ONNXRuntime"
#Attribute scale-factor 0.00392156862
Attribute model "$CURRENT_PATH$/../models/ViT16-light-augmentation-jitter_test_convert.onnx"
Attribute input-nodes input:256,256,3
Attribute dimension-ordering "channel-first"
Input 0 patch 0

ProcessObject stitcher PatchStitcher
Input 0 network 0

ProcessObject stitcher1 PatchStitcher
Input 0 network 1

ProcessObject stitcher2 PatchStitcher
Input 0 network 2

ProcessObject stitcher3 PatchStitcher
Input 0 network 3

ProcessObject stitcher4 PatchStitcher
Input 0 network 4
ProcessObject stitcher5 PatchStitcher
Input 0 network 5
ProcessObject stitcher6 PatchStitcher
Input 0 network 6
ProcessObject stitcher7 PatchStitcher
Input 0 network 7
ProcessObject stitcher8 PatchStitcher
Input 0 network 8

### Renderers
Renderer imgRenderer ImagePyramidRenderer
Input 0 WSI

Renderer heatmap HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "green"
Input 0 stitcher 0

Renderer heatmap1 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "blue"
Input 0 stitcher1 0

Renderer heatmap2 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "yellow"
Input 0 stitcher2 0

Renderer heatmap3 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "black"
Input 0 stitcher3 0

Renderer heatmap4 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "magenta"
Input 0 stitcher4 0

Renderer heatmap5 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "white"
Input 0 stitcher5 0

Renderer heatmap6 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "cyan"
Input 0 stitcher6 0

Renderer heatmap7 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "brown"
Input 0 stitcher7 0

Renderer heatmap8 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "red"
Input 0 stitcher8 0

Answered by smistad

Sep 11, 2023

Yes, but TensorRT requires you to set the min, mean and max shapes which is not possible to do with text pipelines yet. With OpenVINO you are recommended to set these limits, but you don't have to.

View full answer

smistad · 2023-05-03T13:17:44Z

smistad
May 3, 2023
Maintainer

The support for dynamic input shapes is a bit buggy. It has been improved in FAST version 4.7 which is not released yet. You could download one of the latest artifacts and test

9 replies

MarkusDrange May 3, 2023
Author

We are running one of the latest artifacts of FastPathology, and a very recent version of FAST, hence this likely still an issue. Most likely a bug that appears when using dynamic shape and channel first.

smistad May 11, 2023
Maintainer

You can try and turn on verbose when running FastPathology. "./fastpathology --verbose" and look at the output when loading the network. It should say what it guesses the dimension ordering to be.
But yes I think you are right about this bug: FAST guesses the ordering to be channel last, because the width and dimensions are set to -1: https://github.com/smistad/FAST/blob/master/source/FAST/Algorithms/NeuralNetwork/InferenceEngines/OpenVINOEngine.cpp#L91

andreped Sep 10, 2023

Has this been resolved in pyFAST==4.7.1, @smistad?
I might give it a go soon to integrate @MarkusDrange's model into FP.

smistad Sep 11, 2023
Maintainer

Yes, but TensorRT requires you to set the min, mean and max shapes which is not possible to do with text pipelines yet. With OpenVINO you are recommended to set these limits, but you don't have to.

Answer selected by MarkusDrange

andreped Sep 11, 2023

@smistad OK, then I can run a test using OpenVINO to see if I can deploy @MarkusDrange's model in FP.

smistad Sep 11, 2023
Maintainer

Also the model has to be exported with dynamic dimensions (-1).

andreped Sep 11, 2023

Just tried now. His model works just fine on macOS with AMD GPU with the latest FP release (v1.1.2). Don't have a machine currently which supports TensorRT.

The preprocessing is performed within the model, but the network should have dynamic input shape, so that seems to work as intended :]

I also tried changing the input size to the network, and this seems to work as intended, at least with OpenVINO IE. Great!

For anyone that might be interested, this was the final FPL used in this project:

PipelineName "Tissue multiclass"
PipelineDescription "Multi task patch-wise image classification trained from ADP dataset"
PipelineInputData WSI "Whole-slide image"
PipelineOutputData Epithelial stitcher 0
PipelineOutputData Connective stitcher1 0
PipelineOutputData Blood stitcher2 0
PipelineOutputData Skeletal stitcher3 0
PipelineOutputData Adipose stitcher4 0
PipelineOutputData Muscular stitcher5 0
PipelineOutputData Nervous stitcher6 0
PipelineOutputData Glandular stitcher7 0
PipelineOutputData TransportVessel stitcher8 0

Attribute classes "Tissue"

### Processing chain

ProcessObject tissueSeg TissueSegmentation
Attribute threshold 65
Input 0 WSI

ProcessObject patch PatchGenerator
Attribute patch-size 136 136
Attribute patch-magnification 20
Attribute patch-overlap 0.0
Attribute mask-threshold 0.05
Input 0 WSI
Input 1 tissueSeg 0

ProcessObject network NeuralNetwork
Attribute inference-engine "OpenVINO"
Attribute model "$CURRENT_PATH$/../models/MobileViT_multiclass_model.onnx"
Attribute input-nodes input:3,272,272
Attribute dimension-ordering "channel-first"
Input 0 patch 0

ProcessObject stitcher PatchStitcher
Input 0 network 0

ProcessObject stitcher1 PatchStitcher
Input 0 network 1

ProcessObject stitcher2 PatchStitcher
Input 0 network 2

ProcessObject stitcher3 PatchStitcher
Input 0 network 3

ProcessObject stitcher4 PatchStitcher
Input 0 network 4

ProcessObject stitcher5 PatchStitcher
Input 0 network 5

ProcessObject stitcher6 PatchStitcher
Input 0 network 6

ProcessObject stitcher7 PatchStitcher
Input 0 network 7

ProcessObject stitcher8 PatchStitcher
Input 0 network 8


### Renderers
Renderer imgRenderer ImagePyramidRenderer
Input 0 WSI

Renderer heatmap HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "green"
Input 0 stitcher 0

Renderer heatmap1 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "blue"
Input 0 stitcher1 0

Renderer heatmap2 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "yellow"
Input 0 stitcher2 0

Renderer heatmap3 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "black"
Input 0 stitcher3 0

Renderer heatmap4 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "magenta"
Input 0 stitcher4 0

Renderer heatmap5 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "white"
Input 0 stitcher5 0

Renderer heatmap6 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "cyan"
Input 0 stitcher6 0

Renderer heatmap7 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "brown"
Input 0 stitcher7 0

Renderer heatmap8 HeatmapRenderer
Attribute interpolation false
Attribute channel-colors "0" "red"
Input 0 stitcher8 0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of image size agnostic model in FastPathology #173

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Implementation of image size agnostic model in FastPathology #173

MarkusDrange May 3, 2023

Replies: 1 comment · 9 replies

smistad May 3, 2023 Maintainer

MarkusDrange May 3, 2023 Author

smistad May 11, 2023 Maintainer

andreped Sep 10, 2023

smistad Sep 11, 2023 Maintainer

andreped Sep 11, 2023

smistad Sep 11, 2023 Maintainer

andreped Sep 11, 2023

MarkusDrange
May 3, 2023

Replies: 1 comment 9 replies

smistad
May 3, 2023
Maintainer

MarkusDrange May 3, 2023
Author

smistad May 11, 2023
Maintainer

smistad Sep 11, 2023
Maintainer

smistad Sep 11, 2023
Maintainer