Methodology behind 3D DeepGrow

[Zrrr1997]

Hello,

First of all - this is a really cool and helpful project for the scientific community! Thank you!

I'm currently doing research into interactive segmentation models to save annotation time in a clinical setting. I have a brief question about DeepGrow's 3D architecture. The original paper doesn't mention how to extend the approach to 3D volumes but rather iterates over each (image slice + pos/neg click guidance maps). I suppose one could do that and just concatenate the individual segmentations to obtain a 3D volume, albeit with spatial incoherences as each slice is processed independently.

I was wondering whether that's the methodology implemented for extending DeepGrow to 3D, or whether a 3D-UNet is used as a backbone? I am still going through the code, but would really appreciate it (and maybe others who are interested) if you could confirm this.

P.S. If this is the case, how would one handle slices which do not have guidance maps? Are the labels propagated to the unannotated slices with e.g. dense CRFs?

Best,
Zdravko

[SachidanandAlle]

following pointers might be useful for you deep dive:
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/deepgrow.py#L109-L111
https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/deepgrow/transforms.py

[diazandr3s]

Hello,

First of all - this is a really cool and helpful project for the scientific community! Thank you!

I'm currently doing research into interactive segmentation models to save annotation time in a clinical setting. I have a brief question about DeepGrow's 3D architecture. The original paper doesn't mention how to extend the approach to 3D volumes but rather iterates over each (image slice + pos/neg click guidance maps). I suppose one could do that and just concatenate the individual segmentations to obtain a 3D volume, albeit with spatial incoherences as each slice is processed independently.

I was wondering whether that's the methodology implemented for extending DeepGrow to 3D, or whether a 3D-UNet is used as a backbone? I am still going through the code, but would really appreciate it (and maybe others who are interested) if you could confirm this.

P.S. If this is the case, how would one handle slices which do not have guidance maps? Are the labels propagated to the unannotated slices with e.g. dense CRFs?

Best, Zdravko

Hi @Zrrr1997,

Thanks for opening this discussion.
In addition to what @SachidanandAlle commented out, 3D Deepgrow uses a 3D-UNET. You can see here the network definition for this model.

There is also another model that generalizes the DeepGrow model, the DeepEdit. With the DeepEdit model, we can do both automatic segmentation and interactive segmentation.

Here are more details of how this model works.

Regarding this question:

how would one handle slices which do not have guidance maps?

In this transform, we select the slices that we can use to simulate the guidance.

During training, it is actually a combination of several transforms: find the slices, created the guidance seeds, and add the guidance into a tensor: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/deepedit.py#L113

Hope this helps,

[Zrrr1997]

Great! This answers both my questions.

It makes sense to use a 3D-U-Net, otherwise processing each slice one by one would produce independent segmentation maps, which might not be contiguous. A 3D-Conv layer would capture the spatial context between the slices.

I was only asking because Sakinis's paper only introduces how to do this for 2D images. I've seen other methods like MIDeepSeg and DeepIGeoS which tackle 3D volumes but was always wondering how the guidance maps are implemented in practice in 3D, especially since the majority of the slices are unannotated. The transforms.py script answers my question quite precisely.

Keep up the good work!