Contrast-agnostic model on a lumbar SCI patient outputs incorrect segmentation

[naga-karthik]

@valosekj tried the contrast-agnostic v2.4 model on a lumbar SCI patient from the sci-zurich dataset. The model even segmented regions outside the cord.

RESULTS FROM contrast-agnostic model (version: 2.4)

Segmentation outside of the cord:

Undersegmented cord at the level of the lesion:

red - lesion
blue - cord

Issue: the cord should cover also the region at and below the lesion

[naga-karthik]

Even though the these datasets were used for training (including SCI and lumbar datasets), I think that the subject above is a corner case (i.e. lumbar SCI with metal hardware) and is quite difficult to segment given that we don't have that many lumbar scans and scans with hardware (compared to healthy and MS subjects).

Therefore, one possible option I recommend is to consider also using the SCIseg model for cord segmentation.

Reason: It is difficult for one model to be able to reliably segment any cord, given the heterogeneity in the scans, resolutions and image quality. So, if a user has a challenging test case (like the one @valosekj had), we could ask the user to also consider trying SCIseg model if the contrast-agnostic model doesn't work well. The specialist model might be better at such extreme cases than the generalist model that works reasonably well on any/all contrasts.

[jcohenadad]

@naga-karthik do you think we should consider adding this dataset in the contrast-agnostic model? if it makes the model more "agnostic" to pathologies, then it's a big plus for that model. Ultimately, the more cases it can cover, the more it will be used (and cited).

[naga-karthik]

hey @jcohenadad, are you referring to a particular dataset? Maybe I'm missing something, here are the datasets currently included in the model training.

[valosekj]

hey @jcohenadad, are you referring to a particular dataset? Maybe I'm missing something, here are the datasets currently included in the model training.

Thanks for the clarification, Naga! I probably introduced some confusion here during yesterday's meeting, as I thought that only SCIsegV2 included the sci-zurich dataset for training. And I did not realize that contrast-agnostic was actually also trained on this dataset.

[naga-karthik]

[IMPORTANT] hey @valosekj, I made a mistake! Turns out that SCI datasets were not included in the contrast-agnostic v2.4 version. For reference, here's a summary of what datasets were included!

Datasets used in v2.4

Datasets used (n=7):

• datasplit_basel-mp2rage_seed50.json
• datasplit_canproco_seed50.json
• datasplit_data-multi-subject_seed50.json
• datasplit_dcm-zurich_seed50.json
• datasplit_lumbar-epfl_seed50.json
• datasplit_lumbar-vanderbilt_seed50.json
• datasplit_sct-testing-large_seed50.json

Training/Validation/Test splits across all datasets:

• Number of training samples (i.e. images, not subjects): 2700
• Number of validation samples (i.e. images, not subjects): 503
• Number of testing samples (i.e. images, not subjects): 616

Training/Validation/Test across each contrast (n=9):

Contrast	#train_images	#validation_images	#test_images
dwi	172	38	49
mt-off	158	36	49
mt-on	168	37	49
psir	217	49	67
stir	56	15	18
t1w	197	40	54
t2star	633	87	106
t2w	1018	190	213
unit1	81	11	11
TOTAL	2700	503	616

Model name for internal/dev purposes:

nnunet_seed=50_ndata=7_ncont=9_pad=zero_nf=32_opt=adam_lr=0.001_AdapW_bs=2_20240425-170840

I was losing track of what datasets were being added in each release so I create a summary like the one above and upload it with every release. They can found in datasplits.md if you manually download the release folder. Turns out this was indeed the saviour! :)

That said, the current un-released version of the model is trained on all SCI datasets and the DCM datasets. Would you like to try it out internally on the same subject (linked in my 1st comment)? I sent the location on the latest model on Slack.

[naga-karthik]

I went ahead and tested the latest model (v2.5) on the same subject. The issue is only partly resolved. With the latest model, we don't get any segmentations outside the SC (as reported in the original issue -- see screenshot in this comment) but the model is still unable to segment the cord at the site of the implant.

I tested the model on T1w and ax_T2w images as well. The model does slightly better on T1w (at least we see some prediction after the implant) for ax_T2w it works perfectly fine! I know that this does not resolve the issue as far as computing sct_analyze_lesion is concerned. Do you think we can have a workaround based on these additional segmentations? (namely, the axial)

sag T2w pred

sag T1w pred

[naga-karthik]

I found something interesting, thank to @valosekj ! My testing script by default had the keep-largest-component enabled. As a result, we lost some prediction after the metal implant.

When I re-tested by keeping everything, we have a better prediction now. (and there are still no predictions outside the cord)

disabling keep-largest-component

yellow: prediction with the largest component
blue: prediction without the largest component (i.e. keep everything). note that blue is overlaid on yellow i.e. we still have the complete segmentation for blue as well (just that it's hidden behind yellow)

@valosekj I think you can maybe try it on a few other subjects? Please keep in mind the train/test split used for sci-zurich. You might see a difference in performance if you're running it on the entire dataset depending on which subjects were included in training/testing.

[jcohenadad]

I was losing track of what datasets were being added in each release so I create a summary like the one above and upload it with every release. They can found in datasplits.md if you manually download the release folder.

@naga-karthik this shows the importance of creating a script that automatically fetches the dataset and generate a log listing the data used to produce the release. How can everyone be sure that the manually created file datasplits.md has no error? As requested in the old issue #15, it becomes increasingly urgent to perform training in a transparent and reproducible way. The training script should generate the list of training dataset (not you manually) and package that list alongside the generated model, that is ultimately uploaded on github release. Thank you.

[naga-karthik]

oops, sorry for the misunderstanding! I am not doing this manually! it's automatically output from this function when I start the training and the output is stored in the results folder (which is simply uploaded to the GH release) :)

[valosekj]

Thanks for digging in, @naga-karthik!

@valosekj I think you can maybe try it on a few other subjects?

Sure! Do you think it would be possible to zip the v2.5 model and make a pre-release? By doing so, I could easily reuse my existing bash script just by adding sct_deepseg -custom-url <URL_TO_V2.5>.

UPDATE: v2.4.1-beta pre-release created --> Jan TODO: test it on sci-zurich

[naga-karthik]

@valosekj can I close this issue?