Move derivatives from spine-generic-processed back to data-multi-subject

[sandrinebedard]

Context

A processed version of this dataset was created --> spine-generic-processed because of incomptibility of the derivatives and the raw data (mainly for training) Related issue. The same strategy was used for UK-Biobank --> issue.

Problems with the -processedapproach

• We have a duplicate of derivatives (mainly spinal cord segmentations) which makes it confusing and the manual corrections are mainly in spine-generic-processed --> related issue
• The derivatives are in the internal dataset --> better to be in public dataset

Note: This wasn't an issue for uk-biobank since there are no derivatives in the uk-biobank dataset, only in uk-biobank-processed

Solution: Move back all the derivatives to data-multi-subject

The proposed solution is to move back the derivatives in data-multi-subject. The incompatibilities will be addressed in a processing pipeline for each individual projects. The suffixes will be added back so the processing that was done is explicit.

Some concerns about moving it back:

• The suffixes _RPI_r ... are not BIDS compatible --> Maybe we don't keep them?
• For the long term, some derivatives could be from various pipelines --> can get confusing when someone wants to use some derivatives and find the procesing script.

Related issues:

• Upload manual segmentations #94
• UK - Biobank New BIDS dataset sct-pipeline/ukbiobank-spinalcord-csa#29
• Training on spinegeneric dataset gives size mismatches between image and ground truth ivadomed/ivadomed#797

[jcohenadad]

The suffixes _RPI_R ... are not BIDS compatible --> Maybe we don't keep them?

This is a good point indeed. For reference, here is the BIDS definition for derivatives. Another thing I am thinking of, is that in case the preprocessing script changes (eg: we don't reorient anymore), the derivative name will still have '_RPI' in it, which might cause confusion (because nowhere in the preprocessing script there will be a mention of this reorientation).

For the long term, some derivatives could be from various pipelines --> can get confusing when someone wants to use some derivatives and find the procesing script.

Good point, although I wouldn't worry too much at this point. I see more pros than cons, mostly driven by pragmatic concerns: we are currently struggling by not having the derivatives centralized, where we don't know if maybe within the next 5 years other derivatives will possibly conflict with the existing derivatives. I would say, if that happens, we will find a solution (eg: create another folder under derivatives, labels_pipelineX/)

[sandrinebedard]

This is a good point indeed. For reference, here is the BIDS definition for derivatives. Another thing I am thinking of, is that in case the preprocessing script changes (eg: we don't reorient anymore), the derivative name will still have '_RPI' in it, which might cause confusion (because nowhere in the preprocessing script there will be a mention of this reorientation).

Ok, so I will continue with the merging, but without any suffix, is that right (except -manual etc.)?