Move derivatives from spine-generic-processed

[sandrinebedard]

Context

Currenlty, we have a processed version of this dataset (spine-generic-processed) where we have derivatives, but some are duplicated. See #121.

This PR moves the derivatives from spine-generic-processed to data-multi-subjects.

The derivatives are now BIDS compatible, they inlcude:

• spinal cord segmentation (under derivatives/labels)
• softsegmentation (under derivatives/lables_softseg) -->https://github.com/sct-pipeline/contrast-agnostic-softseg-spinalcord

Related issues

fixes #121

What is missing

• since the processing script of the derivatives is not merged yet, in the README.txt under derivatives/labels_softseg, the link needs to be updated. Modify README.txt in derivatives to explain processing #124
• Add a readme for segmentations in /derivatives/labels to explain what are the processing steps and link to the processing script Modify README.txt in derivatives to explain processing #124
• Delete spine-generic-processed

[sandrinebedard]

@mguaypaq , could you take a look at the PR. Thank you!

[mguaypaq]

Sure, I'm pretty busy today but I'll take a look tomorrow.

[sandrinebedard]

I had some extra modifications to add, but I was unable to push to git annex:

(base) sebeda@joplin:~/data_nvme_sebeda/data-multi-subject/derivatives/labels/sub-perform02$ git push origin git-annex:git-annex
Username for 'https://github.com': sandrinebedard
Password for 'https://sandrinebedard@github.com':
To https://github.com/spine-generic/data-multi-subject
 ! [rejected]            git-annex -> git-annex (non-fast-forward)
error: failed to push some refs to 'https://github.com/spine-generic/data-multi-subject'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details

[kousu]

@sandrinebedard please try

git annex sync --no-content --only-annex --cleanup # upload mirror contents
git push

The wiki docs seem to have had a bad merge at some point (this point). The last step, git push origin git-annex:git-annex, is redundant. And also impossible, it turns out, if you're collaborating with others who've also been uploading to the annex.

The lab manual docs have the same instructions but without the final step. I've updated the wiki to match.

[sandrinebedard]

Thanks @kousu ! It seams t have worked!

[sandrinebedard]

@mguaypaq I will need finally to modify some extra data. I think you can wait to check this PR until then. I'll ping you when it's ready!

[sandrinebedard]

Hello @mguaypaq ! I've finished the extra modification, you can now take a look, thank you!

[mguaypaq]

It looks like the following 45 files are missing from amazon:

derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/anat/sub-beijingGE02_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingGE02/dwi/sub-beijingGE02_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio01/anat/sub-beijingVerio01_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio02/anat/sub-beijingVerio02_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/anat/sub-beijingVerio03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-beijingVerio03/dwi/sub-beijingVerio03_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/anat/sub-ubc01_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc01/dwi/sub-ubc01_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc03/anat/sub-ubc03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/anat/sub-ubc05_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ubc05/dwi/sub-ubc05_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/anat/sub-ucdavis03_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis03/dwi/sub-ucdavis03_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/anat/sub-ucdavis05_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis05/dwi/sub-ucdavis05_rec-average_dwi_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis06/anat/sub-ucdavis06_T2w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T1w_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T2star_softseg.nii.gz
derivatives/labels_softseg/sub-ucdavis07/anat/sub-ucdavis07_T2w_softseg.nii.gz

The file metadata already exists on your branch sb/121-move-derivatives, so it should be enough to upload the file contents to amazon and update the git-annex branch, like this (taken from step 6 of the wiki instructions):

git annex enableremote amazon public=no
git annex copy derivatives/labels_softseg/sub-{beijingGE02,beijingVerio01,beijingVerio02,beijingVerio03,ubc01,ubc03,ubc05,ucdavis03,ucdavis05,ucdavis06,ucdavis07}/*/*.nii.gz --to amazon
git annex enableremote amazon public=yes
git annex sync --no-content --only-annex

[sandrinebedard]

OK I've added them to amazon!

[mguaypaq]

Since data validation passes, and this (giant) PR is mostly about moving files from another dataset back into this repository, I'm going to approve and merge.