Test+Docker profile fails: "ValueError: call_variants_outputs did not pass sanity check."

[sitems]

Description of the bug

Aug-12 14:30:26.359 [TaskFinalizer-10] ERROR nextflow.processor.TaskProcessor - Error executing process > 'NFCORE_RAREDISEASE:RAREDISEASE:CALL_SNV:CALL_SNV_DEEPVARIANT:DEEPVARIANT (hugelymodelbat)'

Caused by:
Process NFCORE_RAREDISEASE:RAREDISEASE:CALL_SNV:CALL_SNV_DEEPVARIANT:DEEPVARIANT (hugelymodelbat) terminated with an error exit status (1)

Command executed:

/opt/deepvariant/bin/run_deepvariant
--ref=reference.fasta
--reads=hugelymodelbat_sorted_md.bam
--output_vcf=hugelymodelbat_deepvar.vcf.gz
--output_gvcf=hugelymodelbat_deepvar.g.vcf.gz
--model_type=WGS

--intermediate_results_dir=tmp
--num_shards=2

cat <<-END_VERSIONS > versions.yml
"NFCORE_RAREDISEASE:RAREDISEASE:CALL_SNV:CALL_SNV_DEEPVARIANT:DEEPVARIANT":
deepvariant: 1.6.1
END_VERSIONS

Command exit status:
1

Command output:
call_set_name: "hugelymodelbat"
}
end: 10821750
reference_name: "21"
start: 10821734
is [[0], [1], [2]], which is invalid.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1125, in _mappable_transform_call_variant_group_to_output_variant
return _transform_call_variant_group_to_output_variant(**kwargs)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1030, in _transform_call_variant_group_to_output_variant
canonical_variant, predictions = merge_predictions(
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 891, in merge_predictions
raise ValueError('call_variants_outputs did not pass sanity check.')
ValueError: call_variants_outputs did not pass sanity check.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1419, in
app.run(main)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/absl_py/absl/app.py", line 312, in run
_run_main(main, args)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/absl_py/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1385, in main
tmp_variant_file = dump_variants_to_temp_file(variant_generator)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1067, in dump_variants_to_temp_file
tfrecord.write_tfrecords(variant_protos, temp.name)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/third_party/nucleus/io/tfrecord.py", line 190, in write_tfrecords
for proto in protos:
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 91, in maybe_resolve_conflicting_variants
for overlapping_candidates in _group_overlapping_variants(sorted_variants):
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 111, in _group_overlapping_variants
for variant in sorted_variants:
File "/usr/lib/python3.8/multiprocessing/pool.py", line 420, in
return (item for chunk in result for item in chunk)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
ValueError: call_variants_outputs did not pass sanity check.

real 0m5.169s
user 0m2.696s
sys 0m2.628s

Command error:
call_set_name: "hugelymodelbat"
}
end: 10821750
reference_name: "21"
start: 10821734
is [[0], [1], [2]], which is invalid.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1125, in _mappable_transform_call_variant_group_to_output_variant
return _transform_call_variant_group_to_output_variant(**kwargs)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1030, in _transform_call_variant_group_to_output_variant
canonical_variant, predictions = merge_predictions(
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 891, in merge_predictions
raise ValueError('call_variants_outputs did not pass sanity check.')
ValueError: call_variants_outputs did not pass sanity check.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1419, in
app.run(main)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/absl_py/absl/app.py", line 312, in run
_run_main(main, args)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/absl_py/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1385, in main
tmp_variant_file = dump_variants_to_temp_file(variant_generator)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1067, in dump_variants_to_temp_file
tfrecord.write_tfrecords(variant_protos, temp.name)
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/third_party/nucleus/io/tfrecord.py", line 190, in write_tfrecords
for proto in protos:
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 91, in maybe_resolve_conflicting_variants
for overlapping_candidates in _group_overlapping_variants(sorted_variants):
File "/tmp/Bazel.runfiles_hekmsdb9/runfiles/com_google_deepvariant/deepvariant/haplotypes.py", line 111, in _group_overlapping_variants
for variant in sorted_variants:
File "/usr/lib/python3.8/multiprocessing/pool.py", line 420, in
return (item for chunk in result for item in chunk)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
ValueError: call_variants_outputs did not pass sanity check.

real 0m5.169s
user 0m2.696s
sys 0m2.628s

Work dir:
shortened/raredisease_test/work/b9/4ceb969843a4bba123687b3dfc7a11

Command used and terminal output

nextflow run nf-core/raredisease -resume -revision dev -profile test,docker -config test.config --outdir out

Relevant files

dag.overwrite = true

process {
withName:"DEEPVARIANT" {
//cpus = 20
//memory = 55.GB
time = 2.d
}
withName:"GLNEXUS" {
//cpus = 20
//memory = 55.GB
time = 16.h
}

}

System information

Nextflow version: 24.04.4.5917
Hardware: Desktop
Executor: local
Container engine: Docker
OS: Ubuntu 22.04
Version of nf-core/raredisease: dev (2024-08-12)

[ramprasadn]

Hi @sitems! It looks like this might be a deepvariant problem. google/deepvariant#517.

Can you try running deepvariant outside of nextflow and see if you still get the error?