gatk-framework: command not found

[romanzac]

Hello,

I am running bcbio-nextgen-vm on sample data and I have experienced problem with gatk-framework even it is already installed. Any suggestion please ?

[2018-10-28T02:05Z] 2018-10-28 02:05:57 cgmh-cn05 ERROR [bcbio.run.itx] -
[2018-10-28T02:05Z] �[1;31mjava.lang.Exception�[m: �[3mShell command failed: gatk-framework -Xms250m -Xmx2g -XX:+UseSerialGC -Djava.io.tmpdir=/mnt/work/joint/freebayes-joint/ceph-bwa-j/1/union/tmp1560029062785942045 -T CombineVariants -R /usr/local/share/bcbio-nextgen/genomes/Hsapiens/GRCh37/seq/GRCh37.fa -L 1:36733344-53749339 --out /mnt/work/joint/freebayes-joint/ceph-bwa-j/1/union/txtmp8156946724902932091/union-1_36733343_53749339.vcf.gz --minimalVCF --sites_only --genotypemergeoption UNSORTED --suppressCommandLineHeader --setKey null -U LENIENT_VCF_PROCESSING --logging_level ERROR --variant /mnt/work/joint/freebayes-joint/ceph-bwa-j/1/merge/NA12878-3/NA12878-3-qprep-passonly-NA12878-3-1_36733540_53749540.vcf.gz --variant /mnt/work/joint/freebayes-joint/ceph-bwa-j/1/merge/NA12891-3/NA12891-3-qprep-passonly-NA12891-3-1_36733540_53749540.vcf.gz --variant /mnt/work/joint/freebayes-joint/ceph-bwa-j/1/merge/NA12892-3/NA12892-3-qprep-passonly-NA12892-3-1_36733540_53749540.vcf.gz
[2018-10-28T02:05Z] /mnt/work/joint/freebayes-joint/ceph-bwa-j/1/union/txtmp8156946724902932091/run.sh: line 2: gatk-framework: command not found
[2018-10-28T02:05Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)�[m
[2018-10-28T02:05Z] �[33mbcbio.run.itx/�[1;33mcheck-run�[m �[32m itx.clj: 168�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.ensemble.prep/fn/fn/�[1;33mfn�[m �[32m prep.clj: 101�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.ensemble.prep/fn/�[1;33mfn�[m �[32m prep.clj: 101�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.ensemble.prep/�[1;33mfn�[m �[32m prep.clj: 100�[m
[2018-10-28T02:05Z] �[37mclojure.lang.MultiFn.invoke�[m �[32m MultiFn.java: 251�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.square/�[1;33mby-region�[m �[32m square.clj: 220�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.square/combine-vcfs/�[1;33mfn�[m �[32m square.clj: 285�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.merge/prep-by-region/�[1;33mfn�[m �[32m merge.clj: 140�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/map/fn/�[1;33mfn�[m �[32m reducers.clj: 167�[m
[2018-10-28T02:05Z] �[37mclojure.lang.ArrayChunk.reduce�[m �[32mArrayChunk.java: 58�[m
[2018-10-28T02:05Z] �[33mclojure.core.protocols/�[1;33mfn�[m �[32m protocols.clj: 136�[m
[2018-10-28T02:05Z] �[33mclojure.core.protocols/fn/�[1;33mG�[m �[32m protocols.clj: 19�[m
[2018-10-28T02:05Z] �[33mclojure.core.protocols/�[1;33mseq-reduce�[m �[32m protocols.clj: 31�[m
[2018-10-28T02:05Z] �[33mclojure.core.protocols/�[1;33mfn�[m �[32m protocols.clj: 107�[m
[2018-10-28T02:05Z] �[33mclojure.core.protocols/fn/�[1;33mG�[m �[32m protocols.clj: 13�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/�[1;33mreduce�[m �[32m reducers.clj: 79�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/�[1;33mfoldvec�[m �[32m reducers.clj: 335�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/�[1;33mfn�[m �[32m reducers.clj: 362�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/fn/�[1;33mG�[m �[32m reducers.clj: 81�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/folder/reify/�[1;33mcoll-fold�[m �[32m reducers.clj: 130�[m
[2018-10-28T02:05Z] �[33mclojure.core.reducers/�[1;33mfold�[m �[32m reducers.clj: 98�[m
[2018-10-28T02:05Z] �[33mbcbio.run.parallel/�[1;33mrmap�[m �[32m parallel.clj: 17�[m
[2018-10-28T02:05Z] �[33mbcbio.run.parallel/�[1;33mrmap�[m �[32m parallel.clj: 22�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.merge/�[1;33mprep-by-region�[m �[32m merge.clj: 139�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.square/�[1;33mcombine-vcfs�[m �[32m square.clj: 283�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.square/�[1;33m-main�[m �[32m square.clj: 334�[m
[2018-10-28T02:05Z] �[37mclojure.lang.RestFn.applyTo�[m �[32m RestFn.java: 137�[m
[2018-10-28T02:05Z] �[33mclojure.core/�[1;33mapply�[m �[32m core.clj: 630�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.main/-main/�[1;33mfn�[m �[32m main.clj: 34�[m
[2018-10-28T02:05Z] �[33mbcbio.variation.recall.main/�[1;33m-main�[m �[32m main.clj: 33�[m
[2018-10-28T02:05Z] �[37mclojure.lang.RestFn.applyTo�[m �[32m RestFn.java: 137�[m
[2018-10-28T02:05Z] �[37mbcbio.variation.recall.main.main�[m �[32m : �[m
[2018-10-28T02:05Z] 2018-10-28 02:05:57 cgmh-cn05 ERROR [bcbio.run.parallel] -

...

[2018-10-28T02:05Z] 2018-10-28 02:05:58 cgmh-cn05 INFO [bcbio.run.itx] - bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Traceback (most recent call last):
File "/usr/local/bin/bcbio_nextgen.py", line 241, in
main(**kwargs)
File "/usr/local/bin/bcbio_nextgen.py", line 46, in main
run_main(**kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 43, in run_main
fc_dir, run_info_yaml)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 87, in _run_toplevel
for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 156, in variant2pipeline
samples = joint.square_off(samples, run_parallel)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 143, in square_off
"vrn_file", ["region", "sam_ref", "config"])
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/split.py", line 32, in grouped_parallel_split_combine
final_output = parallel_fn(parallel_name, split_args)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
return run_multicore(fn, items, config, parallel=parallel)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
for data in joblib.Parallel(parallel["num_jobs"], batch_size=1)(joblib.delayed(fn)(x) for x in items):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 789, in call
self.retrieve()
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 740, in retrieve
raise exception
joblib.my_exceptions.JoblibCalledProcessError: JoblibCalledProcessError

---

Multiprocessing exception:
...........................................................................
/usr/local/bin/bcbio_nextgen.py in ()
236 if setup_info is None: # no automated run after setup
237 sys.exit(0)
238 workdir, new_kwargs = setup_info
239 os.chdir(workdir)
240 kwargs.update(new_kwargs)
--> 241 main(**kwargs)
...........................................................................
/usr/local/bin/bcbio_nextgen.py in main(**kwargs={'config_file': '/mnt/work/bcbio_system-forvm.yaml', 'fc_dir': None, 'parallel': {'cores': 80, 'local_controller': False, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, ...}, 'run_info_yaml': '/mnt/work/bcbio_sample-forvm.yaml', 'workdir': '/mnt/work', 'workflow': None})
41 from bcbio.graph import graph
42 from bcbio.provenance import programs
43 from bcbio.pipeline import version
44
45 def main(**kwargs):
---> 46 run_main(**kwargs)
kwargs = {'config_file': '/mnt/work/bcbio_system-forvm.yaml', 'fc_dir': None, 'parallel': {'cores': 80, 'local_controller': False, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, ...}, 'run_info_yaml': '/mnt/work/bcbio_sample-forvm.yaml', 'workdir': '/mnt/work', 'workflow': None}
47
48 def parse_cl_args(in_args):
49 """Parse input commandline arguments, handling multiple cases.
50
...........................................................................
/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py in run_main(workdir='/mnt/work', config_file='/mnt/work/bcbio_system-forvm-merged.yaml', fc_dir=None, run_info_yaml='/mnt/work/bcbio_sample-forvm.yaml', parallel={'cores': 80, 'local_controller': False, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, ...}, workflow=None)
38 if config.get("log_dir", None) is None:
39 config["log_dir"] = os.path.join(workdir, DEFAULT_LOG_DIR)
40 if parallel["type"] in ["local", "clusterk"]:
41 _setup_resources()
42 _run_toplevel(config, config_file, workdir, parallel,
---> 43 fc_dir, run_info_yaml)
fc_dir = None
run_info_yaml = '/mnt/work/bcbio_sample-forvm.yaml'
44 elif parallel["type"] == "ipython":
45 assert parallel["scheduler"] is not None, "IPython parallel requires a specified scheduler (-s)"
46 if parallel["scheduler"] != "sge":
47 assert parallel["queue"] is not None, "IPython parallel requires a specified queue (-q)"
...........................................................................
/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py in _run_toplevel(config={'algorithm': {}, 'bcbio_system': '/mnt/work/bcbio_system-forvm-merged.yaml', 'galaxy_config': '/media/nvme0/work/stephen/bcbio-vm/data/galaxy/universe_wsgi.ini', 'log_dir': '/mnt/work/log', 'resources': {'default': {'cores': 16, 'jvm_opts': ['-Xms750m', '-Xmx3500m'], 'memory': '3G'}, 'dexseq': {'memory': '10g'}, 'express': {'memory': '8g'}, 'gatk': {'jvm_opts': ['-Xms500m', '-Xmx3500m']}, 'macs2': {'memory': '8g'}, 'qualimap': {'memory': '4g'}, 'seqcluster': {'memory': '8g'}, 'snpeff': {'jvm_opts': ['-Xms750m', '-Xmx3g']}}}, config_file='/mnt/work/bcbio_system-forvm-merged.yaml', work_dir='/mnt/work', parallel={'cores': 80, 'local_controller': False, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, ...}, fc_dir=None, run_info_yaml='/mnt/work/bcbio_sample-forvm.yaml')
82 pipelines, config = _pair_samples_with_pipelines(run_info_yaml, config)
83 system.write_info(dirs, parallel, config)
84 with tx_tmpdir(config if parallel.get("type") == "local" else None) as tmpdir:
85 tempfile.tempdir = tmpdir
86 for pipeline, samples in pipelines.items():
---> 87 for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
xs = undefined
pipeline =
config = {'algorithm': {}, 'bcbio_system': '/mnt/work/bcbio_system-forvm-merged.yaml', 'galaxy_config': '/media/nvme0/work/stephen/bcbio-vm/data/galaxy/universe_wsgi.ini', 'log_dir': '/mnt/work/log', 'resources': {'default': {'cores': 16, 'jvm_opts': ['-Xms750m', '-Xmx3500m'], 'memory': '3G'}, 'dexseq': {'memory': '10g'}, 'express': {'memory': '8g'}, 'gatk': {'jvm_opts': ['-Xms500m', '-Xmx3500m']}, 'macs2': {'memory': '8g'}, 'qualimap': {'memory': '4g'}, 'seqcluster': {'memory': '8g'}, 'snpeff': {'jvm_opts': ['-Xms750m', '-Xmx3g']}}}
run_info_yaml = '/mnt/work/bcbio_sample-forvm.yaml'
parallel = {'cores': 80, 'local_controller': False, 'module': 'bcbio.distributed', 'queue': None, 'resources': [], 'retries': 0, 'run_local': False, 'scheduler': None, 'tag': '', 'timeout': 15, ...}
dirs = {'config': '/mnt/work', 'fastq': None, 'flowcell': None, 'galaxy': '/media/nvme0/work/stephen/bcbio-vm/data/galaxy', 'work': '/mnt/work'}
samples = [[{'algorithm': {'aligner': 'bwa', 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'remove_lcr': True, 'validate': 'giab-NA12878/truth_small_variants.vcf.gz', 'validate_regions': 'giab-NA12878/truth_regions.bed', 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12878-1', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa', 'sex': 'female'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12891-1', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa', 'sex': 'male'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12892-1', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa', 'sex': 'female'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'jointcaller': ['freebayes-joint', 'gatk-haplotype-joint', 'platypus-joint', 'samtools-joint'], 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'remove_lcr': True, 'validate': 'giab-NA12878/truth_small_variants.vcf.gz', 'validate_regions': 'giab-NA12878/truth_regions.bed', 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12878-3', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa-j', 'sex': 'female'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'jointcaller': ['freebayes-joint', 'gatk-haplotype-joint', 'platypus-joint', 'samtools-joint'], 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12891-3', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa-j', 'sex': 'male'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'jointcaller': ['freebayes-joint', 'gatk-haplotype-joint', 'platypus-joint', 'samtools-joint'], 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12892-3', 'genome_build': 'GRCh37', 'metadata': {'batch': 'ceph-bwa-j', 'sex': 'female'}, 'resources': {}}], [{'algorithm': {'aligner': 'bwa', 'mark_duplicates': True, 'realign': False, 'recalibrate': False, 'remove_lcr': True, 'validate': 'giab-NA12878/truth_small_variants.vcf.gz', 'validate_regions': 'giab-NA12878/truth_regions.bed', 'variantcaller': ['freebayes', 'gatk-haplotype', 'platypus', 'samtools']}, 'analysis': 'variant2', 'description': 'NA12878-5', 'genome_build': 'GRCh37', 'metadata': {'sex': 'female'}, 'resources': {}}]]
88 pass
89
90 # ## Generic pipeline framework
91

[chapmanb]

Roman;
Thanks much for the report and sorry about the issue. The latest release of bcbio-variation-recall (0.1.9) should avoid this issue and no longer use gatk-framework (which has been replaced by GATK4):

https://github.com/chapmanb/bcbio.variation.recall/releases

How are you running your analysis? Is it possible you have an out of date local install with an older version? Or are you using a Docker container here? If the later, we're generally moving to use Common Workflow Language runners like Cromwell for working with containers:

https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html

and moving to that would be a better long term strategy for future compatibility. Happy to provide more advice with some context about your goals and setup. Thanks again for the report.

[romanzac]

chapmanb, thanks a lot for quick response.

I am trying to run docker container version (bcbio-nextgen-vm) . I have two environments. One with container only install (2 months ago) and one with bare metal + container install on the same machine (2 weeks ago). The environment with bare metal + container install runs container version without problem. The environment with container version only has problem with gatk-framework.

If I want to run container only installation, am I supposed to have CWL v1.0.2 installed outside of the container or this component should be in correct version inside of the container ?

Thanks.
R.