diff --git a/README.md b/README.md index 39850e2..fb7a5e8 100644 --- a/README.md +++ b/README.md @@ -39,12 +39,12 @@ The following installations are available at different sites (referred to as `RP - NSCC: `/home/users/astar/gis/gisshared/rpd/pipelines/` Each of these contains one subfolder per pipeline version, -e.g. `$RPD_PIPELINES/pipelines.2017-01` (referred to as +e.g. `$RPD_PIPELINES/pipelines.2017-06` (referred to as `PIPELINE_ROOTDIR` below). Much of this framework assumes a certain setup and services to be present, as is the case in GIS / the NSCC. This repository is -therefore of limited use to the general public. See INSTALL.md for +therefore of limited use to the general public. See `INSTALL.md` for simplistic installation instructions. Some pipelines only work at a certain site (due to system or software @@ -112,23 +112,25 @@ In either case, you must not prefix the script with `python`. | Name | Category | Notes | @GIS | @NSCC | | --- | --- | --- | --- | --- | -| [bcl2fastq](bcl2fastq/README.md) | Production | Not for end-users | Y | Y | -| [ChIP-seq](chromatin-profiling/chipseq/README.md) | Chromatin Profiling | | Y | Y | -| [SG10K](custom/SG10K/README.md) | Custom | Not for end-users | Y | Y | -| [ViPR](germs/vipr/README.md) | GERMS | | Y | Y | -| [BWA-MEM](mapping/BWA-MEM/README.md) | Mapping | | Y | Y | +| [bcl2fastq](bcl2fastq/README.md) | Production | Not for end-users | Y | Y | +| [ATAC-seq](chromatin-profiling/atacseq/README.md) | Chromatin Profiling | | Y | Y | +| [ChIP-seq](chromatin-profiling/chipseq/README.md) | Chromatin Profiling | | Y | Y | +| [SG10K](custom/SG10K/README.md) | Custom | Not for end-users | Y | Y | +| [ViPR](germs/vipr/README.md) | GERMS | | Y | Y | +| [BWA-MEM](mapping/BWA-MEM/README.md) | Mapping | | Y | Y | | [Shotgun Metagenomics](metagenomics/shotgun-metagenomics/README.md) | Metagenomics | | Y | Y | -| [Essential-Genes](metagenomics/essential-genes/README.md) | Metagenomics | Requires ref download | Y | Y | -| [STAR-RSEM](rnaseq/star-rsem/README.md) | RNA-Seq | | Y | Y | -| [Fluidigm-HT-C1-RNASeq](rnaseq/fluidigm-ht-c1-rnaseq/README.md) | RNA-Seq | | Y | N | -| [LoFreq-Somatic](somatic/lofreq-somatic/README.md) | Somatic | | Y | N | -| [Mutect](somatic/mutect/README.md) | Somatic | | Y | Y | -| [GATK](variant-calling/gatk/README.md) | Variant-calling | | Y | Y | -| [Lacer-LoFreq](variant-calling/lacer-lofreq/README.md) | Variant-calling | | Y | N | +| [Essential-Genes](metagenomics/essential-genes/README.md) | Metagenomics | Requires ref download | Y | Y | +| [STAR-RSEM](rnaseq/star-rsem/README.md) | RNA-Seq | | Y | Y | +| [Fluidigm-HT-C1-RNASeq](rnaseq/fluidigm-ht-c1-rnaseq/README.md)| RNA-Seq | | Y | N | +| [Wafergen](rnaseq/wafergen/README.md) | RNA-Seq | Requires cellular barcodes | Y | Y | +| [LoFreq-Somatic](somatic/lofreq-somatic/README.md) | Somatic | | Y | N | +| [Mutect](somatic/mutect/README.md) | Somatic | | Y | Y | +| [GATK](variant-calling/gatk/README.md) | Variant-calling | | Y | Y | +| [Lacer-LoFreq](variant-calling/lacer-lofreq/README.md) | Variant-calling | | Y | N | See `example-dag.pdf` in each pipeline's folder for a visual overview of the workflow. -Note, pipelines start with fastq files as input (a few allow injection of BAM files). +Note, most pipelines start with FastQ files as input, a few allow injection of BAM files. ## How it Works @@ -138,7 +140,7 @@ Note, pipelines start with fastq files as input (a few allow injection of BAM fi `conf.yaml` file) and gets its own readgroup assigned where appropriate. - Software versions are defined in each pipelines' `cfg/modules.yaml` - and loaded via [dotkit](https://computing.llnl.gov/?set=jobs&page=dotkit) + and loaded via [Lmod](http://lmod.readthedocs.io/en/latest/) - Pipeline wrappers create an output directory containing all necessary configuration files, run scripts etc. - After creation of this folder, the analysis run is automatically submitted to the cluster @@ -154,7 +156,6 @@ Note, pipelines start with fastq files as input (a few allow injection of BAM fi First call the wrapper in question with `--no-run`. cd into the given outdir and then - Check the created `conf.yaml` -- Print the DAG: `rm -f logs/snakemake.log; type=pdf; EXTRA_SNAKEMAKE_ARGS="--dag" bash run.sh; cat logs/snakemake.log | dot -T$type > dag.$type` - Execute a dryrun: `rm -f logs/snakemake.log; EXTRA_SNAKEMAKE_ARGS="--dryrun" bash run.sh; cat logs/snakemake.log` - Run locally: `nohup bash run.sh; tail -f logs/snakemake.log` @@ -189,11 +190,11 @@ described in the following: want to set the CSV delimiter with `-d`, e.g. `-d ,` - Use the created yaml file as input for the pipeline wrapper (option `--sample-cfg your.yaml`) -Please note, not all pipelines support this feature (for example the -somatic pipelines don't), but most do, e.g. GATK, Lacer-LoFreq. In -some cases multisample processing can lead to very high memory -consumption by the snakemake master process itself, a side-effect -which is hard to predict. +Please note, not all pipelines support this feature, for example the +Chipseq and all somatic pipelines. In some cases multisample +processing can lead to very high memory consumption by the snakemake +master process itself, a side-effect which is hard to predict (the +master process will be killed). The above configuration can be used for single sample processing as well, however, for single samples the corresponding use of options diff --git a/VERSION b/VERSION index b3985ed..6fe3643 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2017-06.0 +2017-10.0 diff --git a/bcl2fastq/Snakefile b/bcl2fastq/Snakefile index 8aaf732..34f4a8c 100644 --- a/bcl2fastq/Snakefile +++ b/bcl2fastq/Snakefile @@ -23,6 +23,7 @@ from pipelines import send_status_mail from pipelines import path_to_url from pipelines import RPD_SIGNATURE from utils import generate_timestamp +from pipelines import mark_as_completed from elmlogger import ElmLogging, ElmUnit from bcl2fastq_dbupdate import DBUPDATE_TRIGGER_FILE_FMT, DBUPDATE_TRIGGER_FILE_MAXNUM from readunits import sampledir_to_cfg @@ -149,6 +150,8 @@ onsuccess: os.path.abspath(RESULT_OUTDIR), extra_text=extra_text) # cannot talk to mongodb from compute. use trigger file write_db_update_trigger(True) + + mark_as_completed() onerror: diff --git a/changelog.md b/changelog.md index 9a4b906..36d8ffe 100644 --- a/changelog.md +++ b/changelog.md @@ -3,6 +3,26 @@ This change log only lists the major changes between releases. For a full list of changes refer to the commit log. +## 2017-10 + +New pipelines: +- chromatin-profiling/atacseq: runs Bowtie2 and MACS2 to call ATAC-Seq peaks +- rnaseq/wafergen-scrna: analyses WaferGen's single cell sequencing + data, using umis and scRNApipe for the core analysis. + + +Changes to pipelines and framework: +- Major pipelines now produce benchmarking logs +- STAR-RSEM now supports --estimate-rspd and new strandedness option +- Shotgun-metagenomics: Added SRST2, read counting and coverage + threshold for decontamination +- GATK: support for joint variant calling (--joint-calls) and added a + new option (--gvcf-only). Also added a new optional references + config for agressive splitting +- Added --restarts option to control number of automatic restarts in + case of failure +- Minor additions to SG10K and many under the hood changes to bc2lfastq + ## 2017-06 New pipelines: diff --git a/chromatin-profiling/atacseq/example-dag.pdf b/chromatin-profiling/atacseq/example-dag.pdf index ef97537..099a49f 100644 Binary files a/chromatin-profiling/atacseq/example-dag.pdf and b/chromatin-profiling/atacseq/example-dag.pdf differ diff --git a/chromatin-profiling/atacseq/tests.sh b/chromatin-profiling/atacseq/tests.sh index 27015ed..55fa0a4 100644 --- a/chromatin-profiling/atacseq/tests.sh +++ b/chromatin-profiling/atacseq/tests.sh @@ -68,7 +68,7 @@ echo 'FIXME test quality of results' 1>&2 # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: PE two pairs" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/chromatin-profiling/chipseq/example-dag.pdf b/chromatin-profiling/chipseq/example-dag.pdf index ae5fd6e..2bfaeb9 100644 Binary files a/chromatin-profiling/chipseq/example-dag.pdf and b/chromatin-profiling/chipseq/example-dag.pdf differ diff --git a/chromatin-profiling/chipseq/tests.sh b/chromatin-profiling/chipseq/tests.sh index 97b236b..8c3c948 100644 --- a/chromatin-profiling/chipseq/tests.sh +++ b/chromatin-profiling/chipseq/tests.sh @@ -78,7 +78,7 @@ echo 'FIXME bam injection' 1>&2 # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: WES" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/custom/SG10K/Snakefile b/custom/SG10K/Snakefile index dee4060..c90f7a2 100644 --- a/custom/SG10K/Snakefile +++ b/custom/SG10K/Snakefile @@ -134,9 +134,11 @@ rule bam_to_cram: #conda: # "env.yaml" shell: - "module load bamutil/1.0.14-33-nonprimdup;" + "{{ " + " module load bamutil/1.0.14-33-nonprimdup;" " bam squeeze --in {input.bam} --out -.ubam {params.bin_arg} |" - "samtools view -C -T {input.reffa} -@ {threads} -o {output.cram} - >& {log}" + " samtools view -C -T {input.reffa} -@ {threads} -o {output.cram} -;" + " }} >& {log}" localrules: cram_index diff --git a/custom/SG10K/cfg/modules.yaml b/custom/SG10K/cfg/modules.yaml index 3110adc..e36ca35 100644 --- a/custom/SG10K/cfg/modules.yaml +++ b/custom/SG10K/cfg/modules.yaml @@ -7,4 +7,3 @@ datamash: 1.1.0 sg10k-cov: '062017' fastmitocalc: 'default' - diff --git a/custom/SG10K/example-dag.pdf b/custom/SG10K/example-dag.pdf index c3b2940..61b721d 100644 Binary files a/custom/SG10K/example-dag.pdf and b/custom/SG10K/example-dag.pdf differ diff --git a/custom/SG10K/tests.sh b/custom/SG10K/tests.sh index dbcb246..3bb21fd 100755 --- a/custom/SG10K/tests.sh +++ b/custom/SG10K/tests.sh @@ -61,7 +61,7 @@ echo "Check log if the following final message is not printed: \"$COMPLETE_MSG\" # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: $SAMPLE" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/germs/vipr/aux/plot.py b/germs/vipr/aux/plot.py index fd7d19a..5152a99 100755 --- a/germs/vipr/aux/plot.py +++ b/germs/vipr/aux/plot.py @@ -36,7 +36,7 @@ def parse_genomecov(genomecov_gzfile): for line in fh: sq, pos, cov = line.decode().rstrip().split("\t") pos = int(pos)-1 - cov = int(cov) + cov = int(float(cov))# float for support of scientific notation if sq not in genomecov: genomecov[sq] = OrderedDict() genomecov[sq][pos] = cov diff --git a/germs/vipr/example-dag.pdf b/germs/vipr/example-dag.pdf index fae566b..15c69d9 100644 Binary files a/germs/vipr/example-dag.pdf and b/germs/vipr/example-dag.pdf differ diff --git a/germs/vipr/tests.sh b/germs/vipr/tests.sh index be8e106..2b1de08 100644 --- a/germs/vipr/tests.sh +++ b/germs/vipr/tests.sh @@ -69,7 +69,7 @@ base_cmd="$WRAPPER -1 $FQ1 -2 $FQ2 -s test:DENV2-TSV01-PDH203 -r $REF"; # FIXME fix here and elsewhere # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "Creating DAG" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/mapping/BWA-MEM/example-dag.pdf b/mapping/BWA-MEM/example-dag.pdf index 9ce5910..b995c73 100644 Binary files a/mapping/BWA-MEM/example-dag.pdf and b/mapping/BWA-MEM/example-dag.pdf differ diff --git a/mapping/BWA-MEM/tests.sh b/mapping/BWA-MEM/tests.sh index b6a6eb0..b85bc3f 100755 --- a/mapping/BWA-MEM/tests.sh +++ b/mapping/BWA-MEM/tests.sh @@ -75,7 +75,7 @@ echo "Check log if the following final message is not printed: \"$COMPLETE_MSG\" # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: PE through config" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/metagenomics/essential-genes/example-dag.pdf b/metagenomics/essential-genes/example-dag.pdf index 082174b..88533ed 100644 Binary files a/metagenomics/essential-genes/example-dag.pdf and b/metagenomics/essential-genes/example-dag.pdf differ diff --git a/metagenomics/essential-genes/tests.sh b/metagenomics/essential-genes/tests.sh index e2f79d2..3c2e76c 100755 --- a/metagenomics/essential-genes/tests.sh +++ b/metagenomics/essential-genes/tests.sh @@ -73,7 +73,7 @@ echo "Check log if the following final message is not printed: \"$COMPLETE_MSG\" # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/metagenomics/shotgun-metagenomics/example-dag.pdf b/metagenomics/shotgun-metagenomics/example-dag.pdf new file mode 100644 index 0000000..691ddb7 Binary files /dev/null and b/metagenomics/shotgun-metagenomics/example-dag.pdf differ diff --git a/metagenomics/shotgun-metagenomics/tests.sh b/metagenomics/shotgun-metagenomics/tests.sh index f176b82..278997b 100644 --- a/metagenomics/shotgun-metagenomics/tests.sh +++ b/metagenomics/shotgun-metagenomics/tests.sh @@ -64,7 +64,7 @@ base_cmd="$WRAPPER --sample-cfg $SAMPLECFG --name test:shotgun-metagenomics"; # FIXME fix here and elsewhere # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "Creating DAG" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/rnaseq/fluidigm-ht-c1-rnaseq/example-dag.pdf b/rnaseq/fluidigm-ht-c1-rnaseq/example-dag.pdf index c9218e3..4d98138 100644 Binary files a/rnaseq/fluidigm-ht-c1-rnaseq/example-dag.pdf and b/rnaseq/fluidigm-ht-c1-rnaseq/example-dag.pdf differ diff --git a/rnaseq/fluidigm-ht-c1-rnaseq/tests.sh b/rnaseq/fluidigm-ht-c1-rnaseq/tests.sh index 8a908dd..fbbe1d2 100644 --- a/rnaseq/fluidigm-ht-c1-rnaseq/tests.sh +++ b/rnaseq/fluidigm-ht-c1-rnaseq/tests.sh @@ -65,7 +65,7 @@ cmd_base="$WRAPPER -1 $COL1_R1 -2 $COL1_R2 -s COL01 --name 'test:COL01'" # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/rnaseq/star-rsem/example-dag.pdf b/rnaseq/star-rsem/example-dag.pdf index 14248ad..32d06dc 100644 Binary files a/rnaseq/star-rsem/example-dag.pdf and b/rnaseq/star-rsem/example-dag.pdf differ diff --git a/rnaseq/star-rsem/tests.sh b/rnaseq/star-rsem/tests.sh index b228d52..6ec5424 100644 --- a/rnaseq/star-rsem/tests.sh +++ b/rnaseq/star-rsem/tests.sh @@ -84,7 +84,7 @@ CMD_FULL="$WRAPPER -1 $R1_FULL -2 $R2_FULL -s $SAMPLE --name 'test:FULL'" SKIP_REAL_FULL=0 # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: Full" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/rnaseq/wafergen-scrna/README.md b/rnaseq/wafergen-scrna/README.md index 4c01fe9..e364f3a 100644 --- a/rnaseq/wafergen-scrna/README.md +++ b/rnaseq/wafergen-scrna/README.md @@ -54,8 +54,7 @@ Reports: ## Notes -Deduplication can be very slow for large data-sets. If deduplication - is not a must, we recommend to switch it off (`--no-dedup`). +Deduplication can be very slow for large data-sets. We recommend to not use it unless necessary (`--dedup`). ## References diff --git a/rnaseq/wafergen-scrna/Snakefile b/rnaseq/wafergen-scrna/Snakefile index 12da2b0..cb0d232 100644 --- a/rnaseq/wafergen-scrna/Snakefile +++ b/rnaseq/wafergen-scrna/Snakefile @@ -192,6 +192,8 @@ rule bamtag_split: output: taggedbam = '{prefix}/{sample}/bamtag/{sample}_R2.tagged.bam', splitflag = touch('{prefix}/{sample}/bamtag/{sample}_R2.tagsplit.COMPLETE') + log: + '{prefix}/{sample}/bamtag/{sample}_R2.tagged.log' benchmark: '{prefix}/{sample}/bamtag/{sample}_R2.tag.bamtag_split.benchmark.log' params: @@ -201,10 +203,12 @@ rule bamtag_split: threads: 1 shell: - 'umis bamtag {input.bam} | samtools addreplacerg' + '{{' + ' umis bamtag {input.bam} | samtools addreplacerg' ' -r ID:{params.sample} -r LB:{params.sample} -r SM:{params.sample} -r PL:{params.platform} -r PU:1 -r CN:{params.center}' ' -o - - | samtools sort -o {output.taggedbam} -T {output.taggedbam}.tmp -;' - ' bamtools split -tag XC -in {output.taggedbam}' + ' bamtools split -tag XC -in {output.taggedbam};' + ' }} >& {log}' # not guaranteed to create one file per barcode diff --git a/rnaseq/wafergen-scrna/example-dag.pdf b/rnaseq/wafergen-scrna/example-dag.pdf index 988cd96..2cc5418 100644 Binary files a/rnaseq/wafergen-scrna/example-dag.pdf and b/rnaseq/wafergen-scrna/example-dag.pdf differ diff --git a/rnaseq/wafergen-scrna/tests.sh b/rnaseq/wafergen-scrna/tests.sh index e28684e..393ba5d 100644 --- a/rnaseq/wafergen-scrna/tests.sh +++ b/rnaseq/wafergen-scrna/tests.sh @@ -63,7 +63,7 @@ cmd_base="$WRAPPER -c $TEST_CBINDEX -S $TEST_SAMPLECFG" # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) @@ -103,9 +103,9 @@ if [ $skip_real_runs -ne 1 ]; then jid=$(tail -n 1 $odir/logs/submission.log | cut -f 3 -d ' ') echo "Started job $jid writing to $odir. You will receive an email" - echo "Realrun no_dedup" | tee -a $log + echo "Realrun dedup" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) - eval $cmd_base --no-dedup -o $odir -v >> $log 2>&1 + eval $cmd_base --dedup -o $odir -v >> $log 2>&1 # magically works even if line just contains id as in the case of pbspro jid=$(tail -n 1 $odir/logs/submission.log | cut -f 3 -d ' ') echo "Started job $jid writing to $odir. You will receive an email" diff --git a/rnaseq/wafergen-scrna/wafergen-scrna.py b/rnaseq/wafergen-scrna/wafergen-scrna.py index de91de7..266fdf0 100755 --- a/rnaseq/wafergen-scrna/wafergen-scrna.py +++ b/rnaseq/wafergen-scrna/wafergen-scrna.py @@ -78,8 +78,8 @@ def main(): d = 20.0 parser.add_argument('--frag-len-sd', default=d, type=float, help="Estimated fragment length standard deviation (default={})".format(d)) - parser.add_argument('--no-dedup', action="store_true", - help="Skip UMI-based deduplication (can be slow)") + parser.add_argument('--dedup', action="store_true", + help="Run UMI-based deduplication (slow for large data-sets!)") args = parser.parse_args() # Repeateable -v and -q for setting logging level. @@ -139,7 +139,7 @@ def main(): cfg_dict['cell_barcodes'] = os.path.abspath(args.cell_barcodes) cfg_dict['frag_len'] = args.frag_len cfg_dict['frag_len_sd'] = args.frag_len_sd - cfg_dict['no_dedup'] = args.no_dedup + cfg_dict['no_dedup'] = not args.dedup cfg_dict['scrnapipe_transform'] = os.path.abspath(os.path.join( PIPELINE_BASEDIR, 'aux/transform.json')) cfg_dict['scrna_conf_template'] = os.path.abspath(os.path.join( diff --git a/somatic/lofreq-somatic/example-dag.pdf b/somatic/lofreq-somatic/example-dag.pdf index 21a407a..cf91527 100644 Binary files a/somatic/lofreq-somatic/example-dag.pdf and b/somatic/lofreq-somatic/example-dag.pdf differ diff --git a/somatic/lofreq-somatic/tests.sh b/somatic/lofreq-somatic/tests.sh index 05d2554..c6377a4 100755 --- a/somatic/lofreq-somatic/tests.sh +++ b/somatic/lofreq-somatic/tests.sh @@ -76,7 +76,7 @@ wgs_cmd_base="$WRAPPER --normal-bam $DREAM_WGS_NORMAL_BAM --tumor-bam $DREAM_WGS # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: WES" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/somatic/mutect/example-dag.pdf b/somatic/mutect/example-dag.pdf index 6fd8f3e..0aa9365 100644 Binary files a/somatic/mutect/example-dag.pdf and b/somatic/mutect/example-dag.pdf differ diff --git a/somatic/mutect/tests.sh b/somatic/mutect/tests.sh index f8c25d0..704ce4a 100755 --- a/somatic/mutect/tests.sh +++ b/somatic/mutect/tests.sh @@ -78,7 +78,7 @@ wgs_cmd_base="$WRAPPER --normal-bam $DREAM_WGS_NORMAL_BAM --tumor-bam $DREAM_WGS # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: WES" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/tools/pipelint.py b/tools/pipelint.py index 02420be..1bc419c 100755 --- a/tools/pipelint.py +++ b/tools/pipelint.py @@ -111,6 +111,7 @@ def check_modules(pipeline_dir): is_ok = True module_cfgs = glob.glob(os.path.join(pipeline_dir, "cfg/modules.yaml")) + assert len(module_cfgs) > 0 modules = dict() for cfg in module_cfgs: with open(cfg) as fh: @@ -139,7 +140,14 @@ def main(pipelinedirs, logger.warning("include other existing tools here: check_cluster_conf.py...") snakefiles = [os.path.join(d, "Snakefile") for d in pipelinedirs] - + + if not no_modules_check: + for d in pipelinedirs: + if not check_modules(d): + print("FAILED: Modules check for {}".format(d)) + else: + print("OK: Modules check for {}".format(d)) + includes = [] for f in snakefiles: assert os.path.exists(f) @@ -148,12 +156,6 @@ def main(pipelinedirs, else: print("OK: Expected files for {}".format(f)) - if not no_modules_check: - if not check_modules(f): - print("FAILED: Modules check for {}".format(f)) - else: - print("OK: Modules check for {}".format(f)) - includes.extend(get_includes_from_snakefile(f)) diff --git a/variant-calling/gatk/example-dag.pdf b/variant-calling/gatk/example-dag.pdf index 5c52fa8..9a198a6 100644 Binary files a/variant-calling/gatk/example-dag.pdf and b/variant-calling/gatk/example-dag.pdf differ diff --git a/variant-calling/gatk/tests.sh b/variant-calling/gatk/tests.sh index 9b3044f..1d54adb 100644 --- a/variant-calling/gatk/tests.sh +++ b/variant-calling/gatk/tests.sh @@ -80,7 +80,7 @@ wgs_cmd_base="$WRAPPER -1 $WGS_FQ1 -2 $WGS_FQ2 -s NA12878-WGS -t WGS --name 'tes # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: WES" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE) diff --git a/variant-calling/lacer-lofreq/example-dag.pdf b/variant-calling/lacer-lofreq/example-dag.pdf index 0d0af1b..d195407 100644 Binary files a/variant-calling/lacer-lofreq/example-dag.pdf and b/variant-calling/lacer-lofreq/example-dag.pdf differ diff --git a/variant-calling/lacer-lofreq/tests.sh b/variant-calling/lacer-lofreq/tests.sh index 40c85bd..e5636c7 100755 --- a/variant-calling/lacer-lofreq/tests.sh +++ b/variant-calling/lacer-lofreq/tests.sh @@ -76,7 +76,7 @@ wgs_cmd_base="$WRAPPER -1 $WGS_FQ1 -2 $WGS_FQ2 -s NA12878-WGS -t WGS --name 'tes # DAG -SKIP_DAG=0 +SKIP_DAG=1 if [ $SKIP_DAG -eq 0 ]; then echo "DAG: WES" | tee -a $log odir=$($DOWNSTREAM_OUTDIR_PY -r $(whoami) -p $PIPELINE)