Skip to content

Commit

Permalink
fixed a bug with closing multiprocessing pool
Browse files Browse the repository at this point in the history
  • Loading branch information
@umahsn
umahsn committed Dec 15, 2020
1 parent 2546959 commit 6de1b92
Show file tree
Hide file tree
Showing 22 changed files with 168 additions and 150 deletions.
118 changes: 68 additions & 50 deletions sample/log
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,28 @@

2020-11-03 12:54:37.546552: Starting NanoCaller.
2020-12-15 11:51:46.510937: Starting NanoCaller.

Running arguments are saved in the following file: test_run/args

2020-11-03 12:54:44.877308: SNP calling started.
2020-11-03 12:54:46.708984: Coverage=49.63x.
2020-11-03 12:55:07.620834: (1/5) regions completed.
2020-11-03 12:55:21.178505: (2/5) regions completed.
2020-11-03 12:55:32.449525: (3/5) regions completed.
2020-11-03 12:55:44.149472: (4/5) regions completed.
2020-11-03 12:55:55.296145: (5/5) regions completed.
2020-12-15 11:51:54.445175: SNP calling started.

2020-11-03 12:55:55.611312: SNP calling completed for contig chr22. Time taken= 70.7340
2020-12-15 11:51:56.552637: Coverage=49.63x.
2020-12-15 11:52:20.333301: (1/5) regions completed.
2020-12-15 11:52:36.487862: (2/5) regions completed.
2020-12-15 11:52:49.490300: (3/5) regions completed.
2020-12-15 11:53:02.292401: (4/5) regions completed.
2020-12-15 11:53:14.600400: (5/5) regions completed.
Writing to /tmp/bcftools-sort.fnHPHD
Merging 1 temporary files
Cleaning
Done


2020-11-03 12:55:55.611387: ------WhatsHap SNP phasing log------
2020-12-15 11:53:14.967118: SNP calling completed for contig chr22. Time taken= 80.5220


This is WhatsHap 1.0 running under Python 3.6.8
2020-12-15 11:53:14.967204: ------WhatsHap SNP phasing log------

This is WhatsHap 1.0 running under Python 3.6.11
Working on 1 samples from 1 family
======== Working on chromosome 'chr22'
---- Processing individual SAMPLE
Expand All @@ -40,22 +45,23 @@ Largest component contains 1348 variants (100.0% of accessible variants) between
Done writing VCF

== SUMMARY ==
Maximum memory usage: 0.699 GB
Time spent reading BAM/CRAM: 6.2 s
Maximum memory usage: 0.355 GB
Time spent reading BAM/CRAM: 5.0 s
Time spent parsing VCF: 0.0 s
Time spent selecting reads: 0.3 s
Time spent phasing: 5.8 s
Time spent writing VCF: 0.0 s
Time spent selecting reads: 0.5 s
Time spent phasing: 8.1 s
Time spent writing VCF: 0.1 s
Time spent finding components: 0.0 s
Time spent on rest: 0.3 s
Total elapsed time: 12.7 s
Time spent on rest: 0.4 s
Total elapsed time: 14.0 s



2020-11-03 12:56:11.768244: ------SNP phasing completed------

2020-12-15 11:53:33.743588: ------SNP phasing completed------

2020-11-03 12:56:11.768306: ------WhatsHap BAM phasing log------

2020-12-15 11:53:33.743669: ------WhatsHap BAM phasing log------

Found 1 samples in input VCF
Keeping 1 samples for haplo-tagging
Expand All @@ -67,33 +73,45 @@ Found 2474 reads covering 1323 variants
Total alignments processed: 3967
Alignments that could be tagged: 2612
Alignments spanning multiple phase sets: 0
haplotag - total processing time: 8.877756118774414


2020-11-03 12:56:23.551385: ------BAM phasing completed-----

2020-11-03 12:56:23.551511: Indel calling started.
2020-11-03 12:56:37.223048: (1/20) regions completed.
2020-11-03 12:56:49.573645: (2/20) regions completed.
2020-11-03 12:57:08.137466: (3/20) regions completed.
2020-11-03 12:57:30.306719: (4/20) regions completed.
2020-11-03 12:57:46.430516: (5/20) regions completed.
2020-11-03 12:58:01.844967: (6/20) regions completed.
2020-11-03 12:58:11.831042: (7/20) regions completed.
2020-11-03 12:58:33.912745: (8/20) regions completed.
2020-11-03 12:58:52.015400: (9/20) regions completed.
2020-11-03 12:59:06.197186: (10/20) regions completed.
2020-11-03 12:59:24.105673: (11/20) regions completed.
2020-11-03 13:01:08.098350: (12/20) regions completed.
2020-11-03 13:01:19.541288: (13/20) regions completed.
2020-11-03 13:01:33.742843: (14/20) regions completed.
2020-11-03 13:01:46.493032: (15/20) regions completed.
2020-11-03 13:02:00.343629: (16/20) regions completed.
2020-11-03 13:02:15.169278: (17/20) regions completed.
2020-11-03 13:02:29.679240: (18/20) regions completed.
2020-11-03 13:02:47.559419: (19/20) regions completed.
2020-11-03 13:03:06.198732: (20/20) regions completed.
2020-11-03 13:03:06.488109: Indel calling completed for contig chr22. Time taken= 402.9366
2020-11-03 13:03:06.488197: Post processing

2020-11-03 13:03:11.648681: Total Time Elapsed: 514.12 seconds
haplotag - total processing time: 7.949145078659058




2020-12-15 11:53:45.776510: ------BAM phasing completed-----

2020-12-15 11:53:45.776640: Indel calling started.
2020-12-15 11:54:05.073745: (1/20) regions completed.
2020-12-15 11:54:23.467969: (2/20) regions completed.
2020-12-15 11:54:50.175749: (3/20) regions completed.
2020-12-15 11:55:22.075120: (4/20) regions completed.
2020-12-15 11:55:46.103592: (5/20) regions completed.
2020-12-15 11:56:09.731025: (6/20) regions completed.
2020-12-15 11:56:24.271868: (7/20) regions completed.
2020-12-15 11:56:58.973673: (8/20) regions completed.
2020-12-15 11:57:27.175183: (9/20) regions completed.
2020-12-15 11:57:48.967288: (10/20) regions completed.
2020-12-15 11:58:15.602348: (11/20) regions completed.
2020-12-15 12:00:44.880064: (12/20) regions completed.
2020-12-15 12:01:03.013816: (13/20) regions completed.
2020-12-15 12:01:24.416389: (14/20) regions completed.
2020-12-15 12:01:44.582849: (15/20) regions completed.
2020-12-15 12:02:05.310054: (16/20) regions completed.
2020-12-15 12:02:27.753123: (17/20) regions completed.
2020-12-15 12:02:50.474625: (18/20) regions completed.
2020-12-15 12:03:18.572774: (19/20) regions completed.
2020-12-15 12:03:46.566836: (20/20) regions completed.
Writing to /tmp/bcftools-sort.oKHcDK
Merging 1 temporary files
Cleaning
Done

2020-12-15 12:03:46.930933: Indel calling completed for contig chr22. Time taken= 601.1544
2020-12-15 12:03:46.973526: Post processing



Checking the headers and starting positions of 2 files


2020-12-15 12:03:54.094108: Total Time Elapsed: 727.59 seconds
2 changes: 1 addition & 1 deletion sample/sample_call
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
python ../scripts/NanoCaller.py -bam HG002.nanopore.chr22.sample.bam -mode both -seq ont -model NanoCaller1 -vcf test_run -chrom chr22 -start 20000000 -end 21000000 -ref chr22_ref.fa -prefix HG002.chr22.sample -cpu 1 > log
python ../scripts/NanoCaller.py -bam HG002.nanopore.chr22.sample.bam -mode both -seq ont -model NanoCaller1 -o test_run -chrom chr22 -start 20000000 -end 21000000 -ref chr22_ref.fa -prefix HG002.chr22.sample -cpu 1 > log
Binary file modified sample/test_run/HG002.chr22.sample.final.vcf.gz
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.final.vcf.gz.tbi
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.indels.decomposed.vcf.gz
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.indels.decomposed.vcf.gz.tbi
View file
Binary file not shown.
22 changes: 11 additions & 11 deletions sample/test_run/HG002.chr22.sample.indels.vcf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ chr22 20000499 . GCTCCCACCGC G 388.97 PASS . GT:GQ 0|1:388.93
chr22 20003865 . CAG C 287.65 PASS . GT:GQ 0|1:287.65
chr22 20004502 . GAA GA,G 48.87 PASS . GT:GQ 1|2:0.00
chr22 20005372 . TAA T,TA 231.23 PASS . GT:GQ 1|2:0.00
chr22 20006578 . CA C 238.46 PASS . GT:GQ 1|0:237.89
chr22 20006578 . CA C 238.45 PASS . GT:GQ 1|0:237.89
chr22 20007233 . TA T 5.45 PASS . GT:GQ 1/1:0.01
chr22 20007991 . CAG C 237.70 PASS . GT:GQ 1|0:234.79
chr22 20008925 . AG A 18.12 PASS . GT:GQ 1/1:3.55
Expand Down Expand Up @@ -135,7 +135,7 @@ chr22 20163708 . C CACT 289.16 PASS . GT:GQ 0|1:286.06
chr22 20163775 . ACTT A,ACTA 274.96 PASS . GT:GQ 1|2:0.03
chr22 20163790 . ACTG A,ACTCA 289.32 PASS . GT:GQ 1|2:0.00
chr22 20163822 . CATCACTACCACCACCCCACCACTAC CACCACCACCACCACCAT 289.00 PASS . GT:GQ 0|1:244.95
chr22 20163871 . TCACTAC T 351.36 PASS . GT:GQ 0|1:351.30
chr22 20163871 . TCACTAC T 351.35 PASS . GT:GQ 0|1:351.30
chr22 20163970 . T TCACTACCACCACCAC 424.47 PASS . GT:GQ 1/1:410.65
chr22 20164031 . TACC T 403.36 PASS . GT:GQ 1/1:284.80
chr22 20164070 . TACCACCAT TACCACCACCAC 269.85 PASS . GT:GQ 1/1:269.84
Expand Down Expand Up @@ -173,7 +173,7 @@ chr22 20225061 . GCA G 334.75 PASS . GT:GQ 1|0:334.40
chr22 20225424 . CAGACACG CACACA 484.78 PASS . GT:GQ 1/1:483.59
chr22 20225724 . T TCA 376.85 PASS . GT:GQ 1/1:374.30
chr22 20225955 . T TCA 364.60 PASS . GT:GQ 1|0:358.05
chr22 20226408 . GTGGAGACGTGAAGATGGGGAG G 445.08 PASS . GT:GQ 1/1:445.04
chr22 20226408 . GTGGAGACGTGAAGATGGGGAG G 445.09 PASS . GT:GQ 1/1:445.04
chr22 20227666 . TG T 19.24 PASS . GT:GQ 1/1:15.36
chr22 20227853 . G GTGTC 405.26 PASS . GT:GQ 0|1:404.84
chr22 20228168 . ACC A,AC 20.99 PASS . GT:GQ 1|2:-0.00
Expand Down Expand Up @@ -214,7 +214,7 @@ chr22 20302353 . GA G 9.84 PASS . GT:GQ 0|1:9.75
chr22 20303662 . TG T 17.36 PASS . GT:GQ 0|1:17.36
chr22 20307861 . CG C 23.84 PASS . GT:GQ 1/1:8.17
chr22 20313218 . ACCCCCCCGC ACCCCCCGC,ACCCCCG 16.01 PASS . GT:GQ 1|2:-0.00
chr22 20314910 . CG C 84.75 PASS . GT:GQ 1/1:21.40
chr22 20314910 . CG C 84.75 PASS . GT:GQ 1/1:21.41
chr22 20316593 . CCA C 360.14 PASS . GT:GQ 1|0:360.13
chr22 20319823 . G GC 43.85 PASS . GT:GQ 1|0:43.63
chr22 20322351 . CGCGCG CG,CACACACA 236.46 PASS . GT:GQ 1|2:0.47
Expand Down Expand Up @@ -364,7 +364,7 @@ chr22 20503551 . T TC 220.64 PASS . GT:GQ 1/1:38.57
chr22 20503580 . GATATTTGAGG GATTGCCTGACC 187.67 PASS . GT:GQ 0|1:187.66
chr22 20503709 . C CAGG 104.28 PASS . GT:GQ 0|1:94.54
chr22 20505106 . T TA 179.50 PASS . GT:GQ 1|0:89.93
chr22 20505559 . C CAA 308.06 PASS . GT:GQ 0|1:0.09
chr22 20505559 . C CAA 308.07 PASS . GT:GQ 0|1:0.09
chr22 20505583 . AAAAAAAACCA AAAAAAAAACA,A 382.10 PASS . GT:GQ 1|2:-0.00
chr22 20506501 . TCCCCCC TCCCC,TCCCCCG 249.62 PASS . GT:GQ 1|2:0.00
chr22 20507118 . T TG 496.06 PASS . GT:GQ 1/1:495.81
Expand Down Expand Up @@ -488,7 +488,7 @@ chr22 20599332 . TCCTCCTCCTCTTCTCACTTCTTTGCATTCTTCTGATCTCTTAAGGGCGTACA CTAATTTTT
chr22 20599386 . AGATTCACATGGATATTAAAGTCAATGGTGATTTTCATAGCTGGAA CTAATTTTTTGTATTTTTAGTAGAGATGGGGTTTCACCATGTTG 257.23 PASS . GT:GQ 1|0:84.73
chr22 20599440 . GGGAGAAAGTAGGAATTCTTTTTTTTTTTTTTTTCTC CTAATTTTTTGTATTTTTAGTAGAGATGGGGTTTCACCATGTTGCTCA 259.05 PASS . GT:GQ 1|0:219.44
chr22 20599494 . TCCCACCTCAGCCTTCTGAGTAGCTGGGACCACAGGTGTGAGCCTCCATGATGGCCTGGC C 402.63 PASS . GT:GQ 1|0:402.64
chr22 20601187 . CTT C 187.42 PASS . GT:GQ 1/1:11.55
chr22 20601187 . CTT C 187.43 PASS . GT:GQ 1/1:11.55
chr22 20602281 . CAAA CAA,C 108.32 PASS . GT:GQ 1|2:0.00
chr22 20602375 . TTC T 273.87 PASS . GT:GQ 1|0:273.85
chr22 20603453 . TA T 76.82 PASS . GT:GQ 1/1:35.78
Expand Down Expand Up @@ -574,7 +574,7 @@ chr22 20734424 . CGGG CGG,G 91.67 PASS . GT:GQ 1|2:-0.00
chr22 20735870 . TG T 7.21 PASS . GT:GQ 1/1:0.00
chr22 20739037 . CAA C 114.57 PASS . GT:GQ 1/1:1.49
chr22 20739181 . TA T 21.00 PASS . GT:GQ 1/1:5.40
chr22 20739469 . GAA G 58.25 PASS . GT:GQ 1/1:0.34
chr22 20739469 . GAA G 58.24 PASS . GT:GQ 1/1:0.34
chr22 20740061 . CAA C 227.13 PASS . GT:GQ 1/1:0.00
chr22 20740265 . TAA T,TA 237.39 PASS . GT:GQ 1|2:0.00
chr22 20744359 . TA T 7.47 PASS . GT:GQ 1/1:2.73
Expand All @@ -586,7 +586,7 @@ chr22 20747363 . TA T 37.17 PASS . GT:GQ 1/1:0.13
chr22 20758364 . CAAAAAA C,CA 75.08 PASS . GT:GQ 1|2:0.00
chr22 20758459 . CTT C,CT 4.53 PASS . GT:GQ 1|2:-0.00
chr22 20759402 . CT C 27.09 PASS . GT:GQ 1/1:5.19
chr22 20761076 . G GC 266.95 PASS . GT:GQ 0|1:265.60
chr22 20761076 . G GC 266.95 PASS . GT:GQ 0|1:265.61
chr22 20762086 . TAA T,TA 43.58 PASS . GT:GQ 1|2:-0.00
chr22 20763017 . TGGGGGGG G 175.13 PASS . GT:GQ 1/1:98.23
chr22 20763094 . ACC A 9.09 PASS . GT:GQ 1/1:8.86
Expand Down Expand Up @@ -706,17 +706,17 @@ chr22 20916056 . CAAAA CAAAAA,C 248.23 PASS . GT:GQ 1|2:0.00
chr22 20917429 . AG A 56.79 PASS . GT:GQ 1/1:53.46
chr22 20918430 . GTTTTT G,GT 216.85 PASS . GT:GQ 1|2:0.00
chr22 20918592 . CTT C 58.44 PASS . GT:GQ 1/1:54.51
chr22 20919841 . GAA G 297.75 PASS . GT:GQ 1/1:247.82
chr22 20919841 . GAA G 297.76 PASS . GT:GQ 1/1:247.83
chr22 20919966 . CTGAT C 351.46 PASS . GT:GQ 0|1:351.03
chr22 20920861 . TA T 277.67 PASS . GT:GQ 1/1:277.65
chr22 20921515 . T TACAGGAGGGACCTGTATA 298.53 PASS . GT:GQ 1/1:298.39
chr22 20921515 . T TACAGGAGGGACCTGTATA 298.52 PASS . GT:GQ 1/1:298.39
chr22 20921718 . CT C 191.43 PASS . GT:GQ 1/1:60.55
chr22 20922006 . CT C 5.12 PASS . GT:GQ 1|0:2.19
chr22 20924556 . GA G 440.26 PASS . GT:GQ 1/1:440.20
chr22 20925186 . C CAA 185.64 PASS . GT:GQ 1/1:93.46
chr22 20925516 . GA G 161.40 PASS . GT:GQ 1/1:76.34
chr22 20927111 . CAAAAAAA C,CAAAAAA 277.50 PASS . GT:GQ 1|2:0.02
chr22 20927192 . A AT 167.21 PASS . GT:GQ 1/1:20.49
chr22 20927192 . A AT 167.21 PASS . GT:GQ 1/1:20.48
chr22 20927513 . CT CG 373.62 PASS . GT:GQ 1/1:371.99
chr22 20928467 . AC AA 248.33 PASS . GT:GQ 1/1:248.30
chr22 20928812 . CAAAA CAA,C 109.37 PASS . GT:GQ 1|2:0.05
Expand Down
Binary file modified sample/test_run/HG002.chr22.sample.indels.vcf.gz
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.indels.vcf.gz.tbi
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.snps.phased.bam.bai
View file
Binary file not shown.
2 changes: 1 addition & 1 deletion sample/test_run/HG002.chr22.sample.snps.phased.preclean.vcf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1106,7 +1106,7 @@ chr22 20384399 . G A 386.526 PASS . GT:DP:FQ:PS 0|1:48:0.6458:20000722
chr22 20384500 . G T 348.168 PASS . GT:DP:FQ:PS 0|1:48:0.5417:20000722
chr22 20384589 . G C 312.418 PASS . GT:DP:FQ:PS 1|0:49:0.2653:20000722
chr22 20384769 . G C 301.072 PASS . GT:DP:FQ:PS 0|1:49:0.3878:20000722
chr22 20385085 . C A 308.108 PASS . GT:DP:FQ:PS 1|0:49:0.3265:20000722
chr22 20385085 . C A 308.114 PASS . GT:DP:FQ:PS 1|0:49:0.3265:20000722
chr22 20385336 . C T 446.884 PASS . GT:DP:FQ:PS 1|0:51:0.3137:20000722
chr22 20386488 . T C 400.461 PASS . GT:DP:FQ:PS 0|1:47:0.5319:20000722
chr22 20386943 . G C 327.13 PASS . GT:DP:FQ:PS 0|1:49:0.5306:20000722
Expand Down
Binary file modified sample/test_run/HG002.chr22.sample.snps.phased.vcf.gz
View file
Binary file not shown.
Binary file modified sample/test_run/HG002.chr22.sample.snps.phased.vcf.gz.tbi
View file
Binary file not shown.
2 changes: 1 addition & 1 deletion sample/test_run/HG002.chr22.sample.snps.vcf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1104,7 +1104,7 @@ chr22 20384399 . G A 386.526 PASS . GT:DP:FQ 1/0:48:0.6458
chr22 20384500 . G T 348.168 PASS . GT:DP:FQ 0/1:48:0.5417
chr22 20384589 . G C 312.418 PASS . GT:DP:FQ 0/1:49:0.2653
chr22 20384769 . G C 301.072 PASS . GT:DP:FQ 0/1:49:0.3878
chr22 20385085 . C A 308.108 PASS . GT:DP:FQ 1/0:49:0.3265
chr22 20385085 . C A 308.114 PASS . GT:DP:FQ 1/0:49:0.3265
chr22 20385336 . C T 446.884 PASS . GT:DP:FQ 1/0:51:0.3137
chr22 20386488 . T C 400.461 PASS . GT:DP:FQ 0/1:47:0.5319
chr22 20386943 . G C 327.130 PASS . GT:DP:FQ 0/1:49:0.5306
Expand Down
Binary file modified sample/test_run/HG002.chr22.sample.snps.vcf.gz
View file
Binary file not shown.
2 changes: 1 addition & 1 deletion sample/test_run/args
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Namespace(bam='HG002.nanopore.chr22.sample.bam', chrom='chr22', cpu=1, del_threshold=0.6, enable_whatshap=False, end=21000000, exclude_bed=None, include_bed=None, ins_threshold=0.4, maxcov=160, min_allele_freq=0.15, min_nbr_sites=1, mincov=8, mode='both', model='NanoCaller1', neighbor_threshold='0.4,0.6', prefix='HG002.chr22.sample', ref='chr22_ref.fa', sample='SAMPLE', sequencing='ont', start=20000000, supplementary=False, vcf='test_run', wgs_contigs_type='with_chr', wgs_print_commands=False)
Namespace(bam='HG002.nanopore.chr22.sample.bam', chrom='chr22', cpu=1, del_threshold=0.6, enable_whatshap=False, end=21000000, exclude_bed=None, include_bed=None, ins_threshold=0.4, keep_bam=False, maxcov=160, min_allele_freq=0.15, min_nbr_sites=1, mincov=8, mode='both', model='NanoCaller1', neighbor_threshold='0.4,0.6', output='test_run', prefix='HG002.chr22.sample', ref='chr22_ref.fa', sample='SAMPLE', sequencing='ont', start=20000000, supplementary=False)
2 changes: 1 addition & 1 deletion sample/test_run/ref.sdf/done
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Finished successfully in 1 s.
Finished successfully in 2 s.
Loading

0 comments on commit 6de1b92

Please sign in to comment.