Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Annotation speedups #53

Closed
wants to merge 80 commits into from
Closed

Annotation speedups #53

wants to merge 80 commits into from

Conversation

Marcel-Mueck
Copy link
Collaborator

What

The new update to the annotation pipeline includes all annotation tools used for the published version of deeprvat, as well as all processing steps needed to create ready-to-run annotation data for the published deeprvat version.
rulegraph
Updates since last push:

  • inclusion of AlphaMissense annotations as plugin in VEP
  • changed DAG in order to optimize parallelisation
  • Allele Frequency, MAF and AF_MB calculated from genotype file
  • pipeline includes filtering step for variants based on distance to the nearest exon
  • selection of required columns for deeprvat as well as renaming of columns to fit required input is now performed
  • abSplice pipeline is now integrated into the DAG
  • spliceAI scores are aggregated to spliceAI_delta_sores

Testing

  • since the annotation tools rely on pre-scored data, automatic tests via GitHub actions is not completely possible
  • the annotation pipeline was instead tested locally using example data

“Marcel-Mueck” and others added 30 commits August 4, 2023 11:44
added support for deepRiPe and deepSea, ability to concat and merge a…
efd46a8 
…nnotations into a single file
changed config file
1545c49
@endast
Create deeprvat_annotations.yml
44702f9
@endast
Update deeprvat_annotations.yml
a5fed75 
add snakemake
cleaned up commented out code in annotations.py
bb66bce
Merge branch 'Annotations' of github.com:PMBio/deeprvat into Annotations
f0e1b86
@endast
Update deeprvat_annotations.yml
98aad4e 
Add snakemake, tensorflow and lock numpy
@endast
Update deeprvat_annotations.yml
a012f2c 
add   nvidia channel
@endast
Update deeprvat_annotations.yml
b58f654 
add python=3.8.17
@endast
Use python=3.9.16
d6ef2f5
@endast
Update deeprvat_annotations.yml
74dd04e 
Remove nvidia channel
increased memory resource for rule concat_deepSea
cfe68ca
Merge branch 'main' into Annotations
deec6b7
Merge branch 'Annotations' of github.com:PMBio/deeprvat into Annotations
6e72396
@endast
Update deeprvat_annotations.yml
5091be0 
Support strict
deleted obsolete environment yaml file; removed absolute path from an…
d8806e2 
…notation config.
Merge branch 'Annotations' of github.com:PMBio/deeprvat into Annotations
b327392
updated environment, config, removed dask dependency
03218c8
changed annotations.py to make concatenation
3aac39a 
function more general
added support for different deepRIPE models (hg2, k5 and parclip)
c987400
fixup! Format Python code with psf/black pull_request
9a8391c
changed some temporary paths
44acb9e
filtering and aggregating vep columns
33a5b78
fixup! Format Python code with psf/black pull_request
e18412c
@Marcel-Mueck
Update README.md
5d2c32c
@Marcel-Mueck
Update README.md
5f9a2bc
@Marcel-Mueck
Update README.md
16206dc
@Marcel-Mueck
Update README.md
f707b9e
@bfclarke
speedups using joblib and batching
5d65cdf
@bfclarke
remove unused function
1fec6f2
endast and others added 28 commits October 15, 2023 12:24
@endast
Fix split of cols
e29fcc0
@endast
Only read the file in one place
7be1915
fixed path names in snakemake rules
132efc4
Merge branch 'deepripe-batch' of github.com:PMBio/deeprvat into deepr…
7cdb6ab 
…ipe-batch

Conflicts:
	pipelines/annotations.snakefile
small changes
22b979a
@endast
Use gz files
5995e8b
@endast
Update annotations.snakefile
c9bb44b 
Fix typo
@endast
cast - as na in vep files
c4b452c
pca object is saved and loaded as np array of components by default.
0599c89
Merge branch 'deepripe-batch' of github.com:PMBio/deeprvat into deepr…
5affade 
…ipe-batch
parallelised deepripe merge and use parquet instead of csv for pca
48944dd
use parquet files for pca input
51a59d9
added docstrings to each function
d7f3d1d
changed vep cmd to include only scores (not class label )for PolyPhen…
f3e3ee2 
… and SIFT
removed redundant vep_repo_path from config
6225278
Merge branch 'main' into annotation-speedups
8ba9c47
added final steps of the annotation processing: calculating af, maf a…
4a47768 
…nd maf_mb scores from the genotype information, filtering variants by distance to nearest exomes and slecting and renaming columns to fit to model
@Marcel-Mueck
Create empty gencode.v44.annotation.gtf.gz
6047dd5 
needed for test run
@Marcel-Mueck
Create genotypes.h5
c5a904a 
added empty genotype file for GitHub tests
@Marcel-Mueck
Update deeprvat_annotation_config.yaml
06f0448 
changed name for fasta name to be in line for the github tests
@Marcel-Mueck
removed commented out lines
763cc54
@Marcel-Mueck
changed path for primateAI File in annotation config
5a1c8dc
@Marcel-Mueck
Removed commented out lines in annotation pipeline and source
2602240
@Marcel-Mueck
removed duplicated 'merge_deepsea_pcas' function
1cffa6c
@Marcel-Mueck
updated annotation rulegraph
8f3560e
@Marcel-Mueck
Update deeprvat_annotation_config.yaml
cd217f5
@Marcel-Mueck
Update absplice_download_Snakefile
d471a0b
fixup! Format Python code with psf/black pull_request
3a05645
@Marcel-Mueck Marcel-Mueck deleted the annotation-speedups branch April 11, 2024 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Marcel-Mueck @endast @bfclarke