See the documentation »
Report Bug · Request Feature
ngs-preprocess is built using Nextflow, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. It is an easy to use pipeline that uses state-of-the-art software for quality check and pre-processing ngs reads of Illumina, Pacbio and Oxford Nanopore Technologies.
It wraps up the following software:
| Step | tools |
|---|---|
| SRA NBCI fetch | Entrez-direct & sra-tools |
| Illumina pre-processing | Fastp |
| Nanopore pre-processing | Porechop, Porechop ABI, pycoQC, NanoPack |
| Pacbio pre-processing | bam2fastx, bax2bam, lima, pacbio ccs |
This pipeline has two complementary pipelines (also written in nextflow) for genome assembly and prokaryotic genome annotation that can give the user a complete workflow for bacterial genomics analyses.
-
Install Nextflow:
curl -s https://get.nextflow.io | bash -
Give it a try:
nextflow run fmalmeida/ngs-preprocess --help
-
Download required tools
-
for docker
# for docker docker pull fmalmeida/ngs-preprocess:v2.7 # run nextflow run fmalmeida/ngs-preprocess -profile docker [options]
-
for singularity
# for singularity # remember to properly set NXF_SINGULARITY_LIBRARYDIR # read more at https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub export NXF_SINGULARITY_LIBRARYDIR=MY_SINGULARITY_IMAGES # your singularity storage dir export NXF_SINGULARITY_CACHEDIR=MY_SINGULARITY_CACHE # your singularity cache dir singularity pull \ --dir $NXF_SINGULARITY_LIBRARYDIR \ fmalmeida-ngs-preprocess-v2.7.img docker://fmalmeida/ngs-preprocess:v2.7 # run nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
-
for conda
# for conda # it is better to create envs with mamba for faster solving wget https://github.com/fmalmeida/ngs-preprocess/raw/master/environment.yml conda env create -f environment.yml # advice: use mamba # must be executed from the base environment # This tells nextflow to load the available ngs-preprocess environment when required nextflow run fmalmeida/ngs-preprocess -profile conda [options]
-
-
Start running your analysis
nextflow run fmalmeida/ngs-preprocess -profile <docker/singularity/conda>
🔥 Please read the documentation below on selecting between conda, docker or singularity profiles, since the tools will be made available differently depending on the profile desired.
Nextflow profiles are a set of "sensible defaults" for the resource requirements of each of the steps in the workflow, that can be enabled with the command line flag -profile. You can learn more about nextflow profiles at:
- https://nf-co.re/usage/configuration#basic-configuration-profiles
- https://www.nextflow.io/docs/latest/config.html#config-profiles
The pipeline have "standard profiles" set to run the workflows with either conda, docker or singularity using the local executor, which is nextflow's default and basically runs the pipeline processes in the computer where Nextflow is launched. If you need to run the pipeline using another executor such as sge, lsf, slurm, etc. you can take a look at nextflow's manual page to proper configure one in a new custom profile set in your personal copy of ngs-preprocess config file and take advantage that nextflow allows multiple profiles to be used at once, e.g. -profile conda,sge.
By default, if no profile is chosen, the pipeline will try to load tools from the local machine $PATH. Available pre-set profiles for this pipeline are: docker/conda/singularity, you can choose between them as follows:
-
conda
# must be executed from the base environment # This tells nextflow to load the available ngs-preprocess environment when required nextflow run fmalmeida/ngs-preprocess -profile conda [options]
-
docker
nextflow run fmalmeida/ngs-preprocess -profile docker [options]
-
singularity
nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
📖 Please use conda as last resource since the packages will not be "frozen and pre-installed", problems may arise.
For understading pipeline usage and configuration, users must read the complete online documentation »
All the parameters showed above can be, and are advised to be, set through the configuration file. When a configuration file is set the pipeline is run by simply executing:
nextflow run fmalmeida/ngs-preprocess -c ./configuration-fileYour configuration file is what will tell to the pipeline the type of data you have, and which processes to execute. Therefore, it needs to be correctly set up.
Create a configuration file in your working directory:
nextflow run fmalmeida/ngs-preprocess [ --get_config ]Nextflow has an awesome feature called NF tower. It allows that users quickly customise and set-up the execution and configuration of cloud enviroments to execute any nextflow pipeline from nf-core, github (this one included), bitbucket, etc. By having a compliant JSON schema for pipeline configuration it means that the configuration of parameters in NF tower will be easier because the system will render an input form.
Checkout more about this feature at: https://seqera.io/blog/orgs-and-launchpad/
Users can trigger a graphical and interactive pipeline configuration and execution by using nf-core launch utility. nf-core launch will start an interactive form in your web browser or command line so you can configure the pipeline step by step and start the execution of the pipeline in the end.
# Install nf-core
pip install nf-core
# Launch the pipeline
nf-core launch fmalmeida/ngs-preprocessIt will result in the following:
In order to cite this pipeline, please refer to:
Almeida FMd, Campos TAd and Pappas Jr GJ. Scalable and versatile container-based pipelines for de novo genome assembly and bacterial annotation. F1000Research 2023, 12:1205 (https://doi.org/10.12688/f1000research.139488.1)
Additionally, archived versions of the pipeline are also found in Zenodo.
This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the GPLv3.
The nf-core framework for community-curated bioinformatics pipelines.
Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.
Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.
In addition, users are encouraged to cite the programs used in this pipeline whenever they are used. Links to resources of tools and data used in this pipeline are as follows: