INSTALL

To install pydpiper and its depending Python packages, both the current source and most recent stable release can be downloaded from github:

python3 -m pip install 'git+https://github.com/Mouse-Imaging-Centre/pydpiper' <...other installation args...>

Alternately you may manually clone the repository or download it as a .tar.gz or .zip file, untar/unzip as applicable, `cd` into the source directory (e.g. `pydpiper/`), and run `python3 setup.py install <...extra args...>`.

You must also install the necessary system (non-Python) dependencies; see below.  Of these, the most important is `libminc`.

System dependencies:

If you use our MICe Singularity/Apptainer container as per Pydpiper's README, you don't need to install minc-toolkit, minc-stuffs, or minc2-simple, only libminc.  This is probably the easiest way to go.

libminc (required if minc-toolkit-v2 is not installed)
minc-toolkit v2 - https://github.com/BIC-MNI/minc-toolkit-v2 (required if not using a Singularity container)
minc-stuffs - https://github.com/Mouse-Imaging-Centre/minc-stuffs (required if not using a Singularity container)
minc2-simple - https://github.com/vfonov/minc2-simple (required if not using a Singularity container)
cctools (required for the Makeflow backend) - https://ccl.cse.nd.edu
Singularity (required to avoid installing several of the above packages) - https://docs.sylabs.io/guides/latest/admin-guide/
graphviz (optional) - http://www.graphviz.org/

Various Python libraries are also required (see `setup.py` for details); these should
be installed automatically by setuptools when the above command is run.

Pydpiper has several options for executing pipelines: locally, via its own server-executor system, and via Makeflow.
We recommend one of the the latter two options for most pipelines.
Pydpiper's server-executor model uses `qbatch`](https://github.com/pipitone/qbatch) for submitting executors to compute grid queues and RPC (specifically Pyro5) for submitting jobs to and communicating with executors.
Installing Pydpiper will install the `qbatch` binary as well, but you must ensure this (and a somewhat recent version of GNU Parallel) is in your path. To configure Pydpiper for your queue using this backend, you may configure qbatch directly using `QBATCH_*` environment variables to whatever extent you desire and further specify or override whatever you choose using Pydpiper flags (or a config file or environment vars; see the README); however, the default is currently local execution so you must tell Pydpiper what `--queue-type` you intend.  (We also have a legacy "headless" mode developed for "capacity" queues such as Compute Canada's SciNet grid; this mode uses `qsub` directly and only works with TORQUE/PBS queues at the moment but may soon be retired.)  The makeflow backend is more versatile and supports a larger set of queues, but does not automatically use a server-executor model; to use this, ensure cctools is installed, start your Pydpiper pipeline with --backend=makeflow and --queue-type=... and then launch an instance of `work_queue_factory` to submit executors.