-- mode: markdown; mode: auto-fill; fill-column: 80 --
README -- HPC @ UL
Time-stamp: <Mer 2013-04-03 17:40 svarrette>
This repository holds a set of launcher scripts to be used on the UL HPC platform. They are provided for users of the infrastructure to make their life easier (and hopefully more efficient) on the following typical workflows:
-
Embarrassingly parallel run for repetitive and/or multi-parametric jobs over a Java/C/C++/Ruby/Perl/Python/R script, corresponding (normally) to the following cases:
-
serial (or sequential) tasks having all similar duration, run on one node
-
serial (or sequential) tasks having varying durations, run on one node
-
serial (or sequential) tasks having varying durations, run on multiple nodes
-
MPI run on n processes (ex: HPL) with abstraction of the MPI stack, MPI script, option to compile the code etc.
-
MPI run on n process per node (ex: OSU Micro-benchmarks)
We propose here two types of contributions:
- a set of
bashscripts examples that users can use as a startup example to adapt for their own workflow - NOT YET IMPLEMENTED a more generic ruby script interfaced by a YAML configuration file which hold the specificity of each users.
The UL HPC platform offers parallel computing resource, so
it's important you make an efficient use of the computing nodes, even when
processing serial jobs.
In particular, you should avoid to submit purely serial jobs to the OAR queue as
it would waste the computational power (11 out of 12 cores is you reserve one
node on gaia for instance).
A bad behaviour in this context is illustrated in
bash/serial/NAIVE_AKA_BAD_launcher_serial.sh where you'll recognize a pattern
you perhaps use in your own script:
for i in `seq 1 ${NB_TASKS}`; do
${TASK} $i
done
If you're more familiar with UNIX, you can perhaps argue we can fork separate
processes using the bash & (ampersand) builtin control operator and the wait
command.
This is illustrated in bash/serial/launcher_serial_ampersand.sh and
corresponds to the following pattern:
for i in `seq 1 ${NB_TASKS}`; do
${TASK} $i &
done
wait
This approach is straightforward and is sufficient assuming (1) you don't have a
huge number of tasks to fork and (2) each tasks has the a similar duration.
For all the other (serial) cases, an approach based on
GNU parallel if more effective as it
permits to easily and efficiently schedule batch of n tasks in parallel
(-j n), where n typically stands for the number of cores of the nodes.
This is illustrated in bash/serial/launcher_serial.sh and corresponds to the
following pattern:
seq ${NB_TASKS} | parallel -u -j 12 ${TASK} {}
Not convinced you have interest to these approaches? Take a look at the
following completion times performed on the chaos cluster for the task
mytask.sh proposed in bash/serial/mytask.sh:
+---------+---------------+--------+--------------+----------------------+-----------+
| NB_TASK | HOSTNAME | #CORES | TASK | APPROACH | TIME |
+---------+---------------+--------+--------------+----------------------+-----------+
| 24 | h-cluster1-32 | 12 | sleep {1-24} | Pure serial | 5m0.483s |
| 24 | h-cluster1-32 | 12 | sleep {1-24} | Ampersand + wait | 0m24.141s |
| 24 | h-cluster1-32 | 12 | sleep {1-24} | GNU Parallel (-j 12) | 0m36.404s |
| 24 | h-cluster1-32 | 12 | sleep {1-24} | GNU Parallel (-j 24) | 0m24.257s |
+---------+---------------+--------+--------------+----------------------+-----------+
The same benchmark performed for the sample argument file (see
bash/serial/mytask.args.example) to perform tasks of similar duration:
+---------+---------------+--------+---------+----------------------+-----------+
| NB_TASK | HOSTNAME | #CORES | TASK | APPROACH | TIME |
+---------+---------------+--------+---------+----------------------+-----------+
| 30 | h-cluster1-32 | 12 | sleep 2 | Pure serial | 1m0.374s |
| 30 | h-cluster1-32 | 12 | sleep 2 | Ampersand + wait | 0m2.217s |
| 30 | h-cluster1-32 | 12 | sleep 2 | GNU Parallel (-j 12) | 0m6.375s |
| 30 | h-cluster1-32 | 12 | sleep 2 | GNU Parallel (-j 24) | 0m4.255s |
+---------+---------------+--------+---------+----------------------+-----------+
Resources:
- Official documentation, crappy yet useful and full of concrete examples.
- Wiki SciNet
- Slides on GNU Parallel for Large Batches of Small Jobs
- GNU Parallel and PBS
- GNU parallel introduction
- Using GNU Parallel to Package Multiple Jobs in a Single PBS Job
If you have hundreds of serial tasks that you want to run concurrently and you reserved more than one nodes, then the approach above, while useful, would require tens of scripts to be submitted in separate OAR jobs (each of them reserving 1 full nodes).
It is also possible to use GNU parallel
in this case, using the --sshlogin options (altered to use the oarsh
connector).
This is illustrated in the generic launcher proposed in `
You'll find an example of launcher script for MPI jobs in
bash/MPI/mpi_launcher.sh.
Examples of usage are proposed in examples/MPI/
You should become familiar (if not yet) with Git. Consider these resources:
The Git branching model for this repository follows the guidelines of gitflow.
In particular, the central repo (on github.com) holds two main branches with an infinite lifetime:
production: the production-ready benchmark datadevel: the main branch where the latest developments interviene. This is the default branch you get when you clone the repo.
This repository is hosted on out GitHub. Once cloned, initiate the potential git submodules etc. by running:
$> cd launcher-scripts
$> make setup