Copyright (c) 2021 UL HPC Team <hpc-team@uni.lu>
GNU Parallel is a tool for executing tasks in parallel, typically on a single machine. When coupled with the Slurm command srun
, parallel becomes a powerful way of distributing a set of tasks amongst a number of workers. This is particularly useful when the number of tasks is significantly larger than the number of available workers (i.e. $SLURM_NTASKS
), and each tasks is independent of the others.
The parallel
command is available at the system level across the ULHPC clusters, yet under a relatively old version:
(access)$ which parallel
/usr/bin/parallel
(access)$ parallel --version
GNU parallel 20190922
[...]
If you want to build a more recent version. The process is quite straight-forward and we will illustrate this process Easybuild (see the Easybuild tutorial and Spack tutorial).
- We are going to extend the current software set to avoid recompiling a toolchain
- The parallel version will be built using GCC-10.2.0
(node)$ resif-load-home-swset-prod
# Check where it will be installed
(node)$ echo $MODULEPATH
# Loading Easybuild
(node)$ module load tools/EasyBuild
(node)$ eb -S parallel
eb -S parallel
== found valid index for /opt/apps/resif/aion/2020b/epyc/software/EasyBuild/4.5.4/easybuild/easyconfigs, so using it...
CFGS1=/opt/apps/resif/aion/2020b/epyc/software/EasyBuild/4.5.4/easybuild/easyconfigs
* $CFGS1/a/Amber/Amber-16-AT-17-fix_missing_do_parallel_in_checkrismunsupported.patch
* $CFGS1/a/Amber/Amber-18-AT-18_fix_missing_do_parallel_in_checkrismunsupported.patch
* $CFGS1/h/HPL/HPL_parallel-make.patch
* $CFGS1/i/ipyparallel/ipyparallel-6.2.2-foss-2018a-Python-3.6.4.eb
* $CFGS1/j/Judy/Judy-1.0.5_parallel-make.patch
* $CFGS1/n/NWChem/NWChem-6.3.revision2-parallelbuild.patch
* $CFGS1/n/NWChem/NWChem-6.5.revision26243-parallelbuild.patch
* $CFGS1/n/NWChem/NWChem-6.6.revision27746-parallelbuild.patch
* $CFGS1/n/netCDF/netCDF-4.3.2-parallel-HDF.patch
* $CFGS1/o/OpenSSL/OpenSSL-1.0.1i-fix_parallel_build-1.patch
* $CFGS1/o/OpenSSL/OpenSSL-1.0.1m_fix-parallel.patch
* $CFGS1/o/OpenSees/OpenSees-3.2.0-add_Makefile_def_parallel.patch
* $CFGS1/o/OpenSees/OpenSees-3.2.0-intel-2020a-Python-3.8.2-parallel.eb
* $CFGS1/p/ParallelIO/ParallelIO-2.2.2a-intel-2017a.eb
* $CFGS1/p/PyTorch/PyTorch-1.7.0_fix_test_DistributedDataParallel.patch
* $CFGS1/p/parallel-fastq-dump/parallel-fastq-dump-0.6.5-GCCcore-8.2.0-Python-3.7.2.eb
* $CFGS1/p/parallel-fastq-dump/parallel-fastq-dump-0.6.6-GCCcore-9.3.0-Python-3.8.2.eb
* $CFGS1/p/parallel/parallel-20141122-GCC-4.9.2.eb
* $CFGS1/p/parallel/parallel-20150322-GCC-4.9.2.eb
* $CFGS1/p/parallel/parallel-20150822-GCC-4.9.2.eb
* $CFGS1/p/parallel/parallel-20160622-foss-2016a.eb
* $CFGS1/p/parallel/parallel-20170822-intel-2017a.eb
* $CFGS1/p/parallel/parallel-20171022-intel-2017b.eb
* $CFGS1/p/parallel/parallel-20171122-foss-2017b.eb
* $CFGS1/p/parallel/parallel-20171122-intel-2017b.eb
* $CFGS1/p/parallel/parallel-20180422-intel-2018a.eb
* $CFGS1/p/parallel/parallel-20180822-foss-2018b.eb
* $CFGS1/p/parallel/parallel-20181222-intel-2018b.eb
* $CFGS1/p/parallel/parallel-20190222-GCCcore-7.3.0.eb
* $CFGS1/p/parallel/parallel-20190622-GCCcore-8.2.0.eb
* $CFGS1/p/parallel/parallel-20190922-GCCcore-8.3.0.eb
* $CFGS1/p/parallel/parallel-20200422-GCCcore-9.3.0.eb
* $CFGS1/p/parallel/parallel-20200522-GCCcore-9.3.0.eb
* $CFGS1/p/parallel/parallel-20210322-GCCcore-10.2.0.eb
* $CFGS1/p/parallel/parallel-20210622-GCCcore-10.3.0.eb
* $CFGS1/p/parallel/parallel-20210722-GCCcore-11.2.0.eb
* $CFGS1/r/R/DMCfun-1.3.0_fix-parallel-detect.patch
* $CFGS1/w/WRF/WRF_parallel_build_fix.patch
* $CFGS1/x/Xmipp/Xmipp-3.19.04-Apollo_add_missing_pthread_to_XmippParallel.patch
Note: 7 matching archived easyconfig(s) found, use --consider-archived-easyconfigs to see them
- Let's install
$CFGS1/p/parallel/parallel-20210322-GCCcore-10.2.0.eb
(node)$ eb parallel-20210322-GCCcore-10.2.0.eb -r
(node)$ module av parallel
-------------------------------------------------------------------------------------------------------------------------- /home/users/ekieffer/.local/easybuild/aion/2020b/epyc/modules/all ---------------------------------------------------------------------------------------------------------------------------
tools/parallel/20210322-GCCcore-10.2.0
If the avail list is too long consider trying:
"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
- If we don't have it installed, please follow first the Spack tutorial
(node)$ spack list parallel
aws-parallelcluster intel-parallel-studio parallel parallel-netcdf parallelio parallelmergetree perl-parallel-forkmanager py-ipyparallel py-pytest-parallel r-biocparallel r-doparallel r-optimparallel r-parallelly r-parallelmap r-rcppparallel
==> 15 packages
(node)$ spack versions parallel
==> Safe versions (already checksummed):
20220522 20220422 20220322 20220222 20220122 20210922 20200822 20190222 20170322 20170122 20160422 20160322
==> Remote versions (not yet checksummed):
20230422 20221222 20220822 20211122 20210622 20210222 20201022 20200522 20200122 20190922 20190522 20181222 20180822 20180422 20171222 20170822 20170422 20161022 20160622 20151222 20150822 20150422 20141122 20140722 20140322 20131122 20130722 20130222 20121022 20120522 20120122
20230322 20221122 20220722 20211022 20210522 20210122 20200922 20200422 20191222 20190822 20190422 20181122 20180722 20180322 20171122 20170722 20170222 20160922 20160522 20151122 20150722 20150322 20141022 20140622 20140222 20131022 20130622 20130122 20120822 20120422
20230222 20221022 20220622 20210822 20210422 20201222 20200722 20200322 20191122 20190722 20190322 20181022 20180622 20180222 20171022 20170622 20161222 20160822 20160222 20151022 20150622 20150222 20140922 20140522 20140122 20130922 20130522 20121222 20120722 20120322
20230122 20220922 20211222 20210722 20210322 20201122 20200622 20200222 20191022 20190622 20190122 20180922 20180522 20180122 20170922 20170522 20161122 20160722 20160122 20150922 20150522 20150122 20140822 20140422 20131222 20130822 20130422 20121122 20120622 20120222
- The most recent and safe version is
20220522
(node)$ spack install parallel@20220522
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/berkeley-db-18.1.40-uw5w4yhzzi2fatjzb72ipgdf3w657tle
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/bzip2-1.0.8-ymcs7cevgovcd3bc5iphzo5ztzv62jue
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/ncurses-6.4-3qm6oylywjcvizw7xyqbkxg33vqtgppp
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/zlib-1.2.13-426hs7tsxcfpebed5uqlogma32dbuvj5
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/readline-8.2-zgvmdyizb6g4ee2ozansvqkxvgbq6a6r
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/gdbm-1.23-seklrqiazmh54ts3tdx6fpnlobviw3ia
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/perl-5.36.0-yvgdvqozt46iimytfldxpjhxxao2gtdy
==> Installing parallel-20220522-ediq6fzty3il5qc3frzlwn45szfvbhek
==> No binary for parallel-20220522-ediq6fzty3il5qc3frzlwn45szfvbhek found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/bb/bb6395f8d964e68f3bdb26a764d3c48b69bc5b759a92ac3ab2bd1895c7fa8c1f.tar.bz2
==> No patches needed for parallel
==> parallel: Executing phase: 'autoreconf'
==> parallel: Executing phase: 'configure'
==> parallel: Executing phase: 'build'
==> parallel: Executing phase: 'install'
==> parallel: Successfully installed parallel-20220522-ediq6fzty3il5qc3frzlwn45szfvbhek
Stage: 2.99s. Autoreconf: 0.00s. Configure: 1.20s. Build: 0.02s. Install: 0.60s. Total: 4.90s
[+] /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/parallel-20220522-ediq6fzty3il5qc3frzlwn45szfvbhek
(node)$ spack find -vpl parallel
-- linux-rhel8-zen / gcc@=8.5.0 ---------------------------------
ediq6fz parallel@20220522 build_system=autotools /mnt/irisgpfs/users/ekieffer/.spack/opt/spack/linux-rhel8-zen/gcc-8.5.0/parallel-20220522-ediq6fzty3il5qc3frzlwn45szfvbhek
==> 1 installed package
The GNU Parallel syntax can be a little distributing, but basically it supports two modes:
-
Reading command arguments on the command line:
parallel [-j N] [OPTIONS] COMMAND {} ::: TASKLIST
-
Reading command arguments from an input file:
parallel –a TASKLIST.LST [-j N] [OPTIONS] COMMAND {} parallel [-j N] [OPTIONS] COMMAND {} :::: TASKLIST.LST
If your COMMAND embed a pipe stage, you have to escape the pipe symbol as follows \|
.
Let's make some tests. The -j <N>
option permits to define the jobs per machine - in particular you may want to use -j 1
to enable a sequential resolution of the parallel command.
In all cases, the parallel
command is available at the system across the ULHPC clusters. Run it once.
(access)$> parallel --version
GNU parallel 20160222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --bibtex'.
If you want to avoid the disclaimer requesting to cite the paper describing GNU parallel, you have to run as indicated parallel --bibtex
, type: 'will cite' and press enter.
Let's play with a TASKLIST from the command line:
(access)$> parallel echo {} ::: A B C
A
B
C
(access)$> parallel echo {} ::: {1..3}
1
2
3
# As above
(access)$> parallel echo {} ::: $(seq 1 3)
1
2
3
# Use index to refer to a given TASKLIST
(access)$> parallel echo {1} {2} ::: A B C ::: {1..3}
A 1
A 2
A 3
B 1
B 2
B 3
C 1
C 2
C 3
# Combine (link) TASKLIST with same size - Use '--link' on more recent parallel version
(access)$> parallel --xapply "echo {1} {2}" ::: A B C ::: {1..3}
A 1
B 2
C 3
# This can be useful to output command text with arguments
(access)$> parallel --xapply echo myapp_need_argument {1} {2} ::: A B C ::: {1..3}
myapp_need_argument A 1
myapp_need_argument B 2
myapp_need_argument C 3
# /!\ IMPORTANT: you can then **execute** these commands as above by removing 'echo'
# DON'T do that unless you know what you're doing
# You can filter out some elements:
(access)$> parallel --xapply echo myapp_need_argument {1} {2} \| grep -v 2 ::: A B C ::: {1..3}
myapp_need_argument A 1
myapp_need_argument C 3
Let's play now with a TASKLIST from an input file.
Let's assume you wish to process some images from the OpenImages V4 data set.
A copy of this data set is available on the ULHPC facility, under /work/projects/bigdata_sets/OpenImages_V4/
.
Let's create a CSV file which contains a random selection of 10 training files within this dataset (prefixed by a line number).
You may want to do it as follows (copy the full command):
# training set select first 10K random sort take only top 10 prefix by line number print to stdout AND in file
# ^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(access)$> find /work/projects/bigdata_sets/OpenImages_V4/train/ -print | head -n 10000 | sort -R | head -n 10 | awk '{ print ++i","$0 }' | tee openimages_v4_filelist.csv
1,/work/projects/bigdata_sets/OpenImages_V4/train/6196380ea79283e0.jpg
2,/work/projects/bigdata_sets/OpenImages_V4/train/7f23f40740731c03.jpg
3,/work/projects/bigdata_sets/OpenImages_V4/train/dbfc1b37f45b3957.jpg
4,/work/projects/bigdata_sets/OpenImages_V4/train/f66087cdf8e172cd.jpg
5,/work/projects/bigdata_sets/OpenImages_V4/train/5efed414dd8b23d0.jpg
6,/work/projects/bigdata_sets/OpenImages_V4/train/1be054cb3021f6aa.jpg
7,/work/projects/bigdata_sets/OpenImages_V4/train/61446dee2ee9eb27.jpg
8,/work/projects/bigdata_sets/OpenImages_V4/train/dba2da75d899c3e7.jpg
9,/work/projects/bigdata_sets/OpenImages_V4/train/7ea06f092abc005e.jpg
10,/work/projects/bigdata_sets/OpenImages_V4/train/2db694eba4d4bb04.jpg
Let's manipulate the file content with parallel (prefer the -a <filename>
syntax):
# simple echo of the file
(access)$> parallel -a openimages_v4_filelist.csv echo {}
1,/work/projects/bigdata_sets/OpenImages_V4/train/6196380ea79283e0.jpg
2,/work/projects/bigdata_sets/OpenImages_V4/train/7f23f40740731c03.jpg
3,/work/projects/bigdata_sets/OpenImages_V4/train/dbfc1b37f45b3957.jpg
[...]
# print specific column of the CSV file
(access)$> parallel --colsep '\,' -a openimages_v4_filelist.csv echo {1}
1
2
3
4
5
6
7
8
9
10
(access)$> parallel --colsep '\,' -a openimages_v4_filelist.csv echo {2}
/work/projects/bigdata_sets/OpenImages_V4/train/6196380ea79283e0.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/7f23f40740731c03.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/dbfc1b37f45b3957.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/f66087cdf8e172cd.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/5efed414dd8b23d0.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/1be054cb3021f6aa.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/61446dee2ee9eb27.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/dba2da75d899c3e7.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/7ea06f092abc005e.jpg
/work/projects/bigdata_sets/OpenImages_V4/train/2db694eba4d4bb04.jpg
# reformat and change order
(access)$> parallel --colsep '\,' -a openimages_v4_filelist.csv echo {2} {1}
/work/projects/bigdata_sets/OpenImages_V4/train/6196380ea79283e0.jpg 1
/work/projects/bigdata_sets/OpenImages_V4/train/7f23f40740731c03.jpg 2
/work/projects/bigdata_sets/OpenImages_V4/train/dbfc1b37f45b3957.jpg 3
/work/projects/bigdata_sets/OpenImages_V4/train/f66087cdf8e172cd.jpg 4
/work/projects/bigdata_sets/OpenImages_V4/train/5efed414dd8b23d0.jpg 5
/work/projects/bigdata_sets/OpenImages_V4/train/1be054cb3021f6aa.jpg 6
/work/projects/bigdata_sets/OpenImages_V4/train/61446dee2ee9eb27.jpg 7
/work/projects/bigdata_sets/OpenImages_V4/train/dba2da75d899c3e7.jpg 8
/work/projects/bigdata_sets/OpenImages_V4/train/7ea06f092abc005e.jpg 9
/work/projects/bigdata_sets/OpenImages_V4/train/2db694eba4d4bb04.jpg 10
The ULHPC team has designed a generic launcher for single node GNU parallel: see ../basics/scripts/launcher.parallel.sh
.