OpenFOAM is a free, open source computational fluid dynamics (CFD) software package released by the OpenFOAM Foundation. It has a large user base across most areas of engineering and science, from both commercial and academic organisations. OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics. OpenFOAM is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems, most prominently including computational fluid dynamics (CFD).
- Introduction
- Part I: Installing EasyBuild and OpenFOAM
- Part II: Test Examples
- Test Example 1: Running OpenFOAM-v2312 motorBike Tutorial on HPC
- Test Example 2: Testing turbulenceModels from planeChannel
- Test Example 3: Test atmDownstreamDevelopment from atmosphericModels
- Test Example 4: ChemFoam Tutorial Setup
- Test Example 5: ChemFoam h2 Tutorial Setup
- Test Example 6: ChemFoam ic8h18 Tutorial Setup
- Test Example 7: ChemFoam nc7h16 Tutorial Setup
- Part III: Reframe Installation
Parsa VARES
Nahid JAVADINARAB
In this project, we're gonna supposed to install OpenFOAM-11 and OpenFOAM-v2312 using EasyBuild 4.9.1 (the latest version).
Scope It is used in research organizations, academic institutes and across many types of industries, for example, automotive, manufacturing, process engineering and environmental engineering.
Purpose The goals of this project are:
- To install recent versions of OpenFOAM on the Uni.lu HPC platform using EasyBuild.
- To set up a regression testing configuration for the installed OpenFOAM using ReFrame.
- Connection to a Compute Node Allocating an interactive job with a high priority for 2 hours, using 1 node, 1 task, and 32 CPU cores.
- Command:
salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
- Installing the latest release of EasyBuild, a software build and installation framework.
- Command:
eb --install-latest-eb-release
- Looking for available EasyBuild modules and their details.
- Command:
module spider easybuild
- Now, by loading EasyBuild version 4.9.1 module, making its commands and environment available.
- Command:
module load tools/EasyBuild/4.9.1
5.The following command Adds the local directory of EasyBuild module files to the module path.
- Command:
module use ~/.local/easyconfig/modules/all
- Running this command Adds a project-specific directory of EasyBuild module files to the module path.
- Command:
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
EasyBuild can submit jobs to different backends including Slurm to install software, to distribute the often time-consuming installation of a set of software applications and the dependencies they require to a cluster.
This is done via the --job
command line option.
It is important to be aware of some details before you start using this, which we'll cover here.
- It Configures EasyBuild to use Slurm as the backend job scheduler for build jobs. The default job backend in EasyBuild
v4.x
is GC3Pie. To let EasyBuild submit jobs to Slurm instead, you should set the job-backend configuration setting toSlurm
, for example by setting the corresponding environment variable:
- Command:
export EASYBUILD_JOB_BACKEND='Slurm'
- This command Sets the base directory for EasyBuild installations and builds to the user's home directory.
- Command:
export EASYBUILD_PREFIX=$HOME/easybuild
- We Set the build path for EasyBuild to the shared memory directory, which can improve build performance.
- Command:
export EASYBUILD_BUILDPATH=/dev/shm/$USER
- This command Checks for and lists any missing dependencies required for building OpenFOAM.
- Command:
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --missing
- This command submits the OpenFOAM build as a job with 32 cores, a maximum wall time of 11 hours, resolving dependencies automatically, and tracing the build process.
- by using
sq
you can see it running - after it finished OpenFOAM .out can be found at /home/users/$USER
- the right output uploaded here you can check it. when you are opening the
.out
file, at the end you should run:
- by using
- Command:
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --job --job-cores 32 --job-max-walltime 11 --robot --trace
By default, EasyBuild will submit single-core jobs requesting for 24
hours of walltime
. You can tweak the requested resources via the job-cores
and job-max-walltime
configuration options. Note that not all job-* configuration settings apply to all job backends, see the EasyBuild documentation for more details.
If one or more dependencies are still missing for the software you want to install, you can combine --job
and --robot
to get EasyBuild to submit a separate job for each of the installations. These jobs will not --robot
, they will each only perform a single installation.
Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs on hold until the jobs that install the required (build) dependencies have completed successfully, and cancel jobs if the job to install a dependency failed for some reason.
== COMPLETED: Installation ended successfully (took 1 hour 3 mins 41 secs)
== Results of the build can be found in the log file(s) /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/eas$
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-j7o7pzoh/eb-txnweqte/easybuild-qnzsrnfi.log* have been removed.
== Temporary directory /tmp/eb-j7o7pzoh/eb-txnweqte has been removed.
- It adds the user's EasyBuild module files directory to the module path.
- Command:
module use /home/users/$USER/easybuild/modules/all
- loading OpenFOAM v2312 module built with the foss-2023a toolchain, making its commands and environment available.
- Command:
module load cae/OpenFOAM/v2312-foss-2023a
13.1 unload previous installed module to install the other OpenFOAM version.
- Command:
module unload cae/OpenFOAM/v2312-foss-2023a
- Now we repeat process again from the step 10th for installing OpenFOAM-11.
- Important: don't forget to do: 1.
export EASYBUILD_JOB_BACKEND='Slurm'
2.export EASYBUILD_PREFIX=$HOME/easybuild
3.export EASYBUILD_BUILDPATH=/dev/shm/$USER
before doing this line.
- Important: don't forget to do: 1.
- Command:
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-11-foss-2023a.eb --job --job-cores 64 --job-max-walltime 3 --robot --trace
- Loading OpenFOAM 11 module built with the foss-2023a toolchain, making its commands and environment available.
- Command:
module load cae/OpenFOAM/11-foss-2023a
-
Download the Tutorial Files
- Command:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/incompressible/simpleFoam/motorBike -O openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip
- Command:
-
Create the Directory Structure
- Command:
mkdir -p $HOME/OpenFOAM
- Command:
-
Upload the File to HPC Description: Use SCP or a similar tool to upload
openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip
to$HOME/OpenFOAM
on the HPC.- Command:
scp openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip $HOME/OpenFOAM
-
Unzip the Tutorial Files
- Command:
unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip -d $HOME/OpenFOAM
- Command:
-
Navigate to the Tutorial Directory
- Command:
cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike/tutorials/incompressible/simpleFoam/motorBike
- Command:
-
Source the OpenFOAM Environment Setup Script
- Command:
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
- Command:
-
Verify the Environment Variable
- Command:
echo $WM_PROJECT_DIR
- Command:
-
Make the
Allrun
Script Executable- Command:
chmod +x Allrun
- Command:
-
Run the
Allrun
Script Usingsrun
- Command:
srun -n 6 -c 6 ./Allrun
- Command:
-
Run the
Allrun
Script Directly (Optional)- Command:
./Allrun
- Command:
- Loading required modules
module load tools/EasyBuild/4.9.1
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
- installing(optional if needed)
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --job --job-cores 32 --job-max-walltime 11 --robot --trace
module use /home/users/$USER/easybuild/modules/all
module load cae/OpenFOAM/v2312-foss-2023a
- Now, we're gonna supposed to download tutorials of openfoam v2312, then upload them in openfoam directory that we have created for motorbike example:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip? path=tutorials/incompressible/simpleFoam -O openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip mkdir -p $HOME/OpenFOAM scp openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip $HOME/OpenFOAM unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip -d $HOME/OpenFOAM cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312/tutorials/verificationAndValidation/turbulenceModels/planeChannel
Source the OpenFOAM Environment Setup
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
- Verifying the Environment Variable
echo $WM_PROJECT_DIR
- Execution and running
chmod +x Allrun
srun -n 6 -c 6 ./Allrun
./Allrun
- Now, after running procedure it is possible to generate plots
./plot
open the plots directory
cd plots
- we see that all of
.png
files are saved in plots directory. (check ValidationVerfication folder for plots)
cd /home/users/$USER/OpenFOAM/openfoam-OpenFOAM-v2312/tutorials/verificationAndValidation/atmosphericModels/atmDownstreamDevelopment/
chmod +x Allrun
srun -n 1 -c 32 ./Allrun
./Allrun
By running ./plot
we generate results of the test.
Now, a subdirectory of plots
has been generated in this directory.
cd kOmegaSST/
cd ..
cd kEpsilon/
Allocate resources on the HPC:
salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
Set up the necessary environment modules:
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3
Download the chemFoam
tutorial files:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/gri -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip
Create the directory structure in your home directory:
mkdir -p $HOME/OpenFOAM
Upload the downloaded zip file to the HPC:
scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip $HOME/OpenFOAM
Unzip the tutorial files:
unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip -d $HOME/OpenFOAM
Navigate to the tutorial directory:
cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri/tutorials/combustion/chemFoam/gri
Source the OpenFOAM environment:
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR
Set execute permissions for the Allrun
script and list the directory contents:
chmod +x Allrun
ls
Expected output:
Allclean Allrun chemkin constant system validation
Run the simulation using srun
:
srun -n 6 -c 6 ./Allrun
./Allrun
For further information, refer to the OpenFOAM documentation:
- chemFoam Tutorial
- Part:
$FOAM_TUTORIALS/combustion/chemFoam/gri
Allocate resources on the HPC:
salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
Set up the necessary environment modules:
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3
Download the chemFoam
tutorial files for the h2
case:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/h2 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip
Create the directory structure in your home directory:
mkdir -p $HOME/OpenFOAM
Upload the downloaded zip file to the HPC:
scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip $HOME/OpenFOAM
Unzip the tutorial files:
unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip -d $HOME/OpenFOAM
Navigate to the tutorial directory:
cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2/tutorials/combustion/chemFoam/h2
Source the OpenFOAM environment:
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR
Set execute permissions for the Allrun
script and list the directory contents:
chmod +x Allrun
ls
Expected output:
Allclean Allrun chemkin constant system validation
Run the simulation using srun
:
srun -n 6 -c 6 ./Allrun
./Allrun
For further information, refer to the OpenFOAM documentation:
- chemFoam Tutorial
- Tutorial location in OpenFOAM:
$FOAM_TUTORIALS/combustion/chemFoam/h2
Allocate resources on the HPC:
salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
Set up the necessary environment modules:
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3
Download the chemFoam
tutorial files for the ic8h18
case:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/ic8h18 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip
Create the directory structure in your home directory:
mkdir -p $HOME/OpenFOAM
Upload the downloaded zip file to the HPC:
scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip $HOME/OpenFOAM
Unzip the tutorial files:
unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip -d $HOME/OpenFOAM
Navigate to the tutorial directory:
cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18/tutorials/combustion/chemFoam/ic8h18
Source the OpenFOAM environment:
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR
Set execute permissions for the Allrun
script and list the directory contents:
chmod +x Allrun
ls
Expected output:
Allclean Allrun chemkin constant system validation
Run the simulation using srun
:
srun -n 6 -c 6 ./Allrun
./Allrun
For further information, refer to the OpenFOAM documentation:
- chemFoam Tutorial
- Tutorial location in OpenFOAM:
$FOAM_TUTORIALS/combustion/chemFoam/ic8h18
Allocate resources on the HPC:
salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
Set up the necessary environment modules:
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3
Download the chemFoam
tutorial files for the nc7h16
case:
wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/nc7h16 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip
Create the directory structure in your home directory:
mkdir -p $HOME/OpenFOAM
Upload the downloaded zip file to the HPC:
scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip $HOME/OpenFOAM
Unzip the tutorial files:
unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip -d $HOME/OpenFOAM
Navigate to the tutorial directory:
cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16/tutorials/combustion/chemFoam/nc7h16
Source the OpenFOAM environment:
source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR
Set execute permissions for the Allrun
script and list the directory contents:
chmod +x Allrun
ls
Expected output:
Allclean Allrun chemkin constant system validation
Run the simulation using srun
:
srun -n 6 -c 6 ./Allrun
./Allrun
For further information, refer to the OpenFOAM documentation:
- chemFoam Tutorial
- Tutorial location in OpenFOAM:
$FOAM_TUTORIALS/combustion/chemFoam/nc7h16
This section provides detail steps required for installing and setting up ReFrame on a system using EasyBuild and the Slurm job scheduler. The setup is intended for conducting HPC tests with specific emphasis on OpenFOAM applications.
Load EasyBuild, a software build and installation framework.
module load tools/EasyBuild/4.9.1
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
Check if ReFrame 4.3.3 is available in the module list.
module spider reframe
If ReFrame 4.3.3 is not found, setup environment variables and build it using EasyBuild.
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/r/ReFrame/ReFrame-4.3.3.eb --job --job-cores 64 --job-max-walltime 3 --robot --trace
module load devel/ReFrame/4.3.3
mkdir -p $HOME/ReFrame_test
cd $HOME/ReFrame_test
git clone https://github.com/reframe-hpc/hpc-tests.git
cd hpc-tests/apps/openfoam