Skip to content

parsavares/HPCSE_End_Project

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Installation and Regression Testing of OpenFOAM

image

OpenFOAM is a free, open source computational fluid dynamics (CFD) software package released by the OpenFOAM Foundation. It has a large user base across most areas of engineering and science, from both commercial and academic organisations. OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics. OpenFOAM is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems, most prominently including computational fluid dynamics (CFD).

Table of Contents

Authors:

Parsa VARES

Nahid JAVADINARAB

Introduction

In this project, we're gonna supposed to install OpenFOAM-11 and OpenFOAM-v2312 using EasyBuild 4.9.1 (the latest version).

Scope It is used in research organizations, academic institutes and across many types of industries, for example, automotive, manufacturing, process engineering and environmental engineering.

Purpose The goals of this project are:

  1. To install recent versions of OpenFOAM on the Uni.lu HPC platform using EasyBuild.
  2. To set up a regression testing configuration for the installed OpenFOAM using ReFrame.

I. Installing latest release of EasyBuild

  1. Connection to a Compute Node Allocating an interactive job with a high priority for 2 hours, using 1 node, 1 task, and 32 CPU cores.
  • Command:
    salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64
  1. Installing the latest release of EasyBuild, a software build and installation framework.
  • Command:
    eb --install-latest-eb-release
  1. Looking for available EasyBuild modules and their details.
  • Command:
    module spider easybuild
  1. Now, by loading EasyBuild version 4.9.1 module, making its commands and environment available.
  • Command:
    module load tools/EasyBuild/4.9.1

5.The following command Adds the local directory of EasyBuild module files to the module path.

  • Command:
    module use ~/.local/easyconfig/modules/all
  1. Running this command Adds a project-specific directory of EasyBuild module files to the module path.
  • Command:
    module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/

Submitting installations as Slurm jobs

EasyBuild can submit jobs to different backends including Slurm to install software, to distribute the often time-consuming installation of a set of software applications and the dependencies they require to a cluster.

This is done via the --job command line option.

It is important to be aware of some details before you start using this, which we'll cover here.

  1. It Configures EasyBuild to use Slurm as the backend job scheduler for build jobs. The default job backend in EasyBuild v4.x is GC3Pie. To let EasyBuild submit jobs to Slurm instead, you should set the job-backend configuration setting to Slurm, for example by setting the corresponding environment variable:
  • Command:
    export EASYBUILD_JOB_BACKEND='Slurm'
  1. This command Sets the base directory for EasyBuild installations and builds to the user's home directory.
  • Command:
    export EASYBUILD_PREFIX=$HOME/easybuild
  1. We Set the build path for EasyBuild to the shared memory directory, which can improve build performance.
  • Command:
    export EASYBUILD_BUILDPATH=/dev/shm/$USER
  1. This command Checks for and lists any missing dependencies required for building OpenFOAM.
  • Command:
    eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --missing
  1. This command submits the OpenFOAM build as a job with 32 cores, a maximum wall time of 11 hours, resolving dependencies automatically, and tracing the build process.
    • by using sq you can see it running
    • after it finished OpenFOAM .out can be found at /home/users/$USER
    • the right output uploaded here you can check it. when you are opening the .out file, at the end you should run:
  • Command:
    eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --job --job-cores 32 --job-max-walltime 11 --robot --trace

By default, EasyBuild will submit single-core jobs requesting for 24 hours of walltime. You can tweak the requested resources via the job-cores and job-max-walltime configuration options. Note that not all job-* configuration settings apply to all job backends, see the EasyBuild documentation for more details.

Combining --job and --robot

If one or more dependencies are still missing for the software you want to install, you can combine --job and --robot to get EasyBuild to submit a separate job for each of the installations. These jobs will not --robot, they will each only perform a single installation.

Dependencies between jobs will be "registered" at submission time, so Slurm will put jobs on hold until the jobs that install the required (build) dependencies have completed successfully, and cancel jobs if the job to install a dependency failed for some reason. == COMPLETED: Installation ended successfully (took 1 hour 3 mins 41 secs)

== Results of the build can be found in the log file(s) /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/eas$

== Build succeeded for 1 out of 1

== Temporary log file(s) /tmp/eb-j7o7pzoh/eb-txnweqte/easybuild-qnzsrnfi.log* have been removed.

== Temporary directory /tmp/eb-j7o7pzoh/eb-txnweqte has been removed.

  1. It adds the user's EasyBuild module files directory to the module path.
  • Command:
    module use /home/users/$USER/easybuild/modules/all
  1. loading OpenFOAM v2312 module built with the foss-2023a toolchain, making its commands and environment available.
  • Command:
    module load cae/OpenFOAM/v2312-foss-2023a

13.1 unload previous installed module to install the other OpenFOAM version.

  • Command:
    module unload cae/OpenFOAM/v2312-foss-2023a
  1. Now we repeat process again from the step 10th for installing OpenFOAM-11.
    • Important: don't forget to do: 1. export EASYBUILD_JOB_BACKEND='Slurm' 2. export EASYBUILD_PREFIX=$HOME/easybuild 3. export EASYBUILD_BUILDPATH=/dev/shm/$USER before doing this line.
  • Command:
    eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-11-foss-2023a.eb --job --job-cores 64 --job-max-walltime 3 --robot --trace
  1. Loading OpenFOAM 11 module built with the foss-2023a toolchain, making its commands and environment available.
  • Command:
    module load cae/OpenFOAM/11-foss-2023a

** II: Running Test Examples**

Test Example 1: Running the OpenFOAM-v2312 motorBike Tutorial on HPC

  1. Download the Tutorial Files

    • Command:
      wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/incompressible/simpleFoam/motorBike -O openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip
  2. Create the Directory Structure

    • Command:
      mkdir -p $HOME/OpenFOAM
  3. Upload the File to HPC Description: Use SCP or a similar tool to upload openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip to $HOME/OpenFOAM on the HPC.

    • Command:
    scp openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip $HOME/OpenFOAM
  4. Unzip the Tutorial Files

    • Command:
      unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike.zip -d $HOME/OpenFOAM
  5. Navigate to the Tutorial Directory

    • Command:
      cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam-motorBike/tutorials/incompressible/simpleFoam/motorBike
  6. Source the OpenFOAM Environment Setup Script

    • Command:
      source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
  7. Verify the Environment Variable

    • Command:
      echo $WM_PROJECT_DIR
  8. Make the Allrun Script Executable

    • Command:
      chmod +x Allrun
  9. Run the Allrun Script Using srun

    • Command:
      srun -n 6 -c 6 ./Allrun
  10. Run the Allrun Script Directly (Optional)

    • Command:
      ./Allrun

9y8wrzf8

Test Example 2: Testing turbulenceModels from planeChannel

  1. Loading required modules
module load tools/EasyBuild/4.9.1
module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
  1. installing(optional if needed)
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2312-foss-2023a.eb --job --job-cores 32 --job-max-walltime 11 --robot --trace
module use /home/users/$USER/easybuild/modules/all
module load cae/OpenFOAM/v2312-foss-2023a
  1. Now, we're gonna supposed to download tutorials of openfoam v2312, then upload them in openfoam directory that we have created for motorbike example:
    wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?     path=tutorials/incompressible/simpleFoam -O openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip
    mkdir -p $HOME/OpenFOAM
    scp openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip $HOME/OpenFOAM
    unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-incompressible-simpleFoam.zip -d $HOME/OpenFOAM
    cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312/tutorials/verificationAndValidation/turbulenceModels/planeChannel

Source the OpenFOAM Environment Setup

source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
  1. Verifying the Environment Variable
echo $WM_PROJECT_DIR

image

  1. Execution and running
chmod +x Allrun
srun -n 6 -c 6 ./Allrun
./Allrun

image image

  1. Now, after running procedure it is possible to generate plots
./plot

open the plots directory

cd plots

image

  1. we see that all of .png files are saved in plots directory. (check ValidationVerfication folder for plots)

Test Example 3: Test atmDownstreamDevelopment from atmosphericModels

image

cd /home/users/$USER/OpenFOAM/openfoam-OpenFOAM-v2312/tutorials/verificationAndValidation/atmosphericModels/atmDownstreamDevelopment/
 

image

chmod +x Allrun
srun -n 1 -c 32 ./Allrun
./Allrun

By running ./plot we generate results of the test.

image

Now, a subdirectory of plots has been generated in this directory.

image

cd kOmegaSST/
cd ..
cd kEpsilon/

image

Test Example 4: ChemFoam-gri Tutorial Setup

Pre-settings

Resource Allocation

Allocate resources on the HPC:

salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64

Load Environment Modules

Set up the necessary environment modules:

module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3

Setup

Download the Tutorial Files

Download the chemFoam tutorial files:

wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/gri -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip

Create Directory Structure

Create the directory structure in your home directory:

mkdir -p $HOME/OpenFOAM

Upload File to HPC

Upload the downloaded zip file to the HPC:

scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip $HOME/OpenFOAM

Unzip the Tutorial Files

Unzip the tutorial files:

unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri.zip -d $HOME/OpenFOAM

Navigate to Tutorial Directory

Navigate to the tutorial directory:

cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-gri/tutorials/combustion/chemFoam/gri

Source OpenFOAM Environment

Source the OpenFOAM environment:

source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR

Set Execute Permissions and List Files

Set execute permissions for the Allrun script and list the directory contents:

chmod +x Allrun
ls

Expected output:

Allclean  Allrun  chemkin  constant  system  validation

Run the Simulation

Run the simulation using srun:

srun -n 6 -c 6 ./Allrun
./Allrun

333889934-183427e5-80a2-4ed8-bf9a-e158aa776603

Resources

For further information, refer to the OpenFOAM documentation:

Test Example 5: ChemFoam h2 Tutorial Setup

Pre-settings

Resource Allocation

Allocate resources on the HPC:

salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64

Load Environment Modules

Set up the necessary environment modules:

module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3

Setup

Download the Tutorial Files

Download the chemFoam tutorial files for the h2 case:

wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/h2 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip

Create Directory Structure

Create the directory structure in your home directory:

mkdir -p $HOME/OpenFOAM

Upload File to HPC

Upload the downloaded zip file to the HPC:

scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip $HOME/OpenFOAM

Unzip the Tutorial Files

Unzip the tutorial files:

unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2.zip -d $HOME/OpenFOAM

Navigate to Tutorial Directory

Navigate to the tutorial directory:

cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-h2/tutorials/combustion/chemFoam/h2

Source OpenFOAM Environment

Source the OpenFOAM environment:

source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR

Set Execute Permissions and List Files

Set execute permissions for the Allrun script and list the directory contents:

chmod +x Allrun
ls

Expected output:

Allclean  Allrun  chemkin  constant  system  validation

Run the Simulation

Run the simulation using srun:

srun -n 6 -c 6 ./Allrun
./Allrun

6

Resources

For further information, refer to the OpenFOAM documentation:

  • chemFoam Tutorial
  • Tutorial location in OpenFOAM: $FOAM_TUTORIALS/combustion/chemFoam/h2

Test Example 6: ChemFoam ic8h18 Tutorial Setup

Pre-settings

Resource Allocation

Allocate resources on the HPC:

salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64

Load Environment Modules

Set up the necessary environment modules:

module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3

Setup

Download the Tutorial Files

Download the chemFoam tutorial files for the ic8h18 case:

wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/ic8h18 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip

Create Directory Structure

Create the directory structure in your home directory:

mkdir -p $HOME/OpenFOAM

Upload File to HPC

Upload the downloaded zip file to the HPC:

scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip $HOME/OpenFOAM

Unzip the Tutorial Files

Unzip the tutorial files:

unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18.zip -d $HOME/OpenFOAM

Navigate to Tutorial Directory

Navigate to the tutorial directory:

cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-ic8h18/tutorials/combustion/chemFoam/ic8h18

Source OpenFOAM Environment

Source the OpenFOAM environment:

source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR

Set Execute Permissions and List Files

Set execute permissions for the Allrun script and list the directory contents:

chmod +x Allrun
ls

Expected output:

Allclean  Allrun  chemkin  constant  system  validation

Run the Simulation

Run the simulation using srun:

srun -n 6 -c 6 ./Allrun
./Allrun

Resources

For further information, refer to the OpenFOAM documentation:

  • chemFoam Tutorial
  • Tutorial location in OpenFOAM: $FOAM_TUTORIALS/combustion/chemFoam/ic8h18

Test Example 7: ChemFoam nc7h16 Tutorial Setup

Pre-settings

Resource Allocation

Allocate resources on the HPC:

salloc -p interactive --qos debug --time=2:00:00 -N 1 -n 1 -c 64

Load Environment Modules

Set up the necessary environment modules:

module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all
export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
module load tools/EasyBuild/4.9.1
module load cae/OpenFOAM/v2312-foss-2023a
module load devel/ReFrame/4.3.3

Setup

Download the Tutorial Files

Download the chemFoam tutorial files for the nc7h16 case:

wget https://develop.openfoam.com/Development/openfoam/-/archive/OpenFOAM-v2312/openfoam-OpenFOAM-v2312.zip?path=tutorials/combustion/chemFoam/nc7h16 -O openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip

Create Directory Structure

Create the directory structure in your home directory:

mkdir -p $HOME/OpenFOAM

Upload File to HPC

Upload the downloaded zip file to the HPC:

scp openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip $HOME/OpenFOAM

Unzip the Tutorial Files

Unzip the tutorial files:

unzip $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16.zip -d $HOME/OpenFOAM

Navigate to Tutorial Directory

Navigate to the tutorial directory:

cd $HOME/OpenFOAM/openfoam-OpenFOAM-v2312-tutorials-combustion-chemFoam-nc7h16/tutorials/combustion/chemFoam/nc7h16

Source OpenFOAM Environment

Source the OpenFOAM environment:

source /home/users/$USER/easybuild/software/OpenFOAM/v2312-foss-2023a/OpenFOAM-v2312/etc/bashrc
echo $WM_PROJECT_DIR

Set Execute Permissions and List Files

Set execute permissions for the Allrun script and list the directory contents:

chmod +x Allrun
ls

Expected output:

Allclean  Allrun  chemkin  constant  system  validation

Run the Simulation

Run the simulation using srun:

srun -n 6 -c 6 ./Allrun
./Allrun

Resources

For further information, refer to the OpenFOAM documentation:

  • chemFoam Tutorial
  • Tutorial location in OpenFOAM: $FOAM_TUTORIALS/combustion/chemFoam/nc7h16


III. ReFrame Installation and Setup Guide

Introduction

This section provides detail steps required for installing and setting up ReFrame on a system using EasyBuild and the Slurm job scheduler. The setup is intended for conducting HPC tests with specific emphasis on OpenFOAM applications.

Step 1: Load EasyBuild Module

Load EasyBuild, a software build and installation framework.

module load tools/EasyBuild/4.9.1

Step 2: Configure Module Paths

module use ~/.local/easyconfig/modules/all
module use /work/projects/mhpc-softenv/easybuild/aion-epyc-prod-2023a/modules/all/
module use /home/users/$USER/easybuild/modules/all

Step 3: Check ReFrame Version

Check if ReFrame 4.3.3 is available in the module list.

module spider reframe

Step 4: Environment Setup for Building ReFrame

If ReFrame 4.3.3 is not found, setup environment variables and build it using EasyBuild.

export EASYBUILD_JOB_BACKEND='Slurm'
export EASYBUILD_PREFIX=$HOME/easybuild
export EASYBUILD_BUILDPATH=/dev/shm/$USER
eb /home/users/$USER/.local/easybuild/software/EasyBuild/4.9.1/easybuild/easyconfigs/r/ReFrame/ReFrame-4.3.3.eb --job --job-cores 64 --job-max-walltime 3 --robot --trace

Step 5: Load ReFrame

module load devel/ReFrame/4.3.3
mkdir -p $HOME/ReFrame_test
cd $HOME/ReFrame_test
git clone https://github.com/reframe-hpc/hpc-tests.git
cd hpc-tests/apps/openfoam

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages