Skip to content

Commit

Permalink
Merge branch 'staging' of github.com:uvarc/rc-learning into staging
Browse files Browse the repository at this point in the history
  • Loading branch information
Ahmad Sheikhzada committed Jan 26, 2024
2 parents 66950d2 + afd232f commit 9b3258f
Showing 1 changed file with 17 additions and 16 deletions.
33 changes: 17 additions & 16 deletions content/courses/containers-for-hpc/building-apptainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,10 @@ Inspect the runscript before running an image!

1. (Optional) Cache

The default cache directory is `~/.apptainer`. If you are an active container user it can quickly fill up your home. You can define it to your scratch:
The default cache directory is `~/.apptainer`. If you are an active container user it can quickly fill up your home. You can change it to scratch:
{{< code-snippet >}}export APPTAINER_CACHEDIR=/scratch/$USER/.apptainer{{< /code-snippet >}}
or remember to clean up periodically.
1. We have suppressed non-error output from the `apptainer` command. To see the complete output, type `\apptainer`.
Otherwise, remember to clean up periodically.
1. We have suppressed certain output from the `apptainer` command. To see the complete output, type `\apptainer`.
1. Load the Apptainer module: `module load apptainer`

## Definition File
Expand All @@ -74,7 +74,8 @@ The definition file is a set of instructions that is used to build an Apptainer

This is a skeleton:

{{< code-snippet >}}Bootstrap: ... # "Header"
```bash
Bootstrap: ... # "Header"
From: ... #

%files # "Section"
Expand All @@ -94,7 +95,7 @@ From: ... #

%help
...
{{< /code-snippet >}}
```

### Header

Expand Down Expand Up @@ -130,15 +131,16 @@ Copy files into the container.
...
```

- always copied before the `%post` section
- Files are always copied before the `%post` section.

#### `%post`

Installation commands. Example:

```
%post
apt-get update && apt-get -y install lolcat
apt-get update
apt-get install -y lolcat
```

#### `%environment`
Expand Down Expand Up @@ -196,7 +198,7 @@ From: ubuntu:22.04
### Steps 2 & 3: Install software

{{< info >}}
The package manager will take care of the dependencies for us.
For this application the package manager will take care of all the dependencies.
{{< /info >}}

In `%post` specify the actual commands to be executed (as if you were to type them on the command line).
Expand Down Expand Up @@ -232,7 +234,7 @@ From: ubuntu:22.04

This finally works.

```
```bash
$ apptainer build lolcow.sif lolcow.def
$ apptainer run lolcow.sif
```
Expand Down Expand Up @@ -283,13 +285,13 @@ From: ubuntu:22.04

Save this as `lolcow_0.def`, which will be the basis for comparison.

## 2 Best Practices
## Two Best Practices

While our container is functional, there is room for improvement. We shall look at some important best practices.

### 1. Clean up

Almost all package managers leave behind some cache files after installation that can be safely removed. Dependending on your application, they can easily accumulate up to several GBs. Let's see what happens if we try to clean up the cache in a separate `RUN` statement.
Package managers usually leave behind some cache files after installation that can be safely removed. Dependending on your application, they can easily accumulate up to several GBs.

{{< code-snippet >}}Bootstrap: docker
From: ubuntu:22.04
Expand Down Expand Up @@ -478,10 +480,10 @@ While you can create a conda environment locally, you cannot directly migrate it
Your project requires PyTorch 2.1.2, Numpy, Seaborn, and Pandas. Write the corresponding Apptainer definition file.
Hints:
- Is this version already installed on the cluster?
- Find the appropriate base image from [here](https://hub.docker.com/r/pytorch/pytorch/tags).
- Pull the base image and examine it first. Does it already provide some of the packages?
- Apply version pinning.
{{< info >}}While PyTorch runs on a GPU, you do not need to build the container on a GPU.{{< /info >}}
{{< info >}}You will likely run out of memory when building large containers (over a few GBs). Request an [interactive job](https://www.rc.virginia.edu/userinfo/rivanna/slurm/#submitting-an-interactive-job) to build on a compute node in the `largemem` partition.{{< /info >}}
Expand All @@ -493,14 +495,13 @@ Rocker provides many base images for all R versions (see [here](https://rocker-p
- `rocker/tidyverse`: plus tidyverse and dependencies
- `rocker/shiny`: shiny server
If you want to build a custom R container start with one of the Rocker images. Building R, RStudio Server, etc. from source can be very tedious!
{{< info >}}If you want to build a custom R container start with one of the Rocker images. Building R, RStudio Server, etc. from source can be very tedious!{{< /info >}}
#### Exercise
Your project requires R 4.3.2, dplyr, ggplot2, and RcppGSL. Write the corresponding Apptainer definition file.
Hints:
- Is this version already installed on the cluster?
- Find the appropriate base image.
- Pick an appropriate base image - there are two viable choices here.
- Pull the base image and examine it first. Does it already provide some of the packages?
- In the definition file, install CRAN packages via `R -e "install.packages('...')"`.
- Load the three packages. Do they all succeed? If not, how can you fix it?
Expand Down

0 comments on commit 9b3258f

Please sign in to comment.