ACCESS-OM3 Deployment on Gadi Killed during esmf build Phase

[CodeGat]

Pull request in question: #5

Within the above pull request, during the deployment of ACCESS-OM3, our jobs seemingly get killed by Gadi during esmfs build phase. Examples of these runs are at the bottom of the PR:

• https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/8793833929/job/24135789008
• https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/8793833929/job/24133442178
• https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/8793833929/job/24132330321

It may be something to do with an Out Of Memory Error being triggered by Gadi during that phase.

Potential Solutions

• Find a way to parallelize esmf build phase?

Pinging @micaeljtoliveira - have you had any experience with an issue like this when building ACCESS-OM3 on Gadi?

[micaeljtoliveira]

@CodeGat Where is the compilation happening? On a login or a compute node?

Find a way to parallelize esmf build phase?

By default spack does parallelize builds. What it does not by default is to parallelize over builds, but that's a bit more tricky and not relevant here.

[CodeGat]

The compilation is happening on a login node

[micaeljtoliveira]

There are several limits enforced on the login nodes regarding memory usage and CPU time. I strongly recommend you move the builds to a compute node. That will require submitting a job though...

[aidanheerdegen]

That will require submitting a job though...

And attendant wait in the queue for every build job. Would really like to avoid that if possible.

[CodeGat]

Have confirmed that, for install of access-om3 with spack -d install --fresh on Gadi:

• --jobs 8 fails on esmf build stage
• --jobs 4 succeeds

Will need to test how long it takes to install something like om2 with --jobs 4 vs not specifying --jobs.

[CodeGat]

Just to finish up on this conversation thread, it's now a per-model vars.SPACK_INSTALL_PARALLEL_JOBS rather than an explicit, global --jobs 4.