Compilation Issues on Satori System

[kent0]

I'm running into compilation issues when trying to setup nekRS (v21.1) on Satori which is a Power9 based cluster with V100 accelerators.

The first issue I found was a CUDA_CUDA_LIBRARY not found when running ./nrsconfig so I've addressed the issue with this fix. Afterwards, ./nrsconfig and the build process finished without any issues.

I am now attempting to compile the channel case with a modified nrspre script:

#!/bin/bash

date
module list

export NEKRS_HOME=${NEKRS_HOME:="`dirname "$0"`/../"}

if [ $# -eq 0 ] || [ $# -ne 2 ] || [ "$1" == "-h" ] || [ "$1" == "-help" ]; then
  echo "usage: ${0##*/} <casename> <#target procs>"
  exit 0
fi

srun --gres=gpu:1 -N 1 -n 1 -t 00:30:00 $NEKRS_HOME/bin/nekrs --setup $1 --build-only $2

date

and got the following output:

[kaneko@satori-login-001 channel]$ ./nrspre channel 1
Fri May  6 09:59:36 EDT 2022

Currently Loaded Modules:
  1) spack/0.1                  10) hwloc/1.9-cuda-b7o45dk
  2) StdEnv                     11) slurm/20-02-0-1-7ddtsaq
  3) cuda/11.4-5ncz6p4          12) libnl/3.3.0-dvsvfmt
  4) libpciaccess/0.16-h54mcak  13) rdma-core/32.0-mpwfuo4
  5) libiconv/1.16-5sppl6f      14) ucx/1.9.0-55pjrj2
  6) xz/5.2.5-bm6hlup           15) openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron
  7) zlib/1.2.11-42psvbb        16) ncurses/6.2-z3ol2m3
  8) libxml2/2.9.10-ouyxloi     17) openssl/1.1.1h-6l5r4xg
  9) numactl/2.0.14-gkb2ckj     18) cmake/3.18.4-i5o6gfe

 

                 __    ____  _____
   ____   ___   / /__ / __ \/ ___/
  / __ \ / _ \ / //_// /_/ /\__ \ 
 / / / //  __// ,<  / _, _/___/ / 
/_/ /_/ \___//_/|_|/_/ |_|/____/  v21.1 (no sha)

COPYRIGHT (c) 2019-2021 UCHICAGO ARGONNE, LLC

MPI tasks: 1

reading par file ...

using NEKRS_HOME: /home/kaneko/.local/nekrs
using NEKRS_CACHE_DIR: /home/kaneko/.local/nekrs/examples/channel/.cache
using OCCA_CACHE_DIR: /home/kaneko/.local/nekrs/examples/channel/.cache/occa/

jit-compiling for >=1 MPI tasks ...

Initializing device 
active occa mode: CUDA

building udf ... 
[100%] Built target UDF
done (0.626219s)
terminate called after throwing an instance of 'occa::exception'
  what():  
---[ Error ]--------------------------------------------------------------------
    File     : /home/kaneko/Developer/nekRS-21.1/3rd_party/occa/src/occa/internal/modes/cuda/device.cpp
    Line     : 373
    Function : buildOKLKernelFromBinary
    Message  : Kernel [myDummyKernelName]: Loading Module
               CUDA Error [ 222 ]: UNKNOWN ERROR
    Stack
      14 occa::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      13 occa::cuda::error(cudaError_enum, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
      12 occa::cuda::device::buildOKLKernelFromBinary(occa::hash_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::lang::sourceMetadata_t&, occa::lang::sourceMetadata_t&, occa::json const&, occa::io::lock_t)
      11 occa::cuda::device::buildKernelFromProcessedSource(occa::hash_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, occa::lang::sourceMetadata_t&, occa::lang::sourceMetadata_t&, occa::json const&, occa::io::lock_t)
      10 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::hash_t, bool, occa::json const&)
       9 occa::launchedModeDevice_t::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::hash_t, occa::json const&)
       8 occa::device::buildKernel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::json const&) const
       7 occa::device::buildKernelFromString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, occa::json const&) const
       6 /home/kaneko/.local/nekrs/lib/libnekrs.so(+0x17d988)
       5 compileKernels()
       4 nekrs::setup(ompi_communicator_t*, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
       3 /home/kaneko/.local/nekrs/bin/nekrs()
       2 /usr/lib64/power9/libc.so.6(+0x24078)
       1 /usr/lib64/power9/libc.so.6(__libc_start_main+0xb4)
================================================================================

[node0011:3805926] *** Process received signal ***
[node0011:3805926] Signal: Aborted (6)
[node0011:3805926] Signal code:  (-6)
[node0011:3805926] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000504d8]
[node0011:3805926] [ 1] /usr/lib64/power9/libc.so.6(gsignal+0xd8)[0x200002bf35f8]
[node0011:3805926] [ 2] /usr/lib64/power9/libc.so.6(abort+0x164)[0x200002bd3a2c]
[node0011:3805926] [ 3] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200002804a98]
[node0011:3805926] [ 4] /usr/lib64/libstdc++.so.6(+0xaf6d4)[0x2000027ff6d4]
[node0011:3805926] [ 5] /usr/lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x2000027ff790]
[node0011:3805926] [ 6] /usr/lib64/libstdc++.so.6(__cxa_throw+0x7c)[0x2000027ffc6c]
[node0011:3805926] [ 7] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa8_messageERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbS7_S7_iS7_+0xbc)[0x200000b8b9ec]
[node0011:3805926] [ 8] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa5errorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_iS7_+0x88)[0x200000b8bc18]
[node0011:3805926] [ 9] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa4cuda5errorE14cudaError_enumRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_iS9_+0x160)[0x200000b09b5c]
[node0011:3805926] [10] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6device24buildOKLKernelFromBinaryENS_6hash_tERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_RNS_4lang16sourceMetadata_tESD_RKNS_4jsonENS_2io6lock_tE+0x224)[0x200000afea18]
[node0011:3805926] [11] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa4cuda6device30buildKernelFromProcessedSourceENS_6hash_tERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_SA_SA_bRNS_4lang16sourceMetadata_tESD_RKNS_4jsonENS_2io6lock_tE+0xd8)[0x200000afd3a4]
[node0011:3805926] [12] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa20launchedModeDevice_t11buildKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_NS_6hash_tEbRKNS_4jsonE+0x888)[0x200000998b6c]
[node0011:3805926] [13] /home/kaneko/.local/nekrs/lib/libocca.so(_ZN4occa20launchedModeDevice_t11buildKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_NS_6hash_tERKNS_4jsonE+0x98)[0x2000009981b0]
[node0011:3805926] [14] /home/kaneko/.local/nekrs/lib/libocca.so(_ZNK4occa6device11buildKernelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_RKNS_4jsonE+0x1a4)[0x2000009169a4]
[node0011:3805926] [15] /home/kaneko/.local/nekrs/lib/libocca.so(_ZNK4occa6device21buildKernelFromStringERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_RKNS_4jsonE+0x1d8)[0x200000916d2c]
[node0011:3805926] [16] /home/kaneko/.local/nekrs/lib/libnekrs.so(+0x17d988)[0x2000001ed988]
[node0011:3805926] [17] /home/kaneko/.local/nekrs/lib/libnekrs.so(_Z14compileKernelsv+0x2c)[0x2000001edabc]
[node0011:3805926] [18] /home/kaneko/.local/nekrs/lib/libnekrs.so(_ZN5nekrs5setupEP19ompi_communicator_tiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_S7_S7_+0x10c8)[0x20000019edac]
[node0011:3805926] [19] /home/kaneko/.local/nekrs/bin/nekrs[0x100035e8]
[node0011:3805926] [20] /usr/lib64/power9/libc.so.6(+0x24078)[0x200002bd4078]
[node0011:3805926] [21] /usr/lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x200002bd4264]
[node0011:3805926] *** End of error message ***
srun: error: node0011: task 0: Aborted (core dumped)
Fri May  6 09:59:43 EDT 2022

I'd appreciate any feedback / directions that address this issue.

[stgeke]

The CUDA runtime suggest that your CUDA runtime you're using is newer then the installed CUDA driver.

[stgeke]

Are you build nekRS on a compute or login node?

[kent0]

I built nekRS on a login node so I'll see if re-building it on a compute node will fix the issue.

One potential issue I see is that I'm running module load openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron which loads an GPU-aware openmpi module automatically loads CUDA 11.4 . Perhaps if compile with earlier versions of CUDA, that may fix the issue so I'll try that as well.

[stgeke]

I recommend building the code on the compute node.

[kent0]

I was not able to compile on a compute node due to restrictions on network access. I've reached out to support on this issue.