LAMMPS compilation issue with pair_allegro

[yifan-henry-cao]

Hi All!

I have been having some issues during the compilation of LAMMPS with pair_allegro. I am not familiar with cmake or C so I am a bit clueless now as of what might be causing the issue. Details below:

I am compiling the packages on a remote server using a GPU. Using the latest version of pair_allegro and the stable release version of LAMMPS (23 Jun 2022). OpenMPI version 4.0.4 and Cuda version 11.0.2. Following the installation instructions I also installed my own copy of libtorch-1.10.1 (cxx11-abi version) and cudnn v8.0.1.13. And I am compiling LAMMPS with libtorch + kokkos.

Here is my script for patching and compiling LAMMPS:

module purge
module load gpu/0.15.4 slurm openmpi/4.0.4 intel-mkl cuda/11.0.2 cmake

./patch_lammps.sh ../lammps_stable/
cd ../lammps_stable/
mkdir build
cd build
cmake ../cmake -DCMAKE_PREFIX_PATH=/home/yifanc/MD_intro/lib/libtorch \
-DPKG_KOKKOS=ON -DKokkos_ENABLE_CUDA=ON \
-DCUDA_TOOLKIT_ROOT_DIR=/home/yifanc/MD_intro/lib/cuda \
-DCUDNN_LIBRARY_PATH=/home/yifanc/MD_intro/lib/cuda/lib64/libcudnn.so \
-DCUDNN_INCLUDE_PATH=/home/yifanc/MD_intro/lib/cuda/include \
-DMKL_INCLUDE_DIR=$INTEL_MKLHOME/mkl/include
make -j$(nproc)

The patching and configuration works fine, but I get an error during the make LAMMPS command with the following error message:

[ 74%] Building CXX object CMakeFiles/lammps.dir/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp.o
/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(83): error: identifier "MPI_COMM_TYPE_SHARED" is undefined

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(84): error: identifier "MPI_INFO_NULL" is undefined

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(83): error: identifier "MPI_Comm_split_type" is undefined

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(125): error: member "LAMMPS_NS::NeighRequest::half"
/home/yifanc/MD_intro/lammps_stable/src/neigh_request.h(55): here is inaccessible

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(126): error: member "LAMMPS_NS::NeighRequest::full"
/home/yifanc/MD_intro/lammps_stable/src/neigh_request.h(56): here is inaccessible

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(128): error: member "LAMMPS_NS::NeighRequest::ghost"
/home/yifanc/MD_intro/lammps_stable/src/neigh_request.h(69): here is inaccessible

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(406): warning: variable "jtype" was declared but never referenced

/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp(318): warning: variable "newton_pair" was declared but never referenced

6 errors detected in the compilation of "/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp".
make[2]: *** [CMakeFiles/lammps.dir/build.make:4281: CMakeFiles/lammps.dir/home/yifanc/MD_intro/lammps_stable/src/pair_allegro.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:856: CMakeFiles/lammps.dir/all] Error 2
make: *** [Makefile:149: all] Error 2

It seems that the pair_allegro.cpp code fails to access some MPI commands that should have been available in my Openmpi package, which ultimately caused this issue during compilation. But I have tried changing the versions of openmpi package loaded and it couldn't resolve the issue. Can someone let me know what I have been doing wrong here? Thanks a ton!

Here is the build configuration in case needed:

-- <<< Build configuration >>>
   LAMMPS Version:   20220623
   Operating System: Linux Rocky Linux 8.7
   CMake Version:    3.18.2
   Build type:       RelWithDebInfo
   Install path:     /home/yifanc/.local
   Generator:        Unix Makefiles using /usr/bin/gmake
-- Enabled packages: KOKKOS
-- <<< Compilers and Flags: >>>
-- C++ Compiler:     /usr/bin/c++
      Type:          GNU
      Version:       8.5.0
      C++ Flags:     -O2 -g -DNDEBUG
      Defines:       LAMMPS_SMALLBIG;LAMMPS_MEMALIGN=64;LAMMPS_OMP_COMPAT=3;LAMMPS_PNG;LAMMPS_GZIP;$<BUILD_INTERFACE:LMP_KOKKOS>
      Options:       -Xcudafe;--diag_suppress=unrecognized_pragma
-- <<< Linker flags: >>>
-- Executable name:  lmp
-- Static library flags:    
-- Kokkos Devices: CUDA;CUDA_LAMBDA;SERIAL
-- Kokkos Architecture: VOLTA70
-- Caffe2: CUDA detected: 11.0
-- Caffe2: CUDA nvcc is: /cm/shared/apps/spack/gpu/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/cuda-11.0.2-fgi2asgykuc5n3mokcrspzkhtracrxok/bin/nvcc
-- Caffe2: CUDA toolkit directory: /home/yifanc/MD_intro/lib/cuda
-- Caffe2: Header version is: 11.0
-- Found cuDNN: v8.0.1  (include: /home/yifanc/MD_intro/lib/cuda/include, library: /home/yifanc/MD_intro/lib/cuda/lib64/libcudnn.so)
-- /cm/shared/apps/spack/gpu/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/cuda-11.0.2-fgi2asgykuc5n3mokcrspzkhtracrxok/lib64/libnvrtc.so shorthash is b31c2d61
-- Autodetected CUDA architecture(s):  7.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70
-- Configuring done
-- Generating done
-- Build files have been written to: /home/yifanc/MD_intro/lammps_stable/build

[yifan-henry-cao]

Hi all,
I was able to resolve the above problems by changing the following:

1. Make sure to configure BUILD_MPI=ON in CMakeCache.txt
2. Change to an older version of LAMMPS (the newer version of LAMMPS changed some variable definitions from public to private, hence the error)

Right now I can compile my LAMMPS executable without problems but once I call my lmp executable I get the following error message:

[exp-8-60:2668514] *** Process received signal ***
[exp-8-60:2668514] Signal: Floating point exception (8)
[exp-8-60:2668514] Signal code: Integer divide-by-zero (1)
[exp-8-60:2668514] Failing at address: 0x5561bb
[exp-8-60:2668514] [ 0] /usr/lib64/libpthread.so.0(+0x12cf0)[0x155553157cf0]
[exp-8-60:2668514] [ 1] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x5561bb]
[exp-8-60:2668514] [ 2] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x5cf4be]
[exp-8-60:2668514] [ 3] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x5b94b4]
[exp-8-60:2668514] [ 4] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x4bf556]
[exp-8-60:2668514] [ 5] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x4c0542]
[exp-8-60:2668514] [ 6] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x45076f]
[exp-8-60:2668514] [ 7] /usr/lib64/libc.so.6(__libc_start_main+0xe5)[0x1554e834ad85]
[exp-8-60:2668514] [ 8] /home/yifanc/MD_intro/lammps_2021/build/lmp[0x46de9e]
[exp-8-60:2668514] *** End of error message ***

Does anyone have any idea on why this would occur? Thanks a lot.

[yifan-henry-cao]

Hi all,
The above floating point error seems to come from a conflict between the system openmpi and the conda openmp packages. I was able to get rid of them by deactivating my conda environment completely.

I was able to compile LAMMPS with pair_allegro using a system CUDA/11.0.2 and different versions of libtorch, but when I call my lmp executable I get some mysterious error message that changes with different libtorch versions:

CUDA/11.0.2 + libtorch_1.11.0_cu102:

terminate called after throwing an instance of 'c10::Error'
what(): isTuple()INTERNAL ASSERT FAILED at "../aten/src/ATen/core/ivalue_inl.h":1916, please report a bug to PyTorch. Expected Tuple but got String
Exception raised from toTupleRef at ../aten/src/ATen/core/ivalue_inl.h:1916 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x6b (0x155551f0d0eb in /home/yifanc/MD_intro/lib/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xce (0x155551f08abe in /home/yifanc/MD_intro/lib/libtorch/lib/libc10.so)
frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x4e (0x155551f0a85e in /home/yifanc/MD_intro/lib/libtorch/lib/libc10.so)

CUDA/11.0.2 + libtorch_1.12.1_cu102:

terminate called after throwing an instance of 'std::runtime_error'
what(): The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
... File "/home/yifanc/.local/lib/python3.8/site-packages/allegro/nn/_allegro.py", line 531, in AD_logsumexp_backward
# Normal (non-residual) update
# index_copy replaces, unlike index_add
latents = torch.index_copy(latents, 0, active_edges, new_latents)
~~~~~~~~~~~~~~~~ <--- HERE
# From the latents, compute the weights for active edges:
RuntimeError: CUDA out of memory. Tried to allocate 6.59 GiB (GPU 0; 31.75 GiB total capacity; 24.12 GiB already allocated; 6.34 GiB free; 24.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Does anyone know if there is a limit on the version of CUDA drivers and libtorch for pair_allegro? Since I am working on a remote cluster I have limited access to the version of CUDA drivers and I am running out of things to try.

[Linux-cpp-lisp]

Do you need to be using CUDA 10? I think PyTorch generally recommends 11.* and is dropping support for 10.

[yifan-henry-cao]

Hi Alby,
I have tried various versions libtorch, since I was using a system CUDA/11.0.2, I have tried libtorch of version cu102, cu110, cu113 etc. I think all of the cu110 versions and above failed during LAMMPS compilation.

However, I was able to look into the above error message for the libtorch_1.12.1_cu102 version of executable. It actually works after I reduce the number of parameters in my potential. I'm not sure whether reducing the batch size of the allegro potential during training would actually decrease its memory requirements when running LAMMPS. But reducing number of parameters surely does.

Best,
Yifan

[Linux-cpp-lisp]

Batch size in training has no impact on inference time memory demands. Running parallel inference over more GPUs will certainly resolve the issue. But if you still get good enough results with a smaller model, by all means use that.