ufs-weather-model build on MacBook M3 Sonoma 14.5

[barlage]

I'm attempting to build ufs-weather-model on a MacBook M3 Sonoma 14.5 using spack-stack.

I have successfully built spack-stack 1.7.0 using the ufs-weather-model template, though as a side note, the current build of 1.7.0 will not work with current ufs-weather-model develop due to the UFS dependency on the sp library, which will be removed in a current spack-stack build that uses the ip v5 library. I got around this temporarily by pinning the ip library version to 4.3.0 (same IP version as spack-stack 1.6.0). This at least allows the UFS build to start.

Bottom line is I get an error in the final link that looks like this:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

My module loading and environment setup prior to building:

module load stack-apple-clang/15.0.0 stack-openmpi/5.0.1 ufs-weather-model-env/1.0.0
source modulefiles/ufs_macosx.gnu
export CMAKE_FLAGS="-DAPP=ATM -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8"
./build.sh

if I build without 32BIT=ON, I get a similar error:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r8.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r8.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r8.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r8.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r8.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

Just wondering if anyone out there is using Mac/ARM with spack-stack and running into a similar problem. Or has anyone with Mac M chips been able to build ufs-weather-model?

[barlage]

I now see that OPENMP is forced OFF for Mac builds. However, the spack-stack seems to build with openmp for FMS. Is this the issue?

[barlage]

Some additional tests:

Removing the force of OPENMP OFF for clang builds in configure_macosx.gnu.cmake, same error:

Undefined symbols for architecture arm64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture arm64

Attempt to remove openmp from fms build of spack-stack:

CMake Error at /packages/spack-stack/envs/ufs-plus_noopenmp2/install/apple-clang/15.0.0/crtm-2.4.0.1-jhrjxvw/lib/cmake/crtm/crtm-targets.cmake:61 (set_target_properties):
  The link interface of target "crtm::crtm" contains:

    OpenMP::OpenMP_Fortran

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  /packages/spack-stack/envs/ufs-plus_noopenmp2/install/apple-clang/15.0.0/crtm-2.4.0.1-jhrjxvw/lib/cmake/crtm/crtm-config.cmake:30 (include)
  FV3/upp/CMakeLists.txt:51 (find_package)


-- Generating done (0.1s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

A few additional tests of trying to remove openmp from spack-stack always results in CRTM cmake failure, presumably because there is no way to build CRTM without openmp in spack-stack (?).

Additional test on MacBook Pro intel chip with Ventura 13.6.7:

• successfully built spack-stack 1.7.0 (with IP pinned to 4.3.0 as mentioned above)
• UWM build using the same configuration as my original test results in the same FMS error referenced in the first post above

Undefined symbols for architecture x86_64:
  "___kmpc_for_static_fini", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_for_static_init_4", referenced from:
      _.omp_outlined..1 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..3 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..6 in libfms_r4.a(create_xgrid.c.o)
      _.omp_outlined..7 in libfms_r4.a(create_xgrid.c.o)
  "___kmpc_fork_call", referenced from:
      _create_xgrid_2dx2d_order1 in libfms_r4.a(create_xgrid.c.o)
      _create_xgrid_2dx2d_order2 in libfms_r4.a(create_xgrid.c.o)
ld: symbol(s) not found for architecture x86_64

[gspetro-NOAA]

Hi Mike!

I forwarded your question yesterday to the team that usually helps troubleshoot spack-stack dependencies. @natalie-perlin mentioned that she is working on building spack-stack (v1.6.0) on Mac but hasn't succeeded yet. Most of our other team members are on Linux systems, I believe. I will see if I can get any more information from them, but several people are either on PTO or about to travel to UIFCW, so there may be a bit of delay. Didn't want you to think we were ignoring the question though!

Best,
Gillian

[climbfuji]

This will be fixed in FMS with an update to fms@2024.01 or newer. Until then, you need to force linking against libomp.dylib on macOS because FMS doesn't declare its FMS C dependency correctly. Use -DCMAKE_SHARED_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" and the same for CMAKE_EXE_LINKER_FLAGS. See also https://github.com/JCSDA/ufs-bundle/blob/develop/README.md.

[barlage]

Using the suggestion from @climbfuji I was able to compile with this command:

export CMAKE_FLAGS="-DCMAKE_SHARED_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" -DCMAKE_EXE_LINKER_FLAGS="${llvm_openmp_ROOT}/lib/libomp.dylib" -DAPP=ATM -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8"
./build.sh

[barlage]

After the successful compile, the model still doesn't not run. I have confirmed with another Mac user who gets the same error (I believe using spack-stack 1.6.0 on an M2):

* . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .
     PROGRAM ufs-weather-model HAS BEGUN. COMPILED       0.00     ORG: np23
     STARTING DATE-TIME  JUL 22,2024  17:19:27.024  204  MON   2460514


MPI Library = Open MPI v5.0.1, package: Open MPI  Distribution, ident: 5.0.1, repo rev: v5.0.1, Dec 20, 2023
MPI Version = 3.1
libc++abi: terminating due to uncaught exception of type nlohmann::json_abi_v3_11_2::detail::out_of_range: [json.exception.out_of_range.403] key 'NUOPC' not found

Anyone seeing or seen something similar or have any suggestions?

I'm attempting to run an approximation of the control_p8 regression test setup.

[gspetro-NOAA]

@climbfuji Do you happen to have any suggestions for Mike?

[climbfuji]

Talk to the ESMF team. This looks very much like the bug I filed 3+ years ago that the UFS crashes immediately in ESMF on macOS with apple-clang ...

[climbfuji]

The good news is that your crash report at least tells us what the problem is (mine never did)

[gspetro-NOAA]

@barlage Were you able to get in touch with the ESMF folks to work on the issue, or do you need more assistance from us?

[uturuncoglu]

@barlage There is no need to be a super ESMF user. Please open a discussion in ESMf side and add all the information that you could get. Definitely, output of the make info would help. If you could also run the unit tests using make all_tests in your installation, that would help to track down the issue. We also discuss it on ESMF Core call yesterday and we need to have more information to see what is going on. @climbfuji Once @barlage open a discussion, please fell free to add any comment that you think it would be helpful. We have couple of people in the team that has ESMF build on Mac without any issue at this point.

[barlage]

@uturuncoglu maybe "power user" is a bit much, but for example, to do what you asked, I believe I have to rebuild spack-stack and prevent cleaning the build (I currently don't know how to do this) because I don't see where I can run make info or make all_tests in the current installation. I can follow the (excellent) instructions for building spack-stack on Mac but it will take me some time if I need to go off script.

[uturuncoglu]

@barlage Yes, you are right since you are using spack-stack, it would be hard to run tests. I think you could still access to make info information without building and running anything. Just find the path that ESMf installed and there is a .spack/ directory in it. I think spack-build-out.txt file has the make info information. spack-build-env.txt will also help. @climbfuji @theurich I am not sure spack allow it but it would be nice to add variant to esmf spack package to run make all_tests command as a part of installation. Of course this could be issue if the installation node does not allow to run MPI jobs in the compute node. So, maybe we could only run part of it.

[climbfuji]

spack build-env esmf ... would be one way to build and run the tests manually. We should look into running the tests as part of the spack installs (there's a ticket open in spack-stack to expand unit testing of packages during the installation).

To be clear, there is no issue building ESMF on the mac, the problem is launching the ufs-weather-model executable. I vaguely remember that the problem only occurs when using apple-clang, not with gcc - but I am not sure. The issue with that is that it is getting increasingly difficult to build the entire software stack on the mac with gcc.

[gspetro-NOAA]

@barlage I think @natalie-perlin has worked on a Mac with the SRW App, but unfortunately, we have so much to do supporting the Tier-1 systems that any work on a Mac has generally been an afterthought/low priority. I will put in a request to management to provide expanded support for UFS on a Mac, since I think it falls well within our "community engagement" goals, but I'm guessing it will receive low priority. I think the fastest path forward would be, as @uturuncoglu said, to file a ticket with the ESMF team. If you can include more detail/explanation as to why it's important for you (and others) to be able to run on a Mac, I can include that with the request to management. I know you're not the only stakeholder who would like to run on a Mac, so the more of these requests we can gather, the more likely it is to happen, even if it takes a while.

[DusanJovic-NOAA]

@barlage If you don't need all the libraries that spack-stack provides and you don't need the entire SRW app, but just need to be able to compile ufs-weather-model, UFS_UTILS or UPP, on macOS you can take a look at simple-ufs (https://github.com/DusanJovic-NOAA/simple-ufs). I just re-enabled macos github workflow and it seems it built the model and ran a very short c96 test configuration successfully. The github workflow runs on 'macos-14-arm64', which I believe is Sonoma on M chip. I don't know which M chip exactly. See: https://github.com/DusanJovic-NOAA/simple-ufs/actions/runs/10386707623/job/28758562692a