Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MPI parallelization of thermodynamics #331

Merged
merged 21 commits into from
Dec 4, 2023
Merged

MPI parallelization of thermodynamics #331

merged 21 commits into from
Dec 4, 2023

Commits on Dec 4, 2023

  1. MPI-IO does not like ':' in file names so replace it with '_' 

    In most MPI implementation ':' in file name is interpreted as
limit of prefix with hint about underlying file system.
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    817d4cb View commit details
    Browse the repository at this point in the history

  2. MPI initialisation and linking

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    eb0e04b View commit details
    Browse the repository at this point in the history

  3. Read name of file with MPI decomposition from configuration file

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    b2fe706 View commit details
    Browse the repository at this point in the history

  4. Define communicator, its size and rank for MPI run

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    ee378f1 View commit details
    Browse the repository at this point in the history

  5. Pass MPI related arguments to StructureFactory::stateFromFile

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    1de2877 View commit details
    Browse the repository at this point in the history

  6. MPI: pass partitionFile to getModelState

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    e3bd395 View commit details
    Browse the repository at this point in the history

  7. MPI: pass partitionFile argument to RectGridIO::getModelState

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    1a94ed3 View commit details
    Browse the repository at this point in the history

  8. MPI: Write one file per MPI rank for Rectangular grid 

    It is a temporary state for testing before introducing of fully
parallel output.
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    020dc9c View commit details
    Browse the repository at this point in the history

  9. Pass model metadata to RectGridIO::getModelState 

    We will need MPI communicator and rank information to read only
relevant part of data.
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    44d7006 View commit details
    Browse the repository at this point in the history

  10. Add classes for parallel access to netCDF class 

    Currently netcdf-cxx4 does not provide interfaces for parallel access
to netCDF files. Provide ones based on ncFile class. Not all methods
fron ncFile are implemented, only the ones the will be used by nextsimDG
code.
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    dcc8459 View commit details
    Browse the repository at this point in the history

  11. MPI: Working version for parallel reading

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    a17d35b View commit details
    Browse the repository at this point in the history

  12. Get global X and Y dimensions from partition metadata file 

    They are needed to write to a correct location in parallel mode.
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    6b41b8f View commit details
    Browse the repository at this point in the history

  13. Write output netCDF files in parallel

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    c64bb14 View commit details
    Browse the repository at this point in the history

  14. doctest: Update to 2.4.11 with MPI extensions

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    62befe3 View commit details
    Browse the repository at this point in the history

  15. Add testRectGrid_MPI

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    29ecbf8 View commit details
    Browse the repository at this point in the history

  16. RectGridIO: Treat z direction the same way in serial and parallel rea… 

    …ding
    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    338fc5e View commit details
    Browse the repository at this point in the history

  17. Working testRectGrid_MPI for one rank

    @draenog @TomMelt
    draenog authored and TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    5b0fe92 View commit details
    Browse the repository at this point in the history

  18. fix array ordering for mpi_local tests 

    Array indices were re-ordered in the serial version of the code.

I have reflected similar changes in the mpi branch.

This fixes `testRectGrid_MPI`. I have also tested the changes on
`run_simple_example.sh` and it also produces the same result as the
serial code.
    @TomMelt
    TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    7fbb4a5 View commit details
    Browse the repository at this point in the history

  19. disable non-MPI tests when building with MPI 

    disabled the following tests when MPI is enabled:
* testRectGrid
* testParaGrid
* testConfigOutput

The latter two have only been disabled temporarily.

I have raised issue #454 explaining why the tests are disabled.

They will be re-enabled after paragrid is parallelized with MPI.

`testRectGrid` has a serial version and an MPI version
(`testRectGrid_MPI`). We run the MPI version when MPI is enabled and the
serial otherwise.
    @TomMelt
    TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    3c0d866 View commit details
    Browse the repository at this point in the history

  20. Mpi local tests (#432) 

    # Add Tests for MPI version of code

### Task List
- [x] add tests
- [x] rebase to develop
- [x] fix array ordering
- [x] ~update the readthedocs / readme with install instructions~ (moved
to #455)

---
# Change Description

I rebased @draenog's `mpi_local` and `mpi_local_tests` branches. This
introduced a bug due to array reordering that occurred as part of #426.
I have modified the MPI parts of `./core/src/RectGridIO.cpp` to reorder
array so that they now match the order in the serial version of the
code.

---
# Test Description

To run `testRectGrid_MPI` you will need the parallel netcdf config file
`partition_metadata_1.nc`. I don't want to put it in the github repo for
now. As discussed with @timspainNERSC , soon we will hopefully have an
ftp server for netcdf files.

You can generate the input using the following netcdf definition file
`partition_metadata_1.cdl`.
```
// partition_metadata_1.cdl
netcdf partition_metadata_1 {
dimensions:
	P = 1 ;
	L = 1 ;
	globalX = 25 ;
	globalY = 15 ;

group: bounding_boxes {
  variables:
  	int global_x(P) ;
  	int global_y(P) ;
  	int local_extent_x(P) ;
  	int local_extent_y(P) ;
  data:

   global_x = 0 ;

   global_y = 0 ;

   local_extent_x = 25 ;

   local_extent_y = 15 ;
  } // group bounding_boxes
}
```

Then use `ncgen` to make the netcdf file `partition_metadata_1.nc`
```
ncgen partition_metadata_1.cdl -o partition_metadata_1.nc
```
    @TomMelt
    TomMelt authored Dec 4, 2023
    Configuration menu
    Copy the full SHA
    476e2cc View commit details
    Browse the repository at this point in the history

  21. fix linting error with clang-format

    @TomMelt
    TomMelt committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    2017be7 View commit details
    Browse the repository at this point in the history