Skip to content

Commit

Permalink
Upgrade to PDAF V2.3.1 (#7)
Browse files Browse the repository at this point in the history
* Improved handling dimension of Uinv argument in init_ens_pdaf.

* update offline example

* update PDAF version

* improving tutorial for serial DA and improved documentation for related functions
  • Loading branch information
yumengch authored Dec 10, 2024
1 parent aa02782 commit a2ee213
Show file tree
Hide file tree
Showing 164 changed files with 62,802 additions and 57,997 deletions.
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[submodule "PDAF"]
branch = PDAF_V2.3
branch = PDAF_V2.3.1
path = PDAF
url = https://github.com/PDAF/PDAF.git
2 changes: 1 addition & 1 deletion PDAF
Submodule PDAF updated 77 files
+17 −17 external/CG+_mpi/cgfam.f
+2 −2 external/CG+_mpi/cgsearch.f
+55 −0 make.arch/NECSXAurora_nfort_mpi.h
+4 −4 models/classical/lorenz05b/add_obs_error_pdaf.F90
+4 −4 models/classical/lorenz05c/add_obs_error_pdaf.F90
+4 −4 models/classical/lorenz63/add_obs_error_pdaf.F90
+4 −4 models/classical/lorenz96/add_obs_error_pdaf.F90
+6 −6 models/classical/lorenz96/localize_covar_pdaf.F90
+4 −4 models/lorenz63/add_obs_error_pdaf.F90
+28 −1 src/PDAF_3dvar_optim_cgplus.F90
+29 −3 src/PDAF_en3dvar_optim_cgplus.F90
+29 −4 src/PDAF_hyb3dvar_optim_cgplus.F90
+3 −3 src/PDAF_print_version.F90
+437 −151 src/PDAFomi_obs_f.F90
+0 −4 src/PDAFomi_obs_l.F90
+3 −3 templates/classical/offline/add_obs_error_pdaf.F90
+4 −4 templates/classical/offline/localize_covar_pdaf.F90
+5 −5 templates/classical/online/add_obs_error_pdaf.F90
+4 −4 templates/classical/online/localize_covar_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D/add_obs_error_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D/localize_covar_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D_nobs/add_obs_error_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D_si/add_obs_error_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D_si/localize_covar_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D_snglprec/add_obs_error_pdaf.F90
+4 −4 testsuite/src/dummymodel_1D_snglprec/localize_covar_pdaf.F90
+4 −4 testsuite/src/offline_1D/add_obs_error_pdaf.F90
+4 −4 testsuite/src/offline_1D/localize_covar_pdaf.F90
+0 −7 tutorial/3dvar/runtests.sh
+4 −4 tutorial/classical/offline_2D_parallel/add_obs_error_pdaf.F90
+4 −4 tutorial/classical/offline_2D_serial/add_obs_error_pdaf.F90
+4 −4 tutorial/classical/online_2D_parallelmodel/add_obs_error_pdaf.F90
+4 −4 tutorial/classical/online_2D_parallelmodel_fullpar/add_obs_error_pdaf.F90
+4 −4 tutorial/classical/online_2D_parallelmodel_fullpar_1fpe/add_obs_error_pdaf.F90
+4 −4 tutorial/classical/online_2D_serialmodel/add_obs_error_pdaf.F90
+29 −0 tutorial/plotting/plot_obs.py
+8 −8 tutorial/verification/out.offline_2D_openmp
+11 −11 tutorial/verification/out.offline_2D_parallel
+8 −8 tutorial/verification/out.offline_2D_serial
+5 −5 tutorial/verification/out.offline_2D_serial_3denvar_opt2
+12 −12 tutorial/verification/out.offline_2D_serial_3dhybvar_opt2
+9 −9 tutorial/verification/out.offline_2D_serial_3dlenvar_opt2
+7 −7 tutorial/verification/out.offline_2D_serial_3dlhybvar_opt2
+12 −12 tutorial/verification/out.offline_2D_serial_3dv_opt1
+8 −8 tutorial/verification/out.offline_2D_serial_3dv_opt2
+9 −9 tutorial/verification/out.offline_2D_serial_3dv_opt3
+5 −5 tutorial/verification/out.offline_2D_serial_ESTKF
+14 −14 tutorial/verification/out.online_2D_parallelmodel
+41 −41 tutorial/verification/out.online_2D_parallelmodel_3denvar_opt12
+47 −47 tutorial/verification/out.online_2D_parallelmodel_3denvar_opt13
+49 −49 tutorial/verification/out.online_2D_parallelmodel_3dhybvar_opt12
+38 −38 tutorial/verification/out.online_2D_parallelmodel_3dhybvar_opt13
+52 −52 tutorial/verification/out.online_2D_parallelmodel_3dlenvar_opt12
+46 −46 tutorial/verification/out.online_2D_parallelmodel_3dlenvar_opt13
+52 −52 tutorial/verification/out.online_2D_parallelmodel_3dlhybvar_opt12
+50 −50 tutorial/verification/out.online_2D_parallelmodel_3dlhybvar_opt13
+29 −29 tutorial/verification/out.online_2D_parallelmodel_3dv_opt1
+27 −27 tutorial/verification/out.online_2D_parallelmodel_3dv_opt12
+32 −32 tutorial/verification/out.online_2D_parallelmodel_3dv_opt13
+10 −10 tutorial/verification/out.online_2D_parallelmodel_ESTKF
+13 −13 tutorial/verification/out.online_2D_parallelmodel_fullpar
+13 −13 tutorial/verification/out.online_2D_parallelmodel_fullpar_1fpe
+42 −42 tutorial/verification/out.online_2D_serial_3dv_opt1
+25 −25 tutorial/verification/out.online_2D_serial_3dv_opt2
+18 −18 tutorial/verification/out.online_2D_serial_3dv_opt3
+16 −16 tutorial/verification/out.online_2D_serialmodel
+13 −13 tutorial/verification/out.online_2D_serialmodel_2fields
+15 −15 tutorial/verification/out.online_2D_serialmodel_2fields_ESTKF
+47 −47 tutorial/verification/out.online_2D_serialmodel_2fields_obsB
+42 −42 tutorial/verification/out.online_2D_serialmodel_3denvar_opt2
+39 −39 tutorial/verification/out.online_2D_serialmodel_3denvar_opt3
+46 −46 tutorial/verification/out.online_2D_serialmodel_3dhybvar_opt2
+42 −42 tutorial/verification/out.online_2D_serialmodel_3dlenvar_opt2
+48 −48 tutorial/verification/out.online_2D_serialmodel_3dlenvar_opt3
+51 −51 tutorial/verification/out.online_2D_serialmodel_3dlhybvar_opt2
+17 −17 tutorial/verification/out.online_2D_serialmodel_ESTKF
+12 −12 tutorial/verification/out.online_2D_serialmodel_openmp
74 changes: 62 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,81 @@
# pyPDAF - A Python interface to the Fortran-written data assimilation library

pyPDAF provides a Python interface to the established [Parallel Data Assimilation Framework (PDAF)](https://pdaf.awi.de/trac/wiki). The original framework is used with various regional and global climate models including atmosphere, ocean, hydrology, land surface and sea ice models. These models are typically written in Fortran which can be easily used with PDAF. pyPDAF can become useful in the following scenarios:
- With an increasing number of Python-coded numerical models, especially machine learning models, pyPDAF is a convenient tool to implement data assimilation (DA) systems purely in Python.
- Alternatively, pyPDAF can be used to set up offline data assimilation system. In such a system, the model fields in restart files are replaced by analyses generated by pyPDAF. This can be an attractive alternative to the original Fortran implementations considering the simplicity of code implementation and package management in Python.
pyPDAF provides a Python interface to the established
[Parallel Data Assimilation Framework (PDAF)](https://pdaf.awi.de/trac/wiki).
The original framework is used with various regional and global
climate models including atmosphere, ocean, hydrology, land surface
and sea ice models. These models are typically written in Fortran
which can be easily used with PDAF. pyPDAF can become useful
in the following scenarios:
- With an increasing number of Python-coded numerical models,
especially machine learning models, pyPDAF is a convenient tool
to implement data assimilation (DA) systems purely in Python.
- Alternatively, pyPDAF can be used to set up offline
data assimilation system. In such a system, the model fields in
restart files are replaced by analyses generated by pyPDAF.
This can be an attractive alternative to the original Fortran
implementations considering the simplicity of code implementation
and package management in Python.

The interface inherits the efficiency of the data assimilation algorithms in Fortran, and the flexibility to be applied to different models and observations. This means that users of pyPDAF can couple the DA algorithms with any types of model and observations without the need to coding the actual DA algorithms. This allows the users to focus on the specific research problems. The framework includes various ensemble DA algorithms including many variants of ensemble Kalman filters, particle filters and other non-linear filters. It also provides framework for variants of 3DVar. A full list of supported methods can be found [here](https://pdaf.awi.de/trac/wiki/AvailableOptionsforInitPDAF)
The interface inherits the efficiency of the data assimilation
algorithms in Fortran, and the flexibility to be applied to different
models and observations. This means that users of pyPDAF can couple
the DA algorithms with any types of model and observations without
the need to coding the actual DA algorithms. This allows the users
to focus on the specific research problems. The framework includes
various ensemble DA algorithms including many variants of ensemble
Kalman filters, particle filters and other non-linear filters.
It also provides framework for variants of 3DVar. A full list of
supported methods can be found
[here](https://pdaf.awi.de/trac/wiki/AvailableOptionsforInitPDAF)

## Getting Started
It is recommended to install pyPDAF via `conda`:
```bash
conda create -n pypdaf -c conda-forge yumengch::pypdaf==1.0.0
conda create -n pypdaf -c conda-forge yumengch::pypdaf==1.0.1
```
You can also install locally from the source code using `pip` by setting up `setup.cfg` and `cmake` configurations with examples given in [PDAFBuild](PDAFBuild).
You can also install locally from the source code using `pip`
by setting up `setup.cfg` and `cmake` configurations
with examples given in [PDAFBuild](PDAFBuild).

## Building a DA system with pyPDAF
A tutorial on using pyPDAF to build a data assimilation system without parallelisation is available here:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yumengch/pyPDAF/).
For users without prior experience with PDAF, we highly recommend to
start with the tutorial here:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yumengch/pyPDAF/).

To construct a parallel ensemble DA system, an example is provided in [example](example) directory. pyPDAF and PDAF both utilise `Message Passing Interface (MPI)` parallelisation. Hence, to run the example, it needs to be executed from commandline using `mpiexec`. For example,
To construct a parallel ensemble DA system,
in [example](example) directory, we provide both `online`
and `offline` examples.
pyPDAF and PDAF both utilise `Message Passing Interface (MPI)`
parallelisation. Hence, to run the example, it needs to be executed
from commandline using `mpiexec`. For example,
```bash
mpiexec -n 4 python -u example/main.py
mpiexec -n 4 python -u example/online/main.py
```
will run the example with 4 processes. The example is based on the [tutorials](http://pdaf.awi.de/trac/wiki/FirstSteps) of the original PDAF.
will run the example with 4 processes.
The example is based on
the [tutorials](http://pdaf.awi.de/trac/wiki/FirstSteps) of the original PDAF.


## Documentation:
The most up-to-date pyPDAF has interface with ```PDAF-V2.3```. A preliminary [documentation](https://yumengch.github.io/pyPDAF/index.html) is provided. The interface follows the naming convention of PDAF. One major difference is the localisation functions in the [Observation Module Infrastructure (OMI)](https://pdaf.awi.de/trac/wiki/PDAF_OMI_Overview). In PDAF, one can simply call `PDAFomi_init_dim_obs_l` or `PDAFomi_localize_covar`. In pyPDAF, these subroutines are broken into three functions: `pyPDAF.PDAF.omi_init_dim_obs_l_iso`, `pyPDAF.PDAF.omi_init_dim_obs_l_noniso`, `pyPDAF.PDAF.omi_init_dim_obs_l_noniso_locweights` for isotropic, non-isotropic and horizontal and vertically separated non-isotropic localisation. The suffix is applied similarly to `pyPDAF.PDAF.omi_localize_covar`. Details of the application of these localisation for [`PDAFomi_init_dim_obs_l`](https://pdaf.awi.de/trac/wiki/OMI_observation_modules#init_dim_obs_l_OBSTYPE) and [`PDAFomi_localize_covar`](https://pdaf.awi.de/trac/wiki/OMI_observation_modules#localize_covar_OBSTYPE) can be found in PDAF documentation.
The most up-to-date pyPDAF has interface with ```PDAF-V2.3.1```.
A [documentation](https://yumengch.github.io/pyPDAF/index.html) is provided.
The interface follows the naming convention of PDAF.
One major difference is the localisation functions
in the [Observation Module Infrastructure (OMI)](https://pdaf.awi.de/trac/wiki/PDAF_OMI_Overview).
In PDAF, one can simply call `PDAFomi_init_dim_obs_l` or `PDAFomi_localize_covar`.
In pyPDAF, these subroutines are broken into three functions:
`pyPDAF.PDAF.omi_init_dim_obs_l_iso`,
`pyPDAF.PDAF.omi_init_dim_obs_l_noniso`,
`pyPDAF.PDAF.omi_init_dim_obs_l_noniso_locweights` for isotropic,
non-isotropic and horizontal and vertically separated
non-isotropic localisation. The suffix is applied similarly
to `pyPDAF.PDAF.omi_localize_covar`.
Details of the application of these localisation for
[`PDAFomi_init_dim_obs_l`](https://pdaf.awi.de/trac/wiki/OMI_observation_modules#init_dim_obs_l_OBSTYPE)
and
[`PDAFomi_localize_covar`](https://pdaf.awi.de/trac/wiki/OMI_observation_modules#localize_covar_OBSTYPE)
can be found in PDAF documentation.


## Having questions?
Expand Down
2 changes: 1 addition & 1 deletion docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXOPTS ?= -v
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build
Expand Down
Loading

0 comments on commit a2ee213

Please sign in to comment.