Python binding

.pre-commit-config.yaml

@@ -34,6 +34,10 @@ repos:
         - B # flake8-bugbear
         - I # isort
         - NPY # numpy-specific rules
+        - --ignore
+        - F405
+        - --ignore
+        - F403 # ignore import *
 
   - repo: https://github.com/neutrinoceros/inifix
     rev: v5.0.2

CMakeLists.txt

@@ -15,6 +15,7 @@ option(Idefix_HIGH_ORDER_FARGO "Force Fargo to use a PPM reconstruction scheme"
 option(Idefix_DEBUG "Enable Idefix debug features (makes the code very slow)" OFF)
 option(Idefix_RUNTIME_CHECKS "Enable runtime sanity checks" OFF)
 option(Idefix_WERROR "Treat compiler warnings as errors" OFF)
+option(Idefix_PYTHON "Enable python bindings (requires pybind11)" OFF)
 set(Idefix_CXX_FLAGS "" CACHE STRING "Additional compiler/linker flag")
 set(Idefix_DEFS "definitions.hpp" CACHE FILEPATH "Problem definition header file")
 option(Idefix_CUSTOM_EOS "Use custom equation of state" OFF)
@@ -77,7 +78,7 @@ add_subdirectory(src build)
 if(EXISTS ${PROJECT_BINARY_DIR}/setup.cpp)
   target_sources(idefix PUBLIC ${PROJECT_BINARY_DIR}/setup.cpp)
 else()
-  message(WARNING "No specific setup.cpp found in the problem directory")
+  message(WARNING "No specific setup.cpp found in the problem directory (this message can be ignored if using python to define your problem)")
 endif()
 
 # If a CMakeLists.txt is in the problem dir (for problem-specific source files)
@@ -115,12 +116,26 @@ if(Idefix_HDF5)
   )
   find_package(HDF5 REQUIRED)
   target_link_libraries(idefix "${HDF5_LIBRARIES}")
-  target_include_directories(idefix PUBLIC "${HDF5_INCLUDE_DIRS}")
+  target_include_directories(idefix "${HDF5_INCLUDE_DIRS}")
   message(STATUS "XDMF (hdf5+xmf) dumps enabled")
 else()
   set(Idefix_HDF5 OFF)
 endif()
 
+if(Idefix_PYTHON)
+  add_compile_definitions("WITH_PYTHON")
+  if (NOT DEFINED Python_FIND_FRAMEWORK)
+    set(Python_FIND_FRAMEWORK "LAST") # Use Apple's python only at last resort on Macos
+  endif ()
+  set(PYBIND11_FINDPYTHON ON CACHE BOOL "Idefix requires python" FORCE)
+  find_package(pybind11 REQUIRED)
+  target_link_libraries(idefix pybind11::embed)
+  target_sources(idefix
+    PUBLIC src/pydefix.cpp
+    PUBLIC src/pydefix.hpp
+  )
+endif()
+
 if(Idefix_DEBUG)
   add_compile_definitions("DEBUG")
   set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O0 -g")
@@ -232,6 +247,7 @@ else()
 endif()
 message(STATUS "    MPI:  ${Idefix_MPI}")
 message(STATUS "    HDF5: ${Idefix_HDF5}")
+message(STATUS "    Python: ${Idefix_PYTHON}")
 message(STATUS "    Reconstruction: ${Idefix_RECONSTRUCTION}")
 message(STATUS "    Precision: ${Idefix_PRECISION}")
 message(STATUS "    Version: ${Idefix_VERSION}")

doc/source/index.rst

@@ -33,6 +33,10 @@ MPI library
   When using MPI parallelisation, *Idefix* relies on an external MPI library. *Idefix* has been tested successfully with OpenMPI and IntelMPI libraries. When used on GPU architectures, *Idefix* assumes that
   the MPI library is GPU-Aware. If unsure, check this last point with your system administrator.
 
+Python
+  When using *Idefix* with its python interface through the module `Pydefix`, *Idefix* relies on an external python>=3.8 interpreter with the module `pybind11 <https://pybind11.readthedocs.io>`_
+  installed.
+
 ================
 Features
 ================

doc/source/modules.rst

@@ -28,6 +28,9 @@ In this section, you will find a more detailed documentation about each module t
 :ref:`gridCoarseningModule`
   The grid coarsening module, that allows to derefine the grid in particular locations to speed up the computation.
 
+:ref:`pydefixModule`
+  The Python interaction module, that allows Idefix to interact directly with a python interpreter.
+
 
 .. toctree::
    :maxdepth: 2
@@ -40,3 +43,4 @@ In this section, you will find a more detailed documentation about each module t
    modules/selfGravity.rst
    modules/braginskii.rst
    modules/gridCoarsening.rst
+   modules/pydefix.rst

doc/source/modules/pydefix.rst

@@ -0,0 +1,126 @@
+.. _pydefixModule:
+
+Pydefix module
+==============
+
+The Pydefix module allows Idefix to talk directly to a Python interpreter while running. It can be used to create your own initial condition
+and/or for custom outputs produced from Python. Pydefix relies on the pybind11 python package
+
+The essence of Pydefix is to allows the user to have a direct access to Idefix data structure from Python without writing/accessing any file. In particular, IdefixArrays are viewed as numpy arrays in Python.
+Note however that to keep things simple, Pydefix works on the host memory space only, and hence sync data to/from the GPU (if used) before invoking Python functions. Hence, using Pydefix for outputs induces
+a significant loss of performances.
+
+
+Before you start
+----------------
+Pybind11 installation
++++++++++++++++++++++
+
+In order to use pydefix, you need to be working in a python>=3.8 environement that includes `pybind11 <https://pybind11.readthedocs.io>`_. Follow the instruction of your package manager to install pybind11>=2.12.
+
+Pydefix usage
+-------------
+Idefix Configuration
+++++++++++++++++++++
+
+In order to use Pydefix, you need to switch on ``Idefix_PYTHON`` in cmake. This will auto-detect Python and check that pybind11 can be used effectively.
+
+
+Run Idefix with Pydefix
++++++++++++++++++++++++
+
+Pydefix is typically enabled from your input file `idefix.ini` in the block ``[Python]``. The following parameters are available:
+
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+|  Entry name            | Parameter type        | Comment                                                                                                   |
++========================+=======================+===========================================================================================================+
+| script                 | string                | | (Mandatory) Filename (*without ".py"!*) of the python script that Idefix should use.                    |
+|                        |                       | | The script should be in the same location as the Idefix executable file.                                |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+| output_function        | string                | | (Optional) Name of the function that will be called for each output event (the function should be       |
+|                        |                       | | defined in the  python script above). When ommited, pydefix output functions are disabled.              |
+|                        |                       | | The periodicity of the pydefix output routine is set in the block:entry `[Output]:python`               |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+| initflow_function      | string                | | (Optional) Name of the python function that will be called to initialize the flow in place of the C++   |
+|                        |                       | | function ``Setup::InitFlow``. Revert to ``Setup::Initflow`` when ommited.                               |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+
+Python script
++++++++++++++
+
+When using Pydefix, idefix expects a python script to be specified in the input file (see ``script`` parameter above). To be fully functionnal, you should import the ``pydefix`` module at the beginning
+of your python script (you can also import other python module, as any python script).
+
+Your python script should define functions that will be called while Idefix is running:
+* The signature of the ``initflow`` function should be ``(data)`` where ``data`` is a python structure matching Idefix's ``DataBlockHost`` class.
+* The signature of the ``output`` function should be ``(data,grid,n)`` where ``data`` is a python structure matching Idefix's ``DataBlockHost`` class, ``grid`` is Idefix's ``GridHost`` class, and ``n`` is an integer representing the current number of the output
+
+MPI parallelism
++++++++++++++++
+When Idefix runs with MPI parallelism enabled, a python interpreter and script is launched by each MPI process. Each of these script is independent
+and have access to its local ``dataBlockHost``. The `pydefix` module however gives access to the local rank ``prank`` and total MPI size ``psize``. In addition,
+pydefix provides the function ``GatherIdefixArray`` to gather the data distributed among each process without invoking MPI directly in python. This function
+expects a 3D distributed IdefixArray in entry following the signature
+
+.. code-block:: c++
+
+  GatherIdefixArray(IdefixHostArray3D<real> in, // 3D distributed array
+                    DataBlockHost dataHost,     // dataBlock structure
+                    bool keepBoundaries = true, // Whether we keep the ghost zones in the returned array
+                    bool broadcast = true)      // Whether the returned array is available only in proc #0 or in every proc (caution! possibly requires lots of memory)
+
+This function is used as follows:
+
+.. code-block:: python
+
+  import pydefix as pdfx # mandatory
+  import numpy as np
+  import matplotlib.pyplot as plt
+
+  def output(data,grid,n):
+    # Gather pressure field from every process in process #0 (set broadcast to True to distribute the result to all processes)
+    prs = pdfx.GatherIdefixArray(data.Vc[pdfx.PRS,:,:,:],data,broadcast=False)
+
+    # Only root process performs this
+    if pdfx.prank==0:
+      x=grid.x[pdfx.IDIR] # The grid contains the full domain size, while the datablock contains only local information
+      y=grid.x[pdfx.JDIR]
+      plt.figure()
+      plt.pcolormesh(x,y,prs[0,:,:],cmap='plasma')
+
+
+.. note::
+  For more advanced usage, it is also possibly to directly call MPI routines from python using the `Mpi4py <https://pypi.org/project/mpi4py/>`_ module.
+
+Example
++++++++
+
+An example is provided in the directory `test/IO/python`. This example shows how to use Idefix with pure python initial conditions and outputs.
+It reproduces the 2D OrszagTang vortex available in MHD/OrszagTang without requiring any single line of C++ code from the user.
+
+The python script `pydefix_example.py` initializes the initial condition of the OT test (``initflow``) and produces a series of PNG files through matplotlib (`output`).
+
+Troubleshooting
+---------------
+
+It during configuration stage, you get::
+
+  CMake Error at CMakeLists.txt:122 (find_package):
+    By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
+    asked CMake to find a package configuration file provided by "pybind11",
+    but CMake did not find one.
+
+It means that cmake cannot find the location of pybind11 (this typically happens on MacOs). In order to locate pybind11, open a python interpreter, and get pybind11 install dir through:
+
+.. code-block:: python
+
+  import pybind11
+  print(pybind11.__file__)
+
+You can then exit the interpreter and set the pybind11_DIR environement variable to the right path:
+
+.. code-block:: bash
+
+  export pybind11_DIR=env/lib/python3.10/site-packages/pybind11
+
+you can then run cmake which should be able to find pybind11, and compile the code.

doc/source/reference/idefix.ini.rst

@@ -358,6 +358,23 @@ and ``X1-end``, ``X2-end``, ``X3-end`` for the right boundaries. ``X2`` boundari
 |                | | (see :ref:`userdefBoundaries`)                                                                                 |
 +----------------+------------------------------------------------------------------------------------------------------------------+
 
+``Python`` section
+------------------
+
+This section describes the python script and function that can interact with Idefix while running using the Pydefix module (see :ref:`pydefixModule`)
+
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+|  Entry name            | Parameter type        | Comment                                                                                                   |
++========================+=======================+===========================================================================================================+
+| script                 | string                | | (Mandatory) Filename (*without ".py"!*) of the python script that Idefix should use.                    |
+|                        |                       | | The script should be in location of Idefix executable file                                              |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+| output_function        | string                | | (Optional) Name of the function that will be called for each output event (the function should be       |
+|                        |                       | |  defined in the  python script above). When ommited, pydefix output functions are disabled.             |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
+| initflow_function      | string                | | (optional) Name of the python function that will be called to initialize the flow in place of the C++   |
+|                        |                       | | function `Setup::InitFlow`. Revert to `Setup::Initflow`` when ommited.                                  |
++------------------------+-----------------------+-----------------------------------------------------------------------------------------------------------+
 
 .. _outputSection:
 
@@ -411,6 +428,9 @@ This section describes the outputs *Idefix* produces. For more details about eac
 |                |                         | | (see :ref:`functionEnrollment`). The user-defined variables defined by this function           |
 |                |                         | | are then written as new variables in vtk and/or xdmf  outputs.                                 |
 +----------------+-------------------------+--------------------------------------------------------------------------------------------------+
+| python         | float                   | | Time interval between pydefix outputs, in code units.                                          |
+|                |                         | | If negative, periodic pydefix outputs are disabled.                                            |
++----------------+-------------------------+--------------------------------------------------------------------------------------------------+
 
 .. note::
     Even if dumps are not mentionned in your input file (and are therefore disabled), dump files are still produced when *Idefix* captures a signal

doc/source/reference/makefile.rst

@@ -108,18 +108,11 @@ We recommend the following modules and environement variables on AdAstra:
 
 .. code-block:: bash
 
-    module load cpe/23.12
+    module load cpe/24.07
     module load craype-accel-amd-gfx90a craype-x86-trento
     module load PrgEnv-cray
-    module load amd-mixed/5.7.1
-    module load rocm/5.7.1 # nécessaire a cause d'un bug de path pas encore fix..
-    export HIPCC_COMPILE_FLAGS_APPEND="-isystem ${CRAY_MPICH_PREFIX}/include"
-    export HIPCC_LINK_FLAGS_APPEND="-L${CRAY_MPICH_PREFIX}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a} -lstdc++fs"
-    export CXX=hipcc
-    export CC=hipcc
-
-The `-lstdc++fs` option being there to guarantee the link to the HIP library and the access to specific
-C++17 <filesystem> functions.
+    module load amd-mixed
+    module load cray-python/3.11.7
 
 Finally, *Idefix* can be configured to run on Mi250 by enabling HIP and the desired architecture with the following options to ccmake:
 

doc/source/reference/outputs.rst

@@ -7,7 +7,7 @@ Output formats
 --------------
 
 *Idefix* uses several types of outputs you may want for your setup. By default, *Idefix* allows
-for 4 kinds of outputs:
+for 5 kinds of outputs:
 
 * logs which essentially tells the user what *Idefix* is currently doing. When running in serial, logs are sent to stdout, but when
   MPI is enabled, only the logs of the rank 0 process is sent to stdout, and each process (including rank 0) simultaneously writes a
@@ -21,17 +21,20 @@ for 4 kinds of outputs:
   or `Visit <https://wci.llnl.gov/simulation/computer-codes/visit>`_. The XDMF format relies on the HDF5 format and therefore requires *Idefix* to be configured with HDF5 support.
 * user-defined analysis files. These are totally left to the user. They usually consist of ascii tables defined by the user, but they can
   be anything.
+* python script, that relies on the :ref:`pydefixModule`. This launches a user-defined python function fed with Idefix data. One can then directly plot or interact with Idefix outputs from python.
 
 The output periodicity and the userdef variables should all be declared in the input file, as described in :ref:`outputSection`.
 
 Defining your own outputs
 -------------------------
 
-*Idefix* provides two ways to define your own outputs: analysis, which are used to make your
-own output file (e.g. an ascii-tabulated file); and user variables, which are written by *Idefix* output routines.
+*Idefix* provides three ways to define your own outputs: analysis, which are used to make your
+own output file (e.g. an ascii-tabulated file); user variables, which are written by *Idefix* output routines, and python user-defined
+functions that process Idefix's data.
 
 Both analysis and uservar requires the definition of a user function which needs to be enrolled following the procedure described
-in :ref:`functionEnrollment` and using the function signatures declared in `output.hpp`.
+in :ref:`functionEnrollment` and using the function signatures declared in `output.hpp`. The python outputs are described
+in the :ref:`pydefixModule`.
 
 We provide below an example of a setup using both analysis outputs and uservar outputs
 

src/dataBlock/dataBlockHost.cpp

@@ -34,6 +34,9 @@ DataBlockHost::DataBlockHost(DataBlock& datain) {
 
   nghost = data->nghost;
 
+  lbound = data->lbound;
+  rbound = data->rbound;
+
   xbeg = data->xbeg;
   xend = data->xend;
   beg = data->beg;
@@ -95,13 +98,18 @@ DataBlockHost::DataBlockHost(DataBlock& datain) {
   this->haveplanetarySystem = data->haveplanetarySystem;
   this->planetarySystem = data->planetarySystem.get();
 
+  this->t = data->t;
+  this->dt = data->dt;
+
   idfx::popRegion();
 }
 
 // Synchronisation routines of Data (*Only*)
 void DataBlockHost::SyncToDevice() {
   idfx::pushRegion("DataBlockHost::SyncToDevice()");
 
+  data->t = this->t;
+  data->dt = this->dt;
   Kokkos::deep_copy(data->hydro->Vc,Vc);
   Kokkos::deep_copy(data->hydro->InvDt,InvDt);
 
@@ -138,6 +146,9 @@ void DataBlockHost::SyncToDevice() {
 
 void DataBlockHost::SyncFromDevice() {
   idfx::pushRegion("DataBlockHost::SyncFromDevice()");
+  this->t = data->t;
+  this->dt = data->dt;
+
   Kokkos::deep_copy(Vc,data->hydro->Vc);
   Kokkos::deep_copy(InvDt,data->hydro->InvDt);