Skip to content

Conference call notes 20240911

Jump to bottom
Kenneth Hoste edited this page Sep 11, 2024 · 1 revision

(back to Conference calls)

Notes on the 254th EasyBuild conference call, Wednesday 11 Sept 2024 (08:00 UTC)

Attendees

List of attendees (15):

  • Simon Branford (Univ. of Birmingham, UK)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Alexander Grund (ZIH, Dresden, Germany)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Georgios Kafanas (University of Luxembourg)
  • Sébastien Moretti (SIB, Switzerland)
  • Jure Pečar (EMBL, Heidelberg, Germany
  • Jan Reuter (JSC, Germany)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Alexandre Strube (JSC, Germany)
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a/2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.2 (12 June 2024)
  • next EasyBuild release:
    • likely EasyBuild 4.9.3 very soon (days)
      • status:
        • framework: no more changes planned, see 4.9.3 milestone
        • easyblocks: small set of PRs to be merged, see 4.9.3 milestone
        • easyconfigs: only bug fix PRs (and whatever gets merged before 4.9.x branch is updated with current develop to prepare for release), see 4.9.3 milestone
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • allow for case where homepage = None when generating the docs (PR #4626)
    • enhancements
      • add support for --extra-source-urls to fetch sources from additional URLs (PR #4079)
      • only call _sanity_check_step_extensions if --skip-extensions is not set (PR #4620)
      • Add support for --software-commit and an associated template %(software_commit)s (PR #4628)
    • changes
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • enable --module-extensions by default (+ resolve template values used in extension version) (PR #4501)
      • Use default value $XDG_CONFIG_DIRS from XDG basedir spec: /etc/xdg (instead of /etc) (PR #4591)
        • warning is printed if /etc/easybuild.d/*.cfg is found
      • convert template constant lists to dicts and export the constants by name (PR #4595)
      • move verifying of checksums from source to fetch step, to include it with --fetch (PR #4624)
      • changes to support for using Tcl-based Environment Modules as modules tool (instead of Lmod):
        • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • Run unit tests on an updated versions of Modules: v4.5.3 + v5.3.1 #4415 (PR #4415)
        • add check_group support for module files in Tcl syntax (PR #4418)
        • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
        • Use getenv modulefile command with EnvironmentModules >= 4.2.0 (PR #4614)
        • Add module cache build support on EnvironmentModules (PR #4615)
        • derive EnvironmentModules class directly from ModulesTool rather than from to be deprecated EnvironmentModulesTcl (PR #4625)

  • easyblocks (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • update custom easyblock for PETSc to consider include/suitesparse subdirectory for SuiteSparse headers (PR #3391)
      • enhance custom easyblock for GCC to use with-arch option for nvptx with 13.1+ (PR #3396)
      • disable the version check of pip in Python packages by defining $PIP_DISABLE_PIP_VERSION_CHECK (PR #3427)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • use run_shell_cmd in custom easyblock for ABAQUS (PR #3414)
      • use PYPI_SOURCE constant in generic PythonPackage easyblock + custom easyblock for Python (PR #3410)
      • remove (20) unused software-specific easyblocks (PR #3423)
      • remove unused easyblock for BamTools (PR #3424)
      • add check to make sure that easyblocks do not use run_cmd or run_cmd_qa (PR #3426)

  • easyconfigs (merged PRs)

    • ~60 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • disable test step for WIEN2k 23.2 because files required by it can no longer be downloaded (PR #20969)
      • add patch for HDF5 v1.14.3 to suppress fp exceptions (PR #21280)
      • consistently fix homepage + source URL for HDF + h4toh5 (PR #21134)
      • correct extension order in R-bundle-CRAN 2024.06 (PR #21285)
      • update easyconfig for dorado 0.7.3 to properly use openssl from EB and not install external libraries into its own lib directory (PR #21297)
      • use source tarballs from GitHub for recent libdap easyconfigs (PR #21334)
      • remove Highway build dependency in Brunsli easyconfigs, since it's not actually required at all (PR #21366)
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • noteworthy software updates
      • ...
    • changes
      • use stand-alone absl-py as dependency for jax w/ gfbf/2023a + CUDA 12.1.1 (PR #21039)
      • reorganize Flax/JAX stack in 2023a: scvi-tools, Flax, Optax, jax, absl-py (PRs #21038)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • remove unnecessary use of an easyblock for BamTools (PR #21263)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
    • changes
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • project board: https://github.com/orgs/easybuilders/projects/18/views/2
      • Enable module-depends-on by default (PR #4500)
        • TODO: only use depends_on when using Lmod as modules tool, module load is basically equivalent for recent versions of Environment Modules
      • More granular exit codes (PR #4534)
      • prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • we'll probably handle this by introducing an extra EasyBuild configuration option like --c-cxx-header-path, which will be CPATH by default, but can be set to INCLUDE_PATH which implies not updating $CPATH and updating $C_INCLUDE_PATH, $CXX_INCLUDE_PATH and $INCLUDE_PATH instead;
      • Rename source step (issue #4627)

  • easyblocks (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • Update NCL easyblock for intel (PR #3409)
      • enhance OpenFOAM easyblock to also build the plugins for OpenFOAM >= v2406 (PR #3436)
      • allow Python bundles and packages to specify a maximum Python version for the system toolchain (PR #3431)
      • enhance binutils easyblock to explicitely pass msgpack configure option (PR #3438)
    • updates
      • update sanity check in OpenFOAM easyblock, since there's no modifyMesh in OpenFOAM 12 (PR #3434)
      • fix sanity check for Geant4 >= v11.2 (PR #3439)
      • update custom easyblock for CP2K >= v2024 (PR #3433)
      • update AOCC easyblock to work with LLVM 16 and above (PR #3437)
    • changes
      • ...
    • new
      • ...
    • EasyBuild 5.0 (separate 5.0.x branch)
      • Use prefer_ebpythonprefixes flag to use EBPYTHONPREFIXES (PR #3343)
      • enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • run pip check only once for PythonBundle (PR #3428 + PR #3432)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • add alternative checksum for bold 1.3.0 extension in R-bundle-CRAN (PR #21370)
    • enhancements
      • add the plugins to OpenFOAM v2406 (PR #21332)
    • (noteworthy) new software
      • ...
    • software updates
    • changes
      • unbundle tensorboard from TensorFlow v2.15.1 (PR #21337)
    • EasyBuild 5.0 (separate 5.0.x branch)
      • ...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • Alexandre: how does EasyBuild deal with source tarballs being changed in-place upstream (without version bumps)
    • there's support for "alternative checksums" in case the newly served tarball is benign (no code changes), see for example easyconfig PR #21370
    • sources that have disappeared only can be made available via https://sources.easybuild.io (no automation in place there to add stuff, ask @boegel)
      • but this doesn't help with source tarballs that are suddenly different upstream, only for failing downloads since it's used as a last resort;
    • we also report problems with upstream changes in source tarballs, but sometimes without much success, see for example upstream issue for libxc
  • EasyBuild 5.0
    • last bit of effort will focus on extending the documentation covering the changes
      • https://docs.easybuild.io/easybuild-v5 will become a one-page overview with breaking changes, etc. and links to other pages with more details
      • a webinar will be planned shortly after the release of EasyBuild 5.0 in which we'll present in detail what has changed, and how it may affect your EasyBuild setup (or not)
Clone this wiki locally