Skip to content

Conference call notes 20240117

Jump to bottom
Kenneth Hoste edited this page Jan 17, 2024 · 2 revisions

(back to Conference calls)

Notes on the 238th EasyBuild conference call, Wednesday 17 Jan 2024 (16:00 UTC)

Attendees

List of attendees (19):

  • Sebastian Achilles (JSC, Germany)
  • Simon Branford (Univ. of Birmingham, UK)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Markus Geimer (JSC, Germany)
  • Jasper Grimm (University of York, UK)
  • Stefan Harst (Univ. of Oldenburg)
  • Thomas Hayward-Schneider (Max Planck Institute for Plasma Physics, Germany)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Chia-Jung Hsu (Chalmers University of Technology, Sweden)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Sébastien Moretti (SIB, Switzerland)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Bart Oldeman (The Alliance, Canada)
  • Jure Pečar (EMBL, Heidelberg, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Alexandre Strube (JSC, Germany)
  • Lars Viklund (Umeå University, Sweden)

Agenda

  • overview of recent developments
  • quick status update on development for EasyBuild 5.0
  • Clang-based toolchains;
  • supporting OSs with OpenSSL 3.x (like RHEL9);
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.0 (30 Dec 2023)
  • next EasyBuild release: hopefully EasyBuild v5.0.0
    • there will only be an EasyBuild v4.9.1 if there's a clear need for it, or if getting EasyBuild 5.0 is taking too long (way beyond 8 weeks v4.9.0 release)
  • ETA for EasyBuild 5.0 release: hopefully first EasyBuild release of 2024
    • started doing short sprint meetings, each Monday at 15:00 CE(S)T to set next 5 goals to tackle that week
    • monthly sync meetings every 1st Wed of the month at 14:00 CE(S)T: next today Wed 3 Jan'24 at 14:00 CET
    • see also https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
  • recent changes
    • docs (merged PRs)
      • ...
    • framework (merged PRs)
      • bug fixes
        • ...
      • enhancements
        • use --all option with EnvironmentModules v4.6+ to get available hidden modules (PR #4417)
        • add support for appending to path environment variables via modextrapaths_append (and add corresponding allow_append_abs_path) (PR #4436)
        • Add script for updating local git repos with develop branch (PR #4438)
      • changes
        • ...
      • code cleanup
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • bug fixes
          • ...
        • changes
        • enhancements
          • implement support for 'stream_output' option in run_shell_cmd (PR #4423)
          • detect Fortran .mod files in GCCcore installations (PR #4389)
    • easyblocks (merged PRs)
      • bug fixes
        • set $BGW_TEST_MPI_NPROCS in BerkeleyGW test step (PR #2950)
        • fix ELPA preprocessor flag for QuantumESPRESSO >= 7 (PR #3063)
        • fix link order in PETSc easyblock for SCOTCH >= 7.x (PR #3069)
        • add Clang opt binary to RPATH wrapper dir (PR #3076)
      • enhancements
        • add pip_verbose option to PythonPackage and set it to True in PyTorch easyblock, to show build output when using pip (PR #3065)
      • updates
        • update WIEN2k easyblock for v23.2 (PR #3035)
      • changes
        • ...
      • new easyblocks
        • new easyblock for DualSPHysics (PR #3050)
      • EasyBuild 5.0 (to 5.0.x branch)
        • convert all generic easyblocks to run_shell_cmd (PR #3046)
        • disable check for .mod files in Clang, AOCC, AOMP, and CPLEX easyblocks (PR #3067)
        • remove custom easyblock for EggLib (no longer used) (PR #3066)
    • easyconfigs (merged PRs)
      • ~over 125 easyconfig PRs were merged since last conf call
        • and over 175 old easyconfig PRs were closed as is, mostly because they were for easyconfigs using an unsupported or deprecated toolchains (see toolchain support policy)
      • bug fixes
        • add patch to fix flaky scipy build in SciPy-bundle v2023.11 (PR #19481)
        • fix LAPACK test failures in OpenBLAS 0.3.23/24 (PR #19495)
        • add patches for PyTorch v2.1.2 with foss/2022a to fix test failures on non-x86 platforms (PRs #19571, #19572, #19573)
        • use pocl with CUDA support as dependency for PyOpenCL v2023.1.4 (PR #19584)
      • enhancements
        • ...
      • (noteworthy) new software
        • ...
      • noteworthy software updates
        • ...
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • ...
  • work-in-progress
    • docs (open PRs + issues)
    • framework (open PRs + issues)
      • reported bugs / bug fixes
        • Run unit tests on an updated version of Modules 4 (PR #4415)
        • don't hardcode /bin/bash in eb script, RPATH wrapper script, and run_shell_cmd (PR #4441)
          • use /usr/bin/env bash instead in eb + RPATH wrapper script
          • resolve path to bash via which in run_shell_cmd
          • Are there any use cases where you would want/need to not use the first bash in $PATH, but another bash?
            • Do we need to also add a configuration option to specify the path to bash to use?
      • enhancements
        • improve output produced by --check-github (PR #4437)
        • show error when multiple PR options are passed (PR #4440)
      • changes
        • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • use more performant and concise dict construction (PR #4410)
        • clean up & speed up environment checks (PR #4409)
      • EasyBuild 5.0 (to 5.0.x branch)
        • deprecate failure to resolve a template value (PR #3285)
        • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
        • replace parallel easyconfig parameter by maxparallel (PR #4398)
        • rename unclear run methods to install extensions (PR #4400)
          • needs more work, current approach of deprecating run methods & co is not working as intended...
        • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
          • requires that PR #4415 is merged first
        • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
          • requires that implementation of run_shell_cmd is complete, incl. support for asynchronously running commands and running interactive shell commands (WIP)
        • create lib/lib64 symlink before running postinstallcmds (PR #4435)
        • deprecate EnvironmentModulesC and EnvironmentModulesTcl (PR #4439)
        • support uninstalling software + module (issue #590)
        • clean up SLURM/Batchsystem environment before doing builds (issue #4434)
          • unset all $SLURM_* environment variables to avoid trouble with srun/MPI
    • easyblocks (open PRs + issues)
      • bug reports/fixes
        • fix extension filter for Perl packages (PR #2699)
          • easyconfigs PR #18789 is a big step towards merging this, but there are other easyconfigs that needs to be tested first, like XML-LibXML, worker, GD, ...
        • implementation of CMake policy error for Python (PR #3055)
          • see also this issue comment
          • ready for testing
          • will only be included in EasyBuild v5.0, since it's a change in behaviour (but should be harmless, right CMake?)
        • pick Tcl/Tk lib directory correctly in R easyblock, may be either lib or lib64 (PR #3072)
        • use (=|:=) in regex because OpenFOAM rule files use := instead of = since v2212 (PR #3073)
        • update PyTorch easyblock to avoid configure warnings by disabling some options (PR #3070)
      • enhancements
        • update KOKKOS_CPU_MAPPING for AArch64 in LAMMPS (PR #3036)
        • refactor GAMESS-US easyblock to directly write install.info (v2) (PR #3047)
        • Support GROMACS builds with SYCL (AdaptiveCpp implementation) (PR #3077)
      • updates
        • update Clang-AOMP easyblock for ROCm 5.6 (PR #2958)
      • new easyblocks
        • ...
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
          • needs some tests to make sure easyconfigs that are currently not setting use_pip yet are still working
            • that includes PyTorch; see also PR #3079 to disable use_pip for PyTorch < v2.1)
          • testing easyconfig that currently do not set use_pip yet has revealed some problems
            • any easyconfigs using CMakePythonPackage and using use_pip = True are broken, because PythonPackage changes installopts easyconfig parameter like it own's the place => being fixed in PR #3080
          • follow-up is needed in easyconfigs, to:
            • clean up easyconfigs in 5.0.x branch that now set use_pip & co to True (see easyconfigs PR #19265)
            • explicitly set use_pip & co to False in (non-archived) easyconfig that currently don't specify them at all and can't use pip for some reason (which should be a minimal set of easyconfigs)
        • update ConfigureMake easyblock to error out on unknown configure args (PR #3025 + #3026)
        • update PythonPackage easyblock to build wheel first when installing Python package with pip (PoC PR #3068](https://github.com/easybuilders/easybuild-easyblocks/pull/3068))
    • easyconfigs (open PRs + issues)
      • bug fixes/reports
        • ...
      • enhancements
        • ...
      • new software
        • ...
      • noteworthy software updates
        • Qt5 v5.15.11 w/ GCCcore/13.2.0 (for foss/2023b) (PR #19320)
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
      • closed PRs
        • ...

Clang-based toolchains

  • raised by Thomas, cfr. discussion in Slack
  • flang support in Clang makes it more interesting to have a Clang-based toolchain
  • two variants of Flang
    • LLVM-based flang-new (previously F18)
      • this is what you get when installing Clang >= 15.x with EasyBuild (binary named flang-new)
    • classic Fortran Clang compiler
      • based on PGI compilers
      • used by AMD (AOMP (CPU+GPU) + AOCC (CPU-only) + ROCm compilers (amd-clang, etc.)), Arm, etc. compiler suite
      • AOMP's Fortran compiler also included OpenMP offloading
      • current Clang-AOMP easyconfig PR produces flang binary (not flang-new)
    • Fortran mod files produced by these are not compatible
    • completely different compiler options
  • Fortran compiler in Intel oneAPI, NVHPC are a totally different frontend (with LLVM backend)
  • Markus has been playing with Clang-based toolchains for a while, but only for C/C++
  • current toolchains/compiler/clang.py can be updated to also support Fortran compiler (flang-new)
  • additional compiler support can be implemented in additional classes for AOMP, AOCC, etc so they can be used in EasyBuild toolchains
  • combining Clang (Fortran) compiler and Autotools/CMake can be a hassle...
  • battle plan
    • foss/2023b -> replace with GCC with Clang with flang-new included
      • naming? cofbf
      • version: 2023b
    • foss/2023b -> replace GCC with AOMP or ROCm compiler
      • aomp-ofbf
      • aocc-ofbf
      • rocm-ofbf
  • --include-toolchains is useful here, see http://docs.easybuild.io/including-additional-python-modules/#include_toolchains

Supporting OSs with OpenSSL 3.x

  • see also https://github.com/easybuilders/easybuild-easyconfigs/issues/19593
  • like RHEL9, Debian Bookworm (12.x), etc.
  • dealing with OpenSSL/1.1 wrapper
    • wraps around system OpenSSL 1.1.x if available
    • installs OpenSSL 1.1.x from source if not
    • used as dependency for cURL, Python, Perl, Rust, R, TensorFlow, wget,
    • we need to update openssl_wrapper.py easyblock to also support OpenSSL 3.0.x
      • so we can create an OpenSSL/3.0 wrapper
      • => already done by Sebastian a while ago, see also easyconfigs PR #15659
  • for which toolchains to we pick OpenSSL/3.0 instead of OpenSSL/1.1
  • OpenSSL/1.1 => OpenSSL/system
    • OS openssl 1.1 => OpenSSL/1.1
    • openssl 3 => OpenSSL/3.0 (or OpenSSL/3, cfr. their plan with using semantic versioning)
  • Jörg ran into trouble with recent Perl version when building on top of OpenSSL 3.x 
    undefined symbol: OSSL_PROVIDER_get0_name at
  • can test on jsc-zen3 that Sebastian has set up for testing easyconfig PRs
  • The Alliance is using OpenSSL 3.x in their new compat layer (with openssl-compat-1.1.1 also installed)
    • EESSI ran into trouble with OpenSSL 3.x, cfr.
  • openssl3-devel package is available for RHEL8 & friends
  • for testing, we can use the container images provided via https://github.com/easybuilders/easybuild-containers

Q&A / others

  • EasyBuild User Meeting 2024
  • how to update list of Ruby gems?
    • was probably done manually up until now...
    • talk to rubygems.org
    • could do update in a Ruby virtual environment
  • AFNI installing into $HOME/.local?!
Clone this wiki locally