Skip to content

Conference call notes 20170510

Jump to bottom
Kenneth Hoste edited this page May 10, 2017 · 1 revision

(back to Conference calls)

Notes on the 76th EasyBuild conference call, Wednesday May 10th 2017 (5pm - 6pm CET)

Attendees

Alphabetical list of attendees (6):

  • Damian Alvarez (JSC, Germany)
  • Fotis Georgatos (Illumina, UK)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Bart Oldeman (McGill University, Canada)
  • Alan O'Cais (JSC, Germany)
  • Davide Vanzo (Vanderbilt University)

Agenda

Notes

outlook to EasyBuild v3.2.1 (& v3.3.0)
Dealing with manually loaded modules before running EasyBuild
  • Python installation was failing for Davide because wrong ncurses was being picked up (from Core/ rather than Compiler/)
  • feedback on idea in https://github.com/hpcugent/easybuild-framework/issues/153#issuecomment-300244043
    • purge may be too intrusive, especially when certain modules are loaded like GC3Pie
    • purge+reload may load different modules, is that a concern?
    • order of modules has to be correct
  • default should be to purge any non-whitelisted modules, but that breaks backwards compatibility, strictly speaking
    • so just warn for now as default, and move to warn,purge for EasyBuild v4.0?
    • a purge may unload other modules that should be loaded
      • big problem on Cray systems, module purge is a back no-no there...
    • so, smarter way of unloading modules that shouldn't be there is needed?
      • unload anything not white-listed?
    • how do you detect that the set of modules loaded is not 'clean'
    • can we come up with a decent default whitelist for Cray PrgEnv?
Making checksums mandatory
  • soft enforcing of checksums via Travis style check done for contributions?
  • also helps with avoiding collisions on download
  • care will be need to be taken to avoid situations like happened with Homebrew/Handbrake:
  • won't this further block incoming contributions?
    • for all contributions, a 3rd party should 'vet' the checksums?
  • checksums are a great help with detecting corrupted downloads (due to upstream problems, corrupted files, ...)
  • also, which checksum mechanism should we use?
    • MD5 is no longer considered safe?
    • sha256 should be used now (cfr. Homebrew)?
  • verifying of checksum should be optional, should be disabled by request
  • enhance --new-pr to auto-inject checksums (if sources are available)?
  • Damian: implement this gradually, only enforce on new contributions for now?
  • change default to sha256?
    • sha256 is not supported yet now as checksum mechanism
    • Travis should also enforce using sha256 as checksums
    • can we auto-detect whether provided checksum is MD5 or SHA256?
    • e.g. just based on length?
  • also support --enforce-checksums to let EasyBuild enforce the presence of checksums
    • right now: no checksum, no problem...
EasyBuild/Singularity integration
  • example container for EasyBuild @ https://github.com/pescobar/singularity-easybuild
  • easiest way is to use EasyBuild in bootstrap file used to construct container
    • size, how to handle modules, etc.
    • module could be loaded in startup script, or via default module collection
    • modules tool is require anyway for EB
  • using MPI with containers is less trivial
    • using MPI with Singulary: mpirun singularity exec foo.img
    • MPI inside container needs to connect to MPI in host
    • could be done by binding MPI library from host into container
      • but what if container also includes an MPI library...
    • Alan: tuning of MPI is then still done outside container?
Other
Clone this wiki locally