Error running singularity containers

[arrabito]

Dear all,

Our users have troubles in running jobs with singularity containers.

This happens for every kind of image and on every node. It seems to be related to some missing library in DIRACOS.

Here below the error message:
/scratch/local/3Y1LDmQdQT1nLZfWYmas2NHqEbqlNmABFKDmMEFKDmuBFKDmUEkACo/DIRAC_a8j36cpilot/diracos/bin/unsquashfs: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

generated by this simple command line:

singularity exec --no-home docker://centos:7 echo "Hello world!"

A solution is ti use unpacked images pre-installed somewhere, e.g. CVMFS, but for testing purpose this is not practical.

Could you please suggest a proper solution?

Thank you,

Luisa

[sfayer]

Hi Luisa,

We had a similar issue in GridPP this week: I think there is an incompatibility between the way singularity is packaged in DIRACOS2 and the way singularity processes the dependencies for unsquashfs[*].

We generally advise people to use unpacked versions as you mention (you can just create a tar file of an unpacked directory and untar it in a test job).

A quick workaround for testing is to reset the path, although this requires the machine to have mksquashfs and unsquashfs installed locally:

$ PATH="/usr/bin:/usr/sbin" $(command -v singularity)  exec --no-home docker://centos:7 echo "Hello world"
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
INFO:    Creating SIF file...
INFO:    Converting SIF file to temporary sandbox...
Hello world
INFO:    Cleaning up image...

Regards,
Simon

[*] This appears to be caused by the relative RPATH in the unsquashfs executable: singularity reads the library paths and runs unsquashfs in a temporary container with bind mounts for all of the libraries needed. With the relative library paths, the calculated bind mounts contain "/../" which don't seem to work properly and fail, so you get missing libraries. I'll put in an issue with singularity/apptainer and see if this is something that can be fixed properly.

[sfayer]

Hi,

I did some more testing of this and it seems like the 3.8.7 version of singularity is now tagged as broken in conda-forge:
https://anaconda.org/conda-forge/singularity/files
This seems to go against my RPATH theory: The old version also uses relative paths without issue, so I'm not sure what the difference is.

It looks to me like 3.8.6 version works with sif files...

(base) $ micromamba install singularity==3.8.6
(base) $ singularity exec --no-home docker://centos:7 echo "Hello world"
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
Hello world
INFO:    Cleaning up image...

I don't know if it'd be possible for there to be a new DIRACOS2 release with the previous non-broken singularity version @chrisburr ?

Regards,
Simon

[chrisburr]

The DIRACOS2 release was made so I think this should be fine.

In the near term we'll need to move to using Apptainer so the issues can be fixed as part of that. I'll also extend the DIRACOS2 tests to help catch issues before release. See DIRACGrid/DIRACOS2#80.

[sfayer]

I've tested the latest version of DIRACOS2 and it contains singularity 3.8.6, so this should all be fixed now...

Regards,
Simon