Singularity problems

[mflynn-lanl]

I've had problems with two of the clusters I've created. For one, Singularity was not installed. For the second, Singularity was installed but squashfuse was not. I was able to install it manually

[mflynn-lanl]

I was able to install squashfuse on the login node, but not on the compute nodes

[mflynn-lanl]

i added the path to the squashfuse to PATH in /etc/profile.d/custom.sh. When I start an interactive slurm job, I see that the path to the squashfuse directory is in the PATH. However, when I run the Cromwell workflow, I get the same error:

DEBUG   [U=60005,P=15715]  init()                        squashfuse mounting not enabled because: exec: "squashfuse": executable file not found in $PATH

If I launch the sbatch script manually, sbatch cromwell_4f0bd364_int.sl, I do not get the squashfuse error

[mflynn-lanl]

I created a second cluster with the same problem:

INFO:    squashfuse not found, will not be able to mount SIF
FATAL:   container creation failed: mount hook function failure: mount /proc/self/fd/11->/software/r9/apptainer/1.1.6/var/apptainer/mnt/session/rootfs error: while mounting image /proc/self/fd/11: squashfuse not found

[edwins]

Hi @mflynn-lanl ,

Could you tag my username, edwins, for conversations that need my attention? That would ensure I get notifications for future conversations.

As for singularity and other dependencies, I suggest creating a new image with all the tools you want into a base image (Alma9). You can simply launch as basic vm in any of the other JS2 UIs, and then create a "snapshot". By preinstalling your dependencies, this would eliminate the need in using one of the software stacks in the Magic Castle deployment wizard. Unfortunately, CACAO doesn't support creating individual snapshots at this time.

Here are links to instructions in creating snapshots using horizon, openstack cli, or exosphere:

https://docs.jetstream-cloud.org/ui/cli/snapshot-image/#snapshots-and-images
https://docs.jetstream-cloud.org/ui/horizon/manage/
https://docs.jetstream-cloud.org/getting-started/snapshots/

After your image is created, you should be able to select the image in the Boot image drop down.

I hope this helps.

Edwin

[mflynn-lanl]

Hi Edwin, I've tried creating snapshots of my image in the past but I haven't been able to create a VM the deploys successfully Mark

…

________________________________ From: Edwin Skidmore ***@***.***> Sent: Friday, July 19, 2024 1:50 PM To: microbiomedata/nmdc-edge ***@***.***> Cc: Flynn, Mark C ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [microbiomedata/nmdc-edge] Singularity problems (Discussion #258) Hi @mflynn-lanl<https://urldefense.com/v3/__https://github.com/mflynn-lanl__;!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1Nm65D5ly$> , Could you tag my username, edwins, for conversations that need my attention? That would ensure I get notifications for future conversations. As for singularity and other dependencies, I suggest creating a new image with all the tools you want into a base image (Alma9). You can simply launch as basic vm in any of the other JS2 UIs, and then create a "snapshot". By preinstalling your dependencies, this would eliminate the need in using one of the software stacks in the Magic Castle deployment wizard. Unfortunately, CACAO doesn't support creating individual snapshots at this time. Here are links to instructions in creating snapshots using horizon, openstack cli, or exosphere: https://docs.jetstream-cloud.org/ui/cli/snapshot-image/#snapshots-and-images<https://urldefense.com/v3/__https://docs.jetstream-cloud.org/ui/cli/snapshot-image/*snapshots-and-images__;Iw!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1Nu5w6urt$> https://docs.jetstream-cloud.org/ui/horizon/manage/<https://urldefense.com/v3/__https://docs.jetstream-cloud.org/ui/horizon/manage/__;!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1Nja8C_7k$> https://docs.jetstream-cloud.org/getting-started/snapshots/<https://urldefense.com/v3/__https://docs.jetstream-cloud.org/getting-started/snapshots/__;!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1NmBBzDI-$> After your image is created, you should be able to select the image in the Boot image drop down. I hope this helps. Edwin — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microbiomedata/nmdc-edge/discussions/258*discussioncomment-10098851__;Iw!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1Njtd7Zre$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEXIUF2XHMIS363E2OJZ76TZNFUXBAVCNFSM6AAAAABLDMBJN6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMBZHA4DKMI__;!!Bt8fGhp8LhKGRg!GljecYWgIYgvnrdIpCRW6HLAofVzWZhWradEXTnFZbdGzwNjYy5_m_xiajthkM_46psb6tIDTlaqbnI1NoFIWm37$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mflynn-lanl]

@edwins I'm creating a VM using Cacao openstack-single-image that will be used for the snapshot. Can I add user to that VM that will be used by the MC cluster? Do I need to set the UID and GID to what MC uses?

[edwins]

That is a great question. I suspect any system user will remain on that system once the snapshot is created. I would avoid any user or group (by name or id) that is created by magic castle as that might interfere with its operation.

[mflynn-lanl]

I'm worried that unless user is created by MC, that it won't have access to /project

[edwins]

As a clarification, you shouldn't add any users to your snapshot that might conflict with magic castle. Doing so, might prevent MC from working. In other words, I don't recommend adding a user account into snapshot or any user with the same uid.

[mflynn-lanl]

I've tried creating MC clusters using VM snapshots created from openstack, horizon and from exosphere, but all encountered an error during cluster creation

[edwins]

I can take a look. Just a couple of questions before I try to replicate the process:

Are you able to launch and login to a vm snapshot after creating it, using a simple launch (no magic castle)?

Other than installing software, are you making any other changes on the vm?

[mflynn-lanl]

I can ssh to the VM created in exosphere, but not using Cacao simple launch or horizon

…

________________________________ From: Edwin Skidmore ***@***.***> Sent: Monday, July 22, 2024 12:59 PM To: microbiomedata/nmdc-edge ***@***.***> Cc: Flynn, Mark C ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [microbiomedata/nmdc-edge] Singularity problems (Discussion #258) I can take a look. Just a couple of questions before I try to replicate the process: Are you able to launch and login to a vm snapshot after creating it, using a simple launch (no magic castle)? Other than installing software, are you making any other changes on the vm? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microbiomedata/nmdc-edge/discussions/258*discussioncomment-10118179__;Iw!!Bt8fGhp8LhKGRg!G8WyRl5xzqZ8xRebauOmL6wZa4PNYIPo-vuiJvbEydG8wSAmeooHa5_OCtq7jbh3iDUWRLQo9NOAG8TgTKiLLbnM$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEXIUF5W4NBS2UCFGUOZFLLZNVJC7AVCNFSM6AAAAABLDMBJN6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJRHAYTOOI__;!!Bt8fGhp8LhKGRg!G8WyRl5xzqZ8xRebauOmL6wZa4PNYIPo-vuiJvbEydG8wSAmeooHa5_OCtq7jbh3iDUWRLQo9NOAG8TgTH-8De_t$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mflynn-lanl]

I was able to create a MC cluster. However, when I tried running slurm commands I get [user@login1 jar]$ sinfo slurm_load_partitions: Unable to contact slurm controller (connect failure)

[mflynn-lanl]

I ssh'd to the management node and tried to re-start slurmctld but got an error. This is from the slurmctld.log:

[2024-07-24T20:42:23.958] accounting_storage/slurmdbd: init: Accounting storage SLURMDBD plugin loaded
[2024-07-24T20:42:23.958] error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:mgmt1:6819: Connection refused
[2024-07-24T20:42:23.958] error: Sending PersistInit msg: Connection refused
[2024-07-24T20:42:23.958] accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd
[2024-07-24T20:42:23.958] error: Sending PersistInit msg: Connection refused
[2024-07-24T20:42:23.958] error: Association database appears down, reading from state file.
[2024-07-24T20:42:23.958] debug:  create_mmap_buf: Failed to open file `/var/spool/slurm/last_tres`, No such file or directory
[2024-07-24T20:42:23.958] debug:  create_mmap_buf: Failed to open file `/var/spool/slurm/assoc_mgr_state`, No such file or directory
[2024-07-24T20:42:23.958] error: Unable to get any information from the state file
[2024-07-24T20:42:23.958] fatal: slurmdbd and/or database must be up at slurmctld start time
[2024-07-24T20:42:23.959] error: Sending PersistInit msg: Connection refused
[2024-07-24T20:42:23.959] slurmscriptd: debug:  _slurmscriptd_mainloop: finished

[edwins]

I'll try to reproduce from my allocation.

[mflynn-lanl]

[mcflynn@mgmt1 ~]$ mysql -uslurm -p slurm_acct_db --host mgmt1
Enter password:
ERROR 1130 (HY000): Host 'mgmt1.int.open-snap-cluster.mcb180107.jetstream-cloud.org' is not allowed to connect to this MySQL server

[edwins]

I can't reproduce that error with a snapshot I created and slurm seems to work for me. Would you be able to share your snapshot with me?

Here are instructions: https://docs.jetstream-cloud.org/ui/cli/snapshot-image/#sharing-an-image

I can't seem to find instructions on image sharing using Horizon or even Exosphere. If you're ok using the openstack cli, the project to share with is TRA220028.

Alternatively, you're welcome to me to your allocation for testing purposes.

[mflynn-lanl]

Hi Edwin, I added you to our allocation. Mark

…

________________________________ From: Edwin Skidmore ***@***.***> Sent: Wednesday, July 24, 2024 5:57 PM To: microbiomedata/nmdc-edge ***@***.***> Cc: Flynn, Mark C ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [microbiomedata/nmdc-edge] Singularity problems (Discussion #258) I can't reproduce that error with a snapshot I created and slurm seems to work. Would you be able to share your snapshot with me? Here are instructions: https://docs.jetstream-cloud.org/ui/cli/snapshot-image/#sharing-an-image<https://urldefense.com/v3/__https://docs.jetstream-cloud.org/ui/cli/snapshot-image/*sharing-an-image__;Iw!!Bt8fGhp8LhKGRg!CFWLsz6UknuQXX6QluEi7uL-OOGG-Dbd1HOBFNuRbyx5VM3URq5ImIB8Ejxn5cKkvlJU0H1HzdxBFo69rx4VmotU$> I can't seem to find instructions on image sharing using Horizon or even Exosphere. If you're find using the cli, the project to share with is TRA220028. Alternatively, you're welcome to me to your allocation for testing purposes. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microbiomedata/nmdc-edge/discussions/258*discussioncomment-10143261__;Iw!!Bt8fGhp8LhKGRg!CFWLsz6UknuQXX6QluEi7uL-OOGG-Dbd1HOBFNuRbyx5VM3URq5ImIB8Ejxn5cKkvlJU0H1HzdxBFo69rwbS510y$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEXIUFYWOYVQTP2VEXLBT7TZOA5N5AVCNFSM6AAAAABLDMBJN6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMJUGMZDMMI__;!!Bt8fGhp8LhKGRg!CFWLsz6UknuQXX6QluEi7uL-OOGG-Dbd1HOBFNuRbyx5VM3URq5ImIB8Ejxn5cKkvlJU0H1HzdxBFo69r5QqFJaL$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mflynn-lanl]

https://js2.jetstream-cloud.org/ngdetails/OS::Glance::Image/6a69b206-df76-4220-9eda-d87a0fe909c0

[mflynn-lanl]

I created a new MC cluster from a snapshot from a VM where I did not install anything. I just added that file /etc/cloud/cloud.cfg.d/10_js2_metadata.cfg and for this cluster, slurm commands worked. So it looks like something happens when I install the software on the VM

[mflynn-lanl]

I'm trying again with progressively more packages installed, first Java, then mysql

[mflynn-lanl]

If I install everything but MySQL, slurm works. Somehow installing MySQL messes up slurm

[edwins]

oops, it looks like my suspicions were correct (see the message I just posted below).

[mflynn-lanl]

sudo dnf install java-11-openjdk java-11-openjdk-devel -y && \
sudo dnf update -y && sudo dnf install mysql-server epel-release apptainer -y && sudo systemctl enable mysqld && sudo systemctl start mysqld && sudo mysql_secure_installation
mysql -uroot -p -e "create database cromwell;CREATE USER 'cromwell'@'localhost' IDENTIFIED BY '******';GRANT ALL PRIVILEGES ON cromwell.* TO 'cromwell'@'localhost' WITH GRANT OPTION;FLUSH PRIVILEGES;"
sudo bash -c  "echo 'export APPTAINER_CACHEDIR=/project/apptainer/cache' > /etc/profile.d/custom.sh"

[edwins]

Sorry, I misinformed you earlier. You'll need to ensure that the firewall/security group attached to the mysql host has tcp/3306 opened i.e. from the login node to the mysql host. You can probably just open it up to the subnet.

[mflynn-lanl]

I needed to restart the mysql VM and now it works!

[mflynn-lanl]

Cromwell can also connect to MySQL running on the VM so you don't need to do anything so that the login node and management nodes have different software installed during cluster creation

[edwins]

cool, i'm glad you got that working

[mflynn-lanl]

I'm no longer able to ssh to MC cluster login node for any of the clusters I created today, even though I could ssh to ones I created yesterday. For example: deployment-cqkq0j9ogabsfg6erl2g
https://cacao.jetstream-cloud.org/deployments/deployment-cqkq0j9ogabsfg6erl2g

[mflynn-lanl]

(base) mflynn@pn2294255 MIGDAL_simulator % ssh -vvv mcflynn@149.165.154.115
OpenSSH_9.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/mflynn/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug2: resolve_canonicalize: hostname 149.165.154.115 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/mflynn/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/mflynn/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to 149.165.154.115 [149.165.154.115] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48

[mflynn-lanl]

I think the problem was the length of the name of the cluster. I tried with a shorter name and now I can ssh to the MC cluster