Running Cromwell as a systemd process

[mflynn-lanl]

@edwins I am able to run Cromwell as a systemd process, however only as user mcflynn. When I do this, Cromwell is not able to submit slurm jobs. I tried using sudo visudo and adding %user ALL=NOPASSWD: /usr/bin/systemctl start cromwell.service based on the advice from https://unix.stackexchange.com/questions/672206/how-to-run-a-systemctl-command-without-sudo
but that didn't work. It still prompts me for the mcflynn password when I try running systemctl start cromwell.service

[edwins]

Are you able to run as mcflynn_slurm?

[mflynn-lanl]

i tried using mcflynn_slurm but I still get the same error
/scratch/mcflynn_slurm/cromwell/cromwell-executions/main_workflow/bad628b1-49b3-4b69-9843-af3718340da9/call-jgi_metaASM/MetaAssembly.nmdc_edge_assembly/488b3d1a-cf29-4abc-8af5-e1d2561a7df2/call-preprocess/MetaAssembly_preprocess.preprocess/af2de7ca-6654-4e1c-b010-d0bc725159ec/call-gzip_int/execution/script.submit: line 36: sbatch: command not found
I still need to start it as user mcflynn. when I try starting the systemd process as mcflynn_slurm it asks for a password

here's the systemd file:
`[Unit]
Description=Cromwell Workflow Engine service
After=network.target

[Service]
User=mcflynn_slurm
Group=mcflynn_slurm
Restart=always
TimeoutStopSec=10
RestartSec=30
WorkingDirectory=/project/cromwell
Environment=LOG_MODE=standard
Environment=APPTAINER_CACHEDIR=/project/apptainer/cache
ExecStart=/usr/bin/java -Dconfig.file=/project/def-sponsor00/mcflynn_slurm/cromwell.conf -Dsystem.job-shell=/bin/bash -Dwebservice.port=8002 -jar /project/cromwell/jar/cromwell-86.jar server
ProtectSystem=full
PrivateTmp=true

[Install]
WantedBy=multi-user.target`

[edwins]

Can you set PATH to include /opt/software/slurm/bin? This is where sbatch and other slurm utilities are located.

[edwin_slurm@login1 ~]$ which sbatch
/opt/software/slurm/bin/sbatch

[mflynn-lanl]

I added:
export PATH=$PATH:/opt/software/slurm/bin/sbatch
to /etc/profile.d/custom.sh file and I got it working! Then I modified the systemd file to change the location of the cromwell.conf file and the port. After reloading the daemon and restarting the systemd process, I got the same sbatch not found error. I didn't think the changes should have made a difference but I changed the systemd file back and I was correct.
I verified that the changes to the PATH were there:
[mcflynn_slurm@login1 ~]$ echo $PATH /home/mcflynn_slurm/.local/bin:/home/mcflynn_slurm/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/openmpi/4.0.3/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/bin:/cvmfs/soft.computecanada.ca/easybuild/bin:/cvmfs/soft.computecanada.ca/custom/bin:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/sbin:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin:/cvmfs/soft.computecanada.ca/gentoo/2020/sbin:/cvmfs/soft.computecanada.ca/gentoo/2020/bin:/opt/software/slurm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/software/slurm/bin/sbatch:/opt/puppetlabs/bin

the path to sbatch in the PATH environment variable matches the location on the file system:
[mcflynn_slurm@login1 ~]$ which sbatch /opt/software/slurm/bin/sbatch