feature: add a generic slurm cluster submit option #283
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM |
jaamarks
force-pushed
the
issue-268-generic-slurm
branch
from
24a6e48
to
d5b5cd2
Compare
- Incorrect `timedelta` handling caused Slurm job submission failures for workflows exceeding 24 hours. Python's `timedelta` format (e.g., "1 day, 1:00:00") was incompatible with Slurm. - Introduced `formatted_time` to convert `timedelta` objects to Slurm's required format (https://slurm.schedmd.com/sbatch.html#OPT_time). - This fix resolves errors caused by the incompatible time format in Slurm job submissions. - Applies the same time format fix for Slurm compatibility as commit af39a8d
Users specify the name of the Slurm parition (queue) within the config.yml file in the directory they submit from. The queue name provided will be the slurm parition that all non-local jobs are submitted to.
- Requires users of `--slurm` to specify the slurm partition in their config.yml file. - Removed the option to specify the slurm partition via the `--queue` when directly submitting with `--slurm`. - Renamed `--slurm-generic` flag to the simpler `--slurm`.
This commit improves the error message displayed when a user runs the `--slurm` option without specifying the required `slurm_partition` key-value pair in their `config.yml` file. The changes include: * Clearer error message highlighting the missing key-value pair and its purpose. * Color-coded output (if terminal supports) for better visual distinction between error and solution. These improvements make it easier for users to understand the error and take corrective action.
Users can now specify the desired Slurm partition in their config.yml when building with `cgr config --slurm-partition <partition_name>`. This partition is used for jobs submitted with `cgr submit --slurm`.
Add RTI International contributors to the author list.
- Add documentation about the `--slurm-partition` option for the
`cgr config` command.
- Add documentation about the `--slurm` option for the `cgr submit`
command.
- Fix typo in a warning message about sample sheet requirement
in `cgr_gwas_qc/cli/config.py`
`docs/getting_started/running_pipeline.rst` file.
- Fix typo in a message to Biowulf users in the
`docs/getting_started/running_pipeline.rst` file.
In commit bd74c72 we increased the `ibd_pi_hat_min` from 0.05 to 0.12 to focus on closer relatives. This commit aligns our documentation with that new default.
We have the option to submit to a generic slurm cluster profile now with `cgr submit --slurm`, so this commit updates the tests to incorporate this new feature.
jaamarks
force-pushed
the
issue-268-generic-slurm
branch
from
d5b5cd2
to
3beffc6
Compare
|
Fixes #230 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for submitting jobs to a Slurm cluster using the
cgr submit --slurmcommand. It leverages a generic Slurm cluster template provided within the codebase. It also also:Fixes #268