Skip to content

Commit

Permalink
Draft cluster snakefile stuff.
Browse files Browse the repository at this point in the history
The "Snakefile_cluster_iteration" file manages the list of
widths, and can iterate over jobs on the cluster and then
do a (trivial) aggregation step.

The queuing_config.yaml file is a short cluster config that
removes the need to type the cluster stuff into the command
line.  It conventionally goes in
$HOME/.config/snakemake/queuing/config.yaml, and is accessed
by "snakemake <other-args> --profile=queuing".

The Snakefile has mpi stuff for the trivial aggregation rule
because the cluster-ness of it applies to all the rules.
This is clumsy, but not wrong.
  • Loading branch information
reid-a committed Oct 23, 2023
1 parent c6ba6d1 commit 2564dc3
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 0 deletions.
24 changes: 24 additions & 0 deletions episodes/files/Snakefile_cluster_iteration
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#
# Run a bunch of Amdahl jobs and aggregate the output.
#
WIDTHS=[1,2]
#
def getwidth(wildcards):
return wildcards.sample

rule plot:
input: expand('{size}.out',size=WIDTHS)
output: 'done.out'
resources:
mpi="mpirun",
tasks=1
shell: 'echo "{WIDTHS}, Done!" > done.out'
rule iterate:
input:
output: '{sample}.out'
resources:
mpi="mpirun",
tasks=getwidth
shell:
"module load OpenMPI; mpirun -np {resources.tasks} amdahl > {wildcards.sample}.out"

6 changes: 6 additions & 0 deletions episodes/files/queuing_config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# snakemake -j 3 --cluster "sbatch -N 1 -n {resources.tasks} -p node"
cluster:
sbatch
--partition=node
--nodes=1
--tasks={resources.tasks}

0 comments on commit 2564dc3

Please sign in to comment.