Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BREAKING: Remove the LSF functionality from this package (and point users to LSFClusterManagers.jl instead) #231

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ Implemented in this package (the `ClusterManagers.jl` package):

| Job queue system | Command to add processors |
| ---------------- | ------------------------- |
| Load Sharing Facility (LSF) | `addprocs_lsf(np::Integer; bsub_flags=``, ssh_cmd=``)` or `addprocs(LSFManager(np, bsub_flags, ssh_cmd, retry_delays, throttle))` |
| Sun Grid Engine (SGE) via `qsub` | `addprocs_sge(np::Integer; qsub_flags=``)` or `addprocs(SGEManager(np, qsub_flags))` |
| Sun Grid Engine (SGE) via `qrsh` | `addprocs_qrsh(np::Integer; qsub_flags=``)` or `addprocs(QRSHManager(np, qsub_flags))` |
| PBS (Portable Batch System) | `addprocs_pbs(np::Integer; qsub_flags=``)` or `addprocs(PBSManager(np, qsub_flags))` |
Expand All @@ -32,6 +31,7 @@ Implemented in external packages:
| ---------------- | ------------------------- |
| Kubernetes (K8s) via [K8sClusterManagers.jl](https://github.com/beacon-biosignals/K8sClusterManagers.jl) | `addprocs(K8sClusterManager(np; kwargs...))` |
| Azure scale-sets via [AzManagers.jl](https://github.com/ChevronETC/AzManagers.jl) | `addprocs(vmtemplate, n; kwargs...)` |
| Load Sharing Facility (LSF) via [LSFClusterManagers.jl](https://github.com/JuliaParallel/LSFClusterManagers.jl) | `addprocs_lsf(np::Integer; bsub_flags=``, ssh_cmd=``)` or `addprocs(LSFManager(np, bsub_flags, ssh_cmd, retry_delays, throttle))` |

You can also write your own custom cluster manager; see the instructions in the [Julia manual](https://docs.julialang.org/en/v1/manual/distributed-computing/#ClusterManagers).

Expand Down Expand Up @@ -93,7 +93,7 @@ julia> From worker 2: compute-6
From worker 3: compute-6
```

Some clusters require the user to specify a list of required resources.
Some clusters require the user to specify a list of required resources.
For example, it may be necessary to specify how much memory will be needed by the job - see this [issue](https://github.com/JuliaLang/julia/issues/10390).
The keyword `qsub_flags` can be used to specify these and other options.
Additionally the keyword `wd` can be used to specify the working directory (which defaults to `ENV["HOME"]`).
Expand Down Expand Up @@ -132,11 +132,6 @@ Julia workers can frequently timeout waiting for the standard output files to ap
In this case, it's better to use the `QRSHManager`, which uses SGE's `qrsh`
command to bypass the filesystem and captures STDOUT directly.

### Load Sharing Facility (LSF)

`LSFManager` supports IBM's scheduler. See the `addprocs_lsf` docstring
for more information.

### Using `LocalAffinityManager` (for pinning local workers to specific cores)

- Linux only feature.
Expand Down Expand Up @@ -176,10 +171,10 @@ ElasticManager:
Active workers : []
Number of workers to be added : 0
Terminated workers : []
Worker connect command :
Worker connect command :
/home/user/bin/julia --project=/home/user/myproject/Project.toml -e 'using ClusterManagers; ClusterManagers.elastic_worker("4cOSyaYpgSl6BC0C","127.0.1.1",36275)'
```

By default, the printed command uses the absolute path to the current Julia executable and activates the same project as the current session. You can change either of these defaults by passing `printing_kwargs=(absolute_exename=false, same_project=false))` to the first form of the `ElasticManager` constructor.
By default, the printed command uses the absolute path to the current Julia executable and activates the same project as the current session. You can change either of these defaults by passing `printing_kwargs=(absolute_exename=false, same_project=false))` to the first form of the `ElasticManager` constructor.

Once workers are connected, you can print the `em` object again to see them added to the list of active workers.
Once workers are connected, you can print the `em` object again to see them added to the list of active workers.
1 change: 0 additions & 1 deletion src/ClusterManagers.jl
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,5 @@ include("condor.jl")
include("slurm.jl")
include("affinity.jl")
include("elastic.jl")
include("lsf.jl")

end
164 changes: 0 additions & 164 deletions src/lsf.jl

This file was deleted.

10 changes: 0 additions & 10 deletions test/lsf.jl

This file was deleted.

20 changes: 2 additions & 18 deletions test/runtests.jl
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ using Test: @testset, @test, @test_skip
using ClusterManagers: ElasticManager
# Slurm:
using ClusterManagers: addprocs_slurm, SlurmManager
# LSF:
using ClusterManagers: addprocs_lsf, LSFManager
# SGE:
using ClusterManagers: addprocs_sge, SGEManager

Expand All @@ -23,12 +21,11 @@ const test_args = lowercase.(strip.(ARGS))
@info "" test_args

slurm_is_installed() = !isnothing(Sys.which("sbatch"))
lsf_is_installed() = !isnothing(Sys.which("bsub"))
qsub_is_installed() = !isnothing(Sys.which("qsub"))

@testset "ClusterManagers.jl" begin
include("elastic.jl")

if slurm_is_installed()
@info "Running the Slurm tests..." Sys.which("sbatch")
include("slurm.jl")
Expand All @@ -41,20 +38,7 @@ qsub_is_installed() = !isnothing(Sys.which("qsub"))
@test_skip false
end
end

if lsf_is_installed()
@info "Running the LSF tests..." Sys.which("bsub")
include("lsf.jl")
else
if "lsf" in test_args
@error "ERROR: The LSF tests were explicitly requested in ARGS, but bsub was not found, so the LSF tests cannot be run" Sys.which("bsub") test_args
@test false
else
@warn "bsub was not found - LSF tests will be skipped" Sys.which("bsub")
@test_skip false
end
end


if qsub_is_installed()
@info "Running the SGE (via qsub) tests..." Sys.which("qsub")
include("slurm.jl")
Expand Down
Loading