Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CPU and RAM requirements specified by the user for dask workers #830

Closed
Alputer opened this issue Sep 6, 2024 · 1 comment
Closed
Assignees

Comments

@Alputer
Copy link
Member

Alputer commented Sep 6, 2024

Currently, dask cluster is built using a fixed autoscaler and worker blueprints. We need to specify 2 parameters:

  1. CPU and RAM capacity of the workers
  2. Maximum amount of workers in dask autoscaler

Also, we need to decide on how we share the configuration between these 2 options.

  1. We can have heavy workers with fewer number of maximum amount of dask workers
    --- OR ----
  2. Light workers with a higher number of maximum amount of dask workers

We are also facing a possibly major problem in this task, because dask autoscaler does not seem to obey the maximum number of workers rule. In a few compute-intensive tasks, I have observed it exceeds the limit not only by a few workers but even more. If this issue cannot be fixed at all, we might even need to stop using dask autoscaler completely or implement our patched auto-scaling logic.

@Alputer Alputer self-assigned this Sep 6, 2024
@Alputer
Copy link
Member Author

Alputer commented Sep 20, 2024

This issue is already included in the first set of dask integration pr set whose links are given below. Therefore closing the issue.

#821
reanahub/reana-server#701
reanahub/reana-workflow-controller#600
reanahub/reana-job-controller#464
reanahub/reana-commons#467

@Alputer Alputer closed this as completed Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

1 participant