Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RAY] timeout mechanism for job length or long queues #45

Open
ibm-peach-fish opened this issue Oct 24, 2023 · 1 comment · May be fixed by #46
Open

[RAY] timeout mechanism for job length or long queues #45

ibm-peach-fish opened this issue Oct 24, 2023 · 1 comment · May be fixed by #46
Assignees
@ibm-peach-fish
Labels
enhancement New feature or request

Comments

@ibm-peach-fish
Copy link
Collaborator

reflected from private internal issue tracker by @chakrn

When we use caikit-ray-backend to submit a new job, that job can run indefinitely with no timeout. We need a way to have a configurable timeout value and cancel the job after exceeding the time.

After having a quick discussion with Dean, this can probably be done simply by changing the ray.get() to a ray.wait() (which should've been the case anyway). Then poll for status and kill the job after a certain elapsed time.
@ibm-peach-fish ibm-peach-fish self-assigned this Oct 24, 2023
@ibm-peach-fish ibm-peach-fish added the enhancement New feature or request label Oct 24, 2023
@ibm-peach-fish ibm-peach-fish moved this from ToDo to In Progress in caikit ecosystem Oct 24, 2023
@gkumbhat
Copy link

Thanks @ibm-peach-fish

Clarification on following:

Then poll for status and kill the job after a certain elapsed time.

We need to make the time for timeout configurable. We will be using get_config function from caikit for this, example import statement and example usage. This allows configuration of these variables in multiple fashions, i.e one can configure it via config.yml or can set a value via environment variable.

ibm-peach-fish added a commit to ibm-peach-fish/caikit-ray-backend that referenced this issue Oct 24, 2023
@ibm-peach-fish
caikit#45 add timeout to train job
ac3a237 
Signed-off-by: Trevor Grant <trevor.grant@ibm.com>
@ibm-peach-fish ibm-peach-fish linked a pull request Oct 24, 2023 that will close this issue
Open
ibm-peach-fish added a commit to ibm-peach-fish/caikit-ray-backend that referenced this issue Oct 27, 2023
@ibm-peach-fish
Fixes caikit#45
4c894a7 
Signed-off-by: Trevor Grant <trevor.grant@ibm.com>
@chakrn chakrn moved this from In Progress to Ready for Review in caikit ecosystem Oct 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ibm-peach-fish ibm-peach-fish

Labels
enhancement New feature or request
Projects
caikit ecosystem
Status: Ready for Review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#45 add timeout to train job ibm-peach-fish/caikit-ray-backend
2 participants
@gkumbhat @ibm-peach-fish