Skip to content

Commit

Permalink
Fix issue when running
Browse files Browse the repository at this point in the history
    get_mean_squared_value = SubmitRayJob(
        task_id="SubmitRayJob",
        conn_id=CONN_ID,
        entrypoint="python ray_script.py {{ ti.xcom_pull(task_ids='generate_data') | join(' ') }}",
        runtime_env=RAY_RUNTIME_ENV,
        num_cpus=1,
        num_gpus=0,
        memory=0,
        resources={},
        xcom_task_key="SubmitRayJob.dashboard",
        fetch_logs=True,
        wait_for_completion=True,
        job_timeout_seconds=0,
        poll_interval=5,
    )

That resulted

[2024-09-27, 10:55:06 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-09-27, 10:55:06 UTC] {ray.py:219} INFO - Dashboard URL retrieved from XCom: None
[2024-09-27, 10:55:06 UTC] {base.py:84} INFO - Retrieving connection 'ray_conn_2'
[2024-09-27, 10:55:06 UTC] {ray.py:87} INFO - Ray cluster address is: http://172.23.0.3:30487
[2024-09-27, 10:55:06 UTC] {ray.py:155} INFO - Address URL is: http://172.23.0.3:30487
[2024-09-27, 10:55:06 UTC] {ray.py:156} INFO - Dashboard URL is: None
[2024-09-27, 10:55:06 UTC] {ray.py:183} INFO - Submitted job with ID: raysubmit_G22MqPHyvLv8ghRV
[2024-09-27, 10:55:06 UTC] {ray.py:278} INFO - Ray job submitted with id: raysubmit_G22MqPHyvLv8ghRV
[2024-09-27, 10:55:06 UTC] {ray.py:208} INFO - Job raysubmit_G22MqPHyvLv8ghRV status: PENDING
[2024-09-27, 10:55:06 UTC] {ray.py:282} INFO - Current job status for raysubmit_G22MqPHyvLv8ghRV is: PENDING
[2024-09-27, 10:55:06 UTC] {ray.py:290} INFO - Deferring the polling to RayJobTrigger...
[2024-09-27, 10:55:06 UTC] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/ray_provider/operators/ray.py", line 302, in execute
    timeout=job_timeout_seconds,
            ^^^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'job_timeout_seconds' where it is not associated with a value
During handling of the above exception, another exception occurred:
  • Loading branch information
tatiana committed Sep 27, 2024
1 parent 866756d commit 682b5d4
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions ray_provider/operators/ray.py
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ def execute(self, context: Context) -> str:
current_status = self.hook.get_ray_job_status(self.dashboard_url, self.job_id)
self.log.info(f"Current job status for {self.job_id} is: {current_status}")

job_timeout_seconds = timedelta(seconds=0)
if isinstance(self.job_timeout_seconds, timedelta):
job_timeout_seconds = self.job_timeout_seconds
elif self.job_timeout_seconds is not None:
Expand Down

0 comments on commit 682b5d4

Please sign in to comment.