can't successfully run kfp_v2 uats behind proxy #78
Labels
bug
Something isn't working
Comments
|
Thank you for reporting us your feedback! The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5957.
|
|
Hi @nishant-dash ,
In the meantime, we are prioritizing this issue and will try to reproduce it. |
|
for
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[14], line 4
1 # fetch KFP experiment to ensure it exists
2 client.get_experiment(experiment_name=EXPERIMENT_NAME)
----> 4 assert_run_succeeded(client, run.run_id)
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:336, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
334 copy = self.copy()
335 wrapped_f.statistics = copy.statistics # type: ignore[attr-defined]
--> 336 return copy(f, *args, **kw)
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:475, in Retrying.__call__(self, fn, *args, **kwargs)
473 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
474 while True:
--> 475 do = self.iter(retry_state=retry_state)
476 if isinstance(do, DoAttempt):
477 try:
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:376, in BaseRetrying.iter(self, retry_state)
374 result = None
375 for action in self.iter_state.actions:
--> 376 result = action(retry_state)
377 return result
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:418, in BaseRetrying._post_stop_check_actions.<locals>.exc_check(rs)
416 retry_exc = self.retry_error_cls(fut)
417 if self.reraise:
--> 418 raise retry_exc.reraise()
419 raise retry_exc from fut.exception()
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:185, in RetryError.reraise(self)
183 def reraise(self) -> t.NoReturn:
184 if self.last_attempt.failed:
--> 185 raise self.last_attempt.result()
186 raise self
File /opt/conda/lib/python3.11/concurrent/futures/_base.py:449, in Future.result(self, timeout)
447 raise CancelledError()
448 elif self._state == FINISHED:
--> 449 return self.__get_result()
451 self._condition.wait(timeout)
453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
File /opt/conda/lib/python3.11/concurrent/futures/_base.py:401, in Future.__get_result(self)
399 if self._exception:
400 try:
--> 401 raise self._exception
402 finally:
403 # Break a reference cycle with the exception in self._exception
404 self = None
File /opt/conda/lib/python3.11/site-packages/tenacity/__init__.py:478, in Retrying.__call__(self, fn, *args, **kwargs)
476 if isinstance(do, DoAttempt):
477 try:
--> 478 result = fn(*args, **kwargs)
479 except BaseException: # noqa: B902
480 retry_state.set_exception(sys.exc_info()) # type: ignore[arg-type]
Cell In[13], line 9, in assert_run_succeeded(client, run_id)
7 """Wait for the run to complete successfully."""
8 status = client.get_run(run_id).state
----> 9 assert status == "SUCCEEDED", f"KFP run in {status} state."
AssertionError: KFP run in RUNNING state.
Network is unreachable')': /simple/kfp/ |
|
@nishant-dash also for reproducing the issue, could you please tell us which method for running the UATs are you using? It was not clear from the issue description. |
|
I got access from @nishant-dash to the env in question and was able to investigate It is clear in the logs that the container was not able to install The The default base image for pipeline runners does not have |
|
Based on my comment above, it is required in a proxy environment to configure the pipeline to have the proxy environment variables. I was able to do this and run a pipeline successfully behind proxy by modifying the kfp v2 UATs notebook as follows:
@nishant-dash can you try the above and confirm it fixes the issue for you? |
|
Discussed with @nishant-dash |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Description
running kfp_v2 integration test from https://github.com/canonical/charmed-kubeflow-uats/tree/main/tests, commit [0] experiment fails on
To Reproduce
In a kf 1.8 env behind proxy, run the kfp_v2 uats
Environment
Relevant Log Output
Additional Context
No response
The text was updated successfully, but these errors were encountered: