Skip to content

Commit

Permalink
Increase short addons timeouts
Browse files Browse the repository at this point in the history
In some addons we use short timeouts (60 seconds). These timeouts always
work, when they are not, like:

    drenv.commands.Error: Command failed:
       command: ('addons/ocm-cluster/test', 'dr2', 'hub')
       exitcode: 1
       error:
          Traceback (most recent call last):
            File "/home/nsoffer/src/ramen/test/addons/ocm-cluster/test", line 87, in <module>
              wait_for_deployment(cluster, hub)
            File "/home/nsoffer/src/ramen/test/addons/ocm-cluster/test", line 40, in wait_for_deployment
              kubectl.wait(
            File "/home/nsoffer/src/ramen/test/drenv/kubectl.py", line 90, in wait
              _watch("wait", *args, context=context, log=log)
            File "/home/nsoffer/src/ramen/test/drenv/kubectl.py", line 106, in _watch
              for line in commands.watch(*cmd, input=input):
            File "/home/nsoffer/src/ramen/test/drenv/commands.py", line 127, in watch
              raise Error(args, p.returncode, error)
          drenv.commands.Error: Command failed:
             command: ('kubectl', 'wait', '--context', 'dr2', 'deploy/example-deployment',
                 '--for=condition=available', '--timeout=60s')
             exitcode: 1
             error:
                error: timed out waiting for the condition on deployments/example-deployment

I think these timeouts starts to fail more now since we increased
concurrency when adding submariner addon, and the vms may be overloaded
during initial deployment.

Increase the 60 seconds timeouts to 120 seconds to avoid the random
failures.

We have some short timeouts in `test/basic-test/deploy`, but they never
failed, sine they don't run with overloaded clusters.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
  • Loading branch information
nirs committed Jul 9, 2023
1 parent de18cb4 commit 29f8b7b
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 7 deletions.
10 changes: 5 additions & 5 deletions test/addons/ocm-cluster/test
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ def wait_for_work(cluster, hub):
"manifestwork/example-manifestwork",
"--for=condition=applied",
f"--namespace={cluster}",
"--timeout=60s",
"--timeout=120s",
context=hub,
)

Expand All @@ -32,15 +32,15 @@ def wait_for_deployment(cluster, hub):
"manifestwork/example-manifestwork",
"--for=condition=available",
f"--namespace={cluster}",
"--timeout=60s",
"--timeout=120s",
context=hub,
)

print(f"Waiting until deployment is available in cluster '{cluster}'")
kubectl.wait(
"deploy/example-deployment",
"--for=condition=available",
"--timeout=60s",
"--timeout=120s",
context=cluster,
)

Expand All @@ -56,7 +56,7 @@ def wait_for_delete_work(cluster, hub):
"manifestwork/example-manifestwork",
"--for=delete",
f"--namespace={cluster}",
"--timeout=60s",
"--timeout=120s",
context=hub,
)

Expand All @@ -66,7 +66,7 @@ def wait_for_delete_deployment(cluster):
kubectl.wait(
"deploy/example-deployment",
"--for=delete",
"--timeout=60s",
"--timeout=120s",
context=cluster,
)

Expand Down
2 changes: 1 addition & 1 deletion test/addons/rook-pool/start
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ def wait(cluster):
"cephblockpool/replicapool",
output="jsonpath={.status.phase}",
namespace="rook-ceph",
timeout=60,
timeout=120,
profile=cluster,
)
kubectl.wait(
Expand Down
2 changes: 1 addition & 1 deletion test/addons/velero/test
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ def test(cluster):
"status",
"deploy/nginx-deployment",
"--namespace=nginx-example",
"--timeout=60s",
"--timeout=120s",
context=cluster,
)

Expand Down

0 comments on commit 29f8b7b

Please sign in to comment.