Integrate submariner in test environment #917
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
nirs
force-pushed
the
submariner
branch
4 times, most recently
from
657a719
to
3898b81
Compare
nirs
force-pushed
the
submariner
branch
2 times, most recently
from
636ebf2
to
e1e32d3
Compare
|
Testing with minio by adding minio addons to submariner clusters: Tested minor with mc as described here: |
|
Tested submariner with ocm by adding it before ocm-hub and cluster: |
|
Testing submariner with rook using this patch: rbd mirror secrets looks the same: |
|
Testing with regional-dr: Starting environment: Deploying ramen and running basic test: Following DRRP during the test: |
nirs
requested review from
raghavendra-talur,
rakeshgm,
ShyamsundarR and
ELENAGER
nirs
force-pushed
the
submariner
branch
2 times, most recently
from
bebccb7
to
e95c165
Compare
We had 60 seconds timeout when waiting until the rook cluster is created. This tends to fail when the system is overloaded or when using bad network (e.g. Cellular hotspot). Change to use the default timeout (300 seconds). Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Like other tools, a developer need to install it manually. We can improve this later to install the tools automatically. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Using temporary script to deploy submariner until we have a proper addon. All submariner tests pass but they take 560 seconds, way to slow to use as a self test. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
New options: - `service_cluster_ip_range`: Add `--service-cluster-ip-range` argument to `minikube start` command. - `extra_config`: Add `--extra-config` arguments to `minikube start` command. This allows changing the pod network CIDR. The options can be used to avoid overlapping CIDRs when using submariner without globalnet. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Submariner broker should be deployed on a cluster which is accessible to both managed clusters. The deployment includes the submariner-operator, but if I understand correctly, the other clusters use k8s API to access the submariner configuration. Deploying on the hub make more sense both for production (clusters do not depend on other clusters), but it also simplifies the deployment with drenv: we can deploy all clusters concurrently. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Like other tools we depend on, we want a wrapper to make using the tool easier and less error prone. In this case the tool provide a missing feature, storing the broker-info.subm file in a different location. Signed-off-by: Nir Soffer <nsoffer@redhat.com>
The submariner addon accepts 2 arguments:
addons/submariner/start cluster broker
If the cluster name is equal to the broker name, the cluster is is used
as the broker, and we deploy the submariner broker on the cluster. The
broker-info.subm file is stored in the broker cluster config directory
(`~/.config/drenv/hub/submariner/broker-info.subm`).
If the cluster is not the broker, wait until the submariner broker info
file is available and use it to join the cluster to the broker.
The hub cluster is used as the broker, and the managed clusters are
joined to the broker.
The deploy-submariner.sh script is not needed now.
Example run with the submariner development environment:
$ drenv start envs/submariner.yaml
2023-06-14 01:05:53,965 INFO [submariner] Starting environment
2023-06-14 01:05:53,998 INFO [hub] Starting minikube cluster
2023-06-14 01:05:55,082 INFO [dr1] Starting minikube cluster
2023-06-14 01:05:56,031 INFO [dr2] Starting minikube cluster
2023-06-14 01:06:36,614 INFO [hub] Cluster started in 42.62 seconds
2023-06-14 01:06:36,614 INFO [hub/0] Running addons/submariner/start
2023-06-14 01:07:01,921 INFO [hub/0] addons/submariner/start completed in 25.31 seconds
2023-06-14 01:07:01,988 INFO [dr1] Cluster started in 66.91 seconds
2023-06-14 01:07:01,988 INFO [dr1/0] Running addons/submariner/start
2023-06-14 01:07:23,860 INFO [dr2] Cluster started in 87.83 seconds
2023-06-14 01:07:23,861 INFO [dr2/0] Running addons/submariner/start
2023-06-14 01:07:53,647 INFO [dr1/0] addons/submariner/start completed in 51.66 seconds
2023-06-14 01:08:14,601 INFO [dr2/0] addons/submariner/start completed in 50.74 seconds
2023-06-14 01:08:14,602 INFO [submariner] Environment started in 140.64 seconds
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Submariner has `subctl verify`, running the entire submariner end to end
tests on the cluster. This takes about 10 minutes so we cannot use it as
a self test.
Add a quick self test using
subctl show all --context cluster
The command fails when some component is missing, or when clusters are
not connected. Typically it takes few seconds until the command
succeeds, so we run it in a loop until the command succeeds. If the
commands did not succeed in 60 seconds we fail the test.
We don't have a good way to tell if the command failed because we used a
wrong argument or option. In the worst case a bad change in the code
will cause a 60 seconds delay until we discover the issue.
Example run:
2023-06-13 01:08:55,723 INFO [dr1/0] Running addons/submariner/test
2023-06-13 01:08:55,815 DEBUG [dr1/0] Waiting until subctl status is ok
2023-06-13 01:08:55,815 DEBUG [dr1/0] Checking subctl status...
2023-06-13 01:08:55,898 DEBUG [dr1/0] subctl failed with exitcode 1
2023-06-13 01:08:55,898 DEBUG [dr1/0] Retrying in 1 seconds...
2023-06-13 01:08:56,898 DEBUG [dr1/0] Checking subctl status...
2023-06-13 01:08:56,965 DEBUG [dr1/0] subctl failed with exitcode 1
2023-06-13 01:08:56,965 DEBUG [dr1/0] Retrying in 2 seconds...
2023-06-13 01:08:58,965 DEBUG [dr1/0] Checking subctl status...
2023-06-13 01:08:59,043 DEBUG [dr1/0]
2023-06-13 01:08:59,043 DEBUG [dr1/0] GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
2023-06-13 01:08:59,043 DEBUG [dr1/0] dr2 dr2 192.168.122.116 no vxlan 242.1.0.0/16 connected
2023-06-13 01:08:59,043 DEBUG [dr1/0]
2023-06-13 01:08:59,043 DEBUG [dr1/0] CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
2023-06-13 01:08:59,043 DEBUG [dr1/0] dr1 192.168.122.142 46.117.231.61 vxlan local
2023-06-13 01:08:59,043 DEBUG [dr1/0] dr2 192.168.122.116 46.117.231.61 vxlan remote
2023-06-13 01:08:59,043 DEBUG [dr1/0]
2023-06-13 01:08:59,043 DEBUG [dr1/0] NODE HA STATUS SUMMARY
2023-06-13 01:08:59,043 DEBUG [dr1/0] dr1 active All connections (1) are established
2023-06-13 01:08:59,043 DEBUG [dr1/0]
2023-06-13 01:08:59,043 DEBUG [dr1/0] Discovered network details via Submariner:
2023-06-13 01:08:59,043 DEBUG [dr1/0] Network plugin: kindnet
2023-06-13 01:08:59,043 DEBUG [dr1/0] Service CIDRs: [10.96.0.0/12]
2023-06-13 01:08:59,043 DEBUG [dr1/0] Cluster CIDRs: [10.244.0.0/16]
2023-06-13 01:08:59,043 DEBUG [dr1/0] Global CIDR: 242.0.0.0/16
2023-06-13 01:08:59,043 DEBUG [dr1/0]
2023-06-13 01:08:59,043 DEBUG [dr1/0] COMPONENT REPOSITORY VERSION
2023-06-13 01:08:59,043 DEBUG [dr1/0] submariner-gateway quay.io/submariner 0.15.1
2023-06-13 01:08:59,043 DEBUG [dr1/0] submariner-routeagent quay.io/submariner 0.15.1
2023-06-13 01:08:59,043 DEBUG [dr1/0] submariner-globalnet quay.io/submariner 0.15.1
2023-06-13 01:08:59,044 DEBUG [dr1/0] submariner-operator quay.io/submariner 0.15.1
2023-06-13 01:08:59,044 DEBUG [dr1/0] submariner-lighthouse-agent quay.io/submariner 0.15.1
2023-06-13 01:08:59,044 DEBUG [dr1/0] submariner-lighthouse-coredns quay.io/submariner 0.15.1
2023-06-13 01:08:59,044 DEBUG [dr1/0]
2023-06-13 01:08:59,044 DEBUG [dr1/0]
2023-06-13 01:08:59,055 INFO [dr1/0] addons/submariner/test completed in 3.33 seconds
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Testing show that nothing depend on submariner, so we can install it in
any order. However testing submariner on the clusters depends on having
both clusters running, so it works best at the start, when both clusters
start to deploy in the same time when submariner broker is ready on the
hub.
Testing shows that submariner test can take more than 60 seconds, since
both clusters must be deployed when running the test. Increase the test
timeout to 180 seconds to avoid random failures.
Example run focusing on submariner timings:
$ drenv start envs/regional-dr.yaml
2023-06-14 21:03:50,761 INFO [rdr] Starting environment
...
2023-06-14 21:04:28,636 INFO [dr1/1] Running addons/submariner/start
...
2023-06-14 21:04:46,558 INFO [dr2/1] Running addons/submariner/start
...
2023-06-14 21:05:06,390 INFO [hub/0] Running addons/submariner/start
...
2023-06-14 21:05:31,665 INFO [hub/0] addons/submariner/start completed in 25.28 seconds
2023-06-14 21:05:31,665 INFO [hub/0] Running addons/submariner/test
2023-06-14 21:05:31,892 INFO [hub/0] addons/submariner/test completed in 0.23 seconds
...
2023-06-14 21:08:44,596 INFO [dr2/1] addons/submariner/start completed in 238.04 seconds
2023-06-14 21:08:44,596 INFO [dr2/1] Running addons/submariner/test
2023-06-14 21:08:44,753 INFO [dr2/1] addons/submariner/test completed in 0.16 seconds
...
2023-06-14 21:08:49,771 INFO [dr1/1] addons/submariner/start completed in 261.13 seconds
2023-06-14 21:08:49,772 INFO [dr1/1] Running addons/submariner/test
2023-06-14 21:08:50,002 INFO [dr1/1] addons/submariner/test completed in 0.23 seconds
...
2023-06-14 21:11:28,238 INFO [rdr] Environment started in 457.48 seconds
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
|
Reviewed as part of #937. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add submariner addon and integrate it with regional environment.
Fixes #916