Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate volsync in test environment #927

Merged
merged 8 commits into from
Jul 5, 2023
Merged

Integrate volsync in test environment #927

merged 8 commits into from
Jul 5, 2023

Commits on Jul 5, 2023

  1. Increase submariner test timeouts 

    When other vms are running, I see random timeouts in submariner test.
Lets make the test more robust with overloaded machines.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    894b816 View commit details
    Browse the repository at this point in the history

  2. Add helm to required tools 

    Like other tools, developers have to install the tool manually, but helm
is available in package managers so this is trivial.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    b66e687 View commit details
    Browse the repository at this point in the history

  3. Actually deploy volsync with helm 

    Install volsync using helm, removing the manual CRD install.
We use `helm upgrade --install` to make the installation idempotent.

Thanks: Tesshu Flower <tflower@redhat.com>
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    19219b1 View commit details
    Browse the repository at this point in the history

  4. Change volsync addon to accept both clusters 

    To test volsync we need create a replication destination on one cluster
and replication source on the other, and this can be done only after
volsync is installed on both clusters, and since we also depend on
submariner this must be done after submariner is deployed and tested on
both clusters.

The best way to handle these dependencies is to run the addon on the
global workers after all clusters are deployed, similar to the way we
run rbd-mirror. This change convert volsync to accept both clusters and
move it to the global workers.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    0c6f2ba View commit details
    Browse the repository at this point in the history

  5. Add volsync environment for testing 

    This environment can be used to debug the volsync addon.

We add the minikube `volumesnapshots` and `csi-hostpath-driver`
addons[1], so we can create snapshots with minikube builtin `hostpath`
storage.  With both addons we can create a busybox app with
`csi-hostpath-sc` storage class, and replicate it using `volsync`.

We want to add CephFS later, starting with the simplest solution for
testing volsync.

Example run:

    $ drenv start envs/volsync.yaml
    2023-06-28 23:26:24,534 INFO    [volsync] Starting environment
    2023-06-28 23:26:24,592 INFO    [dr1] Starting minikube cluster
    2023-06-28 23:26:25,610 INFO    [dr2] Starting minikube cluster
    2023-06-28 23:28:27,544 INFO    [dr1] Cluster started in 122.95 seconds
    2023-06-28 23:28:46,160 INFO    [dr2] Cluster started in 140.55 seconds
    2023-06-28 23:28:46,161 INFO    [volsync/0] Running addons/volsync/start
    2023-06-28 23:29:08,356 INFO    [volsync/0] addons/volsync/start completed in 22.20 seconds
    2023-06-28 23:29:08,356 INFO    [volsync] Environment started in 163.82 seconds

[1] https://minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    6c7f784 View commit details
    Browse the repository at this point in the history

  6. Add volumesnapshots and csi-hostpath-driver to regional-dr 

    With these addons we can deploy busybox using `csi-hostpath-sc` storage
class and ramen will setup replication using volsync and submariner.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    11c244b View commit details
    Browse the repository at this point in the history

  7. Add submariner to volsync env 

    Volsync creates a service on the destination cluster using ClusterIP or
LoadBalancer. The recommended way to make the service accessible to the
source cluster is using submariner export. Minikube has a tunnel command
that can expose services using LoadBalancer, but it is painful to use.

Since our submariner addon requires the broker on a separate cluster,
add a tiny hub cluster.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    904cc79 View commit details
    Browse the repository at this point in the history

  8. Add volsync self test 

    Testing volsync without ramen is the best way to understand how it works
and how we need to use it when using minikube (or vanilla k8s?).

We use security context to run the busybox app as unprivileged user and
set the ownership of the pvc. Without this the busybox app runs as root
on minikube. It is possible to allow volsync privileged movers using an
annotation on the namespace but this is great pain to use and conflicts
with ocm, and there is no reason to run the busybox app as root.

The test deploys busybox application on the source cluster, and setup
replication to the destination cluster via submariner service export.
When the replication is ready we start one replication and wait until it
completes.

We don't validate the replicated data since the test is complicated and
slow as is and it may be too slow for self test.

Example run with volsync test environment:

    $ drenv start envs/volsync.yaml
    2023-06-29 23:40:32,314 INFO    [volsync] Starting environment
    2023-06-29 23:40:32,371 INFO    [hub] Starting minikube cluster
    2023-06-29 23:40:33,391 INFO    [dr1] Starting minikube cluster
    2023-06-29 23:40:34,383 INFO    [dr2] Starting minikube cluster
    2023-06-29 23:41:14,260 INFO    [hub] Cluster started in 41.89 seconds
    2023-06-29 23:41:14,261 INFO    [hub/0] Running addons/submariner/start
    2023-06-29 23:43:35,144 INFO    [hub/0] addons/submariner/start completed in 140.88 seconds
    2023-06-29 23:43:35,145 INFO    [hub/0] Running addons/submariner/test
    2023-06-29 23:43:55,502 INFO    [dr1] Cluster started in 202.11 seconds
    2023-06-29 23:44:10,336 INFO    [dr2] Cluster started in 215.95 seconds
    2023-06-29 23:44:16,476 INFO    [hub/0] addons/submariner/test completed in 41.33 seconds
    2023-06-29 23:44:16,477 INFO    [volsync/0] Running addons/volsync/start
    2023-06-29 23:44:48,775 INFO    [volsync/0] addons/volsync/start completed in 32.30 seconds
    2023-06-29 23:44:48,775 INFO    [volsync/0] Running addons/volsync/test
    2023-06-29 23:46:08,017 INFO    [volsync/0] addons/volsync/test completed in 79.24 seconds
    2023-06-29 23:46:08,018 INFO    [volsync] Environment started in 335.70 seconds

Example run with regional-dr environment:

    2023-06-30 00:06:13,967 INFO    [rdr/0] Running addons/rbd-mirror/start
    2023-06-30 00:06:13,968 INFO    [rdr/1] Running addons/volsync/start
    2023-06-30 00:06:48,121 INFO    [rdr/1] addons/volsync/start completed in 34.15 seconds
    2023-06-30 00:06:48,121 INFO    [rdr/1] Running addons/volsync/test
    2023-06-30 00:07:11,942 INFO    [rdr/0] addons/rbd-mirror/start completed in 57.98 seconds
    2023-06-30 00:07:11,942 INFO    [rdr/0] Running addons/rbd-mirror/test
    2023-06-30 00:07:24,297 INFO    [rdr/1] addons/volsync/test completed in 36.18 seconds
    2023-06-30 00:07:28,676 INFO    [rdr/0] addons/rbd-mirror/test completed in 16.73 seconds

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    d8f4715 View commit details
    Browse the repository at this point in the history