Skip to content

Commit

Permalink
Add volsync self test
Browse files Browse the repository at this point in the history
Testing volsync without ramen is the best way to understand how it works
and how we need to use it when using minikube (or vanilla k8s?).

We use security context to run the busybox app as unprivileged user and
set the ownership of the pvc. Without this the busybox app runs as root
on minikube. It is possible to allow volsync privileged movers using an
annotation on the namespace but this is great pain to use and conflicts
with ocm, and there is no reason to run the busybox app as root.

The test deploys busybox application on the source cluster, and setup
replication to the destination cluster via submariner service export.
When the replication is ready we start one replication and wait until it
completes.

We don't validate the replicated data since the test is complicated and
slow as is and it may be too slow for self test.

Example run with volsync test environment:

    $ drenv start envs/volsync.yaml
    2023-06-29 23:40:32,314 INFO    [volsync] Starting environment
    2023-06-29 23:40:32,371 INFO    [hub] Starting minikube cluster
    2023-06-29 23:40:33,391 INFO    [dr1] Starting minikube cluster
    2023-06-29 23:40:34,383 INFO    [dr2] Starting minikube cluster
    2023-06-29 23:41:14,260 INFO    [hub] Cluster started in 41.89 seconds
    2023-06-29 23:41:14,261 INFO    [hub/0] Running addons/submariner/start
    2023-06-29 23:43:35,144 INFO    [hub/0] addons/submariner/start completed in 140.88 seconds
    2023-06-29 23:43:35,145 INFO    [hub/0] Running addons/submariner/test
    2023-06-29 23:43:55,502 INFO    [dr1] Cluster started in 202.11 seconds
    2023-06-29 23:44:10,336 INFO    [dr2] Cluster started in 215.95 seconds
    2023-06-29 23:44:16,476 INFO    [hub/0] addons/submariner/test completed in 41.33 seconds
    2023-06-29 23:44:16,477 INFO    [volsync/0] Running addons/volsync/start
    2023-06-29 23:44:48,775 INFO    [volsync/0] addons/volsync/start completed in 32.30 seconds
    2023-06-29 23:44:48,775 INFO    [volsync/0] Running addons/volsync/test
    2023-06-29 23:46:08,017 INFO    [volsync/0] addons/volsync/test completed in 79.24 seconds
    2023-06-29 23:46:08,018 INFO    [volsync] Environment started in 335.70 seconds

Example run with regional-dr environment:

    2023-06-30 00:06:13,967 INFO    [rdr/0] Running addons/rbd-mirror/start
    2023-06-30 00:06:13,968 INFO    [rdr/1] Running addons/volsync/start
    2023-06-30 00:06:48,121 INFO    [rdr/1] addons/volsync/start completed in 34.15 seconds
    2023-06-30 00:06:48,121 INFO    [rdr/1] Running addons/volsync/test
    2023-06-30 00:07:11,942 INFO    [rdr/0] addons/rbd-mirror/start completed in 57.98 seconds
    2023-06-30 00:07:11,942 INFO    [rdr/0] Running addons/rbd-mirror/test
    2023-06-30 00:07:24,297 INFO    [rdr/1] addons/volsync/test completed in 36.18 seconds
    2023-06-30 00:07:28,676 INFO    [rdr/0] addons/rbd-mirror/test completed in 16.73 seconds

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
  • Loading branch information
nirs committed Jul 5, 2023
1 parent 904cc79 commit d8f4715
Show file tree
Hide file tree
Showing 10 changed files with 319 additions and 0 deletions.
7 changes: 7 additions & 0 deletions test/addons/volsync/destination/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
resources:
- namespace.yaml
- replication-dst.yaml
8 changes: 8 additions & 0 deletions test/addons/volsync/destination/namespace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
kind: Namespace
metadata:
name: busybox
20 changes: 20 additions & 0 deletions test/addons/volsync/destination/replication-dst.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
name: busybox-dst
namespace: busybox
spec:
rsyncTLS:
copyMethod: Snapshot
capacity: 1Gi
accessModes: [ReadWriteOnce]
storageClassName: csi-hostpath-sc
volumeSnapshotClassName: csi-hostpath-snapclass
moverSecurityContext:
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
47 changes: 47 additions & 0 deletions test/addons/volsync/source/deploy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
appname: busybox
name: busybox
namespace: busybox
spec:
replicas: 1
selector:
matchLabels:
appname: busybox
template:
metadata:
labels:
appname: busybox
spec:
containers:
- image: docker.io/library/busybox:latest
imagePullPolicy: IfNotPresent
name: busybox
command:
- sh
- -c
- |
trap exit TERM
while true; do
echo $(date) | tee -a /mnt/test/outfile
sync
sleep 10 &
wait
done
volumeMounts:
- name: volume
mountPath: /mnt/test
volumes:
- name: volume
persistentVolumeClaim:
claimName: busybox-pvc
securityContext:
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
8 changes: 8 additions & 0 deletions test/addons/volsync/source/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
resources:
- namespace.yaml
- pvc.yaml
- deploy.yaml
8 changes: 8 additions & 0 deletions test/addons/volsync/source/namespace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
kind: Namespace
metadata:
name: busybox
17 changes: 17 additions & 0 deletions test/addons/volsync/source/pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: busybox-pvc
namespace: busybox
labels:
appname: busybox
spec:
accessModes: [ReadWriteOnce]
storageClassName: csi-hostpath-sc
resources:
requests:
storage: 1Gi
22 changes: 22 additions & 0 deletions test/addons/volsync/source/replication-src.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: busybox-src
namespace: busybox
spec:
sourcePVC: busybox-pvc
trigger:
manual: replication-1
rsyncTLS:
keySecret: volsync-rsync-tls-busybox-dst
address: volsync-rsync-tls-dst-busybox-dst.busybox.svc.clusterset.local
copyMethod: Snapshot
volumeSnapshotClassName: csi-hostpath-snapclass
moverSecurityContext:
runAsUser: 10000
runAsGroup: 10000
fsGroup: 10000
12 changes: 12 additions & 0 deletions test/addons/volsync/source/secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: volsync-rsync-tls-busybox-dst
namespace: busybox
data:
psk.txt: $value
170 changes: 170 additions & 0 deletions test/addons/volsync/test
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
#!/usr/bin/env python3

# SPDX-FileCopyrightText: The RamenDR authors
# SPDX-License-Identifier: Apache-2.0

import json
import os
import sys

import drenv
from drenv import kubectl
from drenv import subctl

DEPLOY = "busybox"
NAMESPACE = "busybox"

VOLSYNC_SECRET = "volsync-rsync-tls-busybox-dst"
VOLSYNC_SERVICE = "volsync-rsync-tls-dst-busybox-dst"


def setup_application(cluster):
print(f"Deploy application on cluster '{cluster}'")
kubectl.apply("--kustomize", "source", context=cluster)


def setup_replication_destination(cluster):
print(f"Create replication destination on cluster '{cluster}'")
kubectl.apply("--kustomize", "destination", context=cluster)


def wait_for_application(cluster):
print(f"Waiting until deploy '{DEPLOY}' is rolled out in cluster '{cluster}'")
kubectl.rollout(
"status",
f"deploy/{DEPLOY}",
f"--namespace={NAMESPACE}",
"--timeout=120s",
context=cluster,
)


def wait_for_replication_destination(cluster):
print(
f"Waiting until replication destination is synchronizing in cluster '{cluster}'"
)
kubectl.wait(
"replicationdestination/busybox-dst",
"--for=condition=Synchronizing=True",
f"--namespace={NAMESPACE}",
"--timeout=120s",
context=cluster,
)


def setup_replication_secret(cluster1, cluster2):
"""
Create a secret in the source cluster using data from the secret created by
volsync on the destiantion cluster.
"""
print(f"Getting volsync secret in cluster '{cluster2}'")
psk_txt = kubectl.get(
f"secret/{VOLSYNC_SECRET}",
f"--namespace={NAMESPACE}",
"--output=jsonpath={.data.psk\\.txt}",
context=cluster2,
)

print(f"Creating volsync secret in cluster '{cluster1}'")
template = drenv.template("source/secret.yaml")
yaml = template.substitute(value=psk_txt)
kubectl.apply("--filename=-", input=yaml, context=cluster1)


def setup_replication_service(cluster1, cluster2):
"""
Export volsync replication service from the destination cluster to the
source cluster using submariner.
"""
print(f"Exporting volsync service in cluster '{cluster2}'")
subctl.export("service", VOLSYNC_SERVICE, cluster2, namespace=NAMESPACE)

print(f"Waiting until service export is synced in cluster '{cluster2}'")
kubectl.wait(
f"serviceexports/{VOLSYNC_SERVICE}",
"--for=condition=Synced=True",
f"--namespace={NAMESPACE}",
"--timeout=120s",
context=cluster2,
)

print(f"Waiting until service import is ready in cluster '{cluster1}'")
drenv.wait_for(
f"serviceimports/{VOLSYNC_SERVICE}",
output="jsonpath={.status.clusters}",
namespace=NAMESPACE,
timeout=120,
profile=cluster1,
)


def run_replication(cluster):
"""
Start replication and wait until replication completes.
"""
print(f"Creating replication source in cluster '{cluster}'")
kubectl.apply("--filename", "source/replication-src.yaml", context=cluster)

print(f"Waiting until replication is completed in cluster '{cluster}'")
kubectl.wait(
"replicationsource/busybox-src",
"--for=jsonpath={.status.lastManualSync}=replication-1",
f"--namespace={NAMESPACE}",
"--timeout=120s",
context=cluster,
)
out = kubectl.get(
"replicationsource/busybox-src",
"--output=jsonpath={.status}",
f"--namespace={NAMESPACE}",
context=cluster,
)
status = json.loads(out)
print("Replication status:")
print(json.dumps(status, indent=2))


def teardown(cluster1, cluster2):
"""
Remove deployments from both clusters. This also deletes additonal
resources created in the same namespace.
"""
print("Cleaning up clusters")
subctl.unexport("service", VOLSYNC_SERVICE, cluster2, namespace=NAMESPACE)
kubectl.delete(
"--kustomize",
"source",
"--ignore-not-found",
"--wait=false",
context=cluster1,
)
kubectl.delete(
"--kustomize",
"destination",
"--ignore-not-found",
"--wait=false",
context=cluster2,
)
kubectl.delete("--kustomize", "source", "--ignore-not-found", context=cluster1)
kubectl.delete("--kustomize", "destination", "--ignore-not-found", context=cluster2)


if len(sys.argv) != 3:
print(f"Usage: {sys.argv[0]} cluster1 cluster2")
sys.exit(1)

os.chdir(os.path.dirname(__file__))
cluster1, cluster2 = sys.argv[1:]

setup_application(cluster1)
setup_replication_destination(cluster2)

wait_for_application(cluster1)
wait_for_replication_destination(cluster2)

setup_replication_secret(cluster1, cluster2)
setup_replication_service(cluster1, cluster2)

run_replication(cluster1)

teardown(cluster1, cluster2)

0 comments on commit d8f4715

Please sign in to comment.