Check VR Validated condition

When deleting a primary VRG, we wait until the VR Completed condition is
met. However if a VR precondition failed, for example using a drpolicy
without flattening enabled when the PVC needs flattening, the VR will
never complete, so the vrg will never be deleted.

In csi-addons 0.10.0 we have a new Validated VR condition, set to true
if pre conditions are met, and false if not. This change handles this
new condition.

This changes modifies deleted VRG processing to check the new VR
Validated status. If the condition exist and the condition status is
false, the VR will never complete and it is safe to delete it. In this
case validateVRStatus() return true, signaling that the VR is in the
desired state, and ramen finish the delete flow.

If the VR does not report the Validated condition or the condition
status is true, we continue in the normal flow. The VR will be deleted
only when the Completed condition status is true.

Tested locally with discovered app with pvc created from a volume
snapshot. Deleting dr is working again.

Issues:
- Many unit tests fail

Signed-off-by: Nir Soffer <nsoffer@redhat.com>

internal/controller/vrg_volrep.go

@@ -1413,12 +1413,23 @@ func (v *VRGInstance) checkVRStatus(pvc *corev1.PersistentVolumeClaim, volRep *v
 
 // validateVRStatus validates if the VolumeReplication resource has the desired status for the
 // current generation and returns true if so, false otherwise
-//   - When replication state is Primary, only Completed condition is checked.
-//   - When replication state is Secondary, all 3 conditions for Completed/Degraded/Resyncing is
-//     checked and ensured healthy.
+//   - When replication state is Primary, Validated and Completed conditions are checked.
+//   - When replication state is Secondary, Completed, Degraded and Resyncing conditions are checked and
+//     ensured healthy.
 func (v *VRGInstance) validateVRStatus(pvc *corev1.PersistentVolumeClaim, volRep *volrep.VolumeReplication,
 	state ramendrv1alpha1.ReplicationState,
 ) bool {
+	// Check validated for primary during VRG deletion.
+	if state == ramendrv1alpha1.Primary && rmnutil.ResourceIsDeleted(v.instance) {
+		validated, ok := v.validateVRValidatedStatus(volRep)
+		if !validated && ok {
+			v.log.Info(fmt.Sprintf("VolumeReplication %s/%s failed validation and can be deleted",
+				volRep.GetName(), volRep.GetNamespace()))
+
+			return true
+		}
+	}
+
 	// Check completed for both primary and secondary.
 	if !v.validateVRCompletedStatus(pvc, volRep, state) {
 		return false
@@ -1441,6 +1452,21 @@ func (v *VRGInstance) validateVRStatus(pvc *corev1.PersistentVolumeClaim, volRep
 	return true
 }
 
+// validateVRValidatedStatus validates that VolumeReplicaion resource was validated.
+// Return 2 booleans
+// - validated: true if the condition is true, otherwise false
+// - ok: true if the check was succeesfull, false if the condition is missing, stale, or unknown.
+func (v *VRGInstance) validateVRValidatedStatus(
+	volRep *volrep.VolumeReplication,
+) (bool, bool) {
+	conditionMet, errorMsg := isVRConditionMet(volRep, volrep.ConditionValidated, metav1.ConditionTrue)
+	if errorMsg != "" {
+		v.log.Info(fmt.Sprintf("%s (VolRep: %s/%s)", errorMsg, volRep.GetName(), volRep.GetNamespace()))
+	}
+
+	return conditionMet, errorMsg == ""
+}
+
 // validateVRCompletedStatus validates if the VolumeReplication resource Completed condition is met and update
 // the PVC DataReady and Protected conditions.
 // Returns true if the condtion is true, false if the condition is missing, stale, ubnknown, of false.