Fix TLS cert creation when Zones are added. (#115)

This PR revamps the main controller to look for changes
in MinIOInstance fields to respond to changes like
addition of zones, image updates.

Fixes #97

README.md

@@ -7,8 +7,8 @@ MinIO-Operator brings native MinIO, [MCS](https://github.com/minio/mcs), [KES](h
 | Feature                 | Reference Document |
 |-------------------------|--------------------|
 | Create and delete highly available distributed MinIO clusters  | [Create a MinIO Instance](https://github.com/minio/minio-operator#create-a-minio-instance). |
-| Expand an existing MinIO cluster                               | [Expand a MinIO Cluster](https://github.com/minio/minio-operator#expand-a-minio-cluster). |
 | Automatic TLS for MinIO                                        | [Automatic TLS for MinIO Instance](https://github.com/minio/minio-operator/blob/master/docs/tls.md#automatic-csr-generation). |
+| Expand an existing MinIO cluster                               | [Expand a MinIO Cluster](https://github.com/minio/minio-operator/blob/master/docs/adding-zones.md). |
 | Deploy MCS with MinIO cluster  | [Deploy MinIO Instance with MCS](https://github.com/minio/minio-operator/blob/master/docs/mcs.md). |
 | Deploy KES with MinIO cluster  | [Deploy MinIO Instance with KES](https://github.com/minio/minio-operator/blob/master/docs/kes.md). |
 | Deploy mc mirror  | [Deploy Mirror Instance](https://github.com/minio/minio-operator/blob/master/docs/mirror.md). |
@@ -44,24 +44,6 @@ Once MinIO-Operator deployment is running, you can create MinIO instances using
 kubectl apply -f https://raw.githubusercontent.com/minio/minio-operator/master/examples/minioinstance.yaml
 ```
 
-### Expand a MinIO cluster
-
-After you have a distributed MinIO Cluster running (zones.server >= 4), you can expand the MinIO cluster using
-
-```
-kubectl patch minioinstances.operator.min.io minio --patch "$(cat examples/patch.yaml)" --type=merge
-```
-
-You can expand an existing cluster by adding new zones to the `patch.yaml` and run the above `kubectl-patch` command.
-
-**NOTE**: Important point to consider _before_ using cluster expansion:
-
-During cluster expansion, MinIO Operator removes the existing StatefulSet and creates a new StatefulSet with required number of Pods. This means, there is a short downtime during expansion, as the pods are terminated and created again.
-
-As existing StatefulSet pods are terminated, its PVCs are also deleted. It is _very important_ to ensure PVs bound to MinIO StatefulSet PVCs are not deleted at this time to avoid data loss. We recommend configuring every PV with reclaim policy [`retain`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain), to ensure the PV is not deleted.
-
-If you attempt cluster expansion while the PV reclaim policy is set to something else, it may lead to data loss. If you have the reclaim policy set to something else, change it as explained in [Kubernetes documents](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/).
-
 ### Access MinIOInstance via Service
 
 Add an [external service](https://kubernetes.io/docs/concepts/services-networking/service/) in MinIOInstance definition to enable Service based access to the MinIOInstance pods. Refer [the example here](https://github.com/minio/minio-operator/blob/master/examples/minioinstance.yaml?raw=true) for details on how to setup service based access for MinIOInstance pods.

docs/adding-zones.md

@@ -0,0 +1,44 @@
+# Adding Zones to a MinIO Cluster
+
+[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
+[![Docker Pulls](https://img.shields.io/docker/pulls/minio/k8s-operator.svg?maxAge=604800)](https://hub.docker.com/r/minio/k8s-operator)
+
+This document explains how to add zones to an existing MinIO Cluster with Operator. This document is only applicable to a MinIO Cluster created by MinIO Operator.
+
+Read more about MinIO Zones design in [MinIO Docs](https://github.com/minio/minio/blob/master/docs/distributed).
+
+## Getting Started
+
+Assuming you have a MinIO cluster with single zone, `zone-0` with 4 drives (as shown in [examples](https://github.com/minio/minio-operator/tree/master/examples)). You can dd a new zone `zone-1` with 4 drives using `kubectl patch` command.
+
+```
+kubectl patch minioinstances.operator.min.io minio --patch "$(cat examples/patch.yaml)" --type=merge
+```
+
+If you're using a custom configuration (e.g. multiple zones or higher number of drives per zone), make sure to change `patch.yaml` accordingly.
+
+**NOTE**: Important points to consider _before_ using cluster expansion:
+
+- During cluster expansion, MinIO Operator removes the existing StatefulSet and creates a new StatefulSet with required number of Pods. This means, there is a short downtime during expansion, as the pods are terminated and created again. As existing StatefulSet pods are terminated, its PVCs are also deleted. It is _very important_ to ensure PVs bound to MinIO StatefulSet PVCs are not deleted at this time to avoid data loss. We recommend configuring every PV with reclaim policy [`retain`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain), to ensure the PV is not deleted. If you attempt cluster expansion while the PV reclaim policy is set to something else, it may lead to data loss. If you have the reclaim policy set to something else, change it as explained in [Kubernetes documents](https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/).
+
+- MinIO server currently doesn't support zone removal. So, please ensure to not remove zones in patch.yaml file while applying the patch. It can have unintended consequences including missing data or failure of MinIO cluster.
+
+## Rules of Adding Zones
+
+Each zone is a self contained entity with same SLA's (read/write quorum) for each object as original cluster. By using the existing namespace for lookup validation MinIO ensures conflicting objects are not created. When no such object exists then MinIO simply uses the least used zone. There are no limits on how many zones can be combined.
+
+There is only one requirement, i.e. based on initial zone's erasure set count (say `n`), new zones are expected to have a minimum of `n` drives to match the original cluster SLA or it should be in multiples of `n`. For example if initial set count is 4, new zones should have at least 4 or multiple of 4 drives.
+
+Read more about MinIO Zones design in [MinIO Docs](https://github.com/minio/minio/blob/master/docs/distributed).
+
+## Effects on KES/TLS Enabled Instance
+
+If your MinIO Operator configuration has [KES](https://github.com/minio/minio-operator/blob/master/docs/kes.md) or [Automatic TLS](https://github.com/minio/minio-operator/blob/master/docs/tls.md#automatic-csr-generation) enabled, there are additional considerations:
+
+- When new zones are added, Operator invalidates older self signed TLS certificates and the related secrets. Operator then creates new certificate signing requests (CSR). This is because there are new MinIO nodes that must be added in certificate DNS names. The administrator must approve these CSRs for MinIO server to be deployed again. Unless the CSR are approved, Operator will not create MinIO StatefulSet pods.
+
+- If you're using your own certificates, as explained [here](https://github.com/minio/minio-operator/blob/master/docs/tls.md#pass-certificate-secret-to-minioinstance), please ensure to use/update proper certificates that allow older and new MinIO nodes.
+
+## Downtime
+
+Since Operator deletes existing StatefulSet and related CSR / Secrets (for TLS enabled setups), before re-creating a new StatefulSet and other resources, there is a downtime involved when adding zones. This downtime is generally few minutes, assuming CSRs are approved quickly and resources are available for new StatefulSet to be created.

examples/minioinstance-kes.yaml

@@ -13,10 +13,11 @@ metadata:
   name: minio-mcs-secret
 type: Opaque
 data:
-  mcshmacjwt: WU9VUkpXVFNJR05JTkdTRUNSRVQ= # base 64 encoded "YOURJWTSIGNINGSECRET" (echo -n 'YOURJWTSIGNINGSECRET' | base64)
-  mcspbkdfpassphrase: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
-  mcspbkdfsalt: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
-  mcssecretkey: WU9VUk1DU1NFQ1JFVA== # base 64 encoded "YOURMCSSECRET" (echo -n 'YOURMCSSECRET' | base64)
+  MCS_HMAC_JWT_SECRET: WU9VUkpXVFNJR05JTkdTRUNSRVQ= # base 64 encoded "YOURJWTSIGNINGSECRET" (echo -n 'YOURJWTSIGNINGSECRET' | base64)
+  MCS_PBKDF_PASSPHRASE: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
+  MCS_PBKDF_SALT: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
+  MCS_ACCESS_KEY: WU9VUk1DU0FDQ0VTUw== # base 64 encoded "YOURMCSACCESS" (echo -n 'YOURMCSACCESS' | base64)
+  MCS_SECRET_KEY: WU9VUk1DU1NFQ1JFVA== # base 64 encoded "YOURMCSSECRET" (echo -n 'YOURMCSSECRET' | base64)
 ---
 apiVersion: v1
 kind: Service
@@ -148,7 +149,6 @@ spec:
   ## restarts the pods if liveness checks fail.
   liveness:
     httpGet:
-      scheme: HTTPS
       path: /minio/health/live
       port: 9000
     initialDelaySeconds: 120
@@ -160,7 +160,6 @@ spec:
   ## Disable this check if you're setting PodManagementPolicy to "OrderedReady".
   readiness:
     httpGet:
-      scheme: HTTPS
       path: /minio/health/ready
       port: 9000
     initialDelaySeconds: 120

examples/minioinstance-mcs.yaml

@@ -137,7 +137,6 @@ spec:
   ## restarts the pods if liveness checks fail.
   liveness:
     httpGet:
-      scheme: HTTPS
       path: /minio/health/live
       port: 9000
     initialDelaySeconds: 120
@@ -149,7 +148,6 @@ spec:
   ## Disable this check if you're setting PodManagementPolicy to "OrderedReady".
   readiness:
     httpGet:
-      scheme: HTTPS
       path: /minio/health/ready
       port: 9000
     initialDelaySeconds: 120

examples/minioinstance.yaml

@@ -8,17 +8,6 @@ data:
   secretkey: bWluaW8xMjM= # based 64 encoded "minio123" (echo -n 'minio123' | base64)
 ---
 apiVersion: v1
-kind: Secret
-metadata:
-  name: minio-mcs-secret
-type: Opaque
-data:
-  mcshmacjwt: WU9VUkpXVFNJR05JTkdTRUNSRVQ= # base 64 encoded "YOURJWTSIGNINGSECRET" (echo -n 'YOURJWTSIGNINGSECRET' | base64)
-  mcspbkdfpassphrase: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
-  mcspbkdfsalt: U0VDUkVU # base 64 encoded "SECRET" (echo -n 'SECRET' | base64)
-  mcssecretkey: WU9VUk1DU1NFQ1JFVA== # base 64 encoded "YOURMCSSECRET" (echo -n 'YOURMCSSECRET' | base64)
----
-apiVersion: v1
 kind: Service
 metadata:
   name: minio-service
@@ -106,16 +95,6 @@ spec:
   #    key: dedicated
   #    operator: Equal
   #    value: storage
-  ## Define configuration for mcs (A graphical user interface for MinIO)
-  mcs:
-    image: minio/mcs:v0.0.4
-    replicas: 2
-    mcsSecret:
-      name: minio-mcs-secret
-  ## Optionally metadata can be attached to the mcs pod
-  # metadata:
-  #   labels:
-  #     extra: labels
   ## Add environment variables to be set in MinIO container (https://github.com/minio/minio/tree/master/docs/config)
   env:
     - name: MINIO_BROWSER

examples/patch.yaml

@@ -5,8 +5,8 @@ metadata:
 spec:
   zones:
     - name: "zone-0"
-      ## Number of MinIO servers/pods in this zone.
+      ## Number of MinIO servers/pods in zone-0.
       servers: 4
     - name: "zone-1"
-      ## Number of MinIO servers/pods in this zone.
+      ## Number of MinIO servers/pods in zone-1.
       servers: 4

pkg/controller/cluster/csr.go

@@ -254,9 +254,7 @@ func (c *Controller) createSecret(ctx context.Context, mi *miniov1.MinIOInstance
 			"public.crt":  certBytes,
 		},
 	}
-	cOpts := metav1.CreateOptions{}
-	_, err := c.kubeClientSet.CoreV1().Secrets(mi.Namespace).Create(ctx, secret, cOpts)
-	if err != nil {
+	if _, err := c.kubeClientSet.CoreV1().Secrets(mi.Namespace).Create(ctx, secret, metav1.CreateOptions{}); err != nil {
 		return err
 	}
 	return nil