Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
cybozu-neco committed Sep 6, 2023
1 parent 97401e8 commit 282b34c
Show file tree
Hide file tree
Showing 4 changed files with 144 additions and 2 deletions.
71 changes: 71 additions & 0 deletions change-pvc-template.html
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,77 @@ <h3 id="metrics-1"><a class="header" href="#metrics-1">Metrics</a></h3>
If fails to volume size changed, the metrics in <code>moco_cluster_volume_resized_errors_total</code> is incremented after each reconcile,
so users can notice anomalies by monitoring this metrics.</p>
<p>See the <a href="./metrics.html">metrics documentation</a> for more details.</p>
<h2 id="volume-reduction"><a class="header" href="#volume-reduction">Volume reduction</a></h2>
<p>MOCO supports PVC reduction, but unlike PVC expansion, the user must perform the operation manually.</p>
<p>The steps are as follows:</p>
<ol>
<li>The user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster and sets a smaller volume size.</li>
<li>MOCO updates the <code>.spec.volumeClaimTemplates</code> of the StatefulSet. This does not propagate to existing Pods, PVCs, or PVs.</li>
<li>The user manually deletes the MySQL Pod &amp; PVC.</li>
<li>Wait for the Pod &amp; PVC to be recreated by the statefulset-controller, and for MOCO to clone the data.</li>
<li>Once the cluster becomes Healthy, the user deletes the next Pod and PVC.</li>
<li>It is completed when all Pods and PVCs are recreated.</li>
</ol>
<h3 id="1-the-user-modifies-the-specvolumeclaimtemplates-of-the-mysqlcluster-and-sets-a-smaller-volume-size"><a class="header" href="#1-the-user-modifies-the-specvolumeclaimtemplates-of-the-mysqlcluster-and-sets-a-smaller-volume-size">1. The user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster and sets a smaller volume size</a></h3>
<p>For example, the user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster as follows:</p>
<pre><code class="language-diff"> apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
namespace: default
name: test
spec:
replicas: 3
podTemplate:
spec:
containers:
- name: mysqld
image: quay.io/cybozu/mysql:8.0.30
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ &quot;ReadWriteOnce&quot; ]
resources:
requests:
- storage: 1Gi
+ storage: 500Mi
</code></pre>
<h3 id="2-moco-updates-the-specvolumeclaimtemplates-of-the-statefulset-this-does-not-propagate-to-existing-pods-pvcs-or-pvs"><a class="header" href="#2-moco-updates-the-specvolumeclaimtemplates-of-the-statefulset-this-does-not-propagate-to-existing-pods-pvcs-or-pvs">2. MOCO updates the <code>.spec.volumeClaimTemplates</code> of the StatefulSet. This does not propagate to existing Pods, PVCs, or PVs</a></h3>
<p>The moco-controller will update the <code>.spec.volumeClaimTemplates</code> of the StatefulSet.
The actual modification of the StatefulSet's <code>.spec.volumeClaimTemplates</code> is not allowed,
so this change is achieved by recreating the StatefulSet.
At this time, only the recreation of StatefulSet is performed, without deleting the Pods and PVCs.</p>
<h3 id="3-the-user-manually-deletes-the-mysql-pod--pvc"><a class="header" href="#3-the-user-manually-deletes-the-mysql-pod--pvc">3. The user manually deletes the MySQL Pod &amp; PVC</a></h3>
<p>The user manually deletes the PVC and Pod.
Use the following command to delete them:</p>
<pre><code class="language-console">$ kubectl delete --wait=false pvc &lt;pvc-name&gt;
$ kubectl delete --grace-period=1 &lt;pod-name&gt;
</code></pre>
<h3 id="4-wait-for-the-pod--pvc-to-be-recreated-by-the-statefulset-controller-and-for-moco-to-clone-the-data"><a class="header" href="#4-wait-for-the-pod--pvc-to-be-recreated-by-the-statefulset-controller-and-for-moco-to-clone-the-data">4. Wait for the Pod &amp; PVC to be recreated by the statefulset-controller, and for MOCO to clone the data</a></h3>
<p>The statefulset-controller recreates Pods and PVCs, creating a new PVC with a reduced size.
Once the MOCO successfully starts a Pod, it begins cloning the data.</p>
<pre><code class="language-console">$ kubectl get mysqlcluster,po,pvc
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
mysqlcluster.moco.cybozu.com/test True False 0 2 &lt;no value&gt;

NAME READY STATUS RESTARTS AGE
pod/moco-test-0 3/3 Running 0 2m14s
pod/moco-test-1 3/3 Running 0 114s
pod/moco-test-2 0/3 Init:1/2 0 7s

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-data-moco-test-0 Bound pvc-03c73525-0d6d-49de-b68a-f8af4c4c7faa 1Gi RWO standard 2m14s
persistentvolumeclaim/mysql-data-moco-test-1 Bound pvc-73c26baa-3432-4c85-b5b6-875ffd2456d9 1Gi RWO standard 114s
persistentvolumeclaim/mysql-data-moco-test-2 Bound pvc-779b5b3c-3efc-4048-a549-a4bd2d74ed4e 500Mi RWO standard 7s
</code></pre>
<h3 id="5-once-the-cluster-becomes-healthy-the-user-deletes-the-next-pod-and-pvc"><a class="header" href="#5-once-the-cluster-becomes-healthy-the-user-deletes-the-next-pod-and-pvc">5. Once the cluster becomes Healthy, the user deletes the next Pod and PVC</a></h3>
<p>The user waits until the MySQLCluster state becomes Healthy, and then deletes the next Pod and PVC.</p>
<pre><code class="language-console">$ kubectl get mysqlcluster
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
mysqlcluster.moco.cybozu.com/test True True 1 3 &lt;no value&gt;
</code></pre>
<h3 id="6-it-is-completed-when-all-pods-and-pvcs-are-recreated"><a class="header" href="#6-it-is-completed-when-all-pods-and-pvcs-are-recreated">6. It is completed when all Pods and PVCs are recreated</a></h3>
<p>Repeat steps 3 to 5 until all Pods and PVCs are recreated.</p>
<h2 id="references"><a class="header" href="#references">References</a></h2>
<ul>
<li><a href="./designdoc/support_apply_pvc_template_changes.html">Design document</a></li>
Expand Down
71 changes: 71 additions & 0 deletions print.html
Original file line number Diff line number Diff line change
Expand Up @@ -1039,6 +1039,77 @@ <h3 id="metrics-2"><a class="header" href="#metrics-2">Metrics</a></h3>
If fails to volume size changed, the metrics in <code>moco_cluster_volume_resized_errors_total</code> is incremented after each reconcile,
so users can notice anomalies by monitoring this metrics.</p>
<p>See the <a href="./metrics.html">metrics documentation</a> for more details.</p>
<h2 id="volume-reduction"><a class="header" href="#volume-reduction">Volume reduction</a></h2>
<p>MOCO supports PVC reduction, but unlike PVC expansion, the user must perform the operation manually.</p>
<p>The steps are as follows:</p>
<ol>
<li>The user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster and sets a smaller volume size.</li>
<li>MOCO updates the <code>.spec.volumeClaimTemplates</code> of the StatefulSet. This does not propagate to existing Pods, PVCs, or PVs.</li>
<li>The user manually deletes the MySQL Pod &amp; PVC.</li>
<li>Wait for the Pod &amp; PVC to be recreated by the statefulset-controller, and for MOCO to clone the data.</li>
<li>Once the cluster becomes Healthy, the user deletes the next Pod and PVC.</li>
<li>It is completed when all Pods and PVCs are recreated.</li>
</ol>
<h3 id="1-the-user-modifies-the-specvolumeclaimtemplates-of-the-mysqlcluster-and-sets-a-smaller-volume-size"><a class="header" href="#1-the-user-modifies-the-specvolumeclaimtemplates-of-the-mysqlcluster-and-sets-a-smaller-volume-size">1. The user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster and sets a smaller volume size</a></h3>
<p>For example, the user modifies the <code>.spec.volumeClaimTemplates</code> of the MySQLCluster as follows:</p>
<pre><code class="language-diff"> apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
namespace: default
name: test
spec:
replicas: 3
podTemplate:
spec:
containers:
- name: mysqld
image: quay.io/cybozu/mysql:8.0.30
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ &quot;ReadWriteOnce&quot; ]
resources:
requests:
- storage: 1Gi
+ storage: 500Mi
</code></pre>
<h3 id="2-moco-updates-the-specvolumeclaimtemplates-of-the-statefulset-this-does-not-propagate-to-existing-pods-pvcs-or-pvs"><a class="header" href="#2-moco-updates-the-specvolumeclaimtemplates-of-the-statefulset-this-does-not-propagate-to-existing-pods-pvcs-or-pvs">2. MOCO updates the <code>.spec.volumeClaimTemplates</code> of the StatefulSet. This does not propagate to existing Pods, PVCs, or PVs</a></h3>
<p>The moco-controller will update the <code>.spec.volumeClaimTemplates</code> of the StatefulSet.
The actual modification of the StatefulSet's <code>.spec.volumeClaimTemplates</code> is not allowed,
so this change is achieved by recreating the StatefulSet.
At this time, only the recreation of StatefulSet is performed, without deleting the Pods and PVCs.</p>
<h3 id="3-the-user-manually-deletes-the-mysql-pod--pvc"><a class="header" href="#3-the-user-manually-deletes-the-mysql-pod--pvc">3. The user manually deletes the MySQL Pod &amp; PVC</a></h3>
<p>The user manually deletes the PVC and Pod.
Use the following command to delete them:</p>
<pre><code class="language-console">$ kubectl delete --wait=false pvc &lt;pvc-name&gt;
$ kubectl delete --grace-period=1 &lt;pod-name&gt;
</code></pre>
<h3 id="4-wait-for-the-pod--pvc-to-be-recreated-by-the-statefulset-controller-and-for-moco-to-clone-the-data"><a class="header" href="#4-wait-for-the-pod--pvc-to-be-recreated-by-the-statefulset-controller-and-for-moco-to-clone-the-data">4. Wait for the Pod &amp; PVC to be recreated by the statefulset-controller, and for MOCO to clone the data</a></h3>
<p>The statefulset-controller recreates Pods and PVCs, creating a new PVC with a reduced size.
Once the MOCO successfully starts a Pod, it begins cloning the data.</p>
<pre><code class="language-console">$ kubectl get mysqlcluster,po,pvc
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
mysqlcluster.moco.cybozu.com/test True False 0 2 &lt;no value&gt;

NAME READY STATUS RESTARTS AGE
pod/moco-test-0 3/3 Running 0 2m14s
pod/moco-test-1 3/3 Running 0 114s
pod/moco-test-2 0/3 Init:1/2 0 7s

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-data-moco-test-0 Bound pvc-03c73525-0d6d-49de-b68a-f8af4c4c7faa 1Gi RWO standard 2m14s
persistentvolumeclaim/mysql-data-moco-test-1 Bound pvc-73c26baa-3432-4c85-b5b6-875ffd2456d9 1Gi RWO standard 114s
persistentvolumeclaim/mysql-data-moco-test-2 Bound pvc-779b5b3c-3efc-4048-a549-a4bd2d74ed4e 500Mi RWO standard 7s
</code></pre>
<h3 id="5-once-the-cluster-becomes-healthy-the-user-deletes-the-next-pod-and-pvc"><a class="header" href="#5-once-the-cluster-becomes-healthy-the-user-deletes-the-next-pod-and-pvc">5. Once the cluster becomes Healthy, the user deletes the next Pod and PVC</a></h3>
<p>The user waits until the MySQLCluster state becomes Healthy, and then deletes the next Pod and PVC.</p>
<pre><code class="language-console">$ kubectl get mysqlcluster
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
mysqlcluster.moco.cybozu.com/test True True 1 3 &lt;no value&gt;
</code></pre>
<h3 id="6-it-is-completed-when-all-pods-and-pvcs-are-recreated"><a class="header" href="#6-it-is-completed-when-all-pods-and-pvcs-are-recreated">6. It is completed when all Pods and PVCs are recreated</a></h3>
<p>Repeat steps 3 to 5 until all Pods and PVCs are recreated.</p>
<h2 id="references"><a class="header" href="#references">References</a></h2>
<ul>
<li><a href="./designdoc/support_apply_pvc_template_changes.html">Design document</a></li>
Expand Down
2 changes: 1 addition & 1 deletion searchindex.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion searchindex.json

Large diffs are not rendered by default.

0 comments on commit 282b34c

Please sign in to comment.