[Improvement] Make InstanceSet controller set ownerReferences on PersistentVolumeClaims

[jackmaninov]

Is your improvement request related to a problem?

After upgrading kubeblocks from 0.8.x to 0.9.0 all my clusters were migrated from StatefulSets to InstanceSets. My existing clusters are working as expected, but when I create new clusters, their PersistentVolumeClaims do not receive ownerReferences pointing to the Component that created them.

Existing PVCs created by StatefulSet Components have ownerReferences like:

  ownerReferences:
  - apiVersion: apps.kubeblocks.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Component
    name: test-db
    uid: ba4cdb71-3ad2-46a8-9095-4fb90b295653

while new PVCs created by InstanceSet Components have no ownerReferences.

Having an ownerReference on a PVC lets continuous delivery tools like argo-cd identify the source of an object, and identify them as belonging to an application set. Without these references, argo-cd identifies these PVCs as stray objects and will attempt to prune them (for example once an attached pod is killed). There are also garbage collection implications to ownerReferences.

According to https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention the StatefulSet controller is responsible for adding these ownerReferences to PVCs normally, so by migrating to InstanceSets we are losing this functionality.

If this is a new function, please describe the motivation and goals.

I am deploying kubeblocks Clusters using argo-cd in projects that require stray resources to be pruned, and I do not want my Cluster PVCs to be pruned.

Describe the solution you'd like

InstanceSet controller should add ownerReferences to PVCs referencing their Components similiar to the StatefulSet controller.

Describe alternatives you've considered

I can disable pruning in my projects, however by their nature they will have other objects that require pruning from time to time, so this is not ideal. I would have to reorganise my projects to remove Clusters to a separate one, which is also not ideal.

For now I am manually adding ownerReferences to the affected PVCs.

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity

[jackmaninov]

Any updates? This is still causing issues for me after migrating to InstanceSets

[free6om]

Any updates? This is still causing issues for me after migrating to InstanceSets

There's a historical reason for not supporting owner reference of PVCs in InstanceSets. Will support in next major version of KubeBlocks which planned to release within 2 months.

[jackmaninov]

If anyone else comes across this issue, a workaround on the Argo CD side is to set application.resourceTrackingMethod: annotation in your argocd-cm configmap. Then the ownership of the PVCs can be tracked correctly without ownership references being included.

[weicao]

@free6om @leon-inf Can we fix it in recent kubeblocks minor upgrade, like v0.9.1 ?

[free6om]

@jackmaninov One more thing incase you don't know yet: the StatefulSet will not add a ownerReference to the PVCs it created by default, it adds the ownerReference when the PersistentVolumeClaim retention policy is set to Delete.
Dig more here.

[jackmaninov]

Yes, after digging deeper the root cause of my issue turns out to be more argo-cd's poor default component tracking, coupled with the change in behaviour of the InstanceSet. The configuration change I posted above seems to be the "correct" fix. Not sure if you want to close this issue.

However, adding an ownerReference to PVCs when the cluster/component's retention policy is Delete also sounds like correct behaviour. Then the API server would garbage collect them automatically, rather than kubeblocks needing to do it.

[MarkKharitonov]

I hit that one too and the annotation based tracking resolved it.