Add a possibility to keep CRDs on Helm uninstall

[bukovjanmic]

Is your feature request related to a problem? Please describe.
Currently, Minio CRDs are installed, upgraded and uninstalled along the operator. When uninstalling the operator, CRDs are removed as well, which results into a loss of all CR instances on the cluster. This is not very flexible, for cases where we need to switch Helm charts, or do other operation on operator installation.

Describe the solution you'd like
It would be great if the CRDs had an option to have a "helm.sh/resource-policy: keep" annotation so that on Helm chart uninstalltion, the CRDs would be kept on the cluster and no loss of CRs would occur.

This way, operator upgrades, future helm chart refactorings etc. could be made safer.

[ramondeklein]

This should be already set on Helm installation, so that's not very practical. If you prefer not to delete the CRDs upon uninstallation, you could set the helm.sh/resource-policy:keep on the CRD yourself before uninstallation.

[bukovjanmic]

Unfortunately this is not possible as we found out the hard way:

helm/helm#10890

The annotations must be set by Helm. If we add the annotations later by hand, it is ignored by Helm (see helm/helm#10890 (comment)).

[ramondeklein]

I think this should be addressed by Helm. There is not a lot we can do. If we add this annotation, then we'll get a complaint that Helm uninstall won't remove the CRDs. Making it optional is also not a real option, because this isn't a setting that you would think of during installation.

[bukovjanmic]

You could make it optional in Helm via values.yaml (lets say something like crds.keep: false) and make it disabled (false) by default, so that only those who know what they are doing can actually do that.

Unfortunately, support in Helm templates is the only way how to make this work, currently.

[ramondeklein]

We'll discuss and prioritize this. You're always welcome to provide a PR...

[ramondeklein]

We just discussed, but we are not going to implement this. We don't want to make our Helm scripts even more complicated. We also believe that nobody will think about setting this upon installation. You only know you needed this when you uninstalled the Helm script and "everything" is gone.

PS: Note that underlying PV is not removed, so there is no data-loss. It may just be a bit harder to reconfigure a tenant that will mount the existing PVs.