Report when all add-ons are deployed in a workload cluster

[gianlucam76]

Sveltos is used to deploy Kubernetes add-ons.

When a ClusterProfile instance is created, Sveltos starts watching for clusters matching ClusterProfile clusterSelector fields. In each matching workload cluster, Sveltos deploys all add-ons (helm charts and/or Kubernetes resource YAMLs).

There might be more than one ClusterProfile instance matching a cluster.

After all add-ons are deployed, other tools might be used to do something else. For instance:

• if a cluster is being instantiated to run a CI/CD pipeline, the CI/CD tests need to run only after all necessary add-ons are deployed;
• if a cluster is being instantiated to be assigned to an engineer, providing the engineer with a Kubeconfig to access the cluster should happen only after all add-ons are properly deployed.

We want Sveltos to easily provide a notification when all add-ons are deployed.
And we want a solution that can easily be extended. For instance, we can initially simply configure Sveltos to generate a Kubernetes event when all add-ons for a cluster are deployed. But later on we might want to support other type of integrations (send a slack notification, etc).

Also potentially we might want Sveltos to report different type of health checks on top of all add-ons are deployed (for instance, send notification when a Deployment is down in a given cluster).

This means we want to:

1. introduce a new CRD where we tell Sveltos what type of notifications we are interested in (to start with add-ons are deployed);
2. what type of notification to send (to start with an event notification);
3. introduce a new micro-service running in the management cluster that can process the new CRD and send appropriate notification(s)

Moving forward we will also need an agent counter-part, i.e., a micro service deployed in each workload cluster that can detect for instance when a deployment is down.

[kprav33n]

After reading the problem statement, this sounds very similar to the startup/readiness/liveness probe use case. However, in this problem the concept has to be extended across multiple Kubernetes resources/namespaces. As most of the community is already used to startup/readiness/liveness probes, extending that concept using a new CRD that lets admin define a custom cluster-wide startup/readiness/liveness probes would be an interesting option to pursue. Once these probes are exposed using a standard URI, there could be other tooling or CRDs incorporated to handle the notification part.

[kprav33n]

Another interesting resource related to this discussion: kubernetes/kubernetes#34363