CR monitoring via KSM

[bavarianbidi]

Is your feature request related to a problem? Please describe.

Beside the core objects (Pods, Services, ...), a "modern" kubernetes cluster consist of a lot of additional information which is stored in CustomResources (CRs).

The configuration/definition of these CRs are having a huge impact on the "behavior" of a cluster.
All the relevant information should be exported as metrics to get used in dashboards and alerts.

E.g. using flux as gitops tool support different configurations such as the remote-branch to use

$ k get gitrepo -n flux-giantswarm giantswarm-config -o yaml | yq ".spec.ref"       
branch: main

The reconciliation of the gitrepo is stored in the .status.conditions field of the gitrepo CR itself:

$ k get gitrepo -n flux-giantswarm giantswarm-config -o yaml | yq '.status.conditions'

- lastTransitionTime: "2023-05-29T11:58:37Z"
  message: stored artifact for revision 'main@sha1:06a6bf1b36e2ca013d655018435fe7a15e36fbe8'
  observedGeneration: 54
  reason: Succeeded
  status: "True"
  type: Ready
- lastTransitionTime: "2023-05-29T11:58:37Z"
  message: stored artifact for revision 'main@sha1:06a6bf1b36e2ca013d655018435fe7a15e36fbe8'
  observedGeneration: 54
  reason: Succeeded
  status: "True"
  type: ArtifactInStorage

Describe the solution you'd like

With kube-state-metrics (KSM) version v2.5.0 a new feature got introduced to convert different type of information from CRs into metrics.

Within Giant Swarm we already use KSM to monitor all the different CAPI related CRs in the so called cluster-api-monitoring-app.
This app was initially build for CAPI only but it's an easy one to monitor other CRs as well (e.g. demo PR to monitor flux CRs).

For that reason i would like to have a generic app within GiantSwarm to monitor all the known CRs which are relevant for day-2 operation.
Instead of writing custom monitoring/metrics exporter i would like to define the generated metrics in the KSM based CustomResourceDefinition format.

Describe alternatives you've considered

Writing own metrics exporter per CRD controller

Additional context

• KSM is working towards an own CRD where every app ship the monitoring-configuration as CR
  • KSM WIP PR
• KSM WIP PR for a metric-gen tool which can be used in upstream + own controllers for the custom-metric-configuration
  • WIP - implement metric-gen tool
• GiantSwarm: Current CAPI Monitoring documentation
• GiantSwarm: Follow up from ClusterAPI custom resource monitoring #37
• GiantSwarm: SIG-Monitoring session about cluster-api-monitoring-app (recording + slides + slack thread)