Wait for the old CRD Manager to stop before starting a new one

[ptodev]

PR Description

This PR fixes an issue which was reported on the community slack. A person ran into this error when doing a config reload:

failed to register service discovery metrics: failed to create service discovery refresh metrics
error running crd manager

I could not reproduce the error, but I think it happens due to the new CRD Manager starting up before the old one has had a chance to stop and unregister its metrics. I'm not sure how to test this in a unit test, since we'd need to make the CRD Manager stop more slowly somehow. We'd probably have to refactor the code to be more unit testable, so for now I hope we can fix the bug without unit testing it.

I tested my change locally with a config like this, just to make sure the waitgroup functions ok:

Alloy config

discovery.kubernetes "nodes" {
  kubeconfig_file = "/Users/paulintodev/.kube/config"
  role = "node"
}

discovery.kubernetes "services" {
  kubeconfig_file = "/Users/paulintodev/.kube/config"
  role = "service"
}

// ServiceMonitor resources
prometheus.operator.servicemonitors "servicemonitors" {
  client {
    kubeconfig_file = "/Users/paulintodev/.kube/config"
  }
  clustering {
    enabled = false
  }
  forward_to = []
}

// PodMonitor resources
prometheus.operator.podmonitors "podmonitors" {
  client {
    kubeconfig_file = "/Users/paulintodev/.kube/config"
  }
  clustering {
    enabled = false
  }
  forward_to = []
}

// Probe resources
prometheus.operator.probes "probes" {
  client {
    kubeconfig_file = "/Users/paulintodev/.kube/config"
  }
  clustering {
    enabled = false
  }
  forward_to = []
}

logging {
  level = "info"
  format = "logfmt"
}

I changed the clustering/enabled value, then triggered a config reload via curl localhost:12345/-/reload. Then I shut down Alloy using Ctrl + C. The reload and shutdown went ok.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[mattdurham]

Is it possible to get a test on this? Other than that looks good.

[ptodev]

I pushed a commit which outlines how a test could potentially work. Would this be ok? I am not sure how else to test it.