fix: [release/v0.14.x] Introduce startup probe to CAREN pod to reduce races

[jimmidyson]

Introduce a more frequent startup probe polling every second, which once
it succeeds will trigger readiness and liveness probes to run at their
default intervals. This reduces chances of races in clusterctl checking
for readiness while keeping the overhead on the kubelet for checking
readiness once the pod is up to a minimum (default probe period).

Wrong timeout

[dkoshkin]

I think this is the wrong timeout, and is being set for the helm registry?

[supershal]

with this it will result in following with defaults

readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readyz
        port: healthz
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1

Once the container starts it will probe every 1 second with timeout of 1 seconds and upto 3 consecutive failures. so I guess CAREN should be up and ready in 3 seconds. CAREN starts within a second.

[jimmidyson]

with this it will result in following with defaults

readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readyz
        port: healthz
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1

Once the container starts it will probe every 1 second with timeout of 1 seconds and upto 3 consecutive failures. so I guess CAREN should be up and ready in 3 seconds. CAREN starts within a second.

While you're right, the failure of a readiness probe just removes it from the service endpoints so the pod keeps running (as opposed to liveness probe where the pod will be killed). The readiness probe will still run every second and after a single success will be marked as ready. So this isn't a problem even if it fails.

[jimmidyson]

I've actually changed the implementation here to use startup probe to have the same effect (fast checking for readiness) but not have the overhead on the kubelet of probing every second once the pod is first set to ready (using default readiness probe period once startup probe has succeeded).

[jimmidyson]

@dkoshkin If you could take a look and approve again if you're happy then we can merge.