Helm Install Timeout Leader Election Multiple Gloo Replicas

[alexstaroselsky]

When trying to install open source gloo edge 1.16.3, helm times out (5 minutes) waiting for the gloo deployment. No pods fail to start, no jobs error. Looking at the logs for one of the two gloo replicas it has the message:

1 leaderelection.go:250] attempting to acquire leader lease gloo-system/gloo...

The other gloo replica indicates successful leader election with message:

successfully acquired lease gloo-system/gloo.

Here are the value where each deployment has multiple replicas. There doesn't seem to be errors in the gloo replicas other than these messages and no errors around the rollout job. Could this attempting to acquire leader lease gloo-system/gloo... be causing the deployment to fail if helm is waiting for success indicators? Are there logs of specific jobs or pods that should be looked to understand why helm is actually timing out?

settings:
  watchNamespaces: 
    - 'default'
    - 'apps'
gloo:
  logLevel: warn
  deployment:
    replicas: 2
    nodeSelector:
      cool-node-group: "true"
    resources:
      requests:
        cpu: 1000m
        memory: 512Mi
      limits:
        memory: 512Mi
discovery:
  logLevel: warn
  serviceAccount: {}
  deployment:
    replicas: 2
    nodeSelector:
      cool-node-group: "true"
    resources:
      requests:
        cpu: 1000m
        memory: 512Mi
      limits:
        memory: 512Mi
gateway:
  certGenJob:
    nodeSelector:
      cool-node-group: "true"
  cleanupJob:
    nodeSelector:
      cool-node-group: "true"
  rolloutJob:
    nodeSelector:
      cool-node-group: "true"
gatewayProxies:
  gatewayProxy:
    envoyLogLevel: warn
    gatewaySettings:
      disableHttpsGateway: true
    service:
      httpsFirst: true
      loadBalancerSourceRanges:
        #- 10.0.0.0/8
      kubeResourceOverride:
        spec:
          ports:
            - name: http
              port: 80
              targetPort: 8080
            - name: https
              port: 443
              targetPort: 8080
      extraAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "external"
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${acm_cert}"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
        service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: sg-somegroup # add a cloudfront only SG
    podTemplate:
      nodeSelector:
        cool-node-group: "true"
      resources:
        requests:
          cpu: 500m
          memory: 512Mi
        limits:
          memory: 512Mi
    horizontalPodAutoscaler:
      apiVersion: autoscaling/v2
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 60