High Availability Considerations docs outdated

[nijave]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.3", GitCommit:"6813625b7cd706db5bc7388921be03071e1a492d", GitTreeState:"clean", BuildDate:"2024-03-15T00:06:16Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

• Cloud provider or hardware configuration:
  Ubuntu VMs
• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel (e.g. uname -a):

Linux vmubtkube-a01 5.15.0-1054-kvm #59-Ubuntu SMP Thu Mar 14 16:03:41 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Container runtime (CRI) (e.g. containerd, cri-o):
  cri-o
• Container networking plugin (CNI) (e.g. Calico, Cilium):
  Calico
• Others:

What happened?

haproxy was still including nodes which returned non 200 health checks. I attempted to troubleshoot but there's been significant changes since haproxy v2.1 so documentation isn't readily available. It seems most likely the health check was running over HTTP (plaintext) and ignoring the returned 400 error code. In addition, haproxy v2.1 is no longer officially supported.

I also observed the very low timeouts in haproxy lead to frequent termination of kubectl ... -w and kubectl logs -f

Edit: I think it may also be possible that ssl-hello-chk option in the guide is overriding httpchk which would also explain the behavior I was seeing. https://cbonte.github.io/haproxy-dconv/2.1/configuration.html#5.2-check

https://endoflife.date/haproxy

What you expected to happen?

The guide to provide a setup using software still supported (patched) by the vendor

How to reproduce it (as minimally and precisely as possible)?

Attempt to use config in haproxy v2.8 docs (currently the default LTS) and it fails due to syntax changes

Anything else we need to know?

For keepalived check, I ended up with

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl -sfk --max-time 2 https://localhost:5443/healthz -o /dev/null   || errorExit "Error GET https://localhost:5443/healthz

It's unclear in the guide why haproxy is only being health checked if it's running on the VIP. The guide configuration could allow keepalived to move the VIP to a node with working API server and broken haproxy which will fail and it will be moved again. Additionally, it doesn't check the health endpoint so the VIP could be moved to a node that's misconfigured but still responds to requests.

For haproxy 2.8, I ended up with the following (static pod)

global
    log stdout format raw local0
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          1800s
    timeout server          1800s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:5443
    mode tcp
    option tcplog
    default_backend apiserverbackend

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserverbackend
    option httpchk

    http-check connect ssl
    http-check send meth GET uri /healthz
    http-check expect status 200

    mode tcp
    balance roundrobin
    server vmubtkube-a01 172.16.1.65:6443 check verify none
    server vmubtkube-a02 172.16.1.66:6443 check verify none
    server vmubtkube-a03 172.16.1.67:6443 check verify none

I don't know much about haproxy, but I'm not sure how the health check worked before unless haproxy v2.1 didn't validate certificates by default.

I reached these conclusions with the following tests:

• kill etcd on a control plane node rendering api-server inoperable (it responds 500 to /healthz)
• kill haproxy on a node while the control plane is running (other haproxy instances should still route to the node since api-server is operable)
• reboot the node with the VIP, assuming it takes a non trivial amount of time (30 seconds or more), VIP should move to a new node and haproxy running on existing nodes should remove the rebooting node api-server. On completion of reboot, haproxy instances should be available on all nodes with all api-server backends active

[nijave]

I can create a PR but a sanity check on the above conclusions first would be appreciated

[neolit123]

@nijave FTR, we haven't gotten complaints from other users about the guide. and yes, it has users.

@mbert do the above suggestions seem good to you?

/kind documentation
/area ha

[sftim]

Is this also relevant to SIG Docs? I can't tell.

[sftim]

If kubeadm is OK and a page inside https://k8s.io/docs/ is out of date: transfer this to k/website and label it for SIG Cluster Lifecycle.

[nijave]

If kubeadm is OK and a page inside https://k8s.io/docs/ is out of date: transfer this to k/website and label it for SIG Cluster Lifecycle.

This one which I think official docs link to (but not seeing the link at the moment) https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

[neolit123]

yes, https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md ended up in this repo.

the link to it is here:

Read the Options for Software Load Balancing guide for more details.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#first-steps-for-both-methods

[sftim]

If we can move that doc into the main website, I think that benefits our users. Not required though.

[neolit123]

moving it might break existing URL references.

also frankly, i would prefer if kubeadm docs eventually start moving in the other direction - i.e. to this repo. if this repo starts hosting kubeadm source code and it has versioned branches.

the kubeadm docs can be hosted similarly to other projects with netlify / hugo:

• https://kind.sigs.k8s.io/
• https://cluster-api.sigs.k8s.io/
• https://minikube.sigs.k8s.io/docs/

but that's a discussion for another time.
and it's a lot of work.

[mbert]

@nijave FTR, we haven't gotten complaints from other users about the guide. and yes, it has users.

@mbert do the above suggestions seem good to you?

/kind documentation /area ha

From first glance (have been unable so far to thoroughly review things in my development environment): it is true that HAProxy has undergone some changes leading to different configuration files. If this turns out to be an issue here, then providing two sets of example configuration would make sense.

As the rest of the report is concerned I will need to take a closer look. Improvements are always welcome. I hope to be able to do this on the weekend.

[neolit123]

thanks @mbert
if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

[mbert]

thanks @mbert if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

Let's see. For now still both versions are used in the field, because the older one is still present in EL distros.

[neolit123]

thanks @mbert if we can include only the latest config, that seems better to me, so that we can avoid being in the business of tracking N versions.

Let's see. For now still both versions are used in the field, because the older one is still present in EL distros.

that's a good point. users sometimes just install from the distro packaging.

[mbert]

I have now had some time to read through everything. First of all: I totally agree with @nijave - the guide is outdated here, and I think a PR with the proposed changes would be very welcome.

Actually the examples in the guide were, IIRC, created using HAProxy 1.8 on an EL7 platform, and given the changes in HAProxy since then the configuration example should really get updated. Since what I still have (I am not actively using the setup in my environment, hence experimenting would require me setting things up again before) is all based on that "ancient" HAProxy I cannot quickly provide a configuration for version 2.1, because I have never had one in use. Long story short: I propose to provide the configuration example for 2.8 as seen above (assuming that it has been tested and works) along with mentioning the version and the fact that this may not work for other versions.

Regarding the health check: again, I totally agree that the HAProxy should be checked on all nodes, not only the one with the VIP. Thank you for noticing!

[neolit123]

@nijave would you send a PR for this as you suggested earlier?

[larsskj]

I'm running HA Proxy 2.4 (default in Ubuntu 22.04) without any custom healthcheck setups - and it works just fine:

frontend kube-cph
        bind :::6443 v4v6
        mode tcp
        timeout client 600s
        option tcplog
        default_backend kube-cph-api

backend kube-cph-api
        mode tcp
        timeout server 600s
        option log-health-checks
        server kube01 kube01:6443 check
        server kube02 kube02:6443 check
        server kube03 kube03:6443 check

I've been running an HA Proxy loadbalancer in front of my K8s clusters with similar configurations for at least four years, on several clusters, never had any problems.

[nijave]

@nijave would you send a PR for this as you suggested earlier?

Yeah, give me a few days

[nijave]

I'm running HA Proxy 2.4 (default in Ubuntu 22.04) without any custom healthcheck setups - and it works just fine:

frontend kube-cph
        bind :::6443 v4v6
        mode tcp
        timeout client 600s
        option tcplog
        default_backend kube-cph-api

backend kube-cph-api
        mode tcp
        timeout server 600s
        option log-health-checks
        server kube01 kube01:6443 check
        server kube02 kube02:6443 check
        server kube03 kube03:6443 check

I've been running an HA Proxy loadbalancer in front of my K8s clusters with similar configurations for at least four years, on several clusters, never had any problems.

I was able to get a working setup when using the current guide, however it didn't handle failures scenarios correctly (so it was load balanced, but not really highly available). With your setup, it doesn't look like you're checking api-server health, only if it accepts TCP connections. If api-server health check is failing, haproxy will still route traffic to those instances. An easy test is killing etcd on a node and observing api-server is still running but returning an error code for /healthz (in which case it should be removed from active backends in haproxy)

[larsskj]

You're probably right. At home I have a bare metal cluster, and I routinely update the nodes, meaning that Kubernetes will be shutdown during reboots.

So far HA Proxy have handled this situation without any hickups - but let me have a look when I return home.

[larsskj]

Did some further testing: This works for me.

backend kube-cph-api
        mode tcp
        timeout server 600s

        option log-health-checks
        option httpchk GET /healthz

        server kube01 kube01:6443 check check-ssl verify none
        server kube02 kube02:6443 check check-ssl verify none
        server kube03 kube03:6443 check check-ssl verify none

[nijave]

Poked around and it looks like Ubuntu and recent version of RHEL (clones) are on HAProxy v2.4 (LTS) which appears to also work with the config I mentioned above

OS - version (EOL date for community support)

RHEL 7 - v1.5.18 (30 Jun 2024)
RHEL 8 - v1.8.27 (31 May 2029)
RHEL 9 - v2.4.22 (31 May 2032)

Ubuntu 20.04 - v2.0.33 (02 Apr 2025)
Ubuntu 22.04 - v2.4.24 (01 Apr 2027)
Ubuntu 24.04 - v2.8.5 (25 Apr 2036)

[PunkFleet]

Official documentation is incorrect on cent stream9

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        retries                 1
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend apiserver
        bind *:5000
        mode tcp
        option tcplog
        default_backend apiserverbackend
backend apiserverbackend
        mode tcp
#       option httpchk GET /healthz
        option httpchk
        http-check connect ssl
        http-check send meth GET uri /healthz
        http-check expect status 200    
        balance     roundrobin
        default-server check inter 3s fall 3 rise 2
        server master1 10.99.99.4:6443 check ssl verify none
        server master2 10.99.99.9:6443 check ssl verify none
        server master3 10.99.99.8:6443 check ssl verify none

When i run haproxy -f /etc/haproxy/haproxy.cfg -c got this:

[NOTICE]   (25791) : haproxy version is 2.8.5-1ubuntu3
[NOTICE]   (25791) : path to executable is /usr/sbin/haproxy
[WARNING]  (25791) : config : 'option forwardfor' ignored for frontend 'apiserver' as it requires HTTP mode.
[WARNING]  (25791) : config : 'option forwardfor' ignored for backend 'apiserverbackend' as it requires HTTP mode.
Warnings were found.
Configuration file is valid

[nijave]

Official documentation is incorrect on cent stream9

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        retries                 1
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend apiserver
        bind *:5000
        mode tcp
        option tcplog
        default_backend apiserverbackend
backend apiserverbackend
        mode tcp
#       option httpchk GET /healthz
        option httpchk
        http-check connect ssl
        http-check send meth GET uri /healthz
        http-check expect status 200    
        balance     roundrobin
        default-server check inter 3s fall 3 rise 2
        server master1 10.99.99.4:6443 check ssl verify none
        server master2 10.99.99.9:6443 check ssl verify none
        server master3 10.99.99.8:6443 check ssl verify none

When i run haproxy -f /etc/haproxy/haproxy.cfg -c got this:

[NOTICE]   (25791) : haproxy version is 2.8.5-1ubuntu3
[NOTICE]   (25791) : path to executable is /usr/sbin/haproxy
[WARNING]  (25791) : config : 'option forwardfor' ignored for frontend 'apiserver' as it requires HTTP mode.
[WARNING]  (25791) : config : 'option forwardfor' ignored for backend 'apiserverbackend' as it requires HTTP mode.
Warnings were found.
Configuration file is valid

What do you mean by "incorrect"? Your output mentions a warning but also says "Configuration file is valid"

[PunkFleet]

Official documentation is incorrect on cent stream9

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        retries                 1
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend apiserver
        bind *:5000
        mode tcp
        option tcplog
        default_backend apiserverbackend
backend apiserverbackend
        mode tcp
#       option httpchk GET /healthz
        option httpchk
        http-check connect ssl
        http-check send meth GET uri /healthz
        http-check expect status 200    
        balance     roundrobin
        default-server check inter 3s fall 3 rise 2
        server master1 10.99.99.4:6443 check ssl verify none
        server master2 10.99.99.9:6443 check ssl verify none
        server master3 10.99.99.8:6443 check ssl verify none

When i run haproxy -f /etc/haproxy/haproxy.cfg -c got this:

[NOTICE]   (25791) : haproxy version is 2.8.5-1ubuntu3
[NOTICE]   (25791) : path to executable is /usr/sbin/haproxy
[WARNING]  (25791) : config : 'option forwardfor' ignored for frontend 'apiserver' as it requires HTTP mode.
[WARNING]  (25791) : config : 'option forwardfor' ignored for backend 'apiserverbackend' as it requires HTTP mode.
Warnings were found.
Configuration file is valid

What do you mean by "incorrect"? Your output mentions a warning but also says "Configuration file is valid"

My bad and my wording was a bit severe. What I mean is that there are alarm messages in the official configuration, which seems a bit disturbing, especially to newbies

[neolit123]

@PunkFleet if you remove the forwardfor line from defaults does it still work and not produce a warning?

[PunkFleet]

@PunkFleet if you remove the forwardfor line from defaults does it still work and not produce a warning?

Yup, It's vaild and not any warning now

[neolit123]

@PunkFleet if you remove the forwardfor line from defaults does it still work and not produce a warning?

Yup, It's vaild and not any warning now

if you find the warning bothersome you can help us by sending a PR for the document.

[nijave]

@PunkFleet if you remove the forwardfor line from defaults does it still work and not produce a warning?

Yup, It's vaild and not any warning now

I'd open a new issue and ideally PR. When I did the updates earlier, I was trying to do the minimal to get certain failure scenarios working and didn't validate all the config (I've never used haproxy before so don't have much experience with it)

Looking over haproxy docs, seems forwardfor is only useful for http proxy mode (this is set to tcp) so probably should be removed. It's possible there's other config that could use updated