nodeSelector always chooses the same node

[adampetrovic]

I have node-role.kubernetes.io/worker=true set as a label on 4 of my 7 nodes:

➜ kubectl get nodes --selector='node-role.kubernetes.io/worker=true'
NAME         STATUS   ROLES    AGE   VERSION
k8s-node-2   Ready    worker   13d   v1.27.3+k3s1
k8s-node-3   Ready    worker   32d   v1.27.3+k3s1
k8s-node-4   Ready    worker   32d   v1.27.3+k3s1
k8s-node-5   Ready    worker   32d   v1.27.3+k3s1

For all services under my media namespace, I have set

    nodeSelector:
      node-role.kubernetes.io/worker: "true"

after deploying these pods, they only ever get scheduled on k8s-node-4, despite there being 3 other worker nodes. I have noticed this behaviour in other namespaces too.

➜ kubectl get pods -n media -o wide
NAME                           READY   STATUS    RESTARTS        AGE     IP            NODE         NOMINATED NODE   READINESS GATES
bazarr-0                       2/2     Running   0               4d21h   10.42.3.42    k8s-node-4   <none>           <none>
overseerr-7fd4b6df9c-x28sn     1/1     Running   0               5d5h    10.42.3.83    k8s-node-4   <none>           <none>
prowlarr-0                     1/1     Running   0               2d4h    10.42.3.220   k8s-node-4   <none>           <none>
qbittorrent-7f84f58ccf-xpr9d   1/1     Running   1 (4d17h ago)   5d17h   10.42.3.188   k8s-node-4   <none>           <none>
radarr-0                       1/1     Running   0               2d4h    10.42.3.12    k8s-node-4   <none>           <none>
radarr-4k-0                    1/1     Running   0               2d4h    10.42.3.193   k8s-node-4   <none>           <none>
sabnzbd-0                      1/1     Running   0               5d4h    10.42.3.128   k8s-node-4   <none>           <none>
sonarr-0                       1/1     Running   0               2d4h    10.42.3.27    k8s-node-4   <none>           <none>
sonarr-4k-0                    1/1     Running   0               16h     10.42.3.212   k8s-node-4   <none>           <none>
tautulli-0                     1/1     Running   0               2d4h    10.42.3.232   k8s-node-4   <none>           <none>
unpackerr-0                    1/1     Running   0               2d4h    10.42.3.233   k8s-node-4   <none>           <none>

Any ideas on why this is happening?

There are no taints or tolerations set on the other (unused) worker nodes. I've also tried removing the quotes around the nodeSelector's true value, but get an error.

[onedr0p]

It sounds like the scheduler really likes k8s-node-4 🙂 I'm pretty sure this is normal, kubernetes doesn't have any smart scheduling to balance pods across the worker nodes. If you cordon/drain k8s-node-4 do the pods get shifted to another node?