Use host networking

[sjpb]

NB: based on feat/hostport as that is required for host networking.

Uses host networking to avoid MPI errors and give better network performance.

Using host networking means the pod's hostname = k8s node name, e.g. sbtest-worker-a438ab51-c69dx. This isn't "slurm hostlist expression compatible" so e.g. sinfo can't contract the nodenames. This PR uses the downward API to inject the pod name (which is hostlist expression compatible e.g. slurmd-0 as using a StateFullSet) into the container's environment vars, and explicitly sets the slurm'd nodename on startup using slurmd -N <nodename>.

Example of performance changes on arcus using portal-internal network (i.e. not RoCE), showing 0-byte, max bandwidth max message size values from srun-launched IMB-MPI1 pingpong using openmpi in image:

Default CNI :

       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000        43.03         0.00
...
      1048576           40      5046.30       207.79
...
      4194304           10     24296.99       172.63

Host networking:

       #bytes #repetitions      t[usec]   Mbytes/sec
            0         1000        34.75         0.00
...
      2097152           20      5929.08       353.71
      4194304           10     12546.33       334.31

(note in both cases the K8s node VMs are on the same hypervisor host).

[sjpb]

@sd109 checked OOD dashboard and jobs etc work anyway.

[sd109]

Nice one, lgtm