Secure forwarding

[natefoo]

I've been using the double-proxy forwarding option for usegalaxy.org to run GxITs on a K8s cluster for a while now. Because that cluster is on the same network as Galaxy itself, the fact that forwarded communication is insecure was not a huge issue. However, we are migrating to a K8s cluster running in Jetstream2, which means that proxy traffic will be routed over the Internet. Thus it becomes much more imperative to secure it.

Longer term we may switch to the ingress-based configuration that @almahmoud developed, but that only works with the Galaxy native K8s runner, not the Pulsar Coexecution runner, which is what we're using. @jmchilton's K8s deployment of the proxy is working fine in testing (as it did on the old internal K8s cluster), but I don't want to put it in production until we can secure it.

Some ideas:

1. Add both client and server SSL support directly to gx-it-proxy, also will need a way to install and/or generate certs in k8s.
2. Run an nginx pod that does cert management, terminates SSL, and proxies the k8s-side gx-it-proxy.
3. Use ingress to expose the k8s-side proxy on port 443, since it already takes care of certs. @almahmoud and I tried this out today and it works with hackery, but the ingress service is wide open to the Internet, and we probably don't want to restrict it down using security groups since ingress also serves Rancher, CloudMan, etc., so that leaves the k8s-side proxy encrypted but insecure.
4. Use ingress as in the previous example, but add a secure token/symmetric key to gx-it-proxy that gets sent and validated when forwarding.
5. Use IngressClasses? I guess?? to run an additional ingress on a different port and then restrict that port with security groups.
6. Move encryption down a layer and use some kind of symmetric key exchange algorithm other than SSL/TLS, which just requires sharing the secret between the two gx-it-proxies and removes the need to generate, update, and validate certs. An encrypted socat or ssh tunnel, for example.

Reading back over this missive, I think the simplest and most broadly useful solution is probably 4, just add a shared secret/token, https client support to gx-it-proxy, and let ingress handle the rest.

[hexylena]

7. Use wireguard

pretty sure there's a way to do that in k8s with a sidecar, and it removes the issue completely. All networking goes over wireguard and your connection from the gx-it-proxy can do that as well.

[natefoo]

You might not even need the k8s-side proxy in that case? I'll have to do some more reading but if wireguard creates a VPN on the Galaxy side such that the k8s network is accessible directly on the Galaxy node, then I think you just need the one proxy.

[hexylena]

yeah it creates a mesh network. you can choose to expose internal networks as well, I believe, so once you expose the k8s subnet from a wireguard instance running in k8s, talking to wireguard on your galaxy side, then it should function as essentially a one way tunnel into k8s. maybe.

[natefoo]

7. Use wireguard

Yeah this is it right here. Using a tailscale subnet router (in k8s), it Just Works™, no double proxy required.