Using Stunner for a webrtc broadcaster

[nunoribeiro-tw]

I am trying to run the following setup
A device A is sending video to a cluster on device B. On this cluster we have a stunner running and a webrtc broadcast that receives the incoming stream from device A and broadcasts it to whomever connects to it afterwards. Currently we have 2 pods in the cluster connecting to the broadcast and consuming the stream without problem. However when we try to add a browser (either on device A or a device C) to consume from the broadcast there are some issues where the stunner doesnt seem to be able to provide a viable path for the ice connection to be established.

Am I missing something in how the stunner is supposed to work? Should this scenario be doable or is there something that makes this undoable?

kubernetes setup:
I am working on a Arm Mac so I am using multipass with kubeadm to run the cluster and portforwarding the service ports for local network access

[levaitamas]

My understanding is: you can connect to your broadcast app in k8s via STUNner from device A. Pods inside your cluster can reach the broadcast app too. The problem is, a webbrowser outside on the cluster cannot reach the broadcast app via STUNner.

If that's the situation, my first assumption is there is an error in the ICE process at your browser side. I recommend to check your ICE candidates with webrtc-internals (chrome://webrtc-internals in Chromium/Chrome) first. BTW what is the broadcast app?

Last but not least, if it is not a secret, mind to tell a bit about your use case. Thanks!

[nunoribeiro-tw]

my stunner.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: stunner-gatewayclass
spec:
  controllerName: "stunner.l7mp.io/gateway-operator"
  parametersRef:
    group: "stunner.l7mp.io"
    kind: GatewayConfig
    name: stunner-gatewayconfig
    namespace: stunner
  description: "STUNner is a WebRTC media gateway for Kubernetes"

---

apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-gatewayconfig
  namespace: stunner
spec:
  realm: stunner.l7mp.io
  authRef:
    name: stunner-auth-secret
    namespace: stunner

---

apiVersion: v1
kind: Secret
metadata:
  name: stunner-auth-secret
  namespace: stunner
type: Opaque
stringData:
  type: static
  username: user-1
  password: pass-1

---

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: udp-gateway
  namespace: stunner
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: udp-listener
      port: 3478
      protocol: TURN-UDP
      

---

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tcp-gateway
  namespace: stunner
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: tcp-listener
      port: 3478
      protocol: TURN-TCP

---

apiVersion: stunner.l7mp.io/v1
kind: UDPRoute
metadata:
  name: media-plane
  namespace: stunner
spec:
  parentRefs:
    - name: udp-gateway
    - name: tcp-gateway
  rules:
    - backendRefs:
        - name: media-plane
          namespace: default

[levaitamas]

So this is one of the ICE candidates we get on the web browser and we also succesfully send it to the broadcast

candidate: 
  RTCIceCandidate
  address: "10.44.0.7"
  candidate:  "candidate:834171966 1 udp 16785407 10.44.0.7 51824 typ relay raddr 0.0.0.0 rport 0 generation 0 ufrag r7VI network-cost 999"
  component: "rtp"
  foundation: "834171966"
  port: 51824
  priority: 16785407
  protocol: "udp"
  relatedAddress: "0.0.0.0"
  relatedPort: 0
  relayProtocol: "tcp"
  sdpMLineIndex: 0
  sdpMid: "0"
  tcpType: null
  type: "relay"
  url: "turn:192.168.1.165:3478?transport=tcp"
  usernameFragment: "r7VI"

This looks good. The type is set to relay, which is correct.

[levaitamas]

Also the STUNer config looks okay. I would recommend to doublecheck the media-plane service and your port forwarding. Can you show the output of kubectl describe svc media-plane?

[nunoribeiro-tw]

I'm not sure what media-plane would be in my case as I don't have a service with that name. I am portforwarding the tcp-gateway using kubectl port-forward --address 0.0.0.0 -n stunner services/tcp-gateway 3478:3478

~ » k get all -A                                                                                                                                                                       
\NAMESPACE        NAME                                                               READY   STATUS    RESTARTS   AGE
default          pod/broadcast-server-bb45b9685-66xvt                               1/1     Running   0          55m
kube-system      pod/coredns-76f75df574-hgf7p                                       1/1     Running   0          9d
kube-system      pod/coredns-76f75df574-nf4mw                                       1/1     Running   0          9d
kube-system      pod/etcd-kubemaster                                                1/1     Running   0          9d
kube-system      pod/kube-apiserver-kubemaster                                      1/1     Running   0          9d
kube-system      pod/kube-controller-manager-kubemaster                             1/1     Running   1          9d
kube-system      pod/kube-proxy-bjpfv                                               1/1     Running   0          9d
kube-system      pod/kube-proxy-sk9jf                                               1/1     Running   0          7d2h
kube-system      pod/kube-scheduler-kubemaster                                      1/1     Running   1          9d
kube-system      pod/weave-net-fwhcr                                                2/2     Running   0          6d21h
kube-system      pod/weave-net-tpmkn                                                2/2     Running   0          6d21h
metallb-system   pod/controller-7674b76b75-nx478                                    1/1     Running   0          6d21h
metallb-system   pod/speaker-cr8lh                                                  1/1     Running   0          6d21h
metallb-system   pod/speaker-stp55                                                  1/1     Running   0          6d21h
stunner-system   pod/stunner-auth-7455878d88-wr2jk                                  1/1     Running   0          7d
stunner-system   pod/stunner-gateway-operator-controller-manager-79495c87bc-6l8pc   2/2     Running   0          7d
stunner          pod/tcp-gateway-8599cc888d-v69f8                                   1/1     Running   0          3d20h
stunner          pod/udp-gateway-7864d8b56-bzqpm                                    1/1     Running   0          3d18h

NAMESPACE        NAME                                                                  TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                  AGE
default          service/broadcast-service                                             LoadBalancer   10.99.99.146     192.168.64.240   50000:32471/TCP          6d21h
default          service/broadcast-service-browser                                     LoadBalancer   10.96.122.134    192.168.64.245   51000:32344/TCP          5d
default          service/broadcast-tcp                                                 ClusterIP      10.109.59.197    <none>           5001/TCP                 7d2h
default          service/kubernetes                                                    ClusterIP      10.96.0.1        <none>           443/TCP                  9d
kube-system      service/kube-dns                                                      ClusterIP      10.96.0.10       <none>           53/UDP,53/TCP,9153/TCP   9d
metallb-system   service/metallb-webhook-service                                       ClusterIP      10.104.6.247     <none>           443/TCP                  7d2h
stunner-system   service/stunner-auth                                                  ClusterIP      10.97.10.47      <none>           8088/TCP                 7d
stunner-system   service/stunner-config-discovery                                      ClusterIP      10.104.84.179    <none>           13478/TCP                7d
stunner-system   service/stunner-gateway-operator-controller-manager-metrics-service   ClusterIP      10.105.164.193   <none>           8443/TCP                 7d
stunner          service/tcp-gateway                                                   LoadBalancer   10.109.191.122   192.168.64.241   3478:31790/TCP           3d20h
stunner          service/udp-gateway                                                   LoadBalancer   10.103.86.120    192.168.64.244   3478:30715/UDP           3d18h

NAMESPACE        NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system      daemonset.apps/kube-proxy   2         2         2       2            2           kubernetes.io/os=linux   9d
kube-system      daemonset.apps/weave-net    2         2         2       2            2           <none>                   9d
metallb-system   daemonset.apps/speaker      2         2         2       2            2           kubernetes.io/os=linux   7d2h

NAMESPACE        NAME                                                          READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/broadcast-server                              1/1     1            1           5d2h
kube-system      deployment.apps/coredns                                       2/2     2            2           9d
metallb-system   deployment.apps/controller                                    1/1     1            1           7d2h
stunner-system   deployment.apps/stunner-auth                                  1/1     1            1           7d
stunner-system   deployment.apps/stunner-gateway-operator-controller-manager   1/1     1            1           7d
stunner          deployment.apps/tcp-gateway                                   1/1     1            1           3d20h
stunner          deployment.apps/udp-gateway                                   1/1     1            1           3d18h

NAMESPACE        NAME                                                                     DESIRED   CURRENT   READY   AGE
default          replicaset.apps/broadcast-server-567697cdc5                              0         0         0       3d
default          replicaset.apps/broadcast-server-5798798794                              0         0         0       3d18h
default          replicaset.apps/broadcast-server-67776c665c                              0         0         0       88m
default          replicaset.apps/broadcast-server-765f568c9c                              0         0         0       3d1h
default          replicaset.apps/broadcast-server-775bcb7cbf                              0         0         0       113m
default          replicaset.apps/broadcast-server-797557d59                               0         0         0       57m
default          replicaset.apps/broadcast-server-798b9d548b                              0         0         0       147m
default          replicaset.apps/broadcast-server-7c88c747bc                              0         0         0       76m
default          replicaset.apps/broadcast-server-975ddcbfb                               0         0         0       3d1h
default          replicaset.apps/broadcast-server-bb45b9685                               1         1         1       55m
default          replicaset.apps/broadcast-server-d86976b7f                               0         0         0       62m
kube-system      replicaset.apps/coredns-76f75df574                                       2         2         2       9d
metallb-system   replicaset.apps/controller-6748bb4969                                    0         0         0       6d23h
metallb-system   replicaset.apps/controller-7674b76b75                                    1         1         1       6d21h
metallb-system   replicaset.apps/controller-7bc8c49b84                                    0         0         0       6d23h
metallb-system   replicaset.apps/controller-86d64fd765                                    0         0         0       6d23h
stunner-system   replicaset.apps/stunner-auth-7455878d88                                  1         1         1       7d
stunner-system   replicaset.apps/stunner-gateway-operator-controller-manager-79495c87bc   1         1         1       7d
stunner          replicaset.apps/tcp-gateway-8599cc888d                                   1         1         1       3d20h
stunner          replicaset.apps/udp-gateway-7864d8b56                                    1         1         1       3d18h

[nunoribeiro-tw]

k describe -n stunner services/tcp-gateway                                nuno@Nunos-MBP
Name:                     tcp-gateway
Namespace:                stunner
Labels:                   stunner.l7mp.io/owned-by=stunner
                          stunner.l7mp.io/related-gateway-name=tcp-gateway
                          stunner.l7mp.io/related-gateway-namespace=stunner
Annotations:              metallb.universe.tf/ip-allocated-from-pool: first-pool
                          stunner.l7mp.io/related-gateway-name: stunner/tcp-gateway
Selector:                 app=stunner,stunner.l7mp.io/related-gateway-name=tcp-gateway,stunner.l7mp.io/related-gateway-namespace=stunner
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.109.191.122
IPs:                      10.109.191.122
LoadBalancer Ingress:     192.168.64.241
Port:                     tcp-listener  3478/TCP
TargetPort:               3478/TCP
NodePort:                 tcp-listener  31790/TCP
Endpoints:                10.44.0.2:3478
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason        Age                  From             Message
  ----    ------        ----                 ----             -------
  Normal  nodeAssigned  39m (x7 over 3d20h)  metallb-speaker  announcing from node "kubeworker01" with protocol "layer2"