ABR supported in WebRTC

[getroot]

Finally, ABR is available for WebRTC.
You can test it with the latest master branch.

WebRTC ABR only works with the latest OvenPlayer.

Please refer to the manual for how to use it.

https://airensoft.gitbook.io/ovenmediaengine/v/master-1/streaming/webrtc-publishing#adaptive-bitrates-streaming-abr

The algorithm of Auto ABR (a function that automatically switches rendition by measuring the bandwidth of the player session on the server) must be continuously improved. This needs more testing. If you would like to contribute to improving this feature, you can start with the code below.

OvenMediaEngine/src/projects/publishers/webrtc/rtc_session.cpp

Line 801 in 0d09b5b

bool RtcSession::ProcessRemb(const std::shared_ptr<RtcpInfo> &rtcp_info)

Thanks a lot for your feedback. (The information "It works well" is also very helpful.)

[danruser]

Hello
I've been testing for several hours now and everything is great. Thank you for the good work

[nums]

Hello,

does WebRTC ABR take advantage of the WebRTC simulcast: https://www.w3.org/TR/webrtc-svc/#simulcasttemporal-example* ?

On SFUs that implement this logic, there is no server-side transcoding, it's the "producer" that sends the same stream with several dimensions. Is is the same here ?

On the other hand, for an RTMP source, I imagine that we take advantage of the server-side transcoding.

[getroot]

@nums
OME's WebRTC Provider (Ingest) does not yet support simulcast. OME's WebRTC Publisher (Streaming) provides ABR with server-side transcoding. The simulcast developed in the future will pass-through server-side transcoding.

Supporting simulcast for WebRTC Provider is a low priority as there are not many cases of using WebRTC input with OME yet. Most OME users still use RTMP, SRT, MPEG-TS and RTSP inputs.

The link you wrote is SVC, not simulcast. SVC is also a standard that should be supported someday.

[nums]

@getroot
Thank you for the clarification.
Indeed it is great for "livestreaming" use.

Sorry, the link anchor was missing, I made the edit in my previous comment.

One last question, is the management of the connection / disconnection to the WebRTC stream no longer dependent on the WebSocket connection? I think there was a feed on it but I can't find it anymore.

I will carry out tests on my side with a hundred simultaneous connections

[getroot]

@nums
The WebRTC player must always maintain a WebSocket connection to OME during playback. OME notifies the ABR rendition switching result via this WebSocket session. OME determines that a player that loses a WebSocket connection has closed the connection and cleans up the session.

[basisbit]

Do there exist any rough estimates how much ABR / the changed code impacts CPU or memory requirements in a use case of few concurrent streams but many concurrent viewers?
For example during the tests I did end of last year, WebRTC required roughly 50 to 100% more CPU power per concurrent viewer, compared to having these viewers watch the same stream over HLS (with encryption enabled, 6 seconds chunks).
I'd love to test and provide test results of OME with WebRTC ABR with thousands of concurrent users, but I currently have no "small" events where I could test this, so I at least need some rough estimates of changes in CPU/memory requirements. Thanks!

[getroot]

@basisbit
The most CPU-intensive logic for viewers in WebRTC is SRTP encryption and the send() system call. (WebRTC makes a lot more send() system calls than HLS.)

WebRTC ABR is not affected by this. So it's probably not much different from what you tested at the end of last year.

However, there have been several patches to improve performance in recent months, so the latest version may have reduced CPU usage.

[davidghukasyan]

We've been testing a couple of days now. It works as intended. As mentioned above will require contineous improvement, however I think with the current implementation it's worth to push an official release.
There will be lots of usecases and senarios to shape out in future , including delivering via 3G/4G networks, but at this stage of testing it works fine. We'll try to do also a stress test to see the impact on ABR, Engine and hardware.

[nums]

I have tested with some different setup, it's working great.
I noticed that the "degraded" stream generated a latency of just under 1 second compared to the original stream.

[mpisat]

I tested it today. FWIW Simulcast based ABR works much better. When there are multiple transcoded tracks available in WebRTC stream, browser handles (?) stream switching automatically and it happens smoothly.
OME ABR is pretty usable in my opinion, but not as good as Simulcast ABR.

[getroot]

@mpisat
Simulcast is a specification for how to transmit multiple video tracks simultaneously from an encoder (browser). Instead, the current OME transcodes itself without receiving multiple encoded videos via simulcast.

And ABR is that the server transmits a suitable track according to the viewer's estimated bandwidth among multiple video tracks. This has nothing to do with simulcast. For bandwidth estimation, use REMB or transport-cc specifications.

What specifically does Simulcast based ABR better mean? video quality? Or an accurate bandwidth estimate?

[mpisat]

@getroot I've seen ABR implementation via MediaSoup at an adult entertainment cam site. Resolution changes were happening without any stutter when you limit your bandwidth with netlimiter (win) or network link conditioner (xcode tool -- mac).

For example:

Stats graphs for RTCInboundRTPAudioStream_557470284 (inbound-rtp)
Stats graphs for RTCInboundRTPAudioStream_846147500 (inbound-rtp)
Stats graphs for RTCInboundRTPVideoStream_1234 (inbound-rtp)
Stats graphs for RTCInboundRTPVideoStream_379218108 (inbound-rtp)
Stats graphs for RTCInboundRTPVideoStream_992103075 (inbound-rtp)
Stats graphs for RTCMediaStreamTrack_receiver_15 (track)
Stats graphs for RTCMediaStreamTrack_receiver_16 (track)
Stats graphs for RTCMediaStreamTrack_receiver_17 (track)
Stats graphs for RTCMediaStreamTrack_receiver_18 (track)
Stats graphs for RTCMediaStreamTrack_receiver_19 (track)

These tracks are transcoded on the server side and fed to MediaSoup, which is the SFU and handles media playback requests from the visitors.

I'm no simulcast expert here, but from what I understand, the browser was selecting the video/audio track according to the browser's estimated bandwidth or network speed and handling video playback.

[getroot]

@mpisat

Perhaps OME's ABR and MediaSoup's ABR will estimate bandwidth and switch tracks in the same way. I don't think it's a problem caused by the difference between the encoder encoding method (Simulcast) and the server encoding method.

Do you have problems with stutter when switching ABR in OME?
Maybe you have a mix of bypass and encoded tracks in your playlist? The encoded track is a bit slower than the bypass track, so you may feel like there is a little stutter when you switch between the two. Try composing all the tracks in your playlist as encoded tracks.

[mpisat]

@getroot I used the sample from WebRTC ABR help page, and it consists of entirely transcoded tracks (video + audio), no bypass (raw) video in the list.

When I inspect webrtc-internals page on Chrome, it only contains one video + audio track (not simulcast)

Stats graphs for RTCInboundRTPAudioStream_2100345181 (inbound-rtp)
Stats graphs for RTCInboundRTPVideoStream_2331789112 (inbound-rtp)
Stats graphs for RTCMediaStreamTrack_receiver_1 (track)
Stats graphs for RTCMediaStreamTrack_receiver_2 (track)
Stats graphs for RTCPeerConnection (peer-connection)
Stats graphs for RTCRemoteOutboundRTPAudioStream_2100345181 (remote-outbound-rtp)
Stats graphs for RTCTransport_aBph23_1 (transport)

The difference between Mediasoup + Simulcast ABR vs OME implementation is the number of audio / video tracks and who is switching them (browser vs frontend?)

Please don't get me wrong, I already love OME WebRTC ABR. One of the biggest players in Streaming doesn't even have this feature built in (Wowza).

[mpisat]

When I tested WebRTC ABR today, the stutter was unnoticeable during stream switching. It happens under 200 ms (give or take) and I'm very satisfied with how it performs. Perhaps I had some connectivity issues when I tested it yesterday. I have 180 ms latency to my server in France so I guess that's the culprit.

[natzakaria]

I am getting issues with web sessions disconnects when selecting 1080 on one browser while another browser on 720 and another at 480.
Any suggestions?

Log:
[2022-08-10 08:56:43.093] W [Rescaler:70] ov.Queue | queue.h:268 | [0x7f879819b6e8] Input queue of Encoder. codec(h264/27) size has exceeded the threshold: queue: 227, threshold: 120, peak: 312

[wongfei2009]

Is it possible to ingest a already encoded ABR stream to OvenMediaEngine?

[getroot]

Unfortunately it is not possible

[bchah]

How about this as a workaround... You could use two instances of OME. The first OME instance receives the input stream and encodes the ABR renditions. Then you can use RTMP Push to send each ABR rendition separately to the second OME instance.

[Remboooo]

WebRTC ABR is giving some weird quality downgrades here. The setup is as follows:
OBS at a remote location streaming 1080p 60fps 6Mbps --Internet--> OvenMediaEngine on Ubuntu --Gb LAN--> Chrome on Windows

I've configured 3 qualities (all with the same opus audio track): bypass / 720p 30fps @ 3.5Mbps / 480p 30fps @ 1.5Mbps

So the 1080p RTMP stream is coming in over the internet, but WebRTC from OvenMediaEngine to OvenPlayer is going through Gbit LAN across the room. Sometimes OvenPlayer will still fall back to a lower quality despite the network being excellent and no apparent bottlenecks in processing anywhere. On the server running OME nvtop shows constant GPU encode/decode utilization at 8%, CPU load stays below 1.0, averaging ~0.5.

Any way I could debug what's causing the downgrades?

[getroot]

OME actually uses the bandwidth estimation value measured by the browser (Chrome). You might want to look at availableIncomingBitarte in chrome://webrtc-internals. Chrome uses various methods to estimate bandwidth, and packet loss is one of the important indicators. If there is packet loss, it is also a good idea to use webrtc/tcp provided by OME. In addition, network jitter may also be an important factor. To find out exactly how Chrome does BWE, the best way is to read the Webrtc Native Code, and you might also want to read the RFC for GCC. https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02

[Remboooo]

Interesting, that indeed shows a very low value while streaming; around 1000000~2500000 (which I assume means 1-2.5Mbps). When I test with iperf3 what the actual available bandwidth is between my OME server and my windows machine, the result is ~930MBps. So something in that estimation is going terribly wrong. I'll have to dig deeper into what gets factored into the availableIncomingBitrate calculation.

EDIT: it seems to me that the availableIncomingBitrate is just.. completely messed up. Look at these graphs:

So the actual bitrate just wiggles around 8Mbps, while the availableIncomingBitrate is just all over the place, predominantly way too low. All the while, the packetsLost statistic for both the audio and video streams is at a solid 0. I'm still digging through the GCC source trying to grok aimd_rate_control.cc, but it looks to me like this is a terrible estimator for basing ABR on.