Roadmap for 2023

[getroot]

Happy new year, everyone.

As many of you know because I've mentioned it several times, we spend the first half of the year working on OvenMediaEngine and the second half doing external development projects for survival. I sincerely ask you to become a Github Sponsor so that OvenMediaEngine project can be boosted. Thanks to your contributions, OvenMediaEngine continues to evolve.

Here are our big plans for 2023.

IPv6 support
This is far too late. We will unconditionally deal with this within the first quarter of 2023.

Stabilizing the WebRTC Provider
Now we can spend some time on this. We will fix everything so that WebRTC input works reliably (especially with OvenLiveKit).

Improving Performance and Stabilizing GPU Encoder
In the meantime, the community has been asking a lot about this. We will look into this closely.

WHIP support
We will support WHIP once the WebRTC Provider is stabilized. The time has come to support this.

Stabilizing the SRT Provider
A number of issues have also been raised about the SRT input. Now we can see this.

Advanced AdmissionWebhooks
I found that there are many cases in which the callback of AdmissionWebhooks is linked to their statistics system. We will add meta, such as passing user-agent information, and solve problems such as disconnect events not occurring.

Check and block input quality
If a stream of several tens of megabits or a stream containing b-frames is input, OME may not operate normally. We will review the ability to limit this. This work is a low priority and the plan may be delayed if it is determined that there are too many side effects after design.

Fallback stream
We will develop a function that can send a video file that can be replaced when the input of the main stream is not live.

Origin Pulling Query String
We will develop a function where Edge creates a stream by reading the origin url from the playback URL's query string and pulling the stream from the origin. For example ws://edge.com/app/stream?originUrl="ovt://origin.com/app/stream" would work.

RTMP Pull Provider
We will develop RTMP Pull Provider function. This will make it easy to serve WebRTC or LLHLS streams using an existing RTMP Server. If used together with the Origin Pulling Query String function above, OvenMediaEngine will be valuable as a proxy server that relays the existing RTMP server.

Encoding Profiles
Encoding profile will be provided so that you can set more diversely.

Various API functions
Several APIs are planned, such as certificate reload API and stream deletion API.

Besides this, we have many small features, bug fixes and performance improvements planned. Which feature are you most looking forward to? Or is there something you think is a must-have feature that isn't listed here? Please feel free to comment as much as possible.

[naanlizard]

Re: blocking streams with too high a bitrate or b-frames, perhaps three levels.

1. No checking of input streams

2. Streams checked but not blocked - generate callback or set flag you can query from API

3. Streams blocked

or more practically, just add an API endpoint to query for various stream stats. This may already exist for some things (bitrate?) but I don’t believe exists for b frames

Another thought on the b frame detection, does that take time from the stream beginning? The ux is maybe bad for users if they connect successfully and they are disconnected after a minute or whatever because of b frames being detected. Unless we can send back a custom message to explain the disconnect? I’m not sure if that is possible

Personally we would be interested in knowing if streams are using b-frames, so we can offer or not offer the webrtc player as appropriate

[getroot]

@naanlizard
Thank you for your comments.
We will add various information to the stream information API, including whether b-frames are included or not. And if you use the stream deletion API that will be added, you will be able to block streams on your own.

[naanlizard]

Is the stream delete api just a forcible disconnect from the server side? Disconnect might be the better term if so. Delete makes me think they can’t reconnect 🤔

[command-tab]

Maybe it is (or, will be) effectively a server-side disconnect but necessarily transmitted as an HTTP DELETE call?

[getroot]

It is correct to forcibly close the connection on the server side. But that's the result of deleting the stream.

Actually I will add API like below (currently only GET <Stream> is provided).

DELETE https://host/v1/vhosts/vhost/apps/app/streams/stream

This API does not care about reconnecting terminated sessions. If the user wants to prevent a disconnected session from reconnecting after an API call, AdmissionWebhooks should be used.

To forcibly disconnect the session from the server(To use the term Disconnect session), the following API should be provided. (The stream will be deleted as a result.)

POST https://host/v1/vhosts/vhost/apps/app/streams/stream:banSession
{
   sessionId: "f9dafjda9121"
}

or

DELETE https://host/v1/vhosts/vhost/apps/app/streams/stream/sessions/<session id>

And this can cause misunderstanding that this function can be used not only in the session (encoder) connected to the provider, but also in the session (player) connected to the publisher.

[nums]

First, a big thank you for the evolution of your product.
The addition of the DVR and the iD3 tags injections will allow us to deploy ovenmedia for several of our customers :)

A feature that would be quite important for us is the ability to create a multi-language stream. Technically we currently do this by sending several audio streams in SRT and we generate an HLS with several audio tracks with FFMPEG.
Some of my clients could fund this feature, maybe we can discuss it by email?

Regarding WebRTC, WHIP support would be perfect, is simulcast support planned? I know that there is ABR for webRTC, but I have a slight latency on "degraded" streams and suddenly a very slight desynchronization between the video and the audio, the simulcast in input might be a way of solve this problem?
and last thing, is the interpretation of PLI requests under consideration on your side?

[basisbit]

This list reads like my xmas wish list. Looking forward to all the good work you all are doing there! 🎉 🥳 ❤️

[mpisat]

If WHIP gets added, perhaps WHEP can be added too? This way OME can support industry standards. Cloudflare started WebRTC beta with WHIP/WHEP recently and it is going to help spread this standard. Then we won't be locked down to specific players I guess?

[getroot]

@mpisat If there is a player that actually supports WHEP, and my time and stamina allow me to do it, I will work on it.

[mpisat]

According to cloudflare
https://developers.cloudflare.com/stream/webrtc-beta/

this one does (TypeScript), there is even a demo page
https://www.npmjs.com/package/@eyevinn/webrtc-player

[getroot]

@nums
If you would like to add features faster or add your own features for a fee, please send your requirements and your budget to support@airensoft.com.

I need to think more about supporting simulcast in 2023. I'll think more about what the OME benefits versus the effort put in, and I'll tell you later.
If OME's ABR's A/V sync doesn't work, that's another issue. If you want to fix it, please post it as a new issue later.

I'm not sure what you mean by "the interpretation of PLI requests". Please elaborate.

[getroot]

@mpisat Thanks for sharing good information. It is of course good news that Cloudflare supports it. Although OvenMediaEngine and Cloudflare's WebRTC service cannot be used together.
I decided to support WHIP because OBS started supporting WebRTC and WHIP. Like this, if major players like EXOPlayer, videojs, and vlc support WebRTC and WHEP, I'll be on fire.

[alexthedamager]

Thank you for your continued hard work. WHIP support is particularly exciting!
Are there any plans to add support for codecs such as AV1 or VP9?

[zapf2000]

AV1 support would be great. Open source codec, very powerful, safes a lot of bandwidth and hardware decoding is already supported on various GPU's. Even hardware encoding is now available on new GPU's. So why not implement that.

[nums]

I will re-check WebRTC ABR AV/Sync with the last version of OME and I let you know in a new issue if the problem still occur.

PLI (Picture Loss Indication) is really not a must have but it might be worth thinking about.
When a viewer has significant packet loss it must wait for the next keyframe:
On a video stream at 2 keyframes per second, we can have 1.9 seconds of freeze if on the client side we have lost too many packets.
On the other hand, if we process the PLI request on the server side, we could generate a keyframe "on the fly" and have as little freeze as possible.
Afterwards, you have to effectively limit the number of keyframes generated because this can have a strong impact on the quality of the stream.

[getroot]

@nums
Ah... you meant RTCP PLI.
The WebRTC Provider already requests the PLI to adjust the keyframe interval.
But on the Publisher side it's impossible to do that. It will be easier to understand if you assume that tens of thousands of viewers are concurrently connected. This will make every frame a keyframe. Also, streams that bypass Transcoder cannot handle this either.
In preparation for such cases, we already provide WebRTC FEC and RTX functions to reduce packet loss, and provide TCP Transport functions. Sending WebRTC over TCP has no packet loss issues, so I recommend you use this.

[basisbit]

WebRTC over TCP just has different packet loss issues. Typically when a peering/transit point is congested and when the distance between edge server and client are big. Additionally, WebRTC over TCP introduces playback delay issues when the playback buffer runs empty. WebRTC over UDP with RED (only on-demand per connection) and RTX and FEC usually is better suited for very low latency playback from my experience.
TCP is good if you use medium or big playback buffers and/or if your edge servers are very close (latency) to the users.

[nums]

Thanks @getroot and @basisbit for your feedbacks.

Actually recovering the RTCP PLI request is earlier for videoconferencing use with few participants. I know Mediasoup has for example added a Max Request Keyframe Interval parameter but yes, I don't think it's very interesting to implement it on OvenMedia.

Otherwise, I re-tested with the latest WebRTC ABR version and I actually have no more desynchronization between the different bitrates. In my opinion WebRTC simulcast is therefore not very relevant in view of the very good working of WebRTC ABR.

Thanks for all, the Roadmap 2023 seems perfect to me. I also sent an email to support regarding the management of multilingual.

[stevespaw]

I would love to have SRT-"output" for an SRT Re-stream. This would simplify our workflow greatly!

[choigilhoon]

The Push Publishing feature for SRT re-streaming has been added to the master branch.
Please check this discussion for more details: #1299

[kakaruta]

Is AV1 support from the 2022 roadmap missing from the 2023 roadmap?

Does OME have plans to support AV1 Codec?

[getroot]

@kakaruta AV1 will of course be implemented in OME. However, it is not yet widely used in a commercial environment, so it has been pushed out of priority.

[getroot]

We've done the following so far: We are working very hard. :-)

• IPv6 support (IPv6 Enabled #1062)
• Improved WebRTC Provider stability (WebRTC Provider #1078)
• WHIP support (Supports WHIP #1063)
• SRT Ingest stability improvement and multi-audio support (LLHLS Multilingual Audio #1131)
• Advancement of AdmissionWebhooks
  • User-Agent, new_url meta (closing status) added
  • Added functions to WHIP, Thumbnail
  • Fixed missing closing status transmission
• Encoding advancement
  • Added various settings
  • Conditional encoding (BypassIfMatch) method added (Conditional Encoding (Prevent upscaling) #1061)
  • Improved OpenH264 encoding quality
• Added various APIs
  • Add/delete stream (API for stream creation & deletion #1072)
  • Certificate reloading (no server downtime)
• Updated many contents of manual (especially API part) (https://airensoft.gitbook.io/ovenmediaengine/)

We are reviewing various opinions (AV1, SRT-output, WHEP) from this thread. thank you

In addition to this, if you have any ideas about what other features media server users need, please share!

[zapf2000]

What I miss in OME is support for classic devices (IPTV Boxes, Smart-TV's, Satelite Receivers). They mostly use RTMP (MPEG-TS) and RTSP. So why not provide a general MPEG-TS streaming via RTMP/RTSP? Just for compatibility issues, not for sub-second latency. I always need to run a second instance of SRS or mediamtx to serve "oldschool" customers. MPEG-TS via RTMP is widely supported and has a poor, but reasonable latency of about 3 seconds. (see https://github.com/aler9/mediamtx or https://github.com/ossrs/srs). Both work with common end-user hardware. LL-HLS is NOT supported by common devices and that's why OME is not used in professional IPTV broadcasts (they mostly use flussonic). Your server is cheaper than flussonic, but only optimized for web environment. It has much more potential. Another issue... ingest MPEG-4 video with b-frames.... transcode to webrtc compliant video without b-frames (you already have transcoding implemented, so why not program a flag to tell the encoder not to use b-frames), but bypass to ll-hls with b-frames for better performance?

[getroot]

I am not intentionally excluding RTMP/RTSP streaming. Everything is a matter of priorities, a matter of choice and focus. We continue to focus on low-latency streaming so far.

It is a bit wrong to say that OvenMediaEngine is cheaper than other streaming servers like Flussonic. OvenMediaEngine is open source and free software. So our investment is limited. Sponsorship will boost our development speed and make more features develop.

LLHLS-optimized encoding options are on our backlog and will be launched soon. The bframes option already exists. This applies when doing hardware encoding. OpenH264 doesn't support this, so it doesn't apply.

https://airensoft.gitbook.io/ovenmediaengine/transcoding#video

P.s. "general MPEG-TS streaming via RTMP/RTSP" seems misleading. RTMP and RTSP do not use the MPEG-TS format. Or are you talking about something else? MPEG-2 TS ingest and push is also supported by OME.

[tvgrandenatalhdtv]

Dear gentlemen
We install OVENMEDIAENGINE in UBUNTU System
the resulting stream worked fine
but only on ios phones
on android phones it didn't work
only works with HTML wrapping
follows the flow

even with port 443 n it worked on this android phones

kleber m. Of Jesus
Natal/RN -Brazil
zap
+55 84 988599596

follows the flow
https://canalteste.duckdns.org:443/app/tvgrandenatalhdtv/llhls.m3u8

[getroot]

It's off topic for this thread, but please refer to the URL below and put your URL in the player to play it.

https://airensoft.gitbook.io/ovenmediaengine/quick-start#llhls-playback

[fairbairn]

We'd love to see the ability to update the configuration from the API, including the outputs for an app, and not require a restart affecting existing streams.

[getroot]

This is almost impossible due to LLHLS. This is because LLHLS has to create synchronized segments for every track.
However, Playlist can be added dynamically. Isn't this enough?

[fairbairn]

I think the theory would be that the changes would apply only to new streams, not existing (live) streams, at the time of the change.

Existing live streams would remain unaffected. If they disconnected the stream and reconnected, then they would use the new/latest parameters. The "config" attaches to the stream for its operation. A new config template can be saved at any time. New streams use the latest template.

This is how Nimble works. You can make changes on the fly, but changes impacting a live stream are ignored. New streams use the new configuration.

There's a clause in your documentation that states that if specific information is changed by the API, it will drop all live streams, and this is something we want to avoid.

There are some limitations/considerations.
If you add/change/delete the settings of the App/Output Profile by invoking the API, the app will be restarted. This means that all sessions connected to the app will be disconnected.

We'd see this instead as existing streams on the App are NOT affected, but instead continue to run per their launch configuration.

Not dire critical, but certainly advantageous once you have loaded up Origin servers with many independent streams. It becomes very difficult to put a revision into production otherwise.

[getroot]

As you said, it would be possible if it does not affect existing streams and only affects streams that will be created in the future. Thanks for the great idea.

[fairbairn]

Along with a similar idea, it would be great if we could update the Server.xml directly and SIGHUP the server to load in the new config, without dropping existing streams.

Similar to how nginx handles reloads.

Existing streams are not affected.

New streams use the latest config.

[visualies]

+1 for AV1 that would be game changing!

[maruf-cs]

+1 for AV1