[IDEA] near-zero latency multiple (IP) video-streams playing

[MarcoRavich]

(preface: the main purpose of this discussion - following the HyMPS's "philosophy" - is to stimulate the confrontation between devs/maintainers of open projects for the goal of their mutual evolution)

For those who need to display multiple (LAN only) video streams in a grid/wall at the lowest possible latency, here are two open software that we have tested and can recommend:

• @iEvgeny's CCTV Viewer: best of all for (hardware-like) latencies, Linux-only
• @tsingsee's EasyPlayer: multiplatform, simplest GUI

It would be great to "combine" these peculiar characteristics to obtain a platform indipendent and easy to use player for live monitoring at lowest-possible latencies,

We've also rounded up many other interesting projects in this (raw, at the moment) list: https://github.com/forart/HyMPS/blob/main/RTSP.md

...what do you think about this ?

[apple-fritter]

Hello! Just checking in. I'll give this post a little more thorough of a once-over later, but wanted to indicate my interest in assisting you in the endeavor.

I may be a moment to reply as I will think of good questions to ask that will expedite my understanding of your intended implementation.

Cheers, and have a nice day!

[MarcoRavich]

Well, as discussed in this issue @iEvgeny suggests:
"the absence of video buffering, which is in every player and introduces a significant delay up to several tens of seconds.
In addition, the new implementation uses the Zero-copy rendering method whenever possible."

We wonder if someone (like @76technician, @Yash-Garg, @onlyet, @janssen70 and hopefully others) could implement this approach in their projects too.

[76technician]

In fact,my player had output buffer in VideoThread，every rtsp player has buffer to ture the audio and video synchronization.
FFmpeg's rtsp receiver may just support tcp,your image will be destory if you use udp.
So,if you want lower delay,you need use ffmpeg's decoder and encoder module to build your transport moudle or try to use webrtc.

[MarcoRavich]

every rtsp player has buffer to ture the audio and video synchronization.

...and to manage sync between multiple streams, we guess.

So,if you want lower delay,you need use ffmpeg's decoder and encoder module to build your transport moudle or try to use webrtc.

Then do you think that HW-decoding (such as @bmegli's hardware-video-streaming libs or @rigaya's - encoding oriented - ones) may help to speedup the whole process ?

Anyway, the "zero-copy video rendering" approach - that @iEvgeny proposed too - sounds suitable to reduce latencies.

Some resources about it that may help:

@OpenVINO-dev-contest's single and multi-source video decode and preprocess and inference to show the device surface sharing (zero copy)

@paolobarbolini's Simple zero copy* streaming multipart decoder implementation

@gnobal's sample of ZeroCopyInputStream/ZeroCopyOutputStream to handle a streaming socket using a pre-allocated buffer

Understanding about Zero copy

"Zero-copy" video rendering with Mali discussion (many repos linked in)

Improving Streaming Application Performance with Zero-copy and Asynchronous TDMS

Hope that inspires !

[MarcoRavich]

Seems that @alexeyvasilyev's rtsp-client-android implements zero-copy in Android too:

Unlike ExoPlayer which also supports RTSP, this library does not make any video buffering. Video frames are shown immidiately when they arrive

[apple-fritter]

Thank you for your understanding. I apologize for the late reply.

I am interested in discussing the implementation of overlay content on the video. Could you please provide a detailed explanation of the information or types of data that you would like to have overlaid?

More specifically, I would like to categorize your various "zones" into classes that can have different sets of attributes. I have a few specific questions in mind: Do you prefer the overlay content to be dynamic or static? Would you like certain elements to update at specific intervals? How would you like to interact with the content in order to make updates?

[MarcoRavich]

Thanks for your active partecipation, @apple-fritter !

We use the "multi RTSP-player" software (specifically @iEvgeny's CCTV Viewer at the moment) to display live video streams from PTZ cameras to let operators remote control.
[we'll post a video soon to have a better idea of the whole system config]

Android tablets (and app) are used as PTZ controllers for the scope, so they need to match this touch gui with monitoring:

For this kind of use, operators needs at least viewfinder/safe area markers (transparent pngs overlay ?):

-> transparent-background viewfinder usable file here

Text labels would be also useful (customizable position, colors and fonts would be great):

EDIT: @pataelmo (PTZCameraController author), @EdwardGoomba (Hawkeye one) or @TheColditz (behind CCTV-Companion) can contribute with more ideas/needs...

...@berglie's gstreamer-d3d11-overlay or @torreyleonard's RTSP-Utilities may help/inspires.

Thanks again for your interest in the discussion.

[MarcoRavich]

( note: images and text overlays should be - of course - treated as different topics so, from now on, we will use "topic tag" before the relative part )

TEXTS OVERLAY

Can you clarify the relationship of the PTZ controls to the overlay content?

Since each PTZ operator moves one camera at a time, the controllers (both hw and sw) they use allows to select which one.
Typically "nametags" are chosen to identify each remote camera ('cause simplifies their management rather than IPs), so these labels must - of course - match both on controllers and monitors.
(we'll post the - infamous - video about this soon: it will be more explanatory than words)

For this purpose we believe that text(s) overlay - therefore editable on the fly - should be the best choice:

• https://hhsprings.bitbucket.io/docs/programming/examples/ffmpeg/drawing_texts/drawtext.html
• https://ottverse.com/ffmpeg-drawtext-filter-dynamic-overlays-timecode-scrolling-text-credits/
• https://transkriptor.com/add-text-to-video-ffmpeg/
• https://stackoverflow.com/questions/54789728/ffmpeg-command-with-overlay-and-drawtext-on-video
• https://superuser.com/questions/939357/how-to-position-drawtext-text
• https://stackoverflow.com/questions/49454740/multiple-overlays-using-ffmpeg
• https://video.stackexchange.com/questions/25574/realtime-time-overlay-with-ffmpeg

IMAGE OVERLAY

We believe that viewfinder (= trasparent-background image) shape/color/thickness/etc should be a sole users' burden: their widest freedom of choice must be retained as much as possible.

For example they could decide to BUY one (or even more) online...

....or, why not, choose to use a (provocatory - even if perfectly usable - of course) kid-freehand-drawned one:

To us RTSP display software must be limited to render the overlay after an image-to-stream resolution adapting (better, of course. if SVG vector files are provided).

It would be mutch useful to let users choose a different viewfinder (= trasparent-background image) for each stream.

In the end yes, KISS is the key.

[MarcoRavich]

OK, we've digged a little more and discovered that a basic one could be "drawable" directly by FFMPEG (using drawgrid and/or drawbox commands):

Source: https://hhsprings.bitbucket.io/docs/programming/examples/ffmpeg/video_data_visualization/drawgrid.html

Source: https://hhsprings.bitbucket.io/docs/programming/examples/ffmpeg/video_data_visualization/drawbox.html

...perhaps @afrum could help to build a right commandline to draw "EBU-compliant video safe areas" (and, possibly, a rudimentary viewfinder) since he has already made several interesting scripts...

We've also opened a question both @ SO / SE where we posted this (1080p) scheme too:

[MarcoRavich]

Here we go, (driven) ChatGPT did it:
ffplay -vf "drawgrid=x=(iw/2):(ih/2):t=3:color=yellow@0.5, drawbox=x=(iw-iw*0.95)/2:y=(ih-ih*0.95)/2:w=iw*0.95:h=ih*0.95:color=red@1, drawbox=x=(iw-iw*0.965)/2:y=(ih-ih*0.965)/2:w=iw*0.965:h=ih*0.965:color=green@1" filename.ext

[MarcoRavich]

Ok, after a quite long chat, here's the updated commandline (with added 4:3 safe area too):

ffplay -vf "drawgrid=x=(iw/2):(ih/2):t=3:color=yellow@0.5, drawbox=x=(iw-(iw*0.675))/2:y=(ih-ih)/2:w=iw*0.675:h=ih:color=black@0.3:t=fill, drawbox=x=(iw-iw*0.95)/2:y=(ih-ih*0.95)/2:w=iw*0.95:h=ih*0.95:color=red@1, drawbox=x=(iw-iw*0.965)/2:y=(ih-ih*0.965)/2:w=iw*0.965:h=ih*0.965:color=green@1" filename.ext

Hope that inspires.

[MarcoRavich]

Can you clarify the relationship of the PTZ controls to the overlay content?

Here's te video (encoded by Canon HF100 camcorder @ 17 MBps):
http://www.forart.it/_priv/PTZ_ctrl-view.mp4

The operator(s) needs to clearly identify the controller-selected camera in preview monitoring(s) to avoid possible mess and/or misunderstanding with the director.
That's why (customizable) labels are needed.

Hope it helps.

[MarcoRavich]

Just found a very interesting - advanced - approach for generating a viewfinder (but needs to be adapted, eventually):
@tiaanduplessis's react-native-view-finder

Usage example:

import {View, StyleSheet} from 'react-native'
import ViewFinder from 'react-native-view-finder'

const Example = () => (
  <View style={styles.container}>
    <ViewFinder backgroundColor="#333" loading />
  </View>
)

const styles = StyleSheet.create({
  container: { flex: 1 }
})

This will result in:

Hope that inspires.

[apple-fritter]

That is an interesting project. If you enjoy the design, that should be easy to accomplish. I'll visit the code tomorrow to see how the strategies used could be transmuted to fit your purposes.

[MarcoRavich]

* @tsingsee's [EasyPlayer](https://github.com/tsingsee/EasyPlayer-RTSP): multiplatform, simplest GUI ![](https://camo.githubusercontent.com/cbf922c1c179aec8eece7d77967ad4452b403f60c07d998c407fe751ffc7af7c/687474703a2f2f7777772e6561737964617277696e2e6f72672f6769746875622f696d616765732f65617379706c6179657232303136303930383137313032372e706e67)

We've just realized that EasyPlayer' screenshots are no longer reachable in repos managed by @babosa, sorry !

EDIT:
Well, we hope other RTSP-streams "grid player" projects' devs (such as @grigory-lobkov, @yuriizubkov and @Mawkey just to cite some) are somewhat interested in this discussion.

[yuriizubkov]

Hi folks!
I'm not sure how much help I can be. I created my "grid player" script specifically for using on Raspberry Pi (about 6 years ago). Script simply controls omxplayer instances (because omxplayer has hardware acceleration for video playback on the Rpi). I don't even know if it works on other platforms. :)

[apple-fritter]

Hi folks!
I'm not sure how much help I can be. I created my "grid player" script specifically for using on Raspberry Pi (about 6 years ago). Script simply controls omxplayer instances (because omxplayer has hardware acceleration for video playback on the Rpi). I don't even know if it works on other platforms. :)

Hi Yuri! I looked at your video wall project... Interesting. I will revisit it later when I have more time. I have a couple RTSP repos up and will be creating my own meta sections and repo for them in the coming days, if you are interested.

[Tom722620]

work

[MarcoRavich]

BUMP

Just discovered @illuspas's NodeMedia Clients (Android and iOS) which implements:

• Hardware Acceleration
• Low latency
• Delay elimination

Both for play and publish.

[bitvilag]

I landed here as https://github.com/Anonymousdog/displaycameras is not supported on Bullseye.
As far as I can see it is related to the missing live555 from debian due to license issues. (I really dont understand why these companies are this needy in this matter).

The project uses omxplayer, but that is deprecated and as far as I can se is deprecated because of the missing package.

What do you see as the closes implementation to this project? I would need a couple rtsp feeds on my RP1 to be presented on the attached monitor.

[MarcoRavich]

What do you see as the closes implementation to this project? I would need a couple rtsp feeds on my RP1 to be presented on the attached monitor.

Well, at the moment the best - and most updated - RTSP "player" is CCTV Viewer without any doubt.

Anyway it runs on a "standard" linux distro, dunno if works on RPiOS.

Try it yourself and report to @iEvgeny, please.

[bitvilag]

I saw somewhere that if I go back to Buster I am good for a while as its suppose to be supported until mid of 2024.
I checked the project and at first the README did not really encourge me thats something easy to setup.

I will check back later when I guess more docs are present, but for now I will try to stick with Buster and omxplayer

[MarcoRavich]

I checked the project and at first the README did not really encourge me thats something easy to setup.

Well, if you search between CCTV Viewer issues, you can find some interesting infos...

• How to compile from source on arm/raspberry PI

@dvision1979 seems the most experienced user about...