LLHLS Publisher errors while WebRTC works

[bfigueir]

Describe the bug
I'm using Yolo to detect persons in a live video feed piping each processed frame to OME via ffmpeg:
live video stream > ffmpeg extract frames > yolo > ffmpeg publish via rtmp > ome

While playing on over player via webRTC (ws://localhost:3333/app/stream_name) it works as expected without issues, if I use VLC with LLHLS (http://localhost:3333/app/stream_name/llhls.m3u8) the stream does not work properly (1 frame every 3/5 sec) and I0m getting errors on OME:

[2023-05-04 10:03:18.329] I [StreamWorker:331] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. LLHLS(2)/Stream total(4)/App total(4)
[2023-05-04 10:03:24.586] I [StreamWorker:334] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. LLHLS(3)/Stream total(5)/App total(5)
[2023-05-04 10:03:29.154] W [StreamWorker:333] LLHLS Publisher | llhls_stream.cpp:579  | Could not find segment for track_id = 0, segment = 1878 (last_segment = 1890)
[2023-05-04 10:03:29.154] E [StreamWorker:333] HttpServer | http_exchange.cpp:42   | 
[Client] HttpConnection(0x7f6d3c201810) : HTTP/1.1 <ClientSocket: 0x7f6d3c225c10, #36, Connected, TCP, Nonblocking, 172.20.0.1:55312> TLS(Disabled) (Elapsed: 0) 
[Request] GET http://localhost:3333/app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac/seg_0_1878_video_3L9iYJXS_llhls.m4s (HTTP/1.1, Request Time: 2023-05-04T10:03:29.154+00:00)
[Response] 404 Not Found (249 bytes sent, Response Time: 2023-05-04T10:03:29.154+00:00)
[2023-05-04 10:03:29.977] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(2)/Stream total(4)/App total(4)]
[2023-05-04 10:03:29.978] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(1)/Stream total(3)/App total(3)]
[2023-05-04 10:03:34.981] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(0)/Stream total(2)/App total(2)]

To Reproduce
Steps to reproduce the behavior:

1. Use OME default settings
2. running with docker compose the :dev tag

version: "3"
services:
  ome:
    image: airensoft/ovenmediaengine:dev # airensoft/ovenmediaengine:latest # GPU https://airensoft.gitbook.io/ovenmediaengine/transcoding/gpu-usage
    ports:
      - "1935:1935/tcp"
      - "3333:3333/tcp"
      - "3334:3334/tcp"
      - "3478:3478/tcp"
      - "8080:8080/tcp"
      - "9000:9000/tcp"
      - "9999:9999/udp"
      - "20080:20080/tcp"
      - "8081:8081/tcp"
      - "4000-4005:4000-4005/udp"
      - "10006-10010:10006-10010/udp"
    environment:
      - OME_ORIGIN_PORT=9000
      - OME_RTMP_PROV_PORT=1935
      - OME_SRT_PROV_PORT=9999
      - OME_MPEGTS_PROV_PORT=4000-4005/udp
      - OME_HLS_STREAM_PORT=8080
      - OME_DASH_STREAM_PORT=8080
      - OME_SIGNALLING_PORT=3333
      - OME_TCP_RELAY_ADDRESS=*:3478
      - OME_ICE_CANDIDATES=*:10006-10010/udp
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

5. stream to OME using:

ffmpeg -i pipe:0 -an -filter:v scale=w=266:h=426 -loglevel error -vcodec libx264 -tune zerolatency -preset ultrafast -qp 0 -movflags +faststart -fflags nobuffer -x264-params keyint=10:scenecut=0 -f flv rtmp://ome:1935/app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308

4. See error

Expected behavior
The FPS should be equal to the webRTC and without errors

Logs

[2023-05-04 10:03:18.329] I [StreamWorker:331] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. LLHLS(2)/Stream total(4)/App total(4)
[2023-05-04 10:03:24.586] I [StreamWorker:334] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. LLHLS(3)/Stream total(5)/App total(5)
[2023-05-04 10:03:29.154] W [StreamWorker:333] LLHLS Publisher | llhls_stream.cpp:579  | Could not find segment for track_id = 0, segment = 1878 (last_segment = 1890)
[2023-05-04 10:03:29.154] E [StreamWorker:333] HttpServer | http_exchange.cpp:42   | 
[Client] HttpConnection(0x7f6d3c201810) : HTTP/1.1 <ClientSocket: 0x7f6d3c225c10, #36, Connected, TCP, Nonblocking, 172.20.0.1:55312> TLS(Disabled) (Elapsed: 0) 
[Request] GET http://localhost:3333/app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac/seg_0_1878_video_3L9iYJXS_llhls.m4s (HTTP/1.1, Request Time: 2023-05-04T10:03:29.154+00:00)
[Response] 404 Not Found (249 bytes sent, Response Time: 2023-05-04T10:03:29.154+00:00)
[2023-05-04 10:03:29.977] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(2)/Stream total(4)/App total(4)]
[2023-05-04 10:03:29.978] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(1)/Stream total(3)/App total(3)]
[2023-05-04 10:03:34.981] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#app/644bef9cc0036ba9c2a38ab6_afb8fa3d-a761-4fce-a44f-9036ce308dac on the LLHLS publisher. Concurrent Viewers[LLHLS(0)/Stream total(2)/App total(2)]

Server (please complete the following information):

• OS: WSL2 Ubuntu
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.1 LTS
  Release: 22.04
  Codename: jammy
• OvenMediaEngine Version: Docker airensoft/ovenmediaengine:dev
• OvenMediaEngine v0.15.7 ((From archive)) is started on [0148609da004] (Linux x86_64 - 5.15.90.1-microsoft-standard-WSL2, docker support #1 SMP Fri Jan 27 02:56:13 UTC 2023)

Player (please complete the following information):

• Device: MSI laptop with RTX3080
• OS: Windows
• Browser edge
• Version 112.0.1722.64
• VLC version 3.0.18 Vetirani

[getroot]

Since VLC does not support LLHLS, it is probably playing in legacy HLS.

Variable FPS on HLS (also LLHLS) will most likely not work as expected. There are various reasons, but if a segment or partial segment is not created for each #EXT-X-TARGETDURATION and #EXT-X-PART-INF:PART-TARGET, the player will judge it as an error. This means that the server must prepare the next segment before the player has finished playing the downloaded segments. However, with variable FPS, it could take more than 6 seconds (assuming a Target Duration of 6 seconds) to generate a segment, especially since segments must only start with a keyframe.

If your encoder (FFMPEG) is sending at a fixed FPS, it's likely to work as expected.

[bfigueir]

ok thanks. Im going to try to see if I can somehow force the ffmpeg to set a fixed FPS.