OME soft locking after some time

[naanlizard]

I'm not 100% sure this isn't our stream forwarding method from nginx-rtmp to OME, but I am pretty confident it is OME because when it happens, if I restart our forwarder, it doesn't fix it, but restarting OME does fix it.

Essentially, this screenshot shows the problem, but I'm not sure if it is cause or effect. CPU usage drops to almost nothing, nobody can play streams, no crash dump is created.

CPU usage doesn't match inside/outside the docker either. When I run docker exec omedocker ps aux I get 223% CPU, but docker stats reports .33%

Those examples were from a time this happened on the 26th of March. Today it happened again just now, hence the discussion.

Here is the relevant info

Server.xml

Dockerfile

Logger.xml

EdgeConfig.xml

Logs from today - crash happened around 05:3X in the timestamps there. - 05:33:45 on my grafana instance, but I'm unsure how accurate that is

[getroot]

Which OME version are you using?

[naanlizard]

Please, ask anything.

I'll review our logs but this has been happening for a while on at least two versions,this happened after only four days uptime. Usually it is not so frequent

[getroot]

I got one clue from your logs.
This issue will be resolved soon.

[naanlizard]

This is very intimidating haha

I'm very curious what the problem is, I'm looking forward to the commit. Thank you for the hard work!

[naanlizard]

Please let me know what docker hash I should pull (or if I should wait for 0.15.8) and I'll test ASAP

[getroot]

It's still just a clue, so I'm not sure. I'll get back to you when the analysis is complete.

[getroot]

02876e4

I can't confirm this yet, but it fixed what I suspect was the cause of your problem. Is it difficult to test the dev version?

The cause of the problem is freezing when attempting to delete an LLHLS stream with a very low probability. I'll look into this further.

[naanlizard]

I'm now running the latest commit (DVR logging change) and will report back if it soft locks again.

We have a lot of streams start/stop all the time (is an input stream stopping the same as deleting?), so I can imagine we would trigger it. Especially with a 16c/32t CPU (if my amateur guess is that more cores = more chances to break)

[getroot]

yes, deleting stream is same as stopping in ome.
Thanks for testing the dev version. Hope this solves your problem!

[naanlizard]

This hasn't happened since I posted I don't think, we can probably consider it solved!