The transcoder does not fully utilize the CPU.

[Bistiara]

Good afternoon,
OME is running in a container - airensoft/ovenmediaengine:0.16.5
The virtual machine has the following specifications: 64GB RAM/32 vCPU (AMD EPYC 9654), network 1Gbps.

The problem is that no matter what settings I use for the transcoder, after about 10-12 streams, I start getting queue errors, which leads to RAM overflow, and eventually, everything crashes due to OOM (Out of Memory). I have tried setting various values for Profile, Preset, and ThreadCount, but nothing helps. What's most interesting is that the CPU is only utilized up to 20%, and I can't manage to make it use more. Please advise what I am doing wrong?

logs+conf.zip

[bchah]

I think software decode is bound to a single thread per instance in OME. I hit this issue all the time with Thumbnail publishers and the only way to prevent OME from consuming all system memory is to use the experimental DeleteLazyStream feature. This is still not ideal because it would stop the transcode even when there is plenty of CPU available.

Is there any difference if you run this against the master branch? I believe they changed the decoder to use to 2 threads in commit 9e9be09.

[Bistiara]

Returned to the problem, unfortunately, there is no difference. Currently on version 0.16.8, the CPU is still only being used at 20%. I tried it on a virtual machine with 64GB RAM/32 vCPU (AMD EPYC 9654) and also tried from the master branch, but the behavior remains the same.

[getroot]

Could you please check which threads need to be distributed using top -H -p command?
Then capture it and upload it so I can figure out what settings you need.

https://airensoft.gitbook.io/ovenmediaengine/performance-tuning#monitoring-the-usage-of-threads

[Keukhan]

@Bistiara

I checked the problem you reported. The reason is that multi-threading is operating differently than expected in OpenH264 encoders.

One approach is to effectively utilize the process CPU by changing the scheduling policy of the process to RR, FIFO. However, there is a problem that it cannot be used without root user. so, I have a plan to replace the H264 encoder with x264. It is expected that the problem will be solved if it is replaced.

I can't replace it because I don't have enough time right now, but I'll make a plan.
Wait for the good news.

Cheers.

[Bistiara]

Thank you for your explanation @Keukhan @getroot . We are open to implementing a temporary solution. Could you provide more details on where exactly we can change the process scheduling policy to RR or FIFO? In which part of the system or code can this be done, and what specific steps need to be taken?

[Bistiara]

Good day, on this issue, perhaps you need some additional information from us? so that you could tell me what temporary solution can be used?

[Keukhan]

@Bistiara

First, I fixed the issue with thread usage not being accurate. I also added the x264 library.
In short, OpenH264 doesn't show the exact number of threads it creates. It looks like all the encoders share the same threads. but, the H264 decoder, scaling filter, and x264 encoder seem to handle CPU thread usage well.

To use x264, use <Codec>h264_x264</Codec>

Patches

• f37ed41
• 76e918f

CPU load balance test result

• https://airensoft.atlassian.net/wiki/external/Y2E1NWNkYjc3ZjcwNGMxNWI5M2ZjNmEzZDE0YzY0NDA

The patched code has not been released yet. The next version will be released soon. thanks.