FFMPEG image smearing with TCP

[dghodgson]

#As in the title, I get severe image smearing after a period of time, and it affects all of my cameras (the outdoor cams may actually be immune, but they're a different model from what appears to be the same manufacturer).

My montage looks like this: https://imgur.com/a/ypWf0

Note, there are 55 cameras on this network (which is dedicated to IPcam traffic), and they are currently streaming 720p @ 11fps to a VM running Video Insight without any issues. ZM is running in a separate VM. System load appears to make no difference, regardless if it's hovering around 3, or 13 (VM has 10 cores allocated). Shared memory is at 6% (out of 11GB according to df -h. VM has 8GB allocated with 16GB swap). I have tested this system in the past with the VM running VI shut off, and I still experience this problem (again, with either 640x480 or 720p, at 11FPS).

Also of note, the camera's web interface allows for the user to set CBR/CVBR/VBR, I-Fram interval, FPS, and bitrate. None of these settings seemed to make any difference (I tested each variable separately, and together, on 5 cameras at once).

ZoneMinder Version (zmaudit.pl -v):
1.31.12~20171108111933-xenial
from the storageareas PPA

Linux Distribution and Version (cat /etc/os-release or cat /etc/redhat-release):
Ubuntu 16.04.3

If the issue concerns a camera, provide the make, model, frame rate, resolution and ZoneMinder Source Type:

Not really sure (client bought cheap off-brand chinese IP cams). Best info I can provide is that the controller board appears to be made by TopSee, and the model number is TH38A2-ONVIF. Supports ONVIF v2.04, encodes video as h.264 (obviously), current stream resolution is 640x480 @ 11FPS (experienced the same issue at 720p as well). Firmware appears to use Linux kernel 3.0.8 for ARMv5.

Source type is FFMPEG, RTSP over TCP (rtpRtsp).

Relevant log lines:

https://pastebin.com/PSHTYDnt

I believe the stream starts smearing around line 1569 (about 9AM my time)

[dghodgson]

New log for camera that's recording, not just monitoring (same model and settings):
https://pastebin.com/hVzgsQNE

lines 91-285 are there as an example of what I cut out of the rest of the log using grep -n -v "duration\|writing" (it's excessively long, and those lines are just noting what's being written to disk). Each line begins with the actual line number from the log.

I'm relatively certain that smearing begins at line 1006 (orig line 56401)