FFMPEPInput subimages

[sobotka]

iv shows the correct number of frames with the FFMPEG extensions, but when flipping through the subimages, only one updated frame for approximately every twelve subimages appears to be displayed.

[lgritz]

What if you use oiiotool to extract individual frames, i.e.

oiiotool foo.mp4 -subimage 4 -o 4.jpg

for a bunch of frames, would you see the same effect? In other words, are we failing to do something correct in the ffmpeg reader that is causing frame-to-frame differences to be dropped other than key frames? As a first step, let's see if we can isolate this to the ffmpeg reader without iv being a potential factor.

[sobotka]

Just tested, and as your suggestion indicates, it is likely something with the reader. It too is off by about the same batch chunk of frames that I was seeing in iv.

I also tested dumping the first five frames from the MOV in question, and the subimages are identical after the second up to five.

I am hesitant to suggest that the seek code is failing due to seeking on DTS not PTS. I'm also under the impression that seeking is impossible on some stream based codecs, doubly so those with B / P frames.

Some additional information for future reference.

[lgritz]

There are other formats we support that don't have true random-access seeking, but rather have to read sequentially. We just handle that automatically in the readers. They keep track of which subimage was read last, and at the next seek, if it's to a later frame, it keeps ticking forward until it gets to the requested one. If it requires seeking to an earlier frame, sometimes we even close the file, reopen, and tick forward again. So we always try to emulate random access, but we don't guarantee that non-sequential subimage access has high performance.

[sobotka]

I have a fix. Still has a bug near the end frame that segfaults and I want to clean it up.

[lgritz]

Was this ever resolved?

[sobotka]

No. For posterity sake, I'll outline the general problem for those folks that are having more time than myself at the current moment.

All codecs can end up being a mish-mash of I (whole), P (predictive based on previous), and B (bi-directional predictive forward and back) frames. Many H264 encoders rely heavily on B frames to optimize for size.

If we assume a trivial example, the problem manifests itself quickly. Let's assume a series:

Frame Number	0	1	2	3	4
Frame Type	I	P	I	B	I

If OIIO seeks to frame 0, no problem as we have a full frame in the decoded buffer. Frame 1, despite being a predictive frame, we have already sifted through the decode and should have enough information to decode the frame¹. Frame 2 is again a fresh start and not a problem as it is a full frame. Frame 3 however, is where our issue crops up.

While we can seek to frame 3, it being a bi-directional predictive frame means we rely on both the previous frame and the next I frame, which in this case is frame 4. We do not yet have enough information to pull a full frame off of the buffer.

In FFMPEG terms, the proper technique would be to find our desired frame. From there, we need to seek to the first I frame previous, which is I believe typical FFMPEG behavior. So in our above example, I believe the seek will return frame 2. Then we must decode past the I frame using avcodec_decode_video2 until we pop a full frame, tracking the decoded frames as we go. Once we hit the next I frame, we can figure out what frame the artist / technician desired and present it accordingly.

As a final aside, the code will also fail on the end of files with these P or B frames. avcodec_decode_video2 uses FFMPEG's internal allocations and will return a fail value on decodes on frames with B frames. All of those frames will accumulate and sit on the stack. If the code, as OIIO's does if memory serves, will leave those final frames on the stack. The proper technique might be to maintain a track of the not-yet-decoded frames and then continue to pop them off the stack until they are finished. See this post for more information and a possible solution to this issue.

Phew. Apologies for the spam.

¹ I believe P frames also can fail if you direct seek to them, but by default if memory serves, FFMPEG will always attempt to return the previous most valid I frame.

[lgritz]

Thanks for clarifying.

[lgritz]

I think we long ago merged a PR that was supposed to fix this (#1288). Closing.