arXiv2020-RIFE problem -> Invalid data found when processing input

[luisarandas]

Hello!
I've been getting some errors when trying to add RIFE interpolation and I'm not sure where this might come from. By running the example code:

if interpolate_with_RIFE:
    video_tools.RIFE_interpolation(input=final_video_filename,
                       output=f'{os.path.splitext(final_video_filename)[0]}_RIFE.mp4',
                       interpolation_factor=4,
                       metadata_title=text_prompts,
                       metadata_comment=metadata_comment)

After the generation, the upscaling and the encoding, I get this error:

Loaded arXiv2020-RIFE v3.x HD model.
output/example_av/example_video.mp4, 70.0 frames in total, 35.0FPS to 140.0FPS
The audio will be merged after interpolation process
  0%|                                                                        | 0/70.0 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/luis/Desktop/clip/arXiv2020-RIFE/inference_video.py", line 217, in <module>
    ssim = ssim_matlab(I0_small[:, :3], I1_small[:, :3])
  File "/home/luis/Desktop/clip/arXiv2020-RIFE/model/pytorch_msssim/__init__.py", line 107, in ssim_matlab
    mu1 = F.conv3d(F.pad(img1, (5, 5, 5, 5, 5, 5), mode='replicate'), window, padding=padd, groups=1)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55ea946d9b40] moov atom not found
output/example_av/example_video_4X_140fps.mp4: Invalid data found when processing input

Any idea what it might be?
Thanks in advance,
Luís

[rkhamilton]

I've never run into that. Is the non-interpolated video in good shape to play? Can you play output/example_av/example_video.mp4? Did you change any of the ffmpeg encoding defaults? I'm wondering if there's something wrong with the input video you generated so that RIFE can't read it, or if there's a problem with RIFE itself running.

Make sure you have updated drivers (if you run nvidia-smi at a command prompt it will tell you useful stuff like your CUDA version, and free VRAM).

I just added an optional verbose flag to my RIFE wrapper. Update the library to get the new code. If you add the argument "verbose=True" to your RIFE function call it will show you the command it's trying to run, which you can copy/paste into a command window to test it outside of python.

The RIFE dev might be able to help if none of that works.

[luisarandas]

@rkhamilton I put this problem aside a bit as I had to uninstall some drivers here on my machine. I think this is a memory allocation problem as with smaller images the error disappears 👍

After running into RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR it wasn't able to process the data.

[rkhamilton]

I'm glad to hear you've figured it out! Cheers

[rkhamilton]

@luisarandas believes the error message is an out-of-memory condition as the problem goes away with smaller images. Marking as answered.