Restyle Video Question

[gateway]

So I have some 4k footage I want to test from a drone flight I did and run it though the restyle system, however with the gpu's we have their is no way that we are going to get a style to be proccssed without running out of memory unless Im misunderstanding the restyle process..

Looking at the extraction if we could get a option to add -vf scale=Width x Height here

vqgan-clip-generator/src/vqgan_clip/_functional.py

Line 354 in 4892574

'-filter:v', 'fps='+str(extraction_framerate),

So maybe the config could be scale_output_images "WxH" when extracting?

Then use Real-ESRGAN https://github.com/xinntao/Real-ESRGAN which is the newer version to upscale (WIP).. they also have a portable executables for Win/Linux and Mac..

Also I havent done enough videos yet with the project but do they need optical flow applied to avoid weird artifacts or flickering.. ?

My plan is to extract 4k down to 900 x 560 (seems to fit for my into the memory of my Titan RTX) then try to upscale back up to 1920 x 1080 and hope it doesn't look crappy.

Thoughts?

[gateway]

Now that I think bout it.. adding an image upscaler for just image output would be awesome as well. I currently own and use GigiaPixel (Not free) from Topaz and usually upscale by 4x using their Art/CG model to render out.

Here is an example

[rkhamilton]

We are on the same page. I hope that today I'll push v1.2 to github which is a major change to video generation syntax and enabled Real-ESRGAN use. It also includes a wrapper function to easily integrate Real-ESRGAN into the workflow with my package. This will let you upsample both still images and generated video. The workflow examples are for upscaling generated single images and folders full of images that were generated by my package, but the same code would work for images from elsewhere.

Solving the problem of flickering video was the original motivation for creating this whole project. I wanted to do some real digging into combining elements of these GAN-based algorithms, and the code / notebooks out there were not structured in a way that was conducive to that. I think a major improvement I've made for flickering is seeding in each new generated frame with a blend of the new source frame and the old generated frame. That means the new generated frame init_image is seeded with elements that CLIP will consider to be a good match. You have to experiment with the right blend ratio to get good results, but it made a big difference in my tests.

As far as your request to downsample images, that should already be happening in the code right now. If you look at engine.convert_image_to_init_image() and engine.encode_and_append_pil_image() you will see there is a resize_image() method that I'm applying. If you find a case where that's not working for some images please open an issue!