[Feature]: Hope to optimize

[designrans]

Expected behavior

When using video guidance for video style rendering, it is common to get stuck and the GPU usage is very high (4090). Can we optimize it.

[continue-revolution]

Do you mean ControlNet V2V? Can you give me more concrete context, like when the VRAM consumption is high? Do you get OOM before generation starts?

[designrans]

Do you mean ControlNet V2V? Can you give me more concrete context, like when the VRAM consumption is high? Do you get OOM before generation starts?

yes V2V,my 4090 ,22-24g,OOM is?

[designrans]

Do you mean ControlNet V2V? Can you give me more concrete context, like when the VRAM consumption is high? Do you get OOM before generation starts?

[designrans]

3 seconds 480p, 24 frames of video, Controlnet is canny, 4090 takes 1 and a half hours. Did I set it incorrectly?

[continue-revolution]

If the frame number is 90, I don’t think it will OOM (cuda out of memory). I have tested locally with 95 frames, 512*512 You may want to use something like sdp or xformers.

If the frame number is ~300, one user reported that OOM will happen before generation starts (most likely because of CN). I will try to address this in #224

If you set output to be 1024*1024, it is absolutely impossible to prevent OOM, unfortunately.

[continue-revolution]

@min-star I‘ve not supported i2i batch. It will be available in v1.10.0 via #224

[min-star]

@continue-revolution thank you for you reply,thus the controlnet v2v is for txt2image,is the understand correct?

[continue-revolution]

For v1.9.2, yes.

For v1.10.0 which will come later this week, not really. ControlNet v2v will be available for both t2i and i2i batch.

[min-star]

@okay,it's cool.

[davyzhang]

With 512x288 sized video, 26 frames. It can barely get through 16GB VRam instance provided by colab.
Without hres fix

2023-10-24 09:27:53,838 - ControlNet - INFO - Loading model: control_v11p_sd15_openpose [cab727d4]
2023-10-24 09:28:02,099 - ControlNet - INFO - Loaded state_dict from [/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd-webui-controlnet/models/control_v11p_sd15_openpose.pth]
2023-10-24 09:28:02,099 - ControlNet - INFO - controlnet_default_config
2023-10-24 09:28:07,696 - ControlNet - INFO - ControlNet model control_v11p_sd15_openpose [cab727d4] loaded.
2023-10-24 09:28:08,027 - ControlNet - INFO - Loading preprocessor: dw_openpose_full
2023-10-24 09:28:08,027 - ControlNet - INFO - preprocessor resolution = 512
2023-10-24 09:31:07,753 - ControlNet - INFO - ControlNet Hooked - Time = 194.59531688690186
100% 20/20 [01:14<00:00,  3.74s/it]
^C

[LIQUIDMIND111]

NOTE – I have an RTX 2060 6 VRAM GPU and i can make 512×768 animations at 8fps and 124 max frames and get great results.... but if i use an 1024×576 model – it will OOM and crash. I can make an animation of 512×512 of 64 frames max in less tan 10 minutes using SDP and reducing BATCH CONTEXT to 8.... in MY case, that works a charm!

here is a sample https://www.youtube.com/shorts/NLt6HqUgBFM

[continue-revolution]

V2V VRAM problem should have been resolved in v1.11.1