Arise RuntimeError: CUDA out of memory for larget image

[XianchaoZhang]

Hi IceClear,
Thank you for your impressive work! I encounter OOM issue when handle large image with clipiqa_single_image_demo.py, such as 8M/12M pictures. I want to know is there any memery limited for the CLIP-IQA model. Or whether my env runtime had configuration issue?

Here's the detail output logs:

clipiqa) PS D:\code\CLIP-IQA> python .\demo\clipiqa_single_image_demo.py --file_path=../dataset/IMG_20230316_105935577.jpg
Traceback (most recent call last):
File ".\demo\clipiqa_single_image_demo.py", line 62, in
main()
File ".\demo\clipiqa_single_image_demo.py", line 38, in main
output, attributes = restoration_inference(model, os.path.join(args.file_path), return_attributes=True)
File "d:\code\clip-iqa\mmedit\apis\restoration_inference.py", line 79, in restoration_inference
result = model(test_mode=True, **data)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\mmcv\runner\fp16_utils.py", line 110, in new_func
return old_func(*args, **kwargs)
File "d:\code\clip-iqa\mmedit\models\restorers\basic_restorer.py", line 79, in forward
return self.forward_test(lq, gt, **kwargs)
File "d:\code\clip-iqa\mmedit\models\restorers\clipiqa.py", line 182, in forward_test
output, attribute_prob = self.generator(lq)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\backbones\sr_backbones\coopclipiqa.py", line 314, in forward
logits_per_image, logits_per_text = self.clip_model(image, self.tokenized_prompts[i].to(image.device), self.pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 373, in forward
image_features = self.encode_image(image, pos_embedding)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 355, in encode_image
return self.visual(image.type(self.dtype), pos_embedding=pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 158, in forward
x = self.attnpool(x, return_token, pos_embedding)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "d:\code\clip-iqa\mmedit\models\components\clip\model.py", line 72, in forward
x, _ = F.multi_head_attention_forward(
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 5101, in multi_head_attention_forward
attn_output, attn_output_weights = _scaled_dot_product_attention(q, k, v, attn_mask, dropout_p)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 4847, in _scaled_dot_product_attention
attn = softmax(attn, dim=-1)
File "C:\Users\A.conda\envs\clipiqa\lib\site-packages\torch\nn\functional.py", line 1680, in softmax
ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 8.86 GiB (GPU 0; 12.00 GiB total capacity; 9.59 GiB already allocated; 0 bytes free; 9.70 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Could u please give me some advice?

[IceClear]

Hi.
I think sometimes resizing the image is necessary given that the input resolution is too large.
Too large inputs may make the IQA process meaningless.

[XianchaoZhang]

Hi IceClear,
Thanks for your reply!
And why meaningless? Did this means that CLIP is only trained on 224x224, it is limited by the pretrainded model?