Custom ViT: shape mismatch error

[Rumsha7]

Hello,

I'm trying to use GradCAM on a custom ViT model. However, I'm getting the following error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (147x1024 and 3072x64)

I'm wondering if I need to transform my image differently. Any thoughts on how to resolve this error?

This is my code:

  transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
    ])
  im = transform(im)
  print(im.size()) #this gives torch.Size([3, 224, 224])
  with GradCAM(model=model, target_layers=target_layers, use_cuda=torch.cuda.is_available(), reshape_transform=reshape_transform) as cam:
    targets = None
    grayscale_cam = cam(input_tensor=im.unsqueeze(1), targets=targets) #einops.EinopsError: Expected 4 dimensions, got 3 --> fixed with unsqueeze(1) and now: RuntimeError: mat1 and mat2 shapes cannot be multiplied (147x1024 and 3072x64)
    grayscale_cam = grayscale_cam[0, :]
    visualization = show_cam_on_image(im, grayscale_cam, use_rgb=True)

This is the reshape_transform function I am using that's provided in this repo's documentation:

def reshape_transform(tensor, height=14, width=14):
    result = tensor[:, 1 :  , :].reshape(tensor.size(0),
        height, width, tensor.size(2))

    # Bring the channels to the first dimension,
    # like in CNNs.
    result = result.transpose(2, 3).transpose(1, 2)
    return result

[jacobgil]

Hi,
input_tensor=im.unsqueeze(1)

Looks like this should've been input_tensor=im.unsqueeze(0), no?

[Rumsha7]

Thank you for the prompt response! I really appreciate it!

I just tried 0 instead of 1, and now get the following error: RuntimeError: shape '[1, 14, 14, 64]' is invalid for input of size 3136

Below is a full Traceback for context:

File "/usr/local/lib/python3.7/dist-packages/gradio/routes.py", line 276, in run_predict
    fn_index, raw_input, username, session_state, iterators
  File "/usr/local/lib/python3.7/dist-packages/gradio/blocks.py", line 760, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "/usr/local/lib/python3.7/dist-packages/gradio/blocks.py", line 672, in call_function
    block_fn.fn, *processed_input, limiter=self.limiter
  File "/usr/local/lib/python3.7/dist-packages/anyio/to_thread.py", line 32, in run_sync
    func, *args, cancellable=cancellable, limiter=limiter
  File "/usr/local/lib/python3.7/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.7/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "<ipython-input-15-23b16914decd>", line 128, in predict11
    grayscale_cam = cam(input_tensor=im.unsqueeze(0), targets=targets) 
  File "/usr/local/lib/python3.7/dist-packages/pytorch_grad_cam/base_cam.py", line 189, in __call__
    targets, eigen_smooth)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_grad_cam/base_cam.py", line 74, in forward
    outputs = self.activations_and_grads(input_tensor)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_grad_cam/activations_and_gradients.py", line 42, in __call__
    return self.model(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/vit_pytorch/vit.py", line 122, in forward
    x = self.transformer(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/vit_pytorch/vit.py", line 78, in forward
    x = attn(x) + x
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1151, in _call_impl
    hook_result = hook(self, input, result)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_grad_cam/activations_and_gradients.py", line 23, in save_activation
    activation = self.reshape_transform(activation)
  File "<ipython-input-15-23b16914decd>", line 192, in reshape_transform
    height, width, tensor.size(2))
RuntimeError: shape '[1, 14, 14, 64]' is invalid for input of size 3136

[Rumsha7]

To add in case it helps, I printed the sizes of the tensor image in reshape_transform. The comments show what was printed before the Traceback and RuntimeError:

def reshape_transform(tensor, height=14, width=14):
    print(tensor.size()) #torch.Size([1, 50, 64])
    print(tensor[:, 1 :  , :].size()) #torch.Size([1, 49, 64])
    result = tensor[:, 1 :  , :].reshape(tensor.size(0),height, width, tensor.size(2))
    print("result = ", result) #Nothing printed due to RuntimeError for line above
    
# Bring the channels to the first dimension,
    # like in CNNs.
    result = result.transpose(2, 3).transpose(1, 2)
    return result

I was wondering why the RuntimeError mentions size 3136, and I'm thinking it's because 49*64 (from the second print statement) = 3136. I feel like your suggestion is one step closer (yay!) but I'm not sure how to resolve this new size mismatch. Any thoughts on what to try next?

[Rumsha7]

Hi @jacobgil, thank you for your help earlier! I'm wondering if you have any insight on this new error I'm getting, described above: RuntimeError: shape '[1, 14, 14, 64]' is invalid for input of size 3136.

Which shape have you usually passed into .reshape? The original input of the tensor into GradCam is [1, 3, 224, 224]. When it is passed into reshape_transform, it seems to be [1, 50, 64]. When running tensor[:, 1 : , :], it becomes [1, 49, 64]. Is the transformation from [1, 3, 224, 224] to [1, 50, 64] expected by GradCam?

Thank you!