Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities [CVPR 2024]

Yiyuan Zhang¹, Xiaohan Ding², Kaixiong Gong¹, Yixiao Ge², Ying Shan², Xiangyu Yue¹

¹ The Chinese University of Hong Kong  ² Tencent AI Lab 

---

Inspiration of Multimodal Pathway

This diagram's composition is inspired by a figure in Jeff Dean's blog post, where he envisions "pathways" as a high-level concept for general AI models. Our proposed Multimodal Pathway Transformer is a novel approach, and we are delighted to discover that some of its effects align with Jeff Dean's high-level vision, such as training a single model to do many things, enabling multiple senses, and making models sparse and efficient. Multimodal Pathway Transformer can be seen as an initial exploration of this "pathways" concept in the context of basic Transformer models and multimodal learning. Read more about Jeff Dean's concept in his blog post.

Abstract

We propose to improve transformers of a specific modality with irrelevant data from other modalities, e.g, improve an ImageNet model with audio or point cloud datasets. We would like to highlight that the data samples of the target modality are irrelevant to the other modalities, which distinguishes our method from other works utilizing paired (e.g. CLIP) or interleaved data of different modalities. We propose a methodology named Multimodal Pathway - given a target modality and a transformer designed for it, we use an auxiliary transformer trained with data of another modality and construct pathways to connect components of the two models so that data of the target modality can be processed by both models. In this way, we utilize the universal sequence-to-sequence modeling abilities of transformers obtained from two modalities. As a concrete implementation, we use a modality-specific tokenizer and task-specific head as usual but utilize the transformer blocks of the auxiliary model via a proposed method named Cross-Modal Re-parameterization, which exploits the auxiliary weights without any inference costs. On the image, point cloud, video, and audio recognition tasks, we observe significant and consistent performance improvements with irrelevant data from other modalities.

Model Zoo

Model	Modality	Pretraining	#Param	Google Drive	Tencent Cloud
ViT-B16	Image	MAE	86M	ckpt	ckpt
Audio ViT-B16	Audio	Audio MAE	86M	ckpt	ckpt
Point ViT-B16	Point	Point MAE	86M	ckpt	ckpt
Video ViT-B16	Video	Video MAE	86M	ckpt	ckpt

Usage

• Demo of use:

  #  Create Model
  import torch,timm
  model = timm.create_model("vit_base_patch16_224",pretrained = False)
  aux_model = timm.create_model("vit_base_patch16_224",pretrained = False)
  #  Load Pretrained Models
  pretrained_state_dict = torch.load("Image_ViT_B16.pth")
  aux_state_dict = torch.load("Audio_ViT_B16.pth")
  model.load_state_dict(pretrained_state_dict, strict=True)
  aux_model.load_state_dict(aux_state_dict, strict=True)
  #  Construct Multimodal Pathway
  from multimodal_pathway import reparameterize_aux_into_target_model
  reparameterize_aux_into_target_model(model, aux_model, layer_names=('attn.qkv', 'attn.proj', 'mlp.fc1','mlp.fc2'))

• For image recognition, please refer to image doc.

• For video recognition, please refer to video doc.

• For point cloud analysis, please refer to pcd doc.

• For audio recognition, please refer to audio doc.

BibTeX

If you find our work useful, please kindly cite:

@article{zhang2024multimodal,
      title={Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities},
      author={Zhang, Yiyuan and Ding, Xiaohan and Gong, Kaixiong and Ge, Yixiao and Shan, Ying and Yue, Xiangyu},
      journal={arXiv preprint arXiv:2401.14405},
      year={2024}
    }