Multimodal Pathway: Improve Transformers with Irrelevant Data from Other
  Modalities
We propose to improve transformers of a specific modality with irrelevant
data from other modalities, e.g., improve an ImageNet model with audio or point
cloud datasets. We would like to highlight that the data samples of the target
modality are irrelevant to the other modalities, which distinguishes our method
from other works utilizing paired (e.g., CLIP) or interleaved data of different
modalities. We propose a methodology named Multimodal Pathway - given a target
modality and a transformer designed for it, we use an auxiliary transformer
trained with data of another modality and construct pathways to connect
components of the two models so that data of the target modality can be
processed by both models. In this way, we utilize the universal
sequence-to-sequence modeling abilities of transformers obtained from two
modalities. As a concrete implementation, we use a modality-specific tokenizer
and task-specific head as usual but utilize the transformer blocks of the
auxiliary model via a proposed method named Cross-Modal Re-parameterization,
which exploits the auxiliary weights without any inference costs. On the image,
point cloud, video, and audio recognition tasks, we observe significant and
consistent performance improvements with irrelevant data from other modalities.
The code and models are available at https://github.com/AILab-CVC/M2PT.
65b338f98c7cfb1b8bc524fa
Yiyuan Zhang
admin_assigned
2024-01-26T09:50:33.141Z
63176933b58b0184630d2c74
https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
Yiyuan Zhang
user
Yiyuan
65b338f98c7cfb1b8bc524fb
Xiaohan Ding
extracted_confirmed
2024-01-26T09:18:37.159Z
6564f049957ac688429a6754
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nE6q5TlyLKlJpbxCKoDN8.png
xiaohanding
user
DingXiaoH
65b338f98c7cfb1b8bc524fc
Kaixiong Gong
admin_assigned
2024-01-26T09:50:40.131Z
642e427f6748dd4f8eeb2f38
/avatars/07158ff6aa1803c846403594c5d55a34.svg
Kaixiong Gong
user
kxgong
65b338f98c7cfb1b8bc524fd
Yixiao Ge
admin_assigned
2024-01-26T09:50:46.635Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
65b338f98c7cfb1b8bc524fe
Ying Shan
admin_assigned
2024-01-26T09:51:17.615Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
65b338f98c7cfb1b8bc524ff
Xiangyu Yue
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other
  Modalities
We propose to improve transformers of a specific modality with irrelevant
data from other modalities, e.g., improve an ImageNet model with audio or point
cloud datasets. We would like to highlight that the data samples of the target
modality are irrelevant to the other modalities, which distinguishes our method
from other works utilizing paired (e.g., CLIP) or interleaved data of different
modalities. We propose a methodology named Multimodal Pathway - given a target
modality and a transformer designed for it, we use an auxiliary transformer
trained with data of another modality and construct pathways to connect
components of the two models so that data of the target modality can be
processed by both models. In this way, we utilize the universal
sequence-to-sequence modeling abilities of transformers obtained from two
modalities. As a concrete implementation, we use a modality-specific tokenizer
and task-specific head as usual but utilize the transformer blocks of the
auxiliary model via a proposed method named Cross-Modal Re-parameterization,
which exploits the auxiliary weights without any inference costs. On the image,
point cloud, video, and audio recognition tasks, we observe significant and
consistent performance improvements with irrelevant data from other modalities.
The code and models are available at https://github.com/AILab-CVC/M2PT.
Multimodal Pathway improves transformer models by incorporating relevant data from other modalities through a novel Cross-Modal Re-parameterization method, yielding consistent performance gains across various recognition tasks.
transformers Multimodal Pathway auxiliary transformer pathways sequence-to-sequence modeling Cross-Modal Re-parameterization