Meta-Transformer: A Unified Framework for Multimodal Learning
Multimodal learning aims to build models that can process and relate
information from multiple modalities. Despite years of development in this
field, it still remains challenging to design a unified network for processing
various modalities (e.g. natural language, 2D images, 3D point
clouds, audio, video, time series, tabular data) due to the inherent gaps among
them. In this work, we propose a framework, named Meta-Transformer, that
leverages a frozen encoder to perform multimodal perception without
any paired multimodal training data. In Meta-Transformer, the raw input data
from various modalities are mapped into a shared token space, allowing a
subsequent encoder with frozen parameters to extract high-level semantic
features of the input data. Composed of three main components: a unified data
tokenizer, a modality-shared encoder, and task-specific heads for downstream
tasks, Meta-Transformer is the first framework to perform unified learning
across 12 modalities with unpaired data. Experiments on different benchmarks
reveal that Meta-Transformer can handle a wide range of tasks including
fundamental perception (text, image, point cloud, audio, video), practical
application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
tabular, and time-series). Meta-Transformer indicates a promising future for
developing unified multimodal intelligence with transformers. Code will be
available at https://github.com/invictus717/MetaTransformer
64ba0b16b7af2cf5151abc21
Yiyuan Zhang
claimed_verified
2024-01-29T08:30:12.378Z
63176933b58b0184630d2c74
https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
Yiyuan Zhang
user
Yiyuan
64ba0b16b7af2cf5151abc22
Kaixiong Gong
extracted_confirmed
2023-07-21T07:20:24.138Z
642e427f6748dd4f8eeb2f38
/avatars/07158ff6aa1803c846403594c5d55a34.svg
Kaixiong Gong
user
kxgong
64ba0b16b7af2cf5151abc23
Kaipeng Zhang
claimed_verified
2024-06-17T07:34:59.993Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
64ba0b16b7af2cf5151abc24
Hongsheng Li
claimed_verified
2024-02-05T10:50:43.920Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
64ba0b16b7af2cf5151abc25
Yu Qiao
64ba0b16b7af2cf5151abc26
Wanli Ouyang
64ba0b16b7af2cf5151abc27
Xiangyu Yue
Meta-Transformer: A Unified Framework for Multimodal Learning
Multimodal learning aims to build models that can process and relate
information from multiple modalities. Despite years of development in this
field, it still remains challenging to design a unified network for processing
various modalities (e.g. natural language, 2D images, 3D point
clouds, audio, video, time series, tabular data) due to the inherent gaps among
them. In this work, we propose a framework, named Meta-Transformer, that
leverages a frozen encoder to perform multimodal perception without
any paired multimodal training data. In Meta-Transformer, the raw input data
from various modalities are mapped into a shared token space, allowing a
subsequent encoder with frozen parameters to extract high-level semantic
features of the input data. Composed of three main components: a unified data
tokenizer, a modality-shared encoder, and task-specific heads for downstream
tasks, Meta-Transformer is the first framework to perform unified learning
across 12 modalities with unpaired data. Experiments on different benchmarks
reveal that Meta-Transformer can handle a wide range of tasks including
fundamental perception (text, image, point cloud, audio, video), practical
application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph,
tabular, and time-series). Meta-Transformer indicates a promising future for
developing unified multimodal intelligence with transformers. Code will be
available at https://github.com/invictus717/MetaTransformer
Meta-Transformer unifies multimodal learning across 12 unpaired modalities using a frozen encoder in a shared token space.
multimodal learning Meta-Transformer frozen encoder shared token space unified data tokenizer modality-shared encoder task-specific heads multimodal perception unified learning