E5-V: Universal Embeddings with Multimodal Large Language Models
Multimodal large language models (MLLMs) have shown promising advancements in
general visual and language understanding. However, the representation of
multimodal information using MLLMs remains largely unexplored. In this work, we
introduce a new framework, E5-V, designed to adapt MLLMs for achieving
universal multimodal embeddings. Our findings highlight the significant
potential of MLLMs in representing multimodal inputs compared to previous
approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the
modality gap between different types of inputs, demonstrating strong
performance in multimodal embeddings even without fine-tuning. We propose a
single modality training approach for E5-V, where the model is trained
exclusively on text pairs. This method demonstrates significant improvements
over traditional multimodal training on image-text pairs, while reducing
training costs by approximately 95%. Additionally, this approach eliminates the
need for costly multimodal training data collection. Extensive experiments
across four types of tasks demonstrate the effectiveness of E5-V. As a
universal multimodal model, E5-V not only achieves but often surpasses
state-of-the-art performance in each task, despite being trained on a single
modality.
66987f3cbc2f8c7459128bf4
Ting Jiang
66987f3cbc2f8c7459128bf5
Minghui Song
66987f3cbc2f8c7459128bf6
Zihan Zhang
66987f3cbc2f8c7459128bf7
Haizhen Huang
claimed_verified
2024-07-22T07:10:45.003Z
6302e8e57e137e3d6b3f5a06
/avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
Huang
user
Haizhen
66987f3cbc2f8c7459128bf8
Weiwei Deng
66987f3cbc2f8c7459128bf9
Feng Sun
66987f3cbc2f8c7459128bfa
Qi Zhang
66987f3cbc2f8c7459128bfb
Deqing Wang
66987f3cbc2f8c7459128bfc
Fuzhen Zhuang
E5-V: Universal Embeddings with Multimodal Large Language Models
Multimodal large language models (MLLMs) have shown promising advancements in
general visual and language understanding. However, the representation of
multimodal information using MLLMs remains largely unexplored. In this work, we
introduce a new framework, E5-V, designed to adapt MLLMs for achieving
universal multimodal embeddings. Our findings highlight the significant
potential of MLLMs in representing multimodal inputs compared to previous
approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the
modality gap between different types of inputs, demonstrating strong
performance in multimodal embeddings even without fine-tuning. We propose a
single modality training approach for E5-V, where the model is trained
exclusively on text pairs. This method demonstrates significant improvements
over traditional multimodal training on image-text pairs, while reducing
training costs by approximately 95%. Additionally, this approach eliminates the
need for costly multimodal training data collection. Extensive experiments
across four types of tasks demonstrate the effectiveness of E5-V. As a
universal multimodal model, E5-V not only achieves but often surpasses
state-of-the-art performance in each task, despite being trained on a single
modality.
E5-V, a novel framework, utilizes multimodal large language models (MLLMs) to achieve universal multimodal embeddings with strong performance and reduced costs through single modality training.
multimodal large language models MLLMs universal multimodal embeddings promps modality gap image-text pairs single modality training state-of-the-art performance