Scaling Language-centric Omni-modal Representation Learning
Chenghao Xiao, Hou Pong (Ken) Chan, Hao Zhang, Weiwen Xu, Mahani Aljunied, Yu Rong
Recent multi-modal embedding approaches leveraging multi-modal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results. However, the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to leverage multimodal information within a shared representation space for generating unimodal outputs. We empirically demonstrate that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a language-centric omnimodal representation learning approach. Extensive experiments across diverse backbones and benchmarks demonstrate the effectiveness of our approach. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), demonstrating that the representational capacity gained through contrastive refinement scales positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for boosting representation quality. Finally, we provide a theoretical explanation for the GRSL based on mutual information discrepancy