MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late
  Interaction
Universal multimodal embedding models have achieved great success in
capturing semantic relevance between queries and candidates. However, current
methods either condense queries and candidates into a single vector,
potentially limiting the expressiveness for fine-grained information, or
produce too many vectors that are prohibitively expensive for multi-vector
retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal
retrieval that rethinks how multimodal embeddings are constructed and
interacted with at scale. During training, a fixed number of learnable Meta
Tokens are appended to the input sequence. At test-time, their last-layer
contextualized representations serve as compact yet expressive multi-vector
embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training,
MetaEmbed learns to organize information by granularity across multiple
vectors. As a result, we enable test-time scaling in multimodal retrieval,
where users can balance retrieval quality against efficiency demands by
selecting the number of tokens used for indexing and retrieval interactions.
Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and
the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed
achieves state-of-the-art retrieval performance while scaling robustly to
models with 32B parameters.
68d22a351ca7156988a8edfa
Zilin Xiao
claimed_verified
2025-09-23T10:05:31.739Z
67eb818141abf40cd87ab303
/avatars/8fd19f5fcac50946be63d55d265e68b0.svg
Zilin Xiao
user
MrZilinXiao
68d22a351ca7156988a8edfb
Qi Ma
68d22a351ca7156988a8edfc
Mengting Gu
68d22a351ca7156988a8edfd
Chun-cheng Jason Chen
68d22a351ca7156988a8edfe
Xintao Chen
68d22a351ca7156988a8edff
Vicente Ordonez
68d22a351ca7156988a8ee00
Vijai Mohan
MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late
  Interaction
Universal multimodal embedding models have achieved great success in
capturing semantic relevance between queries and candidates. However, current
methods either condense queries and candidates into a single vector,
potentially limiting the expressiveness for fine-grained information, or
produce too many vectors that are prohibitively expensive for multi-vector
retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal
retrieval that rethinks how multimodal embeddings are constructed and
interacted with at scale. During training, a fixed number of learnable Meta
Tokens are appended to the input sequence. At test-time, their last-layer
contextualized representations serve as compact yet expressive multi-vector
embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training,
MetaEmbed learns to organize information by granularity across multiple
vectors. As a result, we enable test-time scaling in multimodal retrieval,
where users can balance retrieval quality against efficiency demands by
selecting the number of tokens used for indexing and retrieval interactions.
Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and
the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed
achieves state-of-the-art retrieval performance while scaling robustly to
models with 32B parameters.
MetaEmbed, a new framework for multimodal retrieval, uses learnable Meta Tokens to provide compact yet expressive multi-vector embeddings, enabling scalable and efficient retrieval performance.
MetaEmbed Meta Tokens multimodal embeddings Matryoshka Multi-Vector Retrieval Massive Multimodal Embedding Benchmark Visual Document Retrieval Benchmark