MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Despite the rapidly growing demand for multimodal retrieval, progress in this
field remains severely constrained by a lack of training data. In this paper,
we introduce MegaPairs, a novel data synthesis method that leverages vision
language models (VLMs) and open-domain images, together with a massive
synthetic dataset generated from this method. Our empirical analysis shows that
MegaPairs generates high-quality data, enabling the multimodal retriever to
significantly outperform the baseline model trained on 70times more data
from existing datasets. Moreover, since MegaPairs solely relies on general
image corpora and open-source VLMs, it can be easily scaled up, enabling
continuous improvements in retrieval performance. In this stage, we produced
more than 26 million training instances and trained several models of varying
sizes using this data. These new models achieve state-of-the-art zero-shot
performance across 4 popular composed image retrieval (CIR) benchmarks and the
highest overall performance on the 36 datasets provided by MMEB. They also
demonstrate notable performance improvements with additional downstream
fine-tuning. Our produced dataset, well-trained models, and data synthesis
pipeline will be made publicly available to facilitate the future development
of this field.
6764da4e0044fe72b24b009e
Junjie Zhou
claimed_verified
2024-12-20T08:32:59.134Z
6564a2ceedae9c33b7654a1f
/avatars/42f09356a1282896573ccb44830cd327.svg
JUNJIE ZHOU
user
JUNJIE99
6764da4e0044fe72b24b009f
Zheng Liu
extracted_pending
2024-12-20T02:45:36.277Z
64a38c590111d5ff6c3d5f2b
/avatars/ef13dc7ce243819bc0da9b04e778b432.svg
zhengliu
user
zl101
6764da4e0044fe72b24b00a0
Ze Liu
claimed_verified
2024-12-21T15:19:54.180Z
66164f6245336ca774679611
/avatars/9baf0ab475bc8d5997abda9ffe8cfa28.svg
Ze Liu
user
marsh123
6764da4e0044fe72b24b00a1
Shitao Xiao
admin_assigned
2024-12-20T09:17:36.576Z
62612679bbcbd1c34f1638af
/avatars/c0675d05a52192ee14e9ab1633353956.svg
Xiao
user
Shitao
6764da4e0044fe72b24b00a2
Yueze Wang
admin_assigned
2024-12-20T09:17:27.209Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
6764da4e0044fe72b24b00a3
Bo Zhao
admin_assigned
2024-12-20T09:17:20.202Z
64acafac1aee69ece03af05a
/avatars/6380777852546dbd432fed5544d5db0b.svg
Bo Zhao
user
Z-LIRA
6764da4e0044fe72b24b00a4
Chen Jason Zhang
admin_assigned
2024-12-20T09:17:04.629Z
6465943fe27766e8921bf4b3
/avatars/86cc7f24c2348c634711807c3261961d.svg
chenjason
user
chenjason
6764da4e0044fe72b24b00a5
Defu Lian
admin_assigned
2024-12-20T09:16:55.115Z
66ed026076a8038cb4ae6053
/avatars/99b6527da6b66c6b5df3fc8261587322.svg
Defu Lian
user
dove-ustc
6764da4e0044fe72b24b00a6
Yongping Xiong
MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Despite the rapidly growing demand for multimodal retrieval, progress in this
field remains severely constrained by a lack of training data. In this paper,
we introduce MegaPairs, a novel data synthesis method that leverages vision
language models (VLMs) and open-domain images, together with a massive
synthetic dataset generated from this method. Our empirical analysis shows that
MegaPairs generates high-quality data, enabling the multimodal retriever to
significantly outperform the baseline model trained on 70times more data
from existing datasets. Moreover, since MegaPairs solely relies on general
image corpora and open-source VLMs, it can be easily scaled up, enabling
continuous improvements in retrieval performance. In this stage, we produced
more than 26 million training instances and trained several models of varying
sizes using this data. These new models achieve state-of-the-art zero-shot
performance across 4 popular composed image retrieval (CIR) benchmarks and the
highest overall performance on the 36 datasets provided by MMEB. They also
demonstrate notable performance improvements with additional downstream
fine-tuning. Our produced dataset, well-trained models, and data synthesis
pipeline will be made publicly available to facilitate the future development
of this field.
MegaPairs, a data synthesis method using vision language models and open-domain images, generates high-quality multimodal retrieval data, outperforming larger existing datasets and achieving state-of-the-art zero-shot performance.
vision language models multimodal retrieval data synthesis MegaPairs composed image retrieval MMEB