Train a Unified Multimodal Data Quality Classifier with Synthetic Data
The Multimodal Large Language Models (MLLMs) are continually pre-trained on a
mixture of image-text caption data and interleaved document data, while the
high-quality data filtering towards image-text interleaved document data is
under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal
Data Quality Classifier to Filter both high-quality image-text caption and
interleaved data (UniFilter). To address the challenge of collecting diverse
labeled multimodal data, we introduce a semi-synthetic approach that leverages
readily available raw images and generates corresponding text across four
quality levels. This method enables efficient creation of sample-score pairs
for both caption and interleaved document data to train UniFilter. We apply
UniFilter to curate high-quality caption data from DataComp caption dataset and
interleaved data from the OBELICS image-text interleaved dataset. MLLMs
pre-trained on the filtered data demonstrate significantly enhanced
capabilities compared to those trained on baseline-filtered data, achieving
stronger zero-shot reasoning and in-context learning capabilities. After visual
supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger
performance on various benchmarks, highlighting the downstream benefits of
high-quality multimodal pre-training. We release the synthetic training data
used for training UniFilter, the UniFilter model checkpoints, and the
high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to
the community for reproduction and further development.
68f5b0a68589920bf4d3217d
Weizhi Wang
claimed_verified
2025-10-20T12:10:21.495Z
63d34004b734eaa4d4faeccf
https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg
Weizhi Wang
user
weizhiwang
68f5b0a68589920bf4d3217e
Rongmei Lin
68f5b0a68589920bf4d3217f
Shiyang Li
68f5b0a68589920bf4d32180
Colin Lockard
68f5b0a68589920bf4d32181
Ritesh Sarkhel
68f5b0a68589920bf4d32182
Sanket Lokegaonkar
68f5b0a68589920bf4d32183
Jingbo Shang
68f5b0a68589920bf4d32184
Xifeng Yan
68f5b0a68589920bf4d32185
Nasser Zalmout
68f5b0a68589920bf4d32186
Xian Li
Train a Unified Multimodal Data Quality Classifier with Synthetic Data
The Multimodal Large Language Models (MLLMs) are continually pre-trained on a
mixture of image-text caption data and interleaved document data, while the
high-quality data filtering towards image-text interleaved document data is
under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal
Data Quality Classifier to Filter both high-quality image-text caption and
interleaved data (UniFilter). To address the challenge of collecting diverse
labeled multimodal data, we introduce a semi-synthetic approach that leverages
readily available raw images and generates corresponding text across four
quality levels. This method enables efficient creation of sample-score pairs
for both caption and interleaved document data to train UniFilter. We apply
UniFilter to curate high-quality caption data from DataComp caption dataset and
interleaved data from the OBELICS image-text interleaved dataset. MLLMs
pre-trained on the filtered data demonstrate significantly enhanced
capabilities compared to those trained on baseline-filtered data, achieving
stronger zero-shot reasoning and in-context learning capabilities. After visual
supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger
performance on various benchmarks, highlighting the downstream benefits of
high-quality multimodal pre-training. We release the synthetic training data
used for training UniFilter, the UniFilter model checkpoints, and the
high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to
the community for reproduction and further development.
UniFilter, a Unified Multimodal Data Quality Classifier, enhances Multimodal Large Language Models (MLLMs) by filtering high-quality image-text and interleaved data, leading to improved zero-shot reasoning and in-context learning.
Multimodal Large Language Models MLLMs Unified Multimodal Data Quality Classifier UniFilter semi-synthetic approach zero-shot reasoning in-context learning visual supervised fine-tuning DataComp caption dataset OBELICS image-text interleaved dataset OBELICS-HQ