Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal
  LLMs on Academic Resources
The reproduction of state-of-the-art multimodal LLM pre-training faces
barriers at every stage of the pipeline, including high-quality data filtering,
multimodal data mixture strategies, sequence packing techniques, and training
frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter
Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs
using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic
image resolution and multimodal sequence packing to significantly enhance
pre-training efficiency. The training dataset was carefully curated using both
MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based
filtering methods, substantially improving data quality and training
efficiency. The Open-Qwen2VL pre-training is conducted on academic level
8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T
multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned
Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on
various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista,
indicating the remarkable training efficiency of Open-Qwen2VL. We open-source
all aspects of our work, including compute-efficient and data-efficient
training details, data filtering methods, sequence packing scripts,
pre-training data in WebDataset format, FSDP-based training codebase, and both
base and instruction-tuned model checkpoints. We redefine "fully open" for
multimodal LLMs as the complete release of: 1) the training codebase, 2)
detailed data filtering techniques, and 3) all pre-training and supervised
fine-tuning data used to develop the model.
67ecaf516560da48c5c34106
Weizhi Wang
claimed_verified
2025-04-02T08:22:20.594Z
63d34004b734eaa4d4faeccf
https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
Weizhi Wang
user
weizhiwang
67ecaf516560da48c5c34107
Yu Tian
67ecaf516560da48c5c34108
Linjie Yang
admin_assigned
2025-04-02T09:55:04.382Z
67cd0b291580ba5d5ee65ffd
/avatars/9584a55473868e5ca1fa09b1536ca546.svg
yanglinjie
user
yanglj55
67ecaf516560da48c5c34109
Heng Wang
67ecaf516560da48c5c3410a
Xifeng Yan
admin_assigned
2025-04-02T09:54:38.935Z
65cd4785c40ab294321d610e
/avatars/3f0053aa2b3d90a10b60ab24cf575fd5.svg
Xifeng Yan
user
windwest
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal
  LLMs on Academic Resources
The reproduction of state-of-the-art multimodal LLM pre-training faces
barriers at every stage of the pipeline, including high-quality data filtering,
multimodal data mixture strategies, sequence packing techniques, and training
frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter
Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs
using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic
image resolution and multimodal sequence packing to significantly enhance
pre-training efficiency. The training dataset was carefully curated using both
MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based
filtering methods, substantially improving data quality and training
efficiency. The Open-Qwen2VL pre-training is conducted on academic level
8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T
multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned
Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on
various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista,
indicating the remarkable training efficiency of Open-Qwen2VL. We open-source
all aspects of our work, including compute-efficient and data-efficient
training details, data filtering methods, sequence packing scripts,
pre-training data in WebDataset format, FSDP-based training codebase, and both
base and instruction-tuned model checkpoints. We redefine "fully open" for
multimodal LLMs as the complete release of: 1) the training codebase, 2)
detailed data filtering techniques, and 3) all pre-training and supervised
fine-tuning data used to develop the model.
Open-Qwen2VL is a 2B-parameter multimodal LLM pre-trained efficiently on image-text pairs with improved data filtering and sequence packing, outperforming state-of-the-art models on various benchmarks.
multimodal LLM pre-training image resolution multimodal sequence packing MLM-based filtering CLIP-based filtering academic level 8xA100-40G GPUs FSDP-based training WebDataset MMBench SEEDBench MMstar MathVista compute-efficient data-efficient