MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
  Scale
Open-source multimodal large language models (MLLMs) have shown significant
potential in a broad range of multimodal tasks. However, their reasoning
capabilities remain constrained by existing instruction-tuning datasets, which
were predominately repurposed from academic datasets such as VQA, AI2D, and
ChartQA. These datasets target simplistic tasks, and only provide phrase-level
answers without any intermediate rationales. To address these challenges, we
introduce a scalable and cost-effective method to construct a large-scale
multimodal instruction-tuning dataset with rich intermediate rationales
designed to elicit CoT reasoning. Using only open models, we create a dataset
containing 12M instruction-response pairs to cover diverse, reasoning-intensive
tasks with detailed and faithful rationales. Experiments demonstrate that
training MLLMs on this dataset significantly improves reasoning capabilities,
achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
studies further highlight the importance of key components, such as rewriting
and self-filtering, in the dataset construction process.
6756592b59dd4e7470378f6c
Jarvis Guo
6756592b59dd4e7470378f6d
Tuney Zheng
claimed_verified
2024-12-09T13:26:09.219Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
6756592b59dd4e7470378f6e
Yuelin Bai
6756592b59dd4e7470378f6f
Bo Li
claimed_verified
2024-12-09T11:05:45.993Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
6756592b59dd4e7470378f70
Yubo Wang
claimed_verified
2025-01-30T08:41:23.329Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
6756592b59dd4e7470378f71
King Zhu
6756592b59dd4e7470378f72
Yizhi Li
claimed_verified
2025-07-25T08:40:09.782Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
6756592b59dd4e7470378f73
Graham Neubig
admin_assigned
2024-12-09T14:59:33.210Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
6756592b59dd4e7470378f74
Wenhu Chen
admin_assigned
2024-12-09T14:59:38.971Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
6756592b59dd4e7470378f75
Xiang Yue
claimed_verified
2024-12-09T11:06:01.577Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at
  Scale
Open-source multimodal large language models (MLLMs) have shown significant
potential in a broad range of multimodal tasks. However, their reasoning
capabilities remain constrained by existing instruction-tuning datasets, which
were predominately repurposed from academic datasets such as VQA, AI2D, and
ChartQA. These datasets target simplistic tasks, and only provide phrase-level
answers without any intermediate rationales. To address these challenges, we
introduce a scalable and cost-effective method to construct a large-scale
multimodal instruction-tuning dataset with rich intermediate rationales
designed to elicit CoT reasoning. Using only open models, we create a dataset
containing 12M instruction-response pairs to cover diverse, reasoning-intensive
tasks with detailed and faithful rationales. Experiments demonstrate that
training MLLMs on this dataset significantly improves reasoning capabilities,
achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
studies further highlight the importance of key components, such as rewriting
and self-filtering, in the dataset construction process.
A large-scale multimodal instruction-tuning dataset with intermediate rationales improves reasoning capabilities in large language models, achieving state-of-the-art performance across benchmarks.
multimodal large language models instruction-tuning dataset intermediate rationales CoT reasoning MathVerse MMMU-Pro MuirBench rewriting self-filtering