Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large
  Language Models
Large language models (LLMs) have demonstrated impressive reasoning
capabilities, particularly in textual mathematical problem-solving. However,
existing open-source image instruction fine-tuning datasets, containing limited
question-answer pairs per image, do not fully exploit visual information to
enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs
(MLLMs). To bridge this gap, we address the lack of high-quality, diverse
multimodal mathematical datasets by collecting 40K high-quality images with
question-answer pairs from 24 existing datasets and synthesizing 320K new
pairs, creating the MathV360K dataset, which enhances both the breadth and
depth of multimodal mathematical questions. We introduce Math-LLaVA, a
LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach
significantly improves the multimodal mathematical reasoning capabilities of
LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V
on MathVista's minitest split. Furthermore, Math-LLaVA demonstrates enhanced
generalizability, showing substantial improvements on the MMMU benchmark. Our
research highlights the importance of dataset diversity and synthesis in
advancing MLLMs' mathematical reasoning abilities. The code and data are
available at: https://github.com/HZQ950419/Math-LLaVA.
667c06bf1976894a8bdac826
Wenhao Shi
667c06bf1976894a8bdac827
Zhiqiang Hu
claimed_verified
2024-06-27T09:03:10.563Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
user
Zhiqiang007
667c06bf1976894a8bdac828
Yi Bin
667c06bf1976894a8bdac829
Junhua Liu
667c06bf1976894a8bdac82a
Yang Yang
667c06bf1976894a8bdac82b
See-Kiong Ng
667c06bf1976894a8bdac82c
Lidong Bing
admin_assigned
2024-06-27T15:58:21.608Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
667c06bf1976894a8bdac82d
Roy Ka-Wei Lee
admin_assigned
2024-06-27T15:58:15.887Z
64b94cc0e3d41dbd6974ae45
/avatars/5edb9d9465addfceccef04c4465a34e6.svg
Roy Ka-Wei lee
user
sroylee
Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large
  Language Models
Large language models (LLMs) have demonstrated impressive reasoning
capabilities, particularly in textual mathematical problem-solving. However,
existing open-source image instruction fine-tuning datasets, containing limited
question-answer pairs per image, do not fully exploit visual information to
enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs
(MLLMs). To bridge this gap, we address the lack of high-quality, diverse
multimodal mathematical datasets by collecting 40K high-quality images with
question-answer pairs from 24 existing datasets and synthesizing 320K new
pairs, creating the MathV360K dataset, which enhances both the breadth and
depth of multimodal mathematical questions. We introduce Math-LLaVA, a
LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach
significantly improves the multimodal mathematical reasoning capabilities of
LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V
on MathVista's minitest split. Furthermore, Math-LLaVA demonstrates enhanced
generalizability, showing substantial improvements on the MMMU benchmark. Our
research highlights the importance of dataset diversity and synthesis in
advancing MLLMs' mathematical reasoning abilities. The code and data are
available at: https://github.com/HZQ950419/Math-LLaVA.
The introduction of MathV360K dataset and Math-LLaVA model enhances multimodal mathematical reasoning by increasing dataset diversity and improving generalizability.
Multimodal LLMs MathV360K Math-LLaVA LLaVA-1.5 MathVista MMMU benchmark multimodal mathematical reasoning