GSM8K-V: Can Vision Language Models Solve Grade School Math Word
  Problems in Visual Contexts
Vision language models (VLMs) achieve unified modeling of images and text,
enabling them to accomplish complex real-world tasks through perception,
planning, and reasoning. Among these tasks, reasoning is particularly
representative, with mathematical reasoning serving as a prominent example. It
highlights the high-level capability of VLMs to comprehend mathematical
information in images and to perform sophisticated reasoning. Recently,
numerous visual mathematical reasoning benchmarks have been proposed, but they
are often restricted to geometry, lack coverage of math word problems, and
rarely assess reasoning across multiple images. To address these gaps, we
introduce GSM8K-V, a purely visual multi-image mathematical reasoning
benchmark. GSM8K-V is built by systematically mapping each sample from the
widely used text-based GSM8K into visual form. Through a carefully designed
automated image-generation pipeline combined with meticulous human annotation,
we curate 1,319 high-quality samples. We evaluate a wide range of open-source
and closed-source models on GSM8K-V. Results show that although existing VLMs
have nearly saturated performance on text-based GSM8K, there remains
substantial room for improvement on GSM8K-V. For example, the best-performing
model, Gemini-2.5-Pro, achieves 95.22% accuracy on GSM8K but only 46.93% on
GSM8K-V. We conduct a comprehensive analysis of GSM8K-V, examining the
limitations of current models as well as potential directions for improvement.
GSM8K-V offers a new perspective on visual mathematical reasoning and
establishes a benchmark to guide the development of more robust and
generalizable VLMs.
68db4fc5d2bf1f4b15ec74a4
Fan Yuan
68db4fc5d2bf1f4b15ec74a5
Yuchen Yan
claimed_verified
2025-09-30T06:00:51.986Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
68db4fc5d2bf1f4b15ec74a6
Yifan Jiang
68db4fc5d2bf1f4b15ec74a7
Haoran Zhao
68db4fc5d2bf1f4b15ec74a8
Tao Feng
68db4fc5d2bf1f4b15ec74a9
Jinyan Chen
68db4fc5d2bf1f4b15ec74aa
Yanwei Lou
68db4fc5d2bf1f4b15ec74ab
Wenqi Zhang
68db4fc5d2bf1f4b15ec74ac
Yongliang Shen
claimed_verified
2025-09-30T12:11:41.604Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
68db4fc5d2bf1f4b15ec74ad
Weiming Lu
68db4fc5d2bf1f4b15ec74ae
Jun Xiao
68db4fc5d2bf1f4b15ec74af
Yueting Zhuang
GSM8K-V: Can Vision Language Models Solve Grade School Math Word
  Problems in Visual Contexts
Vision language models (VLMs) achieve unified modeling of images and text,
enabling them to accomplish complex real-world tasks through perception,
planning, and reasoning. Among these tasks, reasoning is particularly
representative, with mathematical reasoning serving as a prominent example. It
highlights the high-level capability of VLMs to comprehend mathematical
information in images and to perform sophisticated reasoning. Recently,
numerous visual mathematical reasoning benchmarks have been proposed, but they
are often restricted to geometry, lack coverage of math word problems, and
rarely assess reasoning across multiple images. To address these gaps, we
introduce GSM8K-V, a purely visual multi-image mathematical reasoning
benchmark. GSM8K-V is built by systematically mapping each sample from the
widely used text-based GSM8K into visual form. Through a carefully designed
automated image-generation pipeline combined with meticulous human annotation,
we curate 1,319 high-quality samples. We evaluate a wide range of open-source
and closed-source models on GSM8K-V. Results show that although existing VLMs
have nearly saturated performance on text-based GSM8K, there remains
substantial room for improvement on GSM8K-V. For example, the best-performing
model, Gemini-2.5-Pro, achieves 95.22% accuracy on GSM8K but only 46.93% on
GSM8K-V. We conduct a comprehensive analysis of GSM8K-V, examining the
limitations of current models as well as potential directions for improvement.
GSM8K-V offers a new perspective on visual mathematical reasoning and
establishes a benchmark to guide the development of more robust and
generalizable VLMs.
GSM8K-V is a new visual multi-image mathematical reasoning benchmark that highlights the limitations of current vision language models in handling visual mathematical problems.
vision language models VLMs mathematical reasoning visual mathematical reasoning GSM8K-V image-generation pipeline Gemini-2.5-Pro