MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math
  Reasoning in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in visual mathematical reasoning across various existing
benchmarks. However, these benchmarks are predominantly based on clean or
processed multimodal inputs, without incorporating the images provided by
real-world Kindergarten through 12th grade (K-12) educational users. To address
this gap, we introduce MathReal, a meticulously curated dataset comprising
2,000 mathematical questions with images captured by handheld mobile devices in
authentic scenarios. Each question is an image, containing the question text
and visual element. We systematically classify the real images into three
primary categories: image quality degradation, perspective variation, and
irrelevant content interference, which are further delineated into 14
subcategories. Additionally, MathReal spans five core knowledge and ability
categories, which encompass three question types and are divided into three
difficulty levels. To comprehensively evaluate the multimodal mathematical
reasoning abilities of state-of-the-art MLLMs in real-world scenarios, we
design six experimental settings that enable a systematic analysis of their
performance. Through extensive experimentation, we find that the
problem-solving abilities of existing MLLMs are significantly challenged in
realistic educational contexts. Based on this, we conduct a thorough analysis
of their performance and error patterns, providing insights into their
recognition, comprehension, and reasoning capabilities, and outlining
directions for future improvements. Data and code:
https://github.com/junfeng0288/MathReal.
68999432f022d141f5d4365d
Jun Feng
claimed_verified
2025-08-13T07:34:25.197Z
6894ed02b1bed8e651573585
/avatars/4532c56a77a87cf45f1c5040857aa6c3.svg
Jun Feng
user
junfeng0288
68999432f022d141f5d4365e
Zixin Wang
68999432f022d141f5d4365f
Zhentao Zhang
68999432f022d141f5d43660
Yue Guo
68999432f022d141f5d43661
Zhihan Zhou
68999432f022d141f5d43662
Xiuyi Chen
68999432f022d141f5d43663
Zhenyang Li
68999432f022d141f5d43664
Dawei Yin
MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math
  Reasoning in Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in visual mathematical reasoning across various existing
benchmarks. However, these benchmarks are predominantly based on clean or
processed multimodal inputs, without incorporating the images provided by
real-world Kindergarten through 12th grade (K-12) educational users. To address
this gap, we introduce MathReal, a meticulously curated dataset comprising
2,000 mathematical questions with images captured by handheld mobile devices in
authentic scenarios. Each question is an image, containing the question text
and visual element. We systematically classify the real images into three
primary categories: image quality degradation, perspective variation, and
irrelevant content interference, which are further delineated into 14
subcategories. Additionally, MathReal spans five core knowledge and ability
categories, which encompass three question types and are divided into three
difficulty levels. To comprehensively evaluate the multimodal mathematical
reasoning abilities of state-of-the-art MLLMs in real-world scenarios, we
design six experimental settings that enable a systematic analysis of their
performance. Through extensive experimentation, we find that the
problem-solving abilities of existing MLLMs are significantly challenged in
realistic educational contexts. Based on this, we conduct a thorough analysis
of their performance and error patterns, providing insights into their
recognition, comprehension, and reasoning capabilities, and outlining
directions for future improvements. Data and code:
https://github.com/junfeng0288/MathReal.
MathReal, a dataset of real-world mathematical questions with images, evaluates the performance of multimodal large language models in authentic educational settings, highlighting challenges and providing insights for improvement.
Multimodal Large Language Models MLLMs visual mathematical reasoning MathReal image quality degradation perspective variation irrelevant content interference multimodal mathematical reasoning experimental settings problem-solving abilities recognition comprehension reasoning capabilities