We-Math: Does Your Large Multimodal Model Achieve Human-like
  Mathematical Reasoning?
Visual mathematical reasoning, as a fundamental visual reasoning ability, has
received widespread attention from the Large Multimodal Models (LMMs)
community. Existing benchmarks, such as MathVista and MathVerse, focus more on
the result-oriented performance but neglect the underlying principles in
knowledge acquisition and generalization. Inspired by human-like mathematical
reasoning, we introduce WE-MATH, the first benchmark specifically designed to
explore the problem-solving principles beyond end-to-end performance. We
meticulously collect and categorize 6.5K visual math problems, spanning 67
hierarchical knowledge concepts and five layers of knowledge granularity. We
decompose composite problems into sub-problems according to the required
knowledge concepts and introduce a novel four-dimensional metric, namely
Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
(CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
existing LMMs in visual mathematical reasoning and reveal a negative
correlation between solving steps and problem-specific performance. We confirm
the IK issue of LMMs can be effectively improved via knowledge augmentation
strategies. More notably, the primary challenge of GPT-4o has significantly
transitioned from IK to IG, establishing it as the first LMM advancing towards
the knowledge generalization stage. In contrast, other LMMs exhibit a marked
inclination towards Rote Memorization - they correctly solve composite problems
involving multiple knowledge concepts yet fail to answer sub-problems. We
anticipate that WE-MATH will open new pathways for advancements in visual
mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
available at https://github.com/We-Math/We-Math.
66835909bad5fded13ebcdcd
Runqi Qiao
claimed_verified
2024-12-02T08:52:11.000Z
6683a05e74fb1736a4b7c934
https://cdn-avatars.huggingface.co/v1/production/uploads/6683a05e74fb1736a4b7c934/eiz6qlqIUjAWGy5zfg8Cs.jpeg
QRQ
user
RichardQRQ
66835909bad5fded13ebcdce
Qiuna Tan
66835909bad5fded13ebcdcf
Guanting Dong
admin_assigned
2024-07-02T08:00:53.082Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
66835909bad5fded13ebcdd0
Minhui Wu
66835909bad5fded13ebcdd1
Chong Sun
admin_assigned
2024-07-02T08:01:24.607Z
657e77a7efc0f84bcb7ff6ce
/avatars/384e0d8e1ffdf99451ab40e9fcba93ec.svg
Chong Sun
user
Csun1992
66835909bad5fded13ebcdd2
Xiaoshuai Song
66835909bad5fded13ebcdd3
Zhuoma GongQue
claimed_verified
2024-12-20T08:34:25.993Z
6762b973686e574cab4717ec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
zhuoma
user
Ethereal1l
66835909bad5fded13ebcdd4
Shanglin Lei
66835909bad5fded13ebcdd5
Zhe Wei
66835909bad5fded13ebcdd6
Miaoxuan Zhang
66835909bad5fded13ebcdd7
Runfeng Qiao
admin_assigned
2024-07-02T08:20:20.931Z
668362a5f8fd8217cc20b493
/avatars/bc40a0aafdcbf22dc018ad8652a272fe.svg
Runfeng Qiao
user
Arsoeir
66835909bad5fded13ebcdd8
Yifan Zhang
admin_assigned
2024-12-18T14:34:59.169Z
66615dd1c8bbe034b4b82029
/avatars/7ef1ddf2fa356efbeb2b3aa12284a1f0.svg
Zhang YiFan
user
zhangyifan666
66835909bad5fded13ebcdd9
Xiao Zong
66835909bad5fded13ebcdda
Yida Xu
admin_assigned
2024-07-02T08:19:19.230Z
662b19df4f711ee4e1f5bf9b
/avatars/d2a0eab67f71d0eab28dffa7214186eb.svg
Yida Xu
user
yidada
66835909bad5fded13ebcddb
Muxi Diao
claimed_verified
2024-12-30T19:40:00.199Z
6768c97367e4b4606a3c9cec
/avatars/5ddafe7a05828366f66c79072556f370.svg
diaomuxi
user
diaomuxi
66835909bad5fded13ebcddc
Zhimin Bao
66835909bad5fded13ebcddd
Chen Li
66835909bad5fded13ebcdde
Honggang Zhang
We-Math: Does Your Large Multimodal Model Achieve Human-like
  Mathematical Reasoning?
Visual mathematical reasoning, as a fundamental visual reasoning ability, has
received widespread attention from the Large Multimodal Models (LMMs)
community. Existing benchmarks, such as MathVista and MathVerse, focus more on
the result-oriented performance but neglect the underlying principles in
knowledge acquisition and generalization. Inspired by human-like mathematical
reasoning, we introduce WE-MATH, the first benchmark specifically designed to
explore the problem-solving principles beyond end-to-end performance. We
meticulously collect and categorize 6.5K visual math problems, spanning 67
hierarchical knowledge concepts and five layers of knowledge granularity. We
decompose composite problems into sub-problems according to the required
knowledge concepts and introduce a novel four-dimensional metric, namely
Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
(CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
existing LMMs in visual mathematical reasoning and reveal a negative
correlation between solving steps and problem-specific performance. We confirm
the IK issue of LMMs can be effectively improved via knowledge augmentation
strategies. More notably, the primary challenge of GPT-4o has significantly
transitioned from IK to IG, establishing it as the first LMM advancing towards
the knowledge generalization stage. In contrast, other LMMs exhibit a marked
inclination towards Rote Memorization - they correctly solve composite problems
involving multiple knowledge concepts yet fail to answer sub-problems. We
anticipate that WE-MATH will open new pathways for advancements in visual
mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
available at https://github.com/We-Math/We-Math.
WE-MATH benchmark evaluates Large Multimodal Models in visual mathematical reasoning, focusing on knowledge acquisition and generalization principles beyond end-to-end performance.
Large Multimodal Models visual mathematical reasoning MathVista MathVerse WE-MATH visual math problems hierarchical knowledge concepts sub-problems Inconsistent Knowledge Inadequate Generalization Complete Mastery Rote Memorization knowledge augmentation strategies GPT-4o