Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning
  Models to Ask for Information
Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving
abilities in mathematics, as evaluated by existing benchmarks exclusively on
well-defined problems. However, such evaluation setup constitutes a critical
gap, since a genuine intelligent agent should not only solve problems (as a
math quiz solver), but also be able~to ask for information when the problems
lack sufficient information, enabling proactivity in responding users'
requests. To bridge such gap, we proposes a new dataset consisting of two types
of incomplete problems with diverse contexts. Based on the dataset, our
systematical evaluation of LRMs reveals their inability in proactively asking
for information. In addition, we uncover the behaviors related to overthinking
and hallucination of LRMs, and highlight the potential and challenges of
supervised fine-tuning in learning such ability. We hope to provide new
insights in developing LRMs with genuine intelligence, rather than just solving
problems.
68a3e308b65388761d0744bb
Youcheng Huang
claimed_verified
2025-08-19T08:04:24.093Z
5fcc4707563427b03e9af25b
/avatars/effcf3bcc9ef7163c454eeb4a90e1b09.svg
YouchengHuang
user
YouchengHuang
68a3e308b65388761d0744bc
Bowen Qin
68a3e308b65388761d0744bd
Chen Huang
68a3e308b65388761d0744be
Duanyu Feng
68a3e308b65388761d0744bf
Xi Yang
claimed_verified
2025-09-25T08:02:31.206Z
637c248f9c470afa387fb699
/avatars/bb8db04ea1444eac0820fee3acd652c1.svg
Xi Yang
user
xiyang99
68a3e308b65388761d0744c0
Wenqiang Lei
Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning
  Models to Ask for Information
Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving
abilities in mathematics, as evaluated by existing benchmarks exclusively on
well-defined problems. However, such evaluation setup constitutes a critical
gap, since a genuine intelligent agent should not only solve problems (as a
math quiz solver), but also be able~to ask for information when the problems
lack sufficient information, enabling proactivity in responding users'
requests. To bridge such gap, we proposes a new dataset consisting of two types
of incomplete problems with diverse contexts. Based on the dataset, our
systematical evaluation of LRMs reveals their inability in proactively asking
for information. In addition, we uncover the behaviors related to overthinking
and hallucination of LRMs, and highlight the potential and challenges of
supervised fine-tuning in learning such ability. We hope to provide new
insights in developing LRMs with genuine intelligence, rather than just solving
problems.
Systematic evaluation of Large Reasoning Models on incomplete problems reveals their inability to proactively seek information, highlighting issues like overthinking and hallucination, and the challenges of supervised fine-tuning for developing genuine intelligence.
Large Reasoning Models incomplete problems proactive information seeking overthinking hallucination supervised fine-tuning