Generative Evaluation of Complex Reasoning in Large Language Models
With powerful large language models (LLMs) demonstrating superhuman reasoning
capabilities, a critical question arises: Do LLMs genuinely reason, or do they
merely recall answers from their extensive, web-scraped training datasets?
Publicly released benchmarks inevitably become contaminated once incorporated
into subsequent LLM training sets, undermining their reliability as faithful
assessments. To address this, we introduce KUMO, a generative evaluation
framework designed specifically for assessing reasoning in LLMs. KUMO
synergistically combines LLMs with symbolic engines to dynamically produce
diverse, multi-turn reasoning tasks that are partially observable and
adjustable in difficulty. Through an automated pipeline, KUMO continuously
generates novel tasks across open-ended domains, compelling models to
demonstrate genuine generalization rather than memorization. We evaluated 23
state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
benchmarking their reasoning abilities against university students. Our
findings reveal that many LLMs have outperformed university-level performance
on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
performance on complex reasoning challenges. Moreover, LLM performance on KUMO
tasks correlates strongly with results on newly released real-world reasoning
benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
genuine LLM reasoning capabilities.
67f099de103cb604facd26cd
Haowei Lin
claimed_verified
2025-04-06T08:11:10.909Z
63453f02a05b51f7ded3c579
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
Andy Lin
user
pkuHaowei
67f099de103cb604facd26ce
Xiangyu Wang
67f099de103cb604facd26cf
Ruilin Yan
admin_assigned
2025-04-09T16:41:34.723Z
66d08a22526dae983ba5da22
/avatars/9744f8801e5b951ce49e5844c180efd7.svg
Ruilin Yan
user
0o0yrl
67f099de103cb604facd26d0
Baizhou Huang
admin_assigned
2025-04-09T16:41:40.792Z
67345e16b96e9dfdbec150af
/avatars/d502e901ccdc408a9f47ab9a664e7c81.svg
Baizhou Huang
user
BasilHCN
67f099de103cb604facd26d1
Haotian Ye
67f099de103cb604facd26d2
Jianhua Zhu
67f099de103cb604facd26d3
Zihao Wang
admin_assigned
2025-04-09T16:42:33.691Z
6757a1fde1a2dd778051b9e2
/avatars/bd2b5bb509902e1ad88a4b63e6cd0187.svg
Zihao Wang
user
ZihaoWang
67f099de103cb604facd26d4
James Zou
67f099de103cb604facd26d5
Jianzhu Ma
67f099de103cb604facd26d6
Yitao Liang
claimed_verified
2025-04-09T07:38:09.311Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
Generative Evaluation of Complex Reasoning in Large Language Models
With powerful large language models (LLMs) demonstrating superhuman reasoning
capabilities, a critical question arises: Do LLMs genuinely reason, or do they
merely recall answers from their extensive, web-scraped training datasets?
Publicly released benchmarks inevitably become contaminated once incorporated
into subsequent LLM training sets, undermining their reliability as faithful
assessments. To address this, we introduce KUMO, a generative evaluation
framework designed specifically for assessing reasoning in LLMs. KUMO
synergistically combines LLMs with symbolic engines to dynamically produce
diverse, multi-turn reasoning tasks that are partially observable and
adjustable in difficulty. Through an automated pipeline, KUMO continuously
generates novel tasks across open-ended domains, compelling models to
demonstrate genuine generalization rather than memorization. We evaluated 23
state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
benchmarking their reasoning abilities against university students. Our
findings reveal that many LLMs have outperformed university-level performance
on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
performance on complex reasoning challenges. Moreover, LLM performance on KUMO
tasks correlates strongly with results on newly released real-world reasoning
benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
genuine LLM reasoning capabilities.
KUMO, a generative evaluation framework combining LLMs with symbolic engines, assesses genuine reasoning in large language models through diverse, dynamically produced tasks.
large language models (LLMs) symbolic engines generative evaluation framework reasoning tasks university students reasoning-scaled LLMs real-world reasoning benchmarks