Generative Evaluation of Complex Reasoning in Large Language Models
With powerful large language models (LLMs) demonstrating superhuman reasoning
capabilities, a critical question arises: Do LLMs genuinely reason, or do they
merely recall answers from their extensive, web-scraped training datasets?
Publicly released benchmarks inevitably become contaminated once incorporated
into subsequent LLM training sets, undermining their reliability as faithful
assessments. To address this, we introduce KUMO, a generative evaluation
framework designed specifically for assessing reasoning in LLMs. KUMO
synergistically combines LLMs with symbolic engines to dynamically produce
diverse, multi-turn reasoning tasks that are partially observable and
adjustable in difficulty. Through an automated pipeline, KUMO continuously
generates novel tasks across open-ended domains, compelling models to
demonstrate genuine generalization rather than memorization. We evaluated 23
state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
benchmarking their reasoning abilities against university students. Our
findings reveal that many LLMs have outperformed university-level performance
on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
performance on complex reasoning challenges. Moreover, LLM performance on KUMO
tasks correlates strongly with results on newly released real-world reasoning
benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
genuine LLM reasoning capabilities.
67f099de103cb604facd26cd
Haowei Lin
claimed_verified
2025-04-06T08:11:10.909Z
63453f02a05b51f7ded3c579
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
Andy Lin
user
pkuHaowei
67f099de103cb604facd26ce
Xiangyu Wang
67f099de103cb604facd26cf
Ruilin Yan
admin_assigned
2025-04-09T16:41:34.723Z
66d08a22526dae983ba5da22
/avatars/9744f8801e5b951ce49e5844c180efd7.svg
Ruilin Yan
user
0o0yrl
67f099de103cb604facd26d0
Baizhou Huang
admin_assigned
2025-04-09T16:41:40.792Z
67345e16b96e9dfdbec150af
/avatars/d502e901ccdc408a9f47ab9a664e7c81.svg
Baizhou Huang
user
BasilHCN
67f099de103cb604facd26d1
Haotian Ye
67f099de103cb604facd26d2
Jianhua Zhu
67f099de103cb604facd26d3
Zihao Wang
admin_assigned
2025-04-09T16:42:33.691Z
6757a1fde1a2dd778051b9e2
/avatars/bd2b5bb509902e1ad88a4b63e6cd0187.svg
Zihao Wang
user
ZihaoWang
67f099de103cb604facd26d4
James Zou
67f099de103cb604facd26d5
Jianzhu Ma
67f099de103cb604facd26d6
Yitao Liang
claimed_verified
2025-04-09T07:38:09.311Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
Generative Evaluation of Complex Reasoning in Large Language Models
With powerful large language models (LLMs) demonstrating superhuman reasoning
capabilities, a critical question arises: Do LLMs genuinely reason, or do they
merely recall answers from their extensive, web-scraped training datasets?
Publicly released benchmarks inevitably become contaminated once incorporated
into subsequent LLM training sets, undermining their reliability as faithful
assessments. To address this, we introduce KUMO, a generative evaluation
framework designed specifically for assessing reasoning in LLMs. KUMO
synergistically combines LLMs with symbolic engines to dynamically produce
diverse, multi-turn reasoning tasks that are partially observable and
adjustable in difficulty. Through an automated pipeline, KUMO continuously
generates novel tasks across open-ended domains, compelling models to
demonstrate genuine generalization rather than memorization. We evaluated 23
state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO,
benchmarking their reasoning abilities against university students. Our
findings reveal that many LLMs have outperformed university-level performance
on easy reasoning tasks, and reasoning-scaled LLMs reach university-level
performance on complex reasoning challenges. Moreover, LLM performance on KUMO
tasks correlates strongly with results on newly released real-world reasoning
benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for
genuine LLM reasoning capabilities.
large language models (LLMs) superhuman reasoning capabilities web-scraped training datasets generative evaluation framework symbolic engines multi-turn reasoning tasks partially observable adjustable in difficulty automated pipeline open-ended domains genuine generalization memorization reasoning abilities reasoning-scaled LLMs university-level performance complex reasoning challenges real-world reasoning benchmarks