HeroBench: A Benchmark for Long-Horizon Planning and Structured
  Reasoning in Virtual Worlds
Large language models (LLMs) have shown remarkable capabilities in isolated
step-by-step reasoning tasks such as mathematics and programming, but their
proficiency in long-horizon planning, where solutions require extended,
structured sequences of interdependent actions, remains underexplored. Existing
benchmarks typically assess LLMs through abstract or low-dimensional
algorithmic tasks, failing to capture the complexity of realistic planning
environments. We introduce HeroBench, a novel benchmark designed specifically
to evaluate long-horizon planning and structured reasoning within complex
RPG-inspired virtual worlds. HeroBench provides a rigorously constructed
dataset of tasks covering a wide range of difficulties, a simulated environment
to execute and validate agent plans, and detailed analytical tools for
evaluating model performance. Tasks challenge models to formulate strategic
plans, efficiently gather resources, master necessary skills, craft equipment,
and defeat adversaries, reflecting practical scenarios' layered dependencies
and constraints. Our extensive evaluation of 25 state-of-the-art LLMs, spanning
both open-source and proprietary models, including the GPT-5 family, reveals
substantial performance disparities rarely observed in conventional reasoning
benchmarks. Detailed error analysis further uncovers specific weaknesses in
current models' abilities to generate robust high-level plans and reliably
execute structured actions. HeroBench thus not only significantly advances the
evaluation of LLM reasoning but also provides a flexible, scalable foundation
for future research into advanced, autonomous planning in virtual environments.
68a42221b65388761d0745b9
Petr Anokhin
claimed_verified
2025-09-12T16:22:18.816Z
64f734b08e7fa6529fb693f1
/avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg
Petr Anokhin
user
petranokhin
68a42221b65388761d0745ba
Roman Khalikov
claimed_verified
2025-08-19T10:48:38.614Z
657332fc3e0cb21bc7ae7772
https://cdn-avatars.huggingface.co/v1/production/uploads/657332fc3e0cb21bc7ae7772/Z0JupacWx9KWXZwCtXy5Q.jpeg
Roman Khalikov
user
roxal
68a42221b65388761d0745bb
Stefan Rebrikov
claimed_verified
2025-09-19T06:50:55.940Z
637db2ce9a5217b88b7978d6
/avatars/77369f616cd729f21e8fc8a4b6563816.svg
Stefan
user
stefanrer
68a42221b65388761d0745bc
Viktor Volkov
claimed_verified
2025-08-19T15:58:46.930Z
67aadd77653ea1bf5ec27740
https://cdn-avatars.huggingface.co/v1/production/uploads/67aadd77653ea1bf5ec27740/kITzDTYje-J7itQTz_gHy.png
Viktor Volkov
user
ShiftingBorder
68a42221b65388761d0745bd
Artyom Sorokin
68a42221b65388761d0745be
Vincent Bissonnette
HeroBench: A Benchmark for Long-Horizon Planning and Structured
  Reasoning in Virtual Worlds
Large language models (LLMs) have shown remarkable capabilities in isolated
step-by-step reasoning tasks such as mathematics and programming, but their
proficiency in long-horizon planning, where solutions require extended,
structured sequences of interdependent actions, remains underexplored. Existing
benchmarks typically assess LLMs through abstract or low-dimensional
algorithmic tasks, failing to capture the complexity of realistic planning
environments. We introduce HeroBench, a novel benchmark designed specifically
to evaluate long-horizon planning and structured reasoning within complex
RPG-inspired virtual worlds. HeroBench provides a rigorously constructed
dataset of tasks covering a wide range of difficulties, a simulated environment
to execute and validate agent plans, and detailed analytical tools for
evaluating model performance. Tasks challenge models to formulate strategic
plans, efficiently gather resources, master necessary skills, craft equipment,
and defeat adversaries, reflecting practical scenarios' layered dependencies
and constraints. Our extensive evaluation of 25 state-of-the-art LLMs, spanning
both open-source and proprietary models, including the GPT-5 family, reveals
substantial performance disparities rarely observed in conventional reasoning
benchmarks. Detailed error analysis further uncovers specific weaknesses in
current models' abilities to generate robust high-level plans and reliably
execute structured actions. HeroBench thus not only significantly advances the
evaluation of LLM reasoning but also provides a flexible, scalable foundation
for future research into advanced, autonomous planning in virtual environments.
HeroBench evaluates long-horizon planning and structured reasoning in complex virtual worlds, revealing performance disparities in state-of-the-art LLMs.
large language models long-horizon planning structured reasoning HeroBench benchmark RPG-inspired virtual worlds strategic plans resource gathering skill mastery equipment crafting adversary defeat error analysis autonomous planning