UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon
  Scenarios
Autonomous agents have recently achieved remarkable progress across diverse
domains, yet most evaluations focus on short-horizon, fully observable tasks.
In contrast, many critical real-world tasks, such as large-scale software
development, commercial investment, and scientific discovery, unfold in
long-horizon and partially observable scenarios where success hinges on
sustained reasoning, planning, memory management, and tool use. Existing
benchmarks rarely capture these long-horizon challenges, leaving a gap in
systematic evaluation. To bridge this gap, we introduce UltraHorizon a
novel benchmark that measures the foundational capabilities essential for
complex real-world challenges. We use exploration as a unifying task across
three distinct environments to validate these core competencies. Agents are
designed in long-horizon discovery tasks where they must iteratively uncover
hidden rules through sustained reasoning, planning, memory and tools
management, and interaction with environments. Under the heaviest scale
setting, trajectories average 200k+ tokens and 400+ tool
calls, whereas in standard configurations they still exceed 35k tokens
and involve more than 60 tool calls on average. Our extensive
experiments reveal that LLM-agents consistently underperform in these settings,
whereas human participants achieve higher scores, underscoring a persistent gap
in agents' long-horizon abilities. We also observe that simple scaling fails in
our task. To better illustrate the failure of agents, we conduct an in-depth
analysis of collected trajectories. We identify eight types of errors and
attribute them to two primary causes: in-context locking and functional
fundamental capability gaps.
https://github.com/StarDewXXX/UltraHorizon{Our code will be available
here.}
68d9f1fa0177a6054b013ae1
Haotian Luo
68d9f1fa0177a6054b013ae2
Huaisong Zhang
68d9f1fa0177a6054b013ae3
Xuelin Zhang
68d9f1fa0177a6054b013ae4
Haoyu Wang
68d9f1fa0177a6054b013ae5
Zeyu Qin
68d9f1fa0177a6054b013ae6
Wenjie Lu
68d9f1fa0177a6054b013ae7
Guozheng Ma
68d9f1fa0177a6054b013ae8
Haiying He
68d9f1fa0177a6054b013ae9
Yingsha Xie
68d9f1fa0177a6054b013aea
Qiyang Zhou
68d9f1fa0177a6054b013aeb
Zixuan Hu
68d9f1fa0177a6054b013aec
Hongze Mi
68d9f1fa0177a6054b013aed
Yibo Wang
68d9f1fa0177a6054b013aee
Naiqiang Tan
68d9f1fa0177a6054b013aef
Hong Chen
68d9f1fa0177a6054b013af0
Yi R. Fung
68d9f1fa0177a6054b013af1
Chun Yuan
68d9f1fa0177a6054b013af2
Li Shen
UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon
  Scenarios
Autonomous agents have recently achieved remarkable progress across diverse
domains, yet most evaluations focus on short-horizon, fully observable tasks.
In contrast, many critical real-world tasks, such as large-scale software
development, commercial investment, and scientific discovery, unfold in
long-horizon and partially observable scenarios where success hinges on
sustained reasoning, planning, memory management, and tool use. Existing
benchmarks rarely capture these long-horizon challenges, leaving a gap in
systematic evaluation. To bridge this gap, we introduce UltraHorizon a
novel benchmark that measures the foundational capabilities essential for
complex real-world challenges. We use exploration as a unifying task across
three distinct environments to validate these core competencies. Agents are
designed in long-horizon discovery tasks where they must iteratively uncover
hidden rules through sustained reasoning, planning, memory and tools
management, and interaction with environments. Under the heaviest scale
setting, trajectories average 200k+ tokens and 400+ tool
calls, whereas in standard configurations they still exceed 35k tokens
and involve more than 60 tool calls on average. Our extensive
experiments reveal that LLM-agents consistently underperform in these settings,
whereas human participants achieve higher scores, underscoring a persistent gap
in agents' long-horizon abilities. We also observe that simple scaling fails in
our task. To better illustrate the failure of agents, we conduct an in-depth
analysis of collected trajectories. We identify eight types of errors and
attribute them to two primary causes: in-context locking and functional
fundamental capability gaps.
https://github.com/StarDewXXX/UltraHorizon{Our code will be available
here.}
UltraHorizon is a new benchmark that evaluates long-horizon and partially observable tasks for autonomous agents, highlighting gaps in their sustained reasoning, planning, memory, and tool use capabilities.
UltraHorizon long-horizon partially observable sustained reasoning planning memory management tool use exploration trajectories tokens tool calls LLM-agents in-context locking functional fundamental capability gaps