TextQuests: How Good are LLMs at Text-Based Video Games?
Evaluating AI agents within complex, interactive environments that mirror
real-world challenges is critical for understanding their practical
capabilities. While existing agent benchmarks effectively assess skills like
tool use or performance on structured tasks, they often do not fully capture an
agent's ability to operate autonomously in exploratory environments that demand
sustained, self-directed reasoning over a long and growing context. To spur the
development of agents capable of more robust intrinsic reasoning over long
horizons, we introduce TextQuests, a benchmark based on the Infocom suite of
interactive fiction games. These text-based adventures, which can take human
players over 30 hours and require hundreds of precise actions to solve, serve
as an effective proxy for evaluating AI agents on focused, stateful tasks. The
benchmark is specifically designed to assess an LLM agent's capacity for
self-contained problem-solving by precluding the use of external tools, thereby
focusing on intrinsic long-context reasoning capabilities in an exploratory
environment characterized by the need for trial-and-error learning and
sustained problem-solving within a single interactive session. We release
TextQuests at https://textquests.ai.
689ab7e4fab6fdd2e52ac49c
Long Phan
claimed_verified
2025-08-13T07:20:06.535Z
62184306e1ef90f6e56dc116
https://cdn-avatars.huggingface.co/v1/production/uploads/1645757180654-noauth.jpeg
Long Phan
user
justinphan3110
689ab7e4fab6fdd2e52ac49d
Mantas Mazeika
689ab7e4fab6fdd2e52ac49e
Andy Zou
689ab7e4fab6fdd2e52ac49f
Dan Hendrycks
TextQuests: How Good are LLMs at Text-Based Video Games?
Evaluating AI agents within complex, interactive environments that mirror
real-world challenges is critical for understanding their practical
capabilities. While existing agent benchmarks effectively assess skills like
tool use or performance on structured tasks, they often do not fully capture an
agent's ability to operate autonomously in exploratory environments that demand
sustained, self-directed reasoning over a long and growing context. To spur the
development of agents capable of more robust intrinsic reasoning over long
horizons, we introduce TextQuests, a benchmark based on the Infocom suite of
interactive fiction games. These text-based adventures, which can take human
players over 30 hours and require hundreds of precise actions to solve, serve
as an effective proxy for evaluating AI agents on focused, stateful tasks. The
benchmark is specifically designed to assess an LLM agent's capacity for
self-contained problem-solving by precluding the use of external tools, thereby
focusing on intrinsic long-context reasoning capabilities in an exploratory
environment characterized by the need for trial-and-error learning and
sustained problem-solving within a single interactive session. We release
TextQuests at https://textquests.ai.
TextQuests evaluates AI agents' intrinsic reasoning and problem-solving capabilities in long, exploratory, text-based interactive fiction environments without external tools.
LLM agent intrinsic reasoning long-context reasoning interactive fiction games trial-and-error learning exploratory environment