UserBench: An Interactive Gym Environment for User-Centric Agents
Large Language Models (LLMs)-based agents have made impressive progress in
reasoning and tool use, enabling them to solve complex tasks. However, their
ability to proactively collaborate with users, especially when goals are vague,
evolving, or indirectly expressed, remains underexplored. To address this gap,
we introduce UserBench, a user-centric benchmark designed to evaluate agents in
multi-turn, preference-driven interactions. UserBench features simulated users
who start with underspecified goals and reveal preferences incrementally,
requiring agents to proactively clarify intent and make grounded decisions with
tools. Our evaluation of leading open- and closed-source LLMs reveals a
significant disconnect between task completion and user alignment. For
instance, models provide answers that fully align with all user intents only
20% of the time on average, and even the most advanced models uncover fewer
than 30% of all user preferences through active interaction. These results
highlight the challenges of building agents that are not just capable task
executors, but true collaborative partners. UserBench offers an interactive
environment to measure and advance this critical capability.
689ac5a8fab6fdd2e52ac4f1
Cheng Qian
689ac5a8fab6fdd2e52ac4f2
Zuxin Liu
689ac5a8fab6fdd2e52ac4f3
Akshara Prabhakar
689ac5a8fab6fdd2e52ac4f4
Zhiwei Liu
689ac5a8fab6fdd2e52ac4f5
Jianguo Zhang
689ac5a8fab6fdd2e52ac4f6
Haolin Chen
689ac5a8fab6fdd2e52ac4f7
Heng Ji
689ac5a8fab6fdd2e52ac4f8
Weiran Yao
claimed_verified
2025-08-13T07:19:43.230Z
632cdea254e2c512c8f95b12
/avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
Weiran Yao
user
weirayao
689ac5a8fab6fdd2e52ac4f9
Shelby Heinecke
689ac5a8fab6fdd2e52ac4fa
Silvio Savarese
689ac5a8fab6fdd2e52ac4fb
Caiming Xiong
689ac5a8fab6fdd2e52ac4fc
Huan Wang
UserBench: An Interactive Gym Environment for User-Centric Agents
Large Language Models (LLMs)-based agents have made impressive progress in
reasoning and tool use, enabling them to solve complex tasks. However, their
ability to proactively collaborate with users, especially when goals are vague,
evolving, or indirectly expressed, remains underexplored. To address this gap,
we introduce UserBench, a user-centric benchmark designed to evaluate agents in
multi-turn, preference-driven interactions. UserBench features simulated users
who start with underspecified goals and reveal preferences incrementally,
requiring agents to proactively clarify intent and make grounded decisions with
tools. Our evaluation of leading open- and closed-source LLMs reveals a
significant disconnect between task completion and user alignment. For
instance, models provide answers that fully align with all user intents only
20% of the time on average, and even the most advanced models uncover fewer
than 30% of all user preferences through active interaction. These results
highlight the challenges of building agents that are not just capable task
executors, but true collaborative partners. UserBench offers an interactive
environment to measure and advance this critical capability.
UserBench evaluates LLM-based agents in multi-turn interactions with simulated users, revealing gaps in task completion and user alignment.
Large Language Models LLMs UserBench multi-turn interactions preference-driven interactions simulated users underspecified goals user alignment task executors collaborative partners