τ^2-Bench: Evaluating Conversational Agents in a Dual-Control
  Environment
Existing benchmarks for conversational AI agents simulate single-control
environments, where only the AI agent can use tools to interact with the world,
while the user remains a passive information provider. This differs from
real-world scenarios like technical support, where users need to actively
participate in modifying the state of the (shared) world. In order to address
this gap, we introduce tau^2-bench, with four key contributions:
  1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both
agent and user make use of tools to act in a shared, dynamic environment that
tests both agent coordination and communication,
  2) A compositional task generator that programmatically creates diverse,
verifiable tasks from atomic components, ensuring domain coverage and
controlled complexity,
  3) A reliable user simulator tightly coupled with the environment, whose
behavior is constrained by tools and observable states, improving simulation
fidelity,
  4) Fine-grained analysis of agent performance through multiple ablations
including separating errors arising from reasoning vs
communication/coordination.
  In particular, our experiments show significant performance drops when agents
shift from no-user to dual-control, highlighting the challenges of guiding
users. Overall, tau^2-bench provides a controlled testbed for agents that
must both reason effectively and guide user actions.
684872f842e4f9106973f165
Victor Barres
684872f842e4f9106973f166
Honghua Dong
684872f842e4f9106973f167
Soham Ray
684872f842e4f9106973f168
Xujie Si
684872f842e4f9106973f169
Karthik Narasimhan
extracted_pending
2025-06-10T18:01:29.285Z
6643e737ba3c6c8d116e1585
/avatars/7a780509f14d86d1e6ec37cb39ea1f5d.svg
Karthik Narasimhan
user
karthikn-sierra
τ^2-Bench: Evaluating Conversational Agents in a Dual-Control
  Environment
Existing benchmarks for conversational AI agents simulate single-control
environments, where only the AI agent can use tools to interact with the world,
while the user remains a passive information provider. This differs from
real-world scenarios like technical support, where users need to actively
participate in modifying the state of the (shared) world. In order to address
this gap, we introduce tau^2-bench, with four key contributions:
  1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both
agent and user make use of tools to act in a shared, dynamic environment that
tests both agent coordination and communication,
  2) A compositional task generator that programmatically creates diverse,
verifiable tasks from atomic components, ensuring domain coverage and
controlled complexity,
  3) A reliable user simulator tightly coupled with the environment, whose
behavior is constrained by tools and observable states, improving simulation
fidelity,
  4) Fine-grained analysis of agent performance through multiple ablations
including separating errors arising from reasoning vs
communication/coordination.
  In particular, our experiments show significant performance drops when agents
shift from no-user to dual-control, highlighting the challenges of guiding
users. Overall, tau^2-bench provides a controlled testbed for agents that
must both reason effectively and guide user actions.
tau²-bench introduces a dual-control benchmark for conversational AI agents in telecommunications, simulating user-agent collaboration and evaluating their performance in reasoning and coordination.
Dec-POMDP compositional task generator user simulator ablations reasoning communication coordination