τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World
  Domains
Existing benchmarks do not test language agents on their interaction with
human users or ability to follow domain-specific rules, both of which are vital
for deploying them in real world applications. We propose tau-bench, a
benchmark emulating dynamic conversations between a user (simulated by language
models) and a language agent provided with domain-specific API tools and policy
guidelines. We employ an efficient and faithful evaluation process that
compares the database state at the end of a conversation with the annotated
goal state. We also propose a new metric (pass^k) to evaluate the reliability
of agent behavior over multiple trials. Our experiments show that even
state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the
tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point
to the need for methods that can improve the ability of agents to act
consistently and follow rules reliably.
6674ffba568251b0b77afccf
Shunyu Yao
6674ffba568251b0b77afcd0
Noah Shinn
admin_assigned
2024-06-21T12:54:46.421Z
63ca0095b047b9b12a1face0
https://cdn-avatars.huggingface.co/v1/production/uploads/63ca0095b047b9b12a1face0/Eq_BCfFPbBig4XxSHs3xW.png
Noah Shinn
user
noahshinn
6674ffba568251b0b77afcd1
Pedram Razavi
6674ffba568251b0b77afcd2
Karthik Narasimhan
admin_assigned
2024-06-21T12:54:32.248Z
6643e737ba3c6c8d116e1585
/avatars/7a780509f14d86d1e6ec37cb39ea1f5d.svg
Karthik Narasimhan
user
karthikn-sierra
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World
  Domains
Existing benchmarks do not test language agents on their interaction with
human users or ability to follow domain-specific rules, both of which are vital
for deploying them in real world applications. We propose tau-bench, a
benchmark emulating dynamic conversations between a user (simulated by language
models) and a language agent provided with domain-specific API tools and policy
guidelines. We employ an efficient and faithful evaluation process that
compares the database state at the end of a conversation with the annotated
goal state. We also propose a new metric (pass^k) to evaluate the reliability
of agent behavior over multiple trials. Our experiments show that even
state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the
tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point
to the need for methods that can improve the ability of agents to act
consistently and follow rules reliably.
Tau-bench tests agents' interaction with simulated users and adherence to domain-specific rules, revealing inconsistencies in function calling agents.
tau-bench dynamic conversations language models domain-specific API tools policy guidelines pass^k function calling agents retail