ToolSandbox: A Stateful, Conversational, Interactive Evaluation
  Benchmark for LLM Tool Use Capabilities
Recent large language models (LLMs) advancements sparked a growing research
interest in tool assisted LLMs solving real-world challenges, which calls for
comprehensive evaluation of tool-use capabilities. While previous works focused
on either evaluating over stateless web services (RESTful API), based on a
single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
includes stateful tool execution, implicit state dependencies between tools, a
built-in user simulator supporting on-policy conversational evaluation and a
dynamic evaluation strategy for intermediate and final milestones over an
arbitrary trajectory. We show that open source and proprietary models have a
significant performance gap, and complex tasks like State Dependency,
Canonicalization and Insufficient Information defined in ToolSandbox are
challenging even the most capable SOTA LLMs, providing brand-new insights into
tool-use LLM capabilities. ToolSandbox evaluation framework is released at
https://github.com/apple/ToolSandbox
66b97d1874d4fb4d570ee3c7
Jiarui Lu
admin_assigned
2024-08-13T11:16:26.743Z
64bad07938953777fe3ed5e9
/avatars/91c07a176d8d51111873751e82a4e0ea.svg
Jiarui Lu
user
jiarui-lu
66b97d1874d4fb4d570ee3c8
Thomas Holleis
66b97d1874d4fb4d570ee3c9
Yizhe Zhang
admin_assigned
2024-08-13T11:16:52.951Z
64d1284c59503263d91b4707
/avatars/12d3799af150b538ad87874fdbf49dd1.svg
Yizhe Zhang
user
yizheapple
66b97d1874d4fb4d570ee3ca
Bernhard Aumayer
66b97d1874d4fb4d570ee3cb
Feng Nan
66b97d1874d4fb4d570ee3cc
Felix Bai
admin_assigned
2024-08-13T11:17:20.075Z
6468582ba9b4610868a654ab
/avatars/1f15b922a45e43e24905f9df98346ca5.svg
Felix Bai
user
fbai
66b97d1874d4fb4d570ee3cd
Shuang Ma
66b97d1874d4fb4d570ee3ce
Shen Ma
66b97d1874d4fb4d570ee3cf
Mengyu Li
admin_assigned
2024-08-13T11:17:44.954Z
6530b8bdcb5ad115c394e7cd
/avatars/5621f2394e6c382c09147185b56d4154.svg
Mengyu Li
user
molee1004
66b97d1874d4fb4d570ee3d0
Guoli Yin
admin_assigned
2024-08-13T11:17:50.580Z
605b7f42935268bc086131ba
/avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
Guoli Yin
user
gyin94
66b97d1874d4fb4d570ee3d1
Zirui Wang
66b97d1874d4fb4d570ee3d2
Ruoming Pang
admin_assigned
2024-08-13T11:18:19.204Z
654ef8ad3fe6c0b1f871942f
/avatars/8d3689b9bf57c7c8060ac510a1839158.svg
Ruoming Pang
user
ruoming
ToolSandbox: A Stateful, Conversational, Interactive Evaluation
  Benchmark for LLM Tool Use Capabilities
Recent large language models (LLMs) advancements sparked a growing research
interest in tool assisted LLMs solving real-world challenges, which calls for
comprehensive evaluation of tool-use capabilities. While previous works focused
on either evaluating over stateless web services (RESTful API), based on a
single turn user prompt, or an off-policy dialog trajectory, ToolSandbox
includes stateful tool execution, implicit state dependencies between tools, a
built-in user simulator supporting on-policy conversational evaluation and a
dynamic evaluation strategy for intermediate and final milestones over an
arbitrary trajectory. We show that open source and proprietary models have a
significant performance gap, and complex tasks like State Dependency,
Canonicalization and Insufficient Information defined in ToolSandbox are
challenging even the most capable SOTA LLMs, providing brand-new insights into
tool-use LLM capabilities. ToolSandbox evaluation framework is released at
https://github.com/apple/ToolSandbox
ToolSandbox evaluates large language models with tool-use capabilities through stateful execution, implicit state dependencies, and conversational simulation, revealing performance gaps and challenges with complex tasks.
stateful tool execution implicit state dependencies user simulator on-policy evaluation conversational evaluation dynamic evaluation strategy state dependency canonicalization insufficient information