Tool Documentation Enables Zero-Shot Tool-Usage with Large Language
  Models
Today, large language models (LLMs) are taught to use new tools by providing
a few demonstrations of the tool's usage. Unfortunately, demonstrations are
hard to acquire, and can result in undesirable biased usage if the wrong
demonstration is chosen. Even in the rare scenario that demonstrations are
readily available, there is no principled selection protocol to determine how
many and which ones to provide. As tasks grow more complex, the selection
search grows combinatorially and invariably becomes intractable. Our work
provides an alternative to demonstrations: tool documentation. We advocate the
use of tool documentation, descriptions for the individual tool usage, over
demonstrations. We substantiate our claim through three main empirical findings
on 6 tasks across both vision and language modalities. First, on existing
benchmarks, zero-shot prompts with only tool documentation are sufficient for
eliciting proper tool usage, achieving performance on par with few-shot
prompts. Second, on a newly collected realistic tool-use dataset with hundreds
of available tool APIs, we show that tool documentation is significantly more
valuable than demonstrations, with zero-shot documentation significantly
outperforming few-shot without documentation. Third, we highlight the benefits
of tool documentations by tackling image generation and video tracking using
just-released unseen state-of-the-art models as tools. Finally, we highlight
the possibility of using tool documentation to automatically enable new
applications: by using nothing more than the documentation of GroundingDino,
Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the
just-released Grounded-SAM and Track Anything models.
64c9c3f2f7f4ccb5ea66fc39
Cheng-Yu Hsieh
claimed_verified
2024-06-25T20:05:16.142Z
6408e8663461c51cf73480e4
/avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
Cheng-Yu Hsieh
user
cydhsieh01
64c9c3f2f7f4ccb5ea66fc3a
Si-An Chen
claimed_verified
2024-10-08T08:17:44.711Z
64cafdf84dcdaead7af64425
/avatars/6888d0eefa84ba218f2151922179f2f5.svg
Si-An Chen
user
sianchen
64c9c3f2f7f4ccb5ea66fc3b
Chun-Liang Li
64c9c3f2f7f4ccb5ea66fc3c
Yasuhisa Fujii
64c9c3f2f7f4ccb5ea66fc3d
Alexander Ratner
64c9c3f2f7f4ccb5ea66fc3e
Chen-Yu Lee
claimed_verified
2023-08-08T13:45:17.962Z
64d1fe784dfd5df7076bfd1a
/avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
Chen-Yu Lee
user
chenyulee
64c9c3f2f7f4ccb5ea66fc3f
Ranjay Krishna
64c9c3f2f7f4ccb5ea66fc40
Tomas Pfister
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language
  Models
Today, large language models (LLMs) are taught to use new tools by providing
a few demonstrations of the tool's usage. Unfortunately, demonstrations are
hard to acquire, and can result in undesirable biased usage if the wrong
demonstration is chosen. Even in the rare scenario that demonstrations are
readily available, there is no principled selection protocol to determine how
many and which ones to provide. As tasks grow more complex, the selection
search grows combinatorially and invariably becomes intractable. Our work
provides an alternative to demonstrations: tool documentation. We advocate the
use of tool documentation, descriptions for the individual tool usage, over
demonstrations. We substantiate our claim through three main empirical findings
on 6 tasks across both vision and language modalities. First, on existing
benchmarks, zero-shot prompts with only tool documentation are sufficient for
eliciting proper tool usage, achieving performance on par with few-shot
prompts. Second, on a newly collected realistic tool-use dataset with hundreds
of available tool APIs, we show that tool documentation is significantly more
valuable than demonstrations, with zero-shot documentation significantly
outperforming few-shot without documentation. Third, we highlight the benefits
of tool documentations by tackling image generation and video tracking using
just-released unseen state-of-the-art models as tools. Finally, we highlight
the possibility of using tool documentation to automatically enable new
applications: by using nothing more than the documentation of GroundingDino,
Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the
just-released Grounded-SAM and Track Anything models.
LLMs can effectively use tool documentation instead of demonstrations to perform complex tasks, achieving performance comparable or superior to few-shot learning on both vision and language benchmarks.
large language models zero-shot prompts few-shot prompts tool documentation tool APIs image generation video tracking GroundingDino Stable Diffusion XMem SAM Grounded-SAM Track Anything