LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Tools are essential for large language models (LLMs) to acquire up-to-date
information and take consequential actions in external environments. Existing
work on tool-augmented LLMs primarily focuses on the broad coverage of tools
and the flexibility of adding new tools. However, a critical aspect that has
surprisingly been understudied is simply how accurately an LLM uses tools for
which it has been trained. We find that existing LLMs, including GPT-4 and
open-source LLMs specifically fine-tuned for tool use, only reach a correctness
rate in the range of 30% to 60%, far from reliable use in practice. We propose
a biologically inspired method for tool-augmented LLMs, simulated trial and
error (STE), that orchestrates three key mechanisms for successful tool use
behaviors in the biological system: trial and error, imagination, and memory.
Specifically, STE leverages an LLM's 'imagination' to simulate plausible
scenarios for using a tool, after which the LLM interacts with the tool to
learn from its execution feedback. Both short-term and long-term memory are
employed to improve the depth and breadth of the exploration, respectively.
Comprehensive experiments on ToolBench show that STE substantially improves
tool learning for LLMs under both in-context learning and fine-tuning settings,
bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
GPT-4. We also show effective continual learning of tools via a simple
experience replay strategy.
65ea7b02c5edf8e4d5c665c0
Boshi Wang
admin_assigned
2024-03-08T09:33:35.193Z
62e5e742972081fe55828530
/avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
Boshi Wang
user
boshiwang
65ea7b02c5edf8e4d5c665c1
Hao Fang
admin_assigned
2024-03-08T09:33:43.076Z
64d3cba0673c5609e8ecb14d
/avatars/8f6c52f06f1f77d16e067a0f7f05e1b1.svg
Hao Fang
user
haofang-msft
65ea7b02c5edf8e4d5c665c2
Jason Eisner
65ea7b02c5edf8e4d5c665c3
Benjamin Van Durme
65ea7b02c5edf8e4d5c665c4
Yu Su
claimed_verified
2024-03-08T08:33:14.514Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
Tools are essential for large language models (LLMs) to acquire up-to-date
information and take consequential actions in external environments. Existing
work on tool-augmented LLMs primarily focuses on the broad coverage of tools
and the flexibility of adding new tools. However, a critical aspect that has
surprisingly been understudied is simply how accurately an LLM uses tools for
which it has been trained. We find that existing LLMs, including GPT-4 and
open-source LLMs specifically fine-tuned for tool use, only reach a correctness
rate in the range of 30% to 60%, far from reliable use in practice. We propose
a biologically inspired method for tool-augmented LLMs, simulated trial and
error (STE), that orchestrates three key mechanisms for successful tool use
behaviors in the biological system: trial and error, imagination, and memory.
Specifically, STE leverages an LLM's 'imagination' to simulate plausible
scenarios for using a tool, after which the LLM interacts with the tool to
learn from its execution feedback. Both short-term and long-term memory are
employed to improve the depth and breadth of the exploration, respectively.
Comprehensive experiments on ToolBench show that STE substantially improves
tool learning for LLMs under both in-context learning and fine-tuning settings,
bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform
GPT-4. We also show effective continual learning of tools via a simple
experience replay strategy.
tool-augmented LLMs trial and error imagination memory short-term memory long-term memory in-context learning fine-tuning Mistral-Instruct-7B ToolBench experience replay