GPT4Tools: Teaching Large Language Model to Use Tools via
  Self-instruction
This paper aims to efficiently enable Large Language Models (LLMs) to use
multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have
shown great potential for tool usage through sophisticated prompt engineering.
Nevertheless, these models typically rely on prohibitive computational costs
and publicly inaccessible data. To address these challenges, we propose the
GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and
OPT, to use tools. It generates an instruction-following dataset by prompting
an advanced teacher with various multi-modal contexts. By using the Low-Rank
Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs
to solve a range of visual problems, including visual comprehension and image
generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to
use tools, which is performed in both zero-shot and fine-tuning ways. Extensive
experiments demonstrate the effectiveness of our method on various language
models, which not only significantly improves the accuracy of invoking seen
tools, but also enables the zero-shot capacity for unseen tools. The code and
demo are available at https://github.com/StevenGrove/GPT4Tools.
6476a1ce3d8d1cb79190b30a
Rui Yang
6476a1ce3d8d1cb79190b30b
Lin Song
claimed_verified
2023-05-31T12:01:01.658Z
6392a269e04c469baca3b154
/avatars/6446beb55a06f5038e0055c4f4ee8c44.svg
linsong
user
stevengrove
6476a1ce3d8d1cb79190b30c
Yanwei Li
6476a1ce3d8d1cb79190b30d
Sijie Zhao
6476a1ce3d8d1cb79190b30e
Yixiao Ge
claimed_verified
2023-06-01T16:30:20.722Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
6476a1ce3d8d1cb79190b30f
Xiu Li
6476a1ce3d8d1cb79190b310
Ying Shan
GPT4Tools: Teaching Large Language Model to Use Tools via
  Self-instruction
This paper aims to efficiently enable Large Language Models (LLMs) to use
multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have
shown great potential for tool usage through sophisticated prompt engineering.
Nevertheless, these models typically rely on prohibitive computational costs
and publicly inaccessible data. To address these challenges, we propose the
GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and
OPT, to use tools. It generates an instruction-following dataset by prompting
an advanced teacher with various multi-modal contexts. By using the Low-Rank
Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs
to solve a range of visual problems, including visual comprehension and image
generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to
use tools, which is performed in both zero-shot and fine-tuning ways. Extensive
experiments demonstrate the effectiveness of our method on various language
models, which not only significantly improves the accuracy of invoking seen
tools, but also enables the zero-shot capacity for unseen tools. The code and
demo are available at https://github.com/StevenGrove/GPT4Tools.
GPT4Tools enables efficient tool usage by open-source LLMs through LoRA optimization, improving accuracy and zero-shot capacity for unseen tools.
Large Language Models LLMs ChatGPT GPT-4 self-instruct LLaMA OPT Low-Rank Adaptation LoRA visual comprehension image generation benchmark zero-shot fine-tuning