Creative Robot Tool Use with Large Language Models
Tool use is a hallmark of advanced intelligence, exemplified in both animal
behavior and robotic capabilities. This paper investigates the feasibility of
imbuing robots with the ability to creatively use tools in tasks that involve
implicit physical constraints and long-term planning. Leveraging Large Language
Models (LLMs), we develop RoboTool, a system that accepts natural language
instructions and outputs executable code for controlling robots in both
simulated and real-world environments. RoboTool incorporates four pivotal
components: (i) an "Analyzer" that interprets natural language to discern key
task-related concepts, (ii) a "Planner" that generates comprehensive strategies
based on the language input and key concepts, (iii) a "Calculator" that
computes parameters for each skill, and (iv) a "Coder" that translates these
plans into executable Python code. Our results show that RoboTool can not only
comprehend explicit or implicit physical constraints and environmental factors
but also demonstrate creative tool use. Unlike traditional Task and Motion
Planning (TAMP) methods that rely on explicit optimization, our LLM-based
system offers a more flexible, efficient, and user-friendly solution for
complex robotics tasks. Through extensive experiments, we validate that
RoboTool is proficient in handling tasks that would otherwise be infeasible
without the creative use of tools, thereby expanding the capabilities of
robotic systems. Demos are available on our project page:
https://creative-robotool.github.io/.
6535da7924c30636968cc691
Mengdi Xu
admin_assigned
2023-10-23T09:46:07.342Z
639187d92e1b430e96d45bc6
/avatars/1e1ee95a75adc4145166c5b50cbd524f.svg
Mengdi Xu
user
mengdi
6535da7924c30636968cc692
Peide Huang
6535da7924c30636968cc693
Wenhao Yu
admin_assigned
2023-10-23T09:46:24.098Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
6535da7924c30636968cc694
Shiqi Liu
6535da7924c30636968cc695
Xilun Zhang
6535da7924c30636968cc696
Yaru Niu
claimed_verified
2023-10-23T14:46:32.276Z
65366f44dea545ecdacf5cc9
/avatars/da6918251907872f446f70ba3bea1d4a.svg
Yaru Niu
user
chrisyrniu
6535da7924c30636968cc697
Tingnan Zhang
6535da7924c30636968cc698
Fei Xia
claimed_verified
2024-01-23T08:09:57.218Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
6535da7924c30636968cc699
Jie Tan
6535da7924c30636968cc69a
Ding Zhao
Creative Robot Tool Use with Large Language Models
Tool use is a hallmark of advanced intelligence, exemplified in both animal
behavior and robotic capabilities. This paper investigates the feasibility of
imbuing robots with the ability to creatively use tools in tasks that involve
implicit physical constraints and long-term planning. Leveraging Large Language
Models (LLMs), we develop RoboTool, a system that accepts natural language
instructions and outputs executable code for controlling robots in both
simulated and real-world environments. RoboTool incorporates four pivotal
components: (i) an "Analyzer" that interprets natural language to discern key
task-related concepts, (ii) a "Planner" that generates comprehensive strategies
based on the language input and key concepts, (iii) a "Calculator" that
computes parameters for each skill, and (iv) a "Coder" that translates these
plans into executable Python code. Our results show that RoboTool can not only
comprehend explicit or implicit physical constraints and environmental factors
but also demonstrate creative tool use. Unlike traditional Task and Motion
Planning (TAMP) methods that rely on explicit optimization, our LLM-based
system offers a more flexible, efficient, and user-friendly solution for
complex robotics tasks. Through extensive experiments, we validate that
RoboTool is proficient in handling tasks that would otherwise be infeasible
without the creative use of tools, thereby expanding the capabilities of
robotic systems. Demos are available on our project page:
https://creative-robotool.github.io/.
RoboTool, a system utilizing Large Language Models, enables robots to creatively use tools by interpreting natural language and generating executable code for complex tasks involving implicit physical constraints.
Large Language Models LLNs Analyzer Planner Calculator Coder Task and Motion Planning TAMP executable Python code