REFLECT: Summarizing Robot Experiences for Failure Explanation and
  Correction
The ability to detect and analyze failed executions automatically is crucial
for an explainable and robust robotic system. Recently, Large Language Models
(LLMs) have demonstrated strong common sense reasoning skills on textual
inputs. To leverage the power of LLM for robot failure explanation, we propose
a framework REFLECT, which converts multi-sensory data into a hierarchical
summary of robot past experiences and queries LLM with a progressive failure
explanation algorithm. Conditioned on the explanation, a failure correction
planner generates an executable plan for the robot to correct the failure and
complete the task. To systematically evaluate the framework, we create the
RoboFail dataset and show that our LLM-based framework is able to generate
informative failure explanations that assist successful correction planning.
Project website: https://roboreflect.github.io/
649cf7e2bc919c0fd7450003
Zeyi Liu
admin_assigned
2023-06-29T11:21:33.733Z
6369a7c856d1f9349812c2ae
/avatars/0e949ce509c13084c56755cbe4e501b4.svg
Liu
user
Zeyi
649cf7e2bc919c0fd7450004
Arpit Bahety
claimed_verified
2023-06-29T20:55:37.683Z
649d94cb9208f27d63257152
/avatars/5518a1ac8827c7c9b1f736ecbcc5e51d.svg
Arpit Bahety
user
abahety
649cf7e2bc919c0fd7450005
Shuran Song
REFLECT: Summarizing Robot Experiences for Failure Explanation and
  Correction
The ability to detect and analyze failed executions automatically is crucial
for an explainable and robust robotic system. Recently, Large Language Models
(LLMs) have demonstrated strong common sense reasoning skills on textual
inputs. To leverage the power of LLM for robot failure explanation, we propose
a framework REFLECT, which converts multi-sensory data into a hierarchical
summary of robot past experiences and queries LLM with a progressive failure
explanation algorithm. Conditioned on the explanation, a failure correction
planner generates an executable plan for the robot to correct the failure and
complete the task. To systematically evaluate the framework, we create the
RoboFail dataset and show that our LLM-based framework is able to generate
informative failure explanations that assist successful correction planning.
Project website: https://roboreflect.github.io/
A framework called REFLECT uses LLMs to generate informative failure explanations from multi-sensory data, aiding in correction planning for robots.
Large Language Models REFLECT multi-sensory data failure explanation algorithm failure correction planner hierarchical summary RoboFail dataset