Experience is the Best Teacher: Grounding VLMs for Robotics through
  Self-Generated Memory
Vision-language models (VLMs) have been widely adopted in robotics to enable
autonomous planning. However, grounding VLMs, originally trained on internet
data, to diverse real-world robots remains a challenge. This paper presents
ExpTeach, a framework that grounds VLMs to physical robots by building a
self-generated memory of real-world experiences. In ExpTeach, the VLM
autonomously plans actions, verifies outcomes, reflects on failures, and adapts
robot behaviors in a closed loop. The self-generated experiences during this
process are then summarized into a long-term memory, enabling retrieval of
learned knowledge to guide future tasks via retrieval-augmented generation
(RAG). Additionally, ExpTeach enhances the spatial understanding of VLMs with
an on-demand image annotation module. In experiments, we show that reflection
improves success rates from 36% to 84% on four challenging robotic tasks and
observe the emergence of intelligent object interactions, including creative
tool use. Across extensive tests on 12 real-world scenarios (including eight
unseen ones), we find that grounding with long-term memory boosts single-trial
success rates from 22% to 80%, demonstrating the effectiveness and
generalizability of ExpTeach.
6880bc6200f4b5a05f2fbf3f
Guowei Lan
6880bc6200f4b5a05f2fbf40
Kaixian Qu
6880bc6200f4b5a05f2fbf41
René Zurbrügg
6880bc6200f4b5a05f2fbf42
Changan Chen
6880bc6200f4b5a05f2fbf43
Christopher E. Mower
6880bc6200f4b5a05f2fbf44
Haitham Bou-Ammar
admin_assigned
2025-07-23T13:37:09.981Z
631c375768f7da9ad2496bf6
https://cdn-avatars.huggingface.co/v1/production/uploads/631c375768f7da9ad2496bf6/1sDOoecA6e1v_hn_VAgUq.jpeg
Haitham Bou Ammar
user
hba123
6880bc6200f4b5a05f2fbf45
Marco Hutter
Experience is the Best Teacher: Grounding VLMs for Robotics through
  Self-Generated Memory
Vision-language models (VLMs) have been widely adopted in robotics to enable
autonomous planning. However, grounding VLMs, originally trained on internet
data, to diverse real-world robots remains a challenge. This paper presents
ExpTeach, a framework that grounds VLMs to physical robots by building a
self-generated memory of real-world experiences. In ExpTeach, the VLM
autonomously plans actions, verifies outcomes, reflects on failures, and adapts
robot behaviors in a closed loop. The self-generated experiences during this
process are then summarized into a long-term memory, enabling retrieval of
learned knowledge to guide future tasks via retrieval-augmented generation
(RAG). Additionally, ExpTeach enhances the spatial understanding of VLMs with
an on-demand image annotation module. In experiments, we show that reflection
improves success rates from 36% to 84% on four challenging robotic tasks and
observe the emergence of intelligent object interactions, including creative
tool use. Across extensive tests on 12 real-world scenarios (including eight
unseen ones), we find that grounding with long-term memory boosts single-trial
success rates from 22% to 80%, demonstrating the effectiveness and
generalizability of ExpTeach.
ExpTeach grounds vision-language models to physical robots through self-generated memory and retrieval-augmented generation, improving success rates and enabling intelligent object interactions.
vision-language models VLMs autonomous planning self-generated memory long-term memory retrieval-augmented generation RAG spatial understanding on-demand image annotation intelligent object interactions creative tool use