This&That: Language-Gesture Controlled Video Generation for Robot
  Planning
We propose a robot learning method for communicating, planning, and executing
a wide range of tasks, dubbed This&That. We achieve robot planning for general
tasks by leveraging the power of video generative models trained on
internet-scale data containing rich physical and semantic context. In this
work, we tackle three fundamental challenges in video-based planning: 1)
unambiguous task communication with simple human instructions, 2) controllable
video generation that respects user intents, and 3) translating visual planning
into robot actions. We propose language-gesture conditioning to generate
videos, which is both simpler and clearer than existing language-only methods,
especially in complex and uncertain environments. We then suggest a behavioral
cloning design that seamlessly incorporates the video plans. This&That
demonstrates state-of-the-art effectiveness in addressing the above three
challenges, and justifies the use of video generation as an intermediate
representation for generalizable task planning and execution. Project website:
https://cfeng16.github.io/this-and-that/.
668d4fd76bacbfd13c62055a
Boyang Wang
claimed_verified
2024-07-11T08:27:15.636Z
64ed876a74d9b58eabc769a4
https://cdn-avatars.huggingface.co/v1/production/uploads/64ed876a74d9b58eabc769a4/K4bJVW0FlqRtAAxJBJifR.jpeg
Boyang Wang
user
HikariDawn
668d4fd76bacbfd13c62055b
Nikhil Sridhar
668d4fd76bacbfd13c62055c
Chao Feng
claimed_verified
2025-04-13T19:28:25.215Z
645ab0b7c266796265baefa4
/avatars/bdac661996b63c4b2a56881707afa01f.svg
Chao Feng
user
chfeng
668d4fd76bacbfd13c62055d
Mark Van der Merwe
668d4fd76bacbfd13c62055e
Adam Fishman
claimed_verified
2024-07-11T17:34:36.934Z
634b84b92cdd783d63177cd4
/avatars/96ab7f0360db943ad053e913fbc7357e.svg
Adam Fishman
user
fishbotics
668d4fd76bacbfd13c62055f
Nima Fazeli
668d4fd76bacbfd13c620560
Jeong Joon Park
claimed_verified
2024-07-11T17:34:35.169Z
6517d7f84fadbeb643c68269
/avatars/ad21288123abcd4488ae1bdcb10b54e7.svg
JJ Park
user
jjparkcv
This&That: Language-Gesture Controlled Video Generation for Robot
  Planning
We propose a robot learning method for communicating, planning, and executing
a wide range of tasks, dubbed This&That. We achieve robot planning for general
tasks by leveraging the power of video generative models trained on
internet-scale data containing rich physical and semantic context. In this
work, we tackle three fundamental challenges in video-based planning: 1)
unambiguous task communication with simple human instructions, 2) controllable
video generation that respects user intents, and 3) translating visual planning
into robot actions. We propose language-gesture conditioning to generate
videos, which is both simpler and clearer than existing language-only methods,
especially in complex and uncertain environments. We then suggest a behavioral
cloning design that seamlessly incorporates the video plans. This&That
demonstrates state-of-the-art effectiveness in addressing the above three
challenges, and justifies the use of video generation as an intermediate
representation for generalizable task planning and execution. Project website:
https://cfeng16.github.io/this-and-that/.
A robot learning method uses video generative models and language-gesture conditioning to plan and execute tasks based on human instructions and visual context.
video generative models language-gesture conditioning behavioral cloning video-based planning task communication controllable video generation visual planning generalizable task planning execution