Vid2Robot: End-to-end Video-conditioned Policy Learning with
  Cross-Attention Transformers
While large-scale robotic systems typically rely on textual instructions for
tasks, this work explores a different approach: can robots infer the task
directly from observing humans? This shift necessitates the robot's ability to
decode human intent and translate it into executable actions within its
physical constraints and environment. We introduce Vid2Robot, a novel
end-to-end video-based learning framework for robots. Given a video
demonstration of a manipulation task and current visual observations, Vid2Robot
directly produces robot actions. This is achieved through a unified
representation model trained on a large dataset of human video and robot
trajectory. The model leverages cross-attention mechanisms to fuse prompt video
features to the robot's current state and generate appropriate actions that
mimic the observed task. To further improve policy performance, we propose
auxiliary contrastive losses that enhance the alignment between human and robot
video representations. We evaluate Vid2Robot on real-world robots,
demonstrating a 20% improvement in performance compared to other
video-conditioned policies when using human demonstration videos. Additionally,
our model exhibits emergent capabilities, such as successfully transferring
observed motions from one object to another, and long-horizon composition, thus
showcasing its potential for real-world applications. Project website:
vid2robot.github.io
65fa61200174fa082183ec31
Vidhi Jain
claimed_verified
2024-04-02T11:04:39.681Z
6335dd2717cccb7421bb0def
https://cdn-avatars.huggingface.co/v1/production/uploads/6335dd2717cccb7421bb0def/yFjtuIjyyh6MY42pKLLjw.png
Vidhi Jain
user
vidhij
65fa61200174fa082183ec32
Maria Attarian
65fa61200174fa082183ec33
Nikhil J Joshi
65fa61200174fa082183ec34
Ayzaan Wahid
admin_assigned
2024-03-21T12:48:51.769Z
63922fc073bf5173e3631bb7
/avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
Ayzaan Wahid
user
ayzaan
65fa61200174fa082183ec35
Danny Driess
65fa61200174fa082183ec36
Quan Vuong
admin_assigned
2024-03-21T12:49:01.834Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65fa61200174fa082183ec37
Pannag R Sanketi
claimed_verified
2024-08-09T07:49:01.288Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
65fa61200174fa082183ec38
Pierre Sermanet
65fa61200174fa082183ec39
Stefan Welker
65fa61200174fa082183ec3a
Christine Chan
65fa61200174fa082183ec3b
Igor Gilitschenski
admin_assigned
2024-03-21T12:49:24.394Z
64893f0b9ab28735f1a1ec0b
https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
Igor Gilitschenski
user
igilitschenski
65fa61200174fa082183ec3c
Yonatan Bisk
admin_assigned
2024-03-21T12:49:31.842Z
63180f3c879073cf83f135b0
https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
Yonatan Bisk
user
ybisk
65fa61200174fa082183ec3d
Debidatta Dwibedi
Vid2Robot: End-to-end Video-conditioned Policy Learning with
  Cross-Attention Transformers
While large-scale robotic systems typically rely on textual instructions for
tasks, this work explores a different approach: can robots infer the task
directly from observing humans? This shift necessitates the robot's ability to
decode human intent and translate it into executable actions within its
physical constraints and environment. We introduce Vid2Robot, a novel
end-to-end video-based learning framework for robots. Given a video
demonstration of a manipulation task and current visual observations, Vid2Robot
directly produces robot actions. This is achieved through a unified
representation model trained on a large dataset of human video and robot
trajectory. The model leverages cross-attention mechanisms to fuse prompt video
features to the robot's current state and generate appropriate actions that
mimic the observed task. To further improve policy performance, we propose
auxiliary contrastive losses that enhance the alignment between human and robot
video representations. We evaluate Vid2Robot on real-world robots,
demonstrating a 20% improvement in performance compared to other
video-conditioned policies when using human demonstration videos. Additionally,
our model exhibits emergent capabilities, such as successfully transferring
observed motions from one object to another, and long-horizon composition, thus
showcasing its potential for real-world applications. Project website:
vid2robot.github.io
Vid2Robot is a framework that uses video input to infer and execute robotic manipulation tasks by leveraging cross-attention and contrastive losses for improved policy performance and emergent capabilities.
Vid2Robot cross-attention mechanisms contrastive losses end-to-end video-based learning