Pre-training Auto-regressive Robotic Models with 4D Representations
Foundation models pre-trained on massive unlabeled datasets have
revolutionized natural language and computer vision, exhibiting remarkable
generalization capabilities, thus highlighting the importance of pre-training.
Yet, efforts in robotics have struggled to achieve similar success, limited by
either the need for costly robotic annotations or the lack of representations
that effectively model the physical world. In this paper, we introduce ARM4R,
an Auto-regressive Robotic Model that leverages low-level 4D Representations
learned from human video data to yield a better pre-trained robotic model.
Specifically, we focus on utilizing 3D point tracking representations from
videos derived by lifting 2D representations into 3D space via monocular depth
estimation across time. These 4D representations maintain a shared geometric
structure between the points and robot state representations up to a linear
transformation, enabling efficient transfer learning from human video data to
low-level robotic control. Our experiments show that ARM4R can transfer
efficiently from human video data to robotics and consistently improves
performance on tasks across various robot environments and configurations.
67b5790132be608036ee94e5
Dantong Niu
admin_assigned
2025-02-19T09:12:28.457Z
65c3fdf79d062be813813e45
/avatars/52528a61abe5bbbef4a4a431944973cd.svg
Dantong Niu
user
NdtSoCool
67b5790132be608036ee94e6
Yuvan Sharma
admin_assigned
2025-02-19T09:12:35.531Z
65406e82deee4716f1c29271
/avatars/25331a773f8125f9ad1c3d6ac3375586.svg
Yuvan Sharma
user
yuvansharma
67b5790132be608036ee94e7
Haoru Xue
67b5790132be608036ee94e8
Giscard Biamby
admin_assigned
2025-02-19T09:12:49.219Z
650bd36a7c99ca283e58e973
https://cdn-avatars.huggingface.co/v1/production/uploads/650bd36a7c99ca283e58e973/9DTIEwjjOVY73soQCrTe4.jpeg
Giscard Biamby
user
gbiamby
67b5790132be608036ee94e9
Junyi Zhang
claimed_verified
2025-02-20T09:37:02.956Z
62f0ecd2700bdc19558360de
/avatars/5325b4b763f30c41f30e3aec0d2b59fa.svg
Junyi Zhang
user
Junyi42
67b5790132be608036ee94ea
Ziteng Ji
admin_assigned
2025-02-19T09:13:13.907Z
66a09aec369dd38cf2113070
/avatars/cc13bdd3dc1271d33b083b61e12f1a05.svg
Ziteng Ji
user
zitengj0618
67b5790132be608036ee94eb
Trevor Darrell
admin_assigned
2025-02-19T09:13:20.379Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
67b5790132be608036ee94ec
Roei Herzig
admin_assigned
2025-02-19T09:13:26.134Z
667c5764186b27ef806636d3
/avatars/5c08f0109bc0e350624112c0aff544f6.svg
Roei Herzig
user
roeiherz
Pre-training Auto-regressive Robotic Models with 4D Representations
Foundation models pre-trained on massive unlabeled datasets have
revolutionized natural language and computer vision, exhibiting remarkable
generalization capabilities, thus highlighting the importance of pre-training.
Yet, efforts in robotics have struggled to achieve similar success, limited by
either the need for costly robotic annotations or the lack of representations
that effectively model the physical world. In this paper, we introduce ARM4R,
an Auto-regressive Robotic Model that leverages low-level 4D Representations
learned from human video data to yield a better pre-trained robotic model.
Specifically, we focus on utilizing 3D point tracking representations from
videos derived by lifting 2D representations into 3D space via monocular depth
estimation across time. These 4D representations maintain a shared geometric
structure between the points and robot state representations up to a linear
transformation, enabling efficient transfer learning from human video data to
low-level robotic control. Our experiments show that ARM4R can transfer
efficiently from human video data to robotics and consistently improves
performance on tasks across various robot environments and configurations.
ARM4R, an Auto-regressive Robotic Model, uses 4D representations from human video data to enhance pre-trained robotic models and improve performance across different robotic tasks.
Auto-regressive Robotic Model 4D Representations 3D point tracking monocular depth estimation transfer learning robot state representations