TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
  Generalist Robotic Policies
Although large vision-language-action (VLA) models pretrained on extensive
robot datasets offer promising generalist policies for robotic learning, they
still struggle with spatial-temporal dynamics in interactive robotics, making
them less effective in handling complex tasks, such as manipulation. In this
work, we introduce visual trace prompting, a simple yet effective approach to
facilitate VLA models' spatial-temporal awareness for action prediction by
encoding state-action trajectories visually. We develop a new TraceVLA model by
finetuning OpenVLA on our own collected dataset of 150K robot manipulation
trajectories using visual trace prompting. Evaluations of TraceVLA across 137
configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
3.5x on real-robot tasks and exhibiting robust generalization across diverse
embodiments and scenarios. To further validate the effectiveness and generality
of our method, we present a compact VLA model based on 4B Phi-3-Vision,
pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
OpenVLA baseline while significantly improving inference efficiency.
6760512bda45b7d6f71baef6
Ruijie Zheng
admin_assigned
2024-12-18T13:41:57.819Z
653b24ef8f8d60f204872f0a
/avatars/45a55219e8a78be53fd32e96ba460282.svg
Ruijie Zheng
user
rzheng12
6760512bda45b7d6f71baef7
Yongyuan Liang
admin_assigned
2024-12-18T13:41:50.019Z
6646d5819bb34d2b6b7455d3
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JFH3ZTPvlaVSg4RJJBb6L.jpeg
Yongyuan Liang
user
cheryyunl
6760512bda45b7d6f71baef8
Shuaiyi Huang
admin_assigned
2024-12-18T13:41:42.477Z
65cad78525a3b8e44dadaf62
/avatars/76466c1a26195a4a1d77189b4758bf7c.svg
Shuaiyi Huang
user
ShirleyHuang
6760512bda45b7d6f71baef9
Jianfeng Gao
admin_assigned
2024-12-18T13:41:35.144Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
6760512bda45b7d6f71baefa
Hal Daumé III
admin_assigned
2024-12-18T13:41:26.931Z
65e0d1a52512b0269dc01b34
/avatars/3068a8af7dc0b2d05625c282ba7a5ffb.svg
Hal Daumé III
user
haldaume3
6760512bda45b7d6f71baefb
Andrey Kolobov
admin_assigned
2024-12-18T13:41:19.350Z
6350478dc5acdbefb8ecf722
/avatars/75f3cdc38f6e67ae0435c30aff50b089.svg
Andrey Kolobov
user
akolobov
6760512bda45b7d6f71baefc
Furong Huang
6760512bda45b7d6f71baefd
Jianwei Yang
admin_assigned
2024-12-18T13:40:59.234Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for
  Generalist Robotic Policies
Although large vision-language-action (VLA) models pretrained on extensive
robot datasets offer promising generalist policies for robotic learning, they
still struggle with spatial-temporal dynamics in interactive robotics, making
them less effective in handling complex tasks, such as manipulation. In this
work, we introduce visual trace prompting, a simple yet effective approach to
facilitate VLA models' spatial-temporal awareness for action prediction by
encoding state-action trajectories visually. We develop a new TraceVLA model by
finetuning OpenVLA on our own collected dataset of 150K robot manipulation
trajectories using visual trace prompting. Evaluations of TraceVLA across 137
configurations in SimplerEnv and 4 tasks on a physical WidowX robot demonstrate
state-of-the-art performance, outperforming OpenVLA by 10% on SimplerEnv and
3.5x on real-robot tasks and exhibiting robust generalization across diverse
embodiments and scenarios. To further validate the effectiveness and generality
of our method, we present a compact VLA model based on 4B Phi-3-Vision,
pretrained on the Open-X-Embodiment and finetuned on our dataset, rivals the 7B
OpenVLA baseline while significantly improving inference efficiency.
Visual trace prompting enhances visual-language-action models' spatial-temporal awareness, leading to state-of-the-art performance in robotic manipulation tasks through fine-tuning on a large trajectory dataset.
vision-language-action models spatial-temporal dynamics interactive robotics action prediction visual trace prompting TraceVLA OpenVLA robot manipulation trajectories state-of-the-art performance simulation real-robot tasks robust generalization compact VLA model Phi-3-Vision Open-X-Embodiment inference efficiency