Video Language Planning
We are interested in enabling visual planning for complex long-horizon tasks
in the space of generated videos and language, leveraging recent advances in
large generative models pretrained on Internet-scale data. To this end, we
present video language planning (VLP), an algorithm that consists of a tree
search procedure, where we train (i) vision-language models to serve as both
policies and value functions, and (ii) text-to-video models as dynamics models.
VLP takes as input a long-horizon task instruction and current image
observation, and outputs a long video plan that provides detailed multimodal
(video and language) specifications that describe how to complete the final
task. VLP scales with increasing computation budget where more computation time
results in improved video plans, and is able to synthesize long-horizon video
plans across different robotics domains: from multi-object rearrangement, to
multi-camera bi-arm dexterous manipulation. Generated video plans can be
translated into real robot actions via goal-conditioned policies, conditioned
on each intermediate frame of the generated video. Experiments show that VLP
substantially improves long-horizon task success rates compared to prior
methods on both simulated and real robots (across 3 hardware platforms).
652df61f54c1441c7edb303d
Yilun Du
admin_assigned
2023-10-17T08:23:28.094Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
652df61f54c1441c7edb303e
Mengjiao Yang
652df61f54c1441c7edb303f
Pete Florence
652df61f54c1441c7edb3040
Fei Xia
claimed_verified
2023-12-15T08:04:20.071Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
652df61f54c1441c7edb3041
Ayzaan Wahid
admin_assigned
2023-10-17T08:23:51.752Z
63922fc073bf5173e3631bb7
/avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
Ayzaan Wahid
user
ayzaan
652df61f54c1441c7edb3042
Brian Ichter
admin_assigned
2023-10-17T08:23:59.211Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
652df61f54c1441c7edb3043
Pierre Sermanet
652df61f54c1441c7edb3044
Tianhe Yu
admin_assigned
2023-10-17T08:24:20.154Z
64c95527d33e6a7f82b8b911
/avatars/5efcf5d5a84766bbd3a5ec2354233f1c.svg
Tianhe Yu
user
keviny66
652df61f54c1441c7edb3045
Pieter Abbeel
652df61f54c1441c7edb3046
Joshua B. Tenenbaum
652df61f54c1441c7edb3047
Leslie Kaelbling
admin_assigned
2023-10-17T08:24:36.617Z
649780c6af212acd36eff177
/avatars/473acf7502774d9ef9b9d1091d211d59.svg
Leslie Kaelbling
user
lpklpk
652df61f54c1441c7edb3048
Andy Zeng
admin_assigned
2023-10-17T08:25:00.446Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
652df61f54c1441c7edb3049
Jonathan Tompson
Video Language Planning
We are interested in enabling visual planning for complex long-horizon tasks
in the space of generated videos and language, leveraging recent advances in
large generative models pretrained on Internet-scale data. To this end, we
present video language planning (VLP), an algorithm that consists of a tree
search procedure, where we train (i) vision-language models to serve as both
policies and value functions, and (ii) text-to-video models as dynamics models.
VLP takes as input a long-horizon task instruction and current image
observation, and outputs a long video plan that provides detailed multimodal
(video and language) specifications that describe how to complete the final
task. VLP scales with increasing computation budget where more computation time
results in improved video plans, and is able to synthesize long-horizon video
plans across different robotics domains: from multi-object rearrangement, to
multi-camera bi-arm dexterous manipulation. Generated video plans can be
translated into real robot actions via goal-conditioned policies, conditioned
on each intermediate frame of the generated video. Experiments show that VLP
substantially improves long-horizon task success rates compared to prior
methods on both simulated and real robots (across 3 hardware platforms).
Video language planning leverages vision-language and text-to-video models to generate detailed long-horizon video plans, improving task success rates across multiple robotics domains.
video language planning tree search procedure vision-language models value functions text-to-video models dynamics models goal-conditioned policies long-horizon tasks multi-object rearrangement multi-camera bi-arm dexterous manipulation robot actions