Video as the New Language for Real-World Decision Making
Both text and video data are abundant on the internet and support large-scale
self-supervised learning through next token or frame prediction. However, they
have not been equally leveraged: language models have had significant
real-world impact, whereas video generation has remained largely limited to
media entertainment. Yet video data captures important information about the
physical world that is difficult to express in language. To address this gap,
we discuss an under-appreciated opportunity to extend video generation to solve
tasks in the real world. We observe how, akin to language, video can serve as a
unified interface that can absorb internet knowledge and represent diverse
tasks. Moreover, we demonstrate how, like language models, video generation can
serve as planners, agents, compute engines, and environment simulators through
techniques such as in-context learning, planning and reinforcement learning. We
identify major impact opportunities in domains such as robotics, self-driving,
and science, supported by recent work that demonstrates how such advanced
capabilities in video generation are plausibly within reach. Lastly, we
identify key challenges in video generation that mitigate progress. Addressing
these challenges will enable video generation models to demonstrate unique
value alongside language models in a wider array of AI applications.
65dea45f6efc2a4ebb07c3b5
Sherry Yang
admin_assigned
2024-02-28T12:26:49.890Z
636a05908d250800ca6de12b
/avatars/549a764e6cbdf2a410329dea0b726d23.svg
Sherry Yang
user
sherryy
65dea45f6efc2a4ebb07c3b6
Jacob Walker
65dea45f6efc2a4ebb07c3b7
Jack Parker-Holder
65dea45f6efc2a4ebb07c3b8
Yilun Du
admin_assigned
2024-02-28T12:27:41.132Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
65dea45f6efc2a4ebb07c3b9
Jake Bruce
65dea45f6efc2a4ebb07c3ba
Andre Barreto
65dea45f6efc2a4ebb07c3bb
Pieter Abbeel
65dea45f6efc2a4ebb07c3bc
Dale Schuurmans
Video as the New Language for Real-World Decision Making
Both text and video data are abundant on the internet and support large-scale
self-supervised learning through next token or frame prediction. However, they
have not been equally leveraged: language models have had significant
real-world impact, whereas video generation has remained largely limited to
media entertainment. Yet video data captures important information about the
physical world that is difficult to express in language. To address this gap,
we discuss an under-appreciated opportunity to extend video generation to solve
tasks in the real world. We observe how, akin to language, video can serve as a
unified interface that can absorb internet knowledge and represent diverse
tasks. Moreover, we demonstrate how, like language models, video generation can
serve as planners, agents, compute engines, and environment simulators through
techniques such as in-context learning, planning and reinforcement learning. We
identify major impact opportunities in domains such as robotics, self-driving,
and science, supported by recent work that demonstrates how such advanced
capabilities in video generation are plausibly within reach. Lastly, we
identify key challenges in video generation that mitigate progress. Addressing
these challenges will enable video generation models to demonstrate unique
value alongside language models in a wider array of AI applications.
Video generation can serve as a unified interface for diverse tasks and advance real-world AI applications through techniques like in-context learning and reinforcement learning, with opportunities in robotics, self-driving, and science.
in-context learning planning reinforcement learning