iVideoGPT: Interactive VideoGPTs are Scalable World Models
World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.
6653f2d561422fc897228332
Jialong Wu
admin_assigned
2024-05-27T09:11:24.622Z
643b866bff50448bcfc7d1d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q12v-BatVKimoi8q-coi-.jpeg
Jialong Wu
user
manchery
6653f2d561422fc897228333
Shaofeng Yin
claimed_verified
2024-05-28T10:01:15.462Z
64e01150607e093241a208e6
/avatars/fd24699fc000778696092383ac3d9e99.svg
Shaofeng Yin
user
operator22
6653f2d561422fc897228334
Ningya Feng
admin_assigned
2024-05-27T09:11:45.223Z
65c25931c36307b08566c746
/avatars/363aa6ed7c55fa77e72f6e6bc1391afb.svg
fengningya
user
fengningya
6653f2d561422fc897228335
Xu He
6653f2d561422fc897228336
Dong Li
6653f2d561422fc897228337
Jianye Hao
6653f2d561422fc897228338
Mingsheng Long
iVideoGPT: Interactive VideoGPTs are Scalable World Models
World models empower model-based agents to interactively explore, reason, and
plan within imagined environments for real-world decision-making. However, the
high demand for interactivity poses challenges in harnessing recent
advancements in video generative models for developing world models at scale.
This work introduces Interactive VideoGPT (iVideoGPT), a scalable
autoregressive transformer framework that integrates multimodal signals--visual
observations, actions, and rewards--into a sequence of tokens, facilitating an
interactive experience of agents via next-token prediction. iVideoGPT features
a novel compressive tokenization technique that efficiently discretizes
high-dimensional visual observations. Leveraging its scalable architecture, we
are able to pre-train iVideoGPT on millions of human and robotic manipulation
trajectories, establishing a versatile foundation that is adaptable to serve as
interactive world models for a wide range of downstream tasks. These include
action-conditioned video prediction, visual planning, and model-based
reinforcement learning, where iVideoGPT achieves competitive performance
compared with state-of-the-art methods. Our work advances the development of
interactive general world models, bridging the gap between generative video
models and practical model-based reinforcement learning applications.
World models model-based agents imagined environments real-world decision-making video generative models scalable autoregressive transformer multimodal signals visual observations actions rewards sequence of tokens next-token prediction compressive tokenization high-dimensional visual observations pre-train human and robotic manipulation trajectories action-conditioned video prediction visual planning model-based reinforcement learning state-of-the-art methods interactive general world models