Loong: Generating Minute-level Long Videos with Autoregressive Language
  Models
It is desirable but challenging to generate content-rich long videos in the
scale of minutes. Autoregressive large language models (LLMs) have achieved
great success in generating coherent and long sequences of tokens in the domain
of natural language processing, while the exploration of autoregressive LLMs
for video generation is limited to generating short videos of several seconds.
In this work, we conduct a deep analysis of the challenges that prevent
autoregressive LLM-based video generators from generating long videos. Based on
the observations and analysis, we propose Loong, a new autoregressive LLM-based
video generator that can generate minute-long videos. Specifically, we model
the text tokens and video tokens as a unified sequence for autoregressive LLMs
and train the model from scratch. We propose progressive short-to-long training
with a loss re-weighting scheme to mitigate the loss imbalance problem for long
video training. We further investigate inference strategies, including video
token re-encoding and sampling strategies, to diminish error accumulation
during inference. Our proposed Loong can be trained on 10-second videos and be
extended to generate minute-level long videos conditioned on text prompts, as
demonstrated by the results. More samples are available at:
https://epiphqny.github.io/Loong-video.
66ff590f1687036580ad3899
Yuqing Wang
claimed_verified
2024-10-04T07:30:54.287Z
63ea23b9dedfeebe54d02bdf
/avatars/4d9f9a546aa8c63e277161ea700075c4.svg
Yuqing Wang
user
Epiphqny
66ff590f1687036580ad389a
Tianwei Xiong
claimed_verified
2024-10-16T09:44:32.472Z
668125557b50b433cda2a211
https://cdn-avatars.huggingface.co/v1/production/uploads/668125557b50b433cda2a211/j3z3wT5Rv9IyUKtbzQpnc.png
Tianwei Xiong
user
YuuTennYi
66ff590f1687036580ad389b
Daquan Zhou
66ff590f1687036580ad389c
Zhijie Lin
admin_assigned
2024-10-04T08:35:48.411Z
64415957bd0c9726529802f6
/avatars/1132d1ee68fb58ec635d57c8175caacd.svg
Zhijie Lin
user
Ikuinen
66ff590f1687036580ad389d
Yang Zhao
66ff590f1687036580ad389e
Bingyi Kang
admin_assigned
2024-10-04T08:36:06.799Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
66ff590f1687036580ad389f
Jiashi Feng
66ff590f1687036580ad38a0
Xihui Liu
admin_assigned
2024-10-04T08:26:25.925Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
Loong: Generating Minute-level Long Videos with Autoregressive Language
  Models
It is desirable but challenging to generate content-rich long videos in the
scale of minutes. Autoregressive large language models (LLMs) have achieved
great success in generating coherent and long sequences of tokens in the domain
of natural language processing, while the exploration of autoregressive LLMs
for video generation is limited to generating short videos of several seconds.
In this work, we conduct a deep analysis of the challenges that prevent
autoregressive LLM-based video generators from generating long videos. Based on
the observations and analysis, we propose Loong, a new autoregressive LLM-based
video generator that can generate minute-long videos. Specifically, we model
the text tokens and video tokens as a unified sequence for autoregressive LLMs
and train the model from scratch. We propose progressive short-to-long training
with a loss re-weighting scheme to mitigate the loss imbalance problem for long
video training. We further investigate inference strategies, including video
token re-encoding and sampling strategies, to diminish error accumulation
during inference. Our proposed Loong can be trained on 10-second videos and be
extended to generate minute-level long videos conditioned on text prompts, as
demonstrated by the results. More samples are available at:
https://epiphqny.github.io/Loong-video.
A novel autoregressive large language model (LLM)-based video generator, Loong, is proposed to produce minute-long videos by addressing challenges related to long video generation.
autoregressive LLMs video generation text tokens video tokens progressive short-to-long training loss re-weighting scheme inference strategies video token re-encoding sampling strategies