Long-Context Autoregressive Video Modeling with Next-Frame Prediction
Long-context autoregressive modeling has significantly advanced language
generation, but video generation still struggles to fully utilize extended
temporal contexts. To investigate long-context video modeling, we introduce
Frame AutoRegressive (FAR), a strong baseline for video autoregressive
modeling. Just as language models learn causal dependencies between tokens
(i.e., Token AR), FAR models temporal causal dependencies between continuous
frames, achieving better convergence than Token AR and video diffusion
transformers. Building on FAR, we observe that long-context vision modeling
faces challenges due to visual redundancy. Existing RoPE lacks effective
temporal decay for remote context and fails to extrapolate well to long video
sequences. Additionally, training on long videos is computationally expensive,
as vision tokens grow much faster than language tokens. To tackle these issues,
we propose balancing locality and long-range dependency. We introduce FlexRoPE,
an test-time technique that adds flexible temporal decay to RoPE, enabling
extrapolation to 16x longer vision contexts. Furthermore, we propose long
short-term context modeling, where a high-resolution short-term context window
ensures fine-grained temporal consistency, while an unlimited long-term context
window encodes long-range information using fewer tokens. With this approach,
we can train on long video sequences with a manageable token context length. We
demonstrate that FAR achieves state-of-the-art performance in both short- and
long-video generation, providing a simple yet effective baseline for video
autoregressive modeling.
67e35f6fc9d8214b5e1c64c3
Yuchao Gu
67e35f6fc9d8214b5e1c64c4
Weijia Mao
67e35f6fc9d8214b5e1c64c5
Mike Zheng Shou
Long-Context Autoregressive Video Modeling with Next-Frame Prediction
Long-context autoregressive modeling has significantly advanced language
generation, but video generation still struggles to fully utilize extended
temporal contexts. To investigate long-context video modeling, we introduce
Frame AutoRegressive (FAR), a strong baseline for video autoregressive
modeling. Just as language models learn causal dependencies between tokens
(i.e., Token AR), FAR models temporal causal dependencies between continuous
frames, achieving better convergence than Token AR and video diffusion
transformers. Building on FAR, we observe that long-context vision modeling
faces challenges due to visual redundancy. Existing RoPE lacks effective
temporal decay for remote context and fails to extrapolate well to long video
sequences. Additionally, training on long videos is computationally expensive,
as vision tokens grow much faster than language tokens. To tackle these issues,
we propose balancing locality and long-range dependency. We introduce FlexRoPE,
an test-time technique that adds flexible temporal decay to RoPE, enabling
extrapolation to 16x longer vision contexts. Furthermore, we propose long
short-term context modeling, where a high-resolution short-term context window
ensures fine-grained temporal consistency, while an unlimited long-term context
window encodes long-range information using fewer tokens. With this approach,
we can train on long video sequences with a manageable token context length. We
demonstrate that FAR achieves state-of-the-art performance in both short- and
long-video generation, providing a simple yet effective baseline for video
autoregressive modeling.
FAR, an autoregressive video model, achieves state-of-the-art results by balancing locality and long-range context with FlexRoPE and long short-term context modeling, addressing challenges in long-context video generation.
Frame AutoRegressive (FAR) video autoregressive modeling causal dependencies Token AR video diffusion transformers long-context vision modeling visual redundancy RoPE temporal decay long video sequences FlexRoPE long short-term context modeling fine-grained temporal consistency long-range information short-video generation long-video generation