Mixture of Contexts for Long Video Generation
Long video generation is fundamentally a long context memory problem: models
must retain and retrieve salient events across a long range without collapsing
or drifting. However, scaling diffusion transformers to generate long-context
videos is fundamentally limited by the quadratic cost of self-attention, which
makes memory and computation intractable and difficult to optimize for long
sequences. We recast long-context video generation as an internal information
retrieval task and propose a simple, learnable sparse attention routing module,
Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.
In MoC, each query dynamically selects a few informative chunks plus mandatory
anchors (caption, local windows) to attend to, with causal routing that
prevents loop closures. As we scale the data and gradually sparsify the
routing, the model allocates compute to salient history, preserving identities,
actions, and scenes over minutes of content. Efficiency follows as a byproduct
of retrieval (near-linear scaling), which enables practical training and
synthesis, and the emergence of memory and consistency at the scale of minutes.
68b10915b19c5400014848fb
Shengqu Cai
68b10915b19c5400014848fc
Ceyuan Yang
68b10915b19c5400014848fd
Lvmin Zhang
68b10915b19c5400014848fe
Yuwei Guo
68b10915b19c5400014848ff
Junfei Xiao
68b10915b19c540001484900
Ziyan Yang
68b10915b19c540001484901
Yinghao Xu
68b10915b19c540001484902
Zhenheng Yang
68b10915b19c540001484903
Alan Yuille
68b10915b19c540001484904
Leonidas Guibas
68b10915b19c540001484905
Maneesh Agrawala
68b10915b19c540001484906
Lu Jiang
68b10915b19c540001484907
Gordon Wetzstein
Mixture of Contexts for Long Video Generation
Long video generation is fundamentally a long context memory problem: models
must retain and retrieve salient events across a long range without collapsing
or drifting. However, scaling diffusion transformers to generate long-context
videos is fundamentally limited by the quadratic cost of self-attention, which
makes memory and computation intractable and difficult to optimize for long
sequences. We recast long-context video generation as an internal information
retrieval task and propose a simple, learnable sparse attention routing module,
Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.
In MoC, each query dynamically selects a few informative chunks plus mandatory
anchors (caption, local windows) to attend to, with causal routing that
prevents loop closures. As we scale the data and gradually sparsify the
routing, the model allocates compute to salient history, preserving identities,
actions, and scenes over minutes of content. Efficiency follows as a byproduct
of retrieval (near-linear scaling), which enables practical training and
synthesis, and the emergence of memory and consistency at the scale of minutes.
Long video generation is addressed by introducing a sparse attention routing module, Mixture of Contexts, to efficiently manage long-term memory and retrieval in diffusion transformers.
diffusion transformers self-attention long-context video generation internal information retrieval Mixture of Contexts sparse attention routing causal routing salient history memory consistency