MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Long video generation with Diffusion Transformers (DiTs) is bottlenecked by
the quadratic scaling of full attention with sequence length. Since attention
is highly redundant, outputs are dominated by a small subset of query-key
pairs. Existing sparse methods rely on blockwise coarse estimation, whose
accuracy-efficiency trade-offs are constrained by block size. This paper
introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention
that uses a lightweight, learnable token router to precisely match tokens
without blockwise estimation. Through semantic-aware routing, MoGA enables
effective long-range interactions. As a kernel-free method, MoGA integrates
seamlessly with modern attention stacks, including FlashAttention and sequence
parallelism. Building on MoGA, we develop an efficient long video generation
model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps,
with a context length of approximately 580k. Comprehensive experiments on
various video generation tasks validate the effectiveness of our approach.
68f839b57669bcaeecce0cb3
Weinan Jia
68f839b57669bcaeecce0cb4
Yuning Lu
claimed_verified
2025-10-22T15:44:04.222Z
64a247040b7851be3f228ee5
/avatars/9d6da9a66380e5dafd55a18d6aec89fc.svg
Yuning Lu
user
dalynlynlyn
68f839b57669bcaeecce0cb5
Mengqi Huang
68f839b57669bcaeecce0cb6
Hualiang Wang
68f839b57669bcaeecce0cb7
Binyuan Huang
68f839b57669bcaeecce0cb8
Nan Chen
claimed_verified
2025-10-22T15:44:00.299Z
6629d7c9fa14eaccf07d8633
/avatars/dceb2f6c804c583adf15a3536c8c995b.svg
Nan Chen
user
CNcreator0331
68f839b57669bcaeecce0cb9
Mu Liu
68f839b57669bcaeecce0cba
Jidong Jiang
68f839b57669bcaeecce0cbb
Zhendong Mao
MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Long video generation with Diffusion Transformers (DiTs) is bottlenecked by
the quadratic scaling of full attention with sequence length. Since attention
is highly redundant, outputs are dominated by a small subset of query-key
pairs. Existing sparse methods rely on blockwise coarse estimation, whose
accuracy-efficiency trade-offs are constrained by block size. This paper
introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention
that uses a lightweight, learnable token router to precisely match tokens
without blockwise estimation. Through semantic-aware routing, MoGA enables
effective long-range interactions. As a kernel-free method, MoGA integrates
seamlessly with modern attention stacks, including FlashAttention and sequence
parallelism. Building on MoGA, we develop an efficient long video generation
model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps,
with a context length of approximately 580k. Comprehensive experiments on
various video generation tasks validate the effectiveness of our approach.
Mixture-of-Groups Attention (MoGA) enables efficient long video generation by addressing the quadratic scaling issue of full attention in Diffusion Transformers.
Diffusion Transformers DiTs full attention sequence length sparse attention token router semantic-aware routing kernel-free method FlashAttention sequence parallelism long video generation minute-level multi-shot 480p 24 fps context length