Radial Attention: O(nlog n) Sparse Attention with Energy Decay for
  Long Video Generation
Recent advances in diffusion models have enabled high-quality video
generation, but the additional temporal dimension significantly increases
computational costs, making training and inference on long videos prohibitively
expensive. In this paper, we identify a phenomenon we term Spatiotemporal
Energy Decay in video diffusion models: post-softmax attention scores diminish
as spatial and temporal distance between tokens increase, akin to the physical
decay of signal or waves over space and time in nature. Motivated by this, we
propose Radial Attention, a scalable sparse attention mechanism with O(n log
n) complexity that translates energy decay into exponentially decaying compute
density, which is significantly more efficient than standard O(n^2) dense
attention and more expressive than linear attention. Specifically, Radial
Attention employs a simple, static attention mask where each token attends to
spatially nearby tokens, with the attention window size shrinking with temporal
distance. Moreover, it allows pre-trained video diffusion models to extend
their generation length with efficient LoRA-based fine-tuning. Extensive
experiments show that Radial Attention maintains video quality across
Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9times speedup
over the original dense attention. With minimal tuning, it enables video
generation up to 4times longer while reducing training costs by up to
4.4times compared to direct fine-tuning and accelerating inference by up to
3.7times compared to dense attention inference.
685d790d61ef876fd500e925
Xingyang Li
685d790d61ef876fd500e926
Muyang Li
claimed_verified
2025-06-27T08:57:19.218Z
63129589bbaa385279d1826e
https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
Muyang Li
user
Lmxyy
685d790d61ef876fd500e927
Tianle Cai
685d790d61ef876fd500e928
Haocheng Xi
claimed_verified
2025-07-03T07:33:10.955Z
66ce751a8ec9fda2cf5a9e85
/avatars/c17093ca81dad007b3e50bae503955a7.svg
Haocheng Xi
user
xihc-ucb
685d790d61ef876fd500e929
Shuo Yang
685d790d61ef876fd500e92a
Yujun Lin
685d790d61ef876fd500e92b
Lvmin Zhang
685d790d61ef876fd500e92c
Songlin Yang
685d790d61ef876fd500e92d
Jinbo Hu
685d790d61ef876fd500e92e
Kelly Peng
685d790d61ef876fd500e92f
Maneesh Agrawala
685d790d61ef876fd500e930
Ion Stoica
685d790d61ef876fd500e931
Kurt Keutzer
685d790d61ef876fd500e932
Song Han
Radial Attention: O(nlog n) Sparse Attention with Energy Decay for
  Long Video Generation
Recent advances in diffusion models have enabled high-quality video
generation, but the additional temporal dimension significantly increases
computational costs, making training and inference on long videos prohibitively
expensive. In this paper, we identify a phenomenon we term Spatiotemporal
Energy Decay in video diffusion models: post-softmax attention scores diminish
as spatial and temporal distance between tokens increase, akin to the physical
decay of signal or waves over space and time in nature. Motivated by this, we
propose Radial Attention, a scalable sparse attention mechanism with O(n log
n) complexity that translates energy decay into exponentially decaying compute
density, which is significantly more efficient than standard O(n^2) dense
attention and more expressive than linear attention. Specifically, Radial
Attention employs a simple, static attention mask where each token attends to
spatially nearby tokens, with the attention window size shrinking with temporal
distance. Moreover, it allows pre-trained video diffusion models to extend
their generation length with efficient LoRA-based fine-tuning. Extensive
experiments show that Radial Attention maintains video quality across
Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9times speedup
over the original dense attention. With minimal tuning, it enables video
generation up to 4times longer while reducing training costs by up to
4.4times compared to direct fine-tuning and accelerating inference by up to
3.7times compared to dense attention inference.
Radial Attention, a scalable sparse attention mechanism, improves efficiency and preserves video quality in diffusion models by leveraging spatiotemporal energy decay.
Spatiotemporal Energy Decay diffusion models radial attention attention scores dense attention linear attention attention mask LoRA-based fine-tuning video quality Wan2.1-14B HunyuanVideo Mochi 1