Fast Video Generation with Sliding Tile Attention
Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art
video generation, but suffer from prohibitive compute cost -- when generating
just a 5-second 720P video, attention alone takes 800 out of 945 seconds of
total inference time. This paper introduces sliding tile attention (STA) to
address this challenge. STA leverages the observation that attention scores in
pretrained video diffusion models predominantly concentrate within localized 3D
windows. By sliding and attending over the local spatial-temporal region, STA
eliminates redundancy from full attention. Unlike traditional token-wise
sliding window attention (SWA), STA operates tile-by-tile with a novel
hardware-aware sliding window design, preserving expressiveness while being
hardware-efficient. With careful kernel-level optimizations, STA offers the
first efficient 2D/3D sliding-window-like attention implementation, achieving
58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over
FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading
video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s
without quality degradation, requiring no training. Enabling finetuning further
lowers latency to 268s with only a 0.09% drop on VBench.
67a98cd1b8b21202c9004628
Peiyuan Zhang
admin_assigned
2025-02-10T16:07:27.309Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
67a98cd1b8b21202c9004629
Yongqi Chen
claimed_verified
2025-02-10T09:49:48.410Z
65416817271d3bc4d70f6745
https://cdn-avatars.huggingface.co/v1/production/uploads/65416817271d3bc4d70f6745/1YkW0MpuufejvxqksVMIx.jpeg
Yongqi Chen
user
BrianChen1129
67a98cd1b8b21202c900462a
Runlong Su
claimed_verified
2025-02-11T10:00:40.942Z
65d7ed4823e83e1591beacc7
/avatars/2a6714a2a7bbd591f6b726a7330bafbc.svg
Su
user
r3su9
67a98cd1b8b21202c900462b
Hangliang Ding
claimed_verified
2025-02-11T07:56:04.110Z
643a451ee2b979ae6141329d
https://cdn-avatars.huggingface.co/v1/production/uploads/643a451ee2b979ae6141329d/HN3M5vyroanQoUEiXJFyB.jpeg
Hangliang Ding
user
foreverpiano
67a98cd1b8b21202c900462c
Ion Stoica
67a98cd1b8b21202c900462d
Zhenghong Liu
67a98cd1b8b21202c900462e
Hao Zhang
claimed_verified
2025-02-11T10:00:38.019Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
Fast Video Generation with Sliding Tile Attention
Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art
video generation, but suffer from prohibitive compute cost -- when generating
just a 5-second 720P video, attention alone takes 800 out of 945 seconds of
total inference time. This paper introduces sliding tile attention (STA) to
address this challenge. STA leverages the observation that attention scores in
pretrained video diffusion models predominantly concentrate within localized 3D
windows. By sliding and attending over the local spatial-temporal region, STA
eliminates redundancy from full attention. Unlike traditional token-wise
sliding window attention (SWA), STA operates tile-by-tile with a novel
hardware-aware sliding window design, preserving expressiveness while being
hardware-efficient. With careful kernel-level optimizations, STA offers the
first efficient 2D/3D sliding-window-like attention implementation, achieving
58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over
FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading
video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s
without quality degradation, requiring no training. Enabling finetuning further
lowers latency to 268s with only a 0.09% drop on VBench.
Sliding tile attention (STA) accelerates 3D attention in Diffusion Transformers (DiTs), reducing video generation latency without quality loss.
diffusion transformers DiTs 3D full attention sliding tile attention STA attention scores hardware-aware sliding window 2D/3D sliding-window-like attention FlashAttention-2 FlashAttention-3 HunyuanVideo VBench