VideoNSA: Native Sparse Attention Scales Video Understanding
Video understanding in multimodal language models remains limited by context
length: models often miss key transition frames and struggle to maintain
coherence across long time scales. To address this, we adapt Native Sparse
Attention (NSA) to video-language models. Our method, VideoNSA, adapts
Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We
employ a hardware-aware hybrid approach to attention, preserving dense
attention for text, while employing NSA for video. Compared to
token-compression and training-free sparse baselines, VideoNSA achieves
improved performance on long-video understanding, temporal reasoning, and
spatial benchmarks. Further ablation analysis reveals four key findings: (1)
reliable scaling to 128K tokens; (2) an optimal global-local attention
allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4)
the learnable combined sparse attention help induce dynamic attention sinks.
68df252adf49fb0df1e03b9e
Enxin Song
68df252adf49fb0df1e03b9f
Wenhao Chai
68df252adf49fb0df1e03ba0
Shusheng Yang
68df252adf49fb0df1e03ba1
Ethan Armand
68df252adf49fb0df1e03ba2
Xiaojun Shan
68df252adf49fb0df1e03ba3
Haiyang Xu
68df252adf49fb0df1e03ba4
Jianwen Xie
68df252adf49fb0df1e03ba5
Zhuowen Tu
VideoNSA: Native Sparse Attention Scales Video Understanding
Video understanding in multimodal language models remains limited by context
length: models often miss key transition frames and struggle to maintain
coherence across long time scales. To address this, we adapt Native Sparse
Attention (NSA) to video-language models. Our method, VideoNSA, adapts
Qwen2.5-VL through end-to-end training on a 216K video instruction dataset. We
employ a hardware-aware hybrid approach to attention, preserving dense
attention for text, while employing NSA for video. Compared to
token-compression and training-free sparse baselines, VideoNSA achieves
improved performance on long-video understanding, temporal reasoning, and
spatial benchmarks. Further ablation analysis reveals four key findings: (1)
reliable scaling to 128K tokens; (2) an optimal global-local attention
allocation at a fixed budget; (3) task-dependent branch usage patterns; and (4)
the learnable combined sparse attention help induce dynamic attention sinks.
VideoNSA, an adaptation of Native Sparse Attention to video-language models, enhances long-video understanding and temporal reasoning through end-to-end training and a hardware-aware hybrid attention approach.
Native Sparse Attention NSA VideoNSA Qwen2.5-VL token-compression sparse baselines long-video understanding temporal reasoning spatial benchmarks global-local attention dynamic attention sinks