MMInference: Accelerating Pre-filling for Long-Context VLMs via
  Modality-Aware Permutation Sparse Attention
The integration of long-context capabilities with visual understanding
unlocks unprecedented potential for Vision Language Models (VLMs). However, the
quadratic attention complexity during the pre-filling phase remains a
significant obstacle to real-world deployment. To overcome this limitation, we
introduce MMInference (Multimodality Million tokens Inference), a dynamic
sparse attention method that accelerates the prefilling stage for long-context
multi-modal inputs. First, our analysis reveals that the temporal and spatial
locality of video input leads to a unique sparse pattern, the Grid pattern.
Simultaneously, VLMs exhibit markedly different sparse distributions across
different modalities. We introduce a permutation-based method to leverage the
unique Grid pattern and handle modality boundary issues. By offline search the
optimal sparse patterns for each head, MMInference constructs the sparse
distribution dynamically based on the input. We also provide optimized GPU
kernels for efficient sparse computations. Notably, MMInference integrates
seamlessly into existing VLM pipelines without any model modifications or
fine-tuning. Experiments on multi-modal benchmarks-including Video QA,
Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art
long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that
MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while
maintaining accuracy. Our code is available at https://aka.ms/MMInference.
68105d8632d635f02bc2976e
Yucheng Li
admin_assigned
2025-04-29T12:26:01.688Z
655ea46bb11e49dd1fd6ce2e
/avatars/671771dc565cf718f677536f91f1e8a3.svg
Yuchengli
user
Yuchengli
68105d8632d635f02bc2976f
Huiqiang Jiang
extracted_confirmed
2025-05-08T06:07:20.459Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
68105d8632d635f02bc29770
Chengruidong Zhang
admin_assigned
2025-04-29T12:26:11.194Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
68105d8632d635f02bc29771
Qianhui Wu
admin_assigned
2025-04-29T12:26:17.854Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
68105d8632d635f02bc29772
Xufang Luo
admin_assigned
2025-04-29T12:26:38.807Z
64b750a2fdb702b3d8619514
/avatars/f09181c0825763dff692c4bc65effc4c.svg
Xufang Luo
user
luoxufang
68105d8632d635f02bc29773
Surin Ahn
68105d8632d635f02bc29774
Amir H. Abdi
68105d8632d635f02bc29775
Dongsheng Li
admin_assigned
2025-04-29T12:27:31.251Z
6524f13821af4b07309015aa
/avatars/f97d6a041428d48305568df7afd83cc8.svg
dongsheng li
user
dongshengli
68105d8632d635f02bc29776
Jianfeng Gao
admin_assigned
2025-04-29T12:27:37.335Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
68105d8632d635f02bc29777
Yuqing Yang
68105d8632d635f02bc29778
Lili Qiu
MMInference: Accelerating Pre-filling for Long-Context VLMs via
  Modality-Aware Permutation Sparse Attention
The integration of long-context capabilities with visual understanding
unlocks unprecedented potential for Vision Language Models (VLMs). However, the
quadratic attention complexity during the pre-filling phase remains a
significant obstacle to real-world deployment. To overcome this limitation, we
introduce MMInference (Multimodality Million tokens Inference), a dynamic
sparse attention method that accelerates the prefilling stage for long-context
multi-modal inputs. First, our analysis reveals that the temporal and spatial
locality of video input leads to a unique sparse pattern, the Grid pattern.
Simultaneously, VLMs exhibit markedly different sparse distributions across
different modalities. We introduce a permutation-based method to leverage the
unique Grid pattern and handle modality boundary issues. By offline search the
optimal sparse patterns for each head, MMInference constructs the sparse
distribution dynamically based on the input. We also provide optimized GPU
kernels for efficient sparse computations. Notably, MMInference integrates
seamlessly into existing VLM pipelines without any model modifications or
fine-tuning. Experiments on multi-modal benchmarks-including Video QA,
Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art
long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that
MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while
maintaining accuracy. Our code is available at https://aka.ms/MMInference.
MMInference, a dynamic sparse attention method, accelerates the pre-filling stage of Vision Language Models by leveraging unique sparse patterns in video input and modality boundary handling, achieving up to 8.3x speedup at 1M tokens.
long-context capabilities visual understanding Vision Language Models VLMs quadratic attention complexity dynamic sparse attention permuted-based method Grid pattern sparse distribution GPU kernels Video QA Captioning VisionNIAH Mixed-Modality NIAH LongVila LlavaVideo VideoChat-Flash Qwen2.5-VL