LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Long-context capability is critical for multi-modal foundation models. We
introduce LongVILA, a full-stack solution for long-context vision-language
models, including system, model training, and dataset development. On the
system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
system that enables long-context training and inference, enabling 2M context
length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
text-only settings. Moreover, it seamlessly integrates with Hugging Face
Transformers. For model training, we propose a five-stage pipeline comprising
alignment, pre-training, context extension, and long-short joint supervised
fine-tuning. Regarding datasets, we meticulously construct large-scale visual
language pre-training datasets and long video instruction-following datasets to
support our multi-stage training process. The full-stack solution extends the
feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
accuracy in 1400-frames video (274k context length) needle in a haystack.
LongVILA-8B also demonstrates a consistent improvement in performance on long
videos within the VideoMME benchmark as the video frames increase.
66c41182ea476bea05da1a4c
Fuzhao Xue
admin_assigned
2024-08-30T16:25:05.182Z
64a52314978242bb77e05790
/avatars/e97206c653de3446266c80c2943c0a6f.svg
Fuzhao Xue
user
fuzhao
66c41182ea476bea05da1a4d
Yukang Chen
66c41182ea476bea05da1a4e
Dacheng Li
66c41182ea476bea05da1a4f
Qinghao Hu
claimed_verified
2024-09-09T06:57:25.574Z
62fb8412610dae1bcd036db7
https://cdn-avatars.huggingface.co/v1/production/uploads/62fb8412610dae1bcd036db7/LIaEoDrvrq8hDl8509ZuF.jpeg
Qinghao Hu
user
Qinghao
66c41182ea476bea05da1a50
Ligeng Zhu
claimed_verified
2024-12-30T19:39:35.010Z
62b4b5beb25cb80fcf278354
https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
Ligeng Zhu
user
Ligeng-Zhu
66c41182ea476bea05da1a51
Xiuyu Li
66c41182ea476bea05da1a52
Yunhao Fang
66c41182ea476bea05da1a53
Haotian Tang
66c41182ea476bea05da1a54
Shang Yang
claimed_verified
2025-02-21T10:00:46.154Z
641d8bacd526196afc12766d
/avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
Shang Yang
user
Shangy
66c41182ea476bea05da1a55
Zhijian Liu
claimed_verified
2025-05-30T06:59:36.699Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
66c41182ea476bea05da1a56
Ethan He
claimed_verified
2024-10-17T11:28:50.068Z
62b20f6df4a72794189248fc
/avatars/87e1125868616d4f7d6ee1e5ec4499b4.svg
Ethan He
user
ethanhe
66c41182ea476bea05da1a57
Hongxu Yin
claimed_verified
2025-03-27T09:06:10.536Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
66c41182ea476bea05da1a58
Pavlo Molchanov
claimed_verified
2024-09-30T08:07:48.737Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
66c41182ea476bea05da1a59
Jan Kautz
66c41182ea476bea05da1a5a
Linxi Fan
66c41182ea476bea05da1a5b
Yuke Zhu
66c41182ea476bea05da1a5c
Yao Lu
66c41182ea476bea05da1a5d
Song Han
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Long-context capability is critical for multi-modal foundation models. We
introduce LongVILA, a full-stack solution for long-context vision-language
models, including system, model training, and dataset development. On the
system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
system that enables long-context training and inference, enabling 2M context
length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
text-only settings. Moreover, it seamlessly integrates with Hugging Face
Transformers. For model training, we propose a five-stage pipeline comprising
alignment, pre-training, context extension, and long-short joint supervised
fine-tuning. Regarding datasets, we meticulously construct large-scale visual
language pre-training datasets and long video instruction-following datasets to
support our multi-stage training process. The full-stack solution extends the
feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
accuracy in 1400-frames video (274k context length) needle in a haystack.
LongVILA-8B also demonstrates a consistent improvement in performance on long
videos within the VideoMME benchmark as the video frames increase.
LongVILA, a full-stack solution for long-context vision-language models, introduces Multi-Modal Sequence Parallelism for efficient training and inference, and a five-stage training pipeline that enhances long video processing and captioning performance.
Multi-Modal Sequence Parallelism MM-SP long-context vision-language models alignment pre-training context extension long-short joint supervised fine-tuning VideoMME benchmark