Visual Context Window Extension: A New Perspective for Long Video
  Understanding
Large Multimodal Models (LMMs) have demonstrated impressive performance in
short video understanding tasks but face great challenges when applied to long
video understanding. In contrast, Large Language Models (LLMs) exhibit
outstanding capabilities in modeling long texts. Existing work attempts to
address this issue by introducing long video-text pairs during training.
However, these approaches require substantial computational and data resources.
In this paper, we tackle the challenge of long video understanding from the
perspective of context windows, aiming to apply LMMs to long video tasks
without retraining on long video datasets. We first conduct an in-depth
analysis of why pretrained LMMs struggle to understand lengthy video content,
identifying that discrepancies between visual and language modalities lead to
different context windows for visual and language tokens, making it difficult
to directly extend the visual tokens to match the language context window.
Based on this, we propose to adapt LMMs for long video understanding tasks by
extending the visual context window, eliminating the need for retraining on
large scalelong video datasets. To further mitigate the significant memory
consumption caused by long sequences, we introduce a progressive pooling
inference strategy that selectively adjusts the spatial resolution of frame
embeddings, reducing the number of visual tokens while retaining important
spatial information. Across multiple long video understanding benchmarks, our
method consistently improves the performance as the number of video frames
increases. On the MLVU benchmark, our method outperforms GPT-4o, even though
our model size is only 7B. Additionally, in the 256-frame setting, our method
reduces memory usage by approximately 45% compared to the baseline, without
introducing any performance loss.
66fd0c5c1a1f05e11daf0883
Hongchen Wei
claimed_verified
2024-10-02T09:13:08.738Z
63f96e99ade090bc87bc2f81
/avatars/0dd0807e5b2cec011e97c8d6a3c61bae.svg
hcwei
user
hcwei
66fd0c5c1a1f05e11daf0884
Zhenzhong Chen
Visual Context Window Extension: A New Perspective for Long Video
  Understanding
Large Multimodal Models (LMMs) have demonstrated impressive performance in
short video understanding tasks but face great challenges when applied to long
video understanding. In contrast, Large Language Models (LLMs) exhibit
outstanding capabilities in modeling long texts. Existing work attempts to
address this issue by introducing long video-text pairs during training.
However, these approaches require substantial computational and data resources.
In this paper, we tackle the challenge of long video understanding from the
perspective of context windows, aiming to apply LMMs to long video tasks
without retraining on long video datasets. We first conduct an in-depth
analysis of why pretrained LMMs struggle to understand lengthy video content,
identifying that discrepancies between visual and language modalities lead to
different context windows for visual and language tokens, making it difficult
to directly extend the visual tokens to match the language context window.
Based on this, we propose to adapt LMMs for long video understanding tasks by
extending the visual context window, eliminating the need for retraining on
large scalelong video datasets. To further mitigate the significant memory
consumption caused by long sequences, we introduce a progressive pooling
inference strategy that selectively adjusts the spatial resolution of frame
embeddings, reducing the number of visual tokens while retaining important
spatial information. Across multiple long video understanding benchmarks, our
method consistently improves the performance as the number of video frames
increases. On the MLVU benchmark, our method outperforms GPT-4o, even though
our model size is only 7B. Additionally, in the 256-frame setting, our method
reduces memory usage by approximately 45% compared to the baseline, without
introducing any performance loss.
A method is proposed to extend Large Multimodal Models for long video understanding by increasing the visual context window and implementing a progressive pooling inference strategy, improving performance and reducing memory usage compared to previous models.
Large Multimodal Models LMMs long video understanding Large Language Models LLMs context windows visual tokens language tokens visual context window progressive pooling MLVU benchmark