VideoLLaMB: Long-context Video Understanding with Recurrent Memory
  Bridges
Recent advancements in large-scale video-language models have shown
significant potential for real-time planning and detailed interactions.
However, their high computational demands and the scarcity of annotated
datasets limit their practicality for academic researchers. In this work, we
introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens
within bridge layers to allow for the encoding of entire video sequences
alongside historical visual data, effectively preserving semantic continuity
and enhancing model performance across various tasks. This approach includes
recurrent memory tokens and a SceneTilling algorithm, which segments videos
into independent semantic units to preserve semantic integrity. Empirically,
VideoLLaMB significantly outstrips existing video-language models,
demonstrating a 5.5 points improvement over its competitors across three
VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive
results on the MVBench show that VideoLLaMB-7B achieves markedly better results
than previous 7B models of same LLM. Remarkably, it maintains robust
performance as PLLaVA even as video length increases up to 8 times. Besides,
the frame retrieval results on our specialized Needle in a Video Haystack
(NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately
identifying specific frames within lengthy videos. Our SceneTilling algorithm
also enables the generation of streaming video captions directly, without
necessitating additional training. In terms of efficiency, VideoLLaMB, trained
on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear
GPU memory scaling, ensuring both high performance and cost-effectiveness,
thereby setting a new foundation for long-form video-language models in both
academic and practical applications.
66d7d91fa2ddade9371e0b28
Yuxuan Wang
claimed_verified
2024-09-04T07:25:24.968Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
66d7d91fa2ddade9371e0b29
Cihang Xie
admin_assigned
2024-09-04T08:23:42.394Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
66d7d91fa2ddade9371e0b2a
Yang Liu
66d7d91fa2ddade9371e0b2b
Zilong Zheng
admin_assigned
2024-09-04T07:14:48.519Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
VideoLLaMB: Long-context Video Understanding with Recurrent Memory
  Bridges
Recent advancements in large-scale video-language models have shown
significant potential for real-time planning and detailed interactions.
However, their high computational demands and the scarcity of annotated
datasets limit their practicality for academic researchers. In this work, we
introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens
within bridge layers to allow for the encoding of entire video sequences
alongside historical visual data, effectively preserving semantic continuity
and enhancing model performance across various tasks. This approach includes
recurrent memory tokens and a SceneTilling algorithm, which segments videos
into independent semantic units to preserve semantic integrity. Empirically,
VideoLLaMB significantly outstrips existing video-language models,
demonstrating a 5.5 points improvement over its competitors across three
VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive
results on the MVBench show that VideoLLaMB-7B achieves markedly better results
than previous 7B models of same LLM. Remarkably, it maintains robust
performance as PLLaVA even as video length increases up to 8 times. Besides,
the frame retrieval results on our specialized Needle in a Video Haystack
(NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately
identifying specific frames within lengthy videos. Our SceneTilling algorithm
also enables the generation of streaming video captions directly, without
necessitating additional training. In terms of efficiency, VideoLLaMB, trained
on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear
GPU memory scaling, ensuring both high performance and cost-effectiveness,
thereby setting a new foundation for long-form video-language models in both
academic and practical applications.
VideoLLaMB enhances video-language model performance through temporal memory tokens and SceneTilling algorithm, enabling accurate frame retrieval and efficient video caption generation without additional training.
VideoLLaMB temporal memory tokens bridge layers SceneTilling algorithm VideoQA benchmarks egocentric planning MVBench VideoLLaMB-7B PLLaVA Needle in a Video Haystack (NIAVH) benchmark GPU memory scaling