Flash-VStream: Memory-Based Real-Time Understanding for Long Video
  Streams
Benefiting from the advancements in large language models and cross-modal
alignment, existing multi-modal video understanding methods have achieved
prominent performance in offline scenario. However, online video streams, as
one of the most common media forms in the real world, have seldom received
attention. Compared to offline videos, the 'dynamic' nature of online video
streams poses challenges for the direct application of existing models and
introduces new problems, such as the storage of extremely long-term
information, interaction between continuous visual content and 'asynchronous'
user questions. Therefore, in this paper we present Flash-VStream, a
video-language model that simulates the memory mechanism of human. Our model is
able to process extremely long video streams in real-time and respond to user
queries simultaneously. Compared to existing models, Flash-VStream achieves
significant reductions in inference latency and VRAM consumption, which is
intimately related to performing understanding of online streaming video. In
addition, given that existing video understanding benchmarks predominantly
concentrate on offline scenario, we propose VStream-QA, a novel question
answering benchmark specifically designed for online video streaming
understanding. Comparisons with popular existing methods on the proposed
benchmark demonstrate the superiority of our method for such challenging
setting. To verify the generalizability of our approach, we further evaluate it
on existing video understanding benchmarks and achieves state-of-the-art
performance in offline scenarios as well. All code, models, and datasets are
available at the https://invinciblewyq.github.io/vstream-page/
666aa4f277d119acc29d4ba0
Haoji Zhang
admin_assigned
2024-07-08T21:14:08.479Z
65253da6a895f70381f35444
/avatars/e5cc8645e949d071887e89af37d1e3a3.svg
Haoji Zhang
user
zhang9302002
666aa4f277d119acc29d4ba1
Yiqin Wang
admin_assigned
2024-07-08T21:14:15.995Z
63e065ae42591dda0b9b74cc
https://cdn-avatars.huggingface.co/v1/production/uploads/63e065ae42591dda0b9b74cc/U1zhG8saFcEo8bNewxugP.jpeg
Yiqin Wang
user
Yiqin
666aa4f277d119acc29d4ba2
Yansong Tang
666aa4f277d119acc29d4ba3
Yong Liu
666aa4f277d119acc29d4ba4
Jiashi Feng
666aa4f277d119acc29d4ba5
Jifeng Dai
admin_assigned
2024-07-08T21:14:44.589Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
666aa4f277d119acc29d4ba6
Xiaojie Jin
Flash-VStream: Memory-Based Real-Time Understanding for Long Video
  Streams
Benefiting from the advancements in large language models and cross-modal
alignment, existing multi-modal video understanding methods have achieved
prominent performance in offline scenario. However, online video streams, as
one of the most common media forms in the real world, have seldom received
attention. Compared to offline videos, the 'dynamic' nature of online video
streams poses challenges for the direct application of existing models and
introduces new problems, such as the storage of extremely long-term
information, interaction between continuous visual content and 'asynchronous'
user questions. Therefore, in this paper we present Flash-VStream, a
video-language model that simulates the memory mechanism of human. Our model is
able to process extremely long video streams in real-time and respond to user
queries simultaneously. Compared to existing models, Flash-VStream achieves
significant reductions in inference latency and VRAM consumption, which is
intimately related to performing understanding of online streaming video. In
addition, given that existing video understanding benchmarks predominantly
concentrate on offline scenario, we propose VStream-QA, a novel question
answering benchmark specifically designed for online video streaming
understanding. Comparisons with popular existing methods on the proposed
benchmark demonstrate the superiority of our method for such challenging
setting. To verify the generalizability of our approach, we further evaluate it
on existing video understanding benchmarks and achieves state-of-the-art
performance in offline scenarios as well. All code, models, and datasets are
available at the https://invinciblewyq.github.io/vstream-page/
Flash-VStream is a video-language model that efficiently processes online video streams in real-time and responds to user queries, achieving superior performance compared to existing methods on both online and offline video understanding benchmarks.
large language models cross-modal alignment online video streams dynamic nature storage asynchronous user questions Flash-VStream memory mechanism real-time processing inference latency VRAM consumption understanding of online streaming video VStream-QA question answering benchmark offline scenario video understanding benchmarks state-of-the-art performance