StreamingVLM: Real-Time Understanding for Infinite Video Streams
Vision-language models (VLMs) could power real-time assistants and autonomous
agents, but they face a critical challenge: understanding near-infinite video
streams without escalating latency and memory usage. Processing entire videos
with full attention leads to quadratic computational costs and poor performance
on long videos. Meanwhile, simple sliding window methods are also flawed, as
they either break coherence or suffer from high latency due to redundant
recomputation. In this paper, we introduce StreamingVLM, a model designed for
real-time, stable understanding of infinite visual input. Our approach is a
unified framework that aligns training with streaming inference. During
inference, we maintain a compact KV cache by reusing states of attention sinks,
a short window of recent vision tokens, and a long window of recent text
tokens. This streaming ability is instilled via a simple supervised fine-tuning
(SFT) strategy that applies full attention on short, overlapped video chunks,
which effectively mimics the inference-time attention pattern without training
on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a
new benchmark with videos averaging over two hours that requires dense,
per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM
achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time
performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy
also enhances general VQA abilities without any VQA-specific fine-tuning,
improving performance on LongVideoBench by +4.30 and OVOBench Realtime by
+5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.
68ec5910cd07fb414898c8e1
Ruyi Xu
68ec5910cd07fb414898c8e2
Guangxuan Xiao
68ec5910cd07fb414898c8e3
Yukang Chen
claimed_verified
2025-10-13T10:06:08.409Z
62919485a29097b211bc7b83
https://cdn-avatars.huggingface.co/v1/production/uploads/62919485a29097b211bc7b83/TX8iBGu5JSuFlrRvjPEBV.png
YukangChen
user
Yukang
68ec5910cd07fb414898c8e4
Liuning He
68ec5910cd07fb414898c8e5
Kelly Peng
68ec5910cd07fb414898c8e6
Yao Lu
68ec5910cd07fb414898c8e7
Song Han
StreamingVLM: Real-Time Understanding for Infinite Video Streams
Vision-language models (VLMs) could power real-time assistants and autonomous
agents, but they face a critical challenge: understanding near-infinite video
streams without escalating latency and memory usage. Processing entire videos
with full attention leads to quadratic computational costs and poor performance
on long videos. Meanwhile, simple sliding window methods are also flawed, as
they either break coherence or suffer from high latency due to redundant
recomputation. In this paper, we introduce StreamingVLM, a model designed for
real-time, stable understanding of infinite visual input. Our approach is a
unified framework that aligns training with streaming inference. During
inference, we maintain a compact KV cache by reusing states of attention sinks,
a short window of recent vision tokens, and a long window of recent text
tokens. This streaming ability is instilled via a simple supervised fine-tuning
(SFT) strategy that applies full attention on short, overlapped video chunks,
which effectively mimics the inference-time attention pattern without training
on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a
new benchmark with videos averaging over two hours that requires dense,
per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM
achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time
performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy
also enhances general VQA abilities without any VQA-specific fine-tuning,
improving performance on LongVideoBench by +4.30 and OVOBench Realtime by
+5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.
StreamingVLM is a real-time vision-language model that efficiently processes infinite video streams using a compact KV cache and supervised fine-tuning, achieving high performance on long videos and diverse benchmarks.
vision-language models VLMs real-time assistants autonomous agents video streams quadratic computational costs sliding window methods attention sinks vision tokens text tokens supervised fine-tuning SFT inference-time attention pattern Inf-Streams-Eval win rate NVIDIA H100 VQA abilities LongVideoBench OVOBench Realtime