TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming
  Videos
The rapid growth of online video platforms, particularly live streaming
services, has created an urgent need for real-time video understanding systems.
These systems must process continuous video streams and respond to user queries
instantaneously, presenting unique challenges for current Video Large Language
Models (VideoLLMs). While existing VideoLLMs excel at processing complete
videos, they face significant limitations in streaming scenarios due to their
inability to handle dense, redundant frames efficiently. We introduce
TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video
interaction. At its core lies our innovative Differential Token Drop (DTD)
module, which addresses the fundamental challenge of visual redundancy in
streaming videos. Drawing inspiration from human visual perception's Change
Blindness phenomenon, DTD preserves meaningful temporal changes while filtering
out static, redundant content between frames. Remarkably, our experiments
demonstrate that DTD achieves an 82.8% reduction in video tokens while
maintaining 98% performance on StreamingBench, revealing that over 80% of
visual content in streaming videos is naturally redundant without requiring
language guidance. To enable seamless real-time interaction, we present
TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse
interaction patterns including backward-tracing, current-perception, and
future-responding scenarios. TimeChat-Online's unique Proactive Response
capability, naturally achieved through continuous monitoring of video scene
transitions via DTD, sets it apart from conventional approaches. Our extensive
evaluation demonstrates TimeChat-Online's superior performance on streaming
benchmarks (StreamingBench and OvOBench) and maintaining competitive results on
long-form video tasks such as Video-MME and MLVU.
680b24471c5fbd15909bf1f9
Linli Yao
680b24471c5fbd15909bf1fa
Yicheng Li
680b24471c5fbd15909bf1fb
Yuancheng Wei
680b24471c5fbd15909bf1fc
Lei Li
claimed_verified
2025-05-13T08:13:14.592Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
680b24471c5fbd15909bf1fd
Shuhuai Ren
680b24471c5fbd15909bf1fe
Yuanxin Liu
680b24471c5fbd15909bf1ff
Kun Ouyang
claimed_verified
2025-04-30T07:56:51.047Z
62cd7aca7a036fc9941bb2b0
/avatars/17a4d27af0243fd7dccf06066f671461.svg
kun ouyang
user
RUBBISHLIKE
680b24471c5fbd15909bf200
Lean Wang
680b24471c5fbd15909bf201
Shicheng Li
claimed_verified
2025-06-05T12:42:22.568Z
64539588fc2b5f69e8faac76
/avatars/111441eeb0dd4d8ad2f0d3f28277952a.svg
Li Shicheng
user
lscpku
680b24471c5fbd15909bf202
Sida Li
680b24471c5fbd15909bf203
Lingpeng Kong
680b24471c5fbd15909bf204
Qi Liu
680b24471c5fbd15909bf205
Yuanxing Zhang
680b24471c5fbd15909bf206
Xu Sun
TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming
  Videos
The rapid growth of online video platforms, particularly live streaming
services, has created an urgent need for real-time video understanding systems.
These systems must process continuous video streams and respond to user queries
instantaneously, presenting unique challenges for current Video Large Language
Models (VideoLLMs). While existing VideoLLMs excel at processing complete
videos, they face significant limitations in streaming scenarios due to their
inability to handle dense, redundant frames efficiently. We introduce
TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video
interaction. At its core lies our innovative Differential Token Drop (DTD)
module, which addresses the fundamental challenge of visual redundancy in
streaming videos. Drawing inspiration from human visual perception's Change
Blindness phenomenon, DTD preserves meaningful temporal changes while filtering
out static, redundant content between frames. Remarkably, our experiments
demonstrate that DTD achieves an 82.8% reduction in video tokens while
maintaining 98% performance on StreamingBench, revealing that over 80% of
visual content in streaming videos is naturally redundant without requiring
language guidance. To enable seamless real-time interaction, we present
TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse
interaction patterns including backward-tracing, current-perception, and
future-responding scenarios. TimeChat-Online's unique Proactive Response
capability, naturally achieved through continuous monitoring of video scene
transitions via DTD, sets it apart from conventional approaches. Our extensive
evaluation demonstrates TimeChat-Online's superior performance on streaming
benchmarks (StreamingBench and OvOBench) and maintaining competitive results on
long-form video tasks such as Video-MME and MLVU.
TimeChat-Online, an online VideoLLM with a Differential Token Drop module, efficiently handles real-time video streaming by reducing redundant frames, achieving superior performance compared to existing models.
Video Large Language Models Differential Token Drop Change Blindness phenomenon StreamingBench OvOBench Video-MME MLVU Proactive Response