An LMM for Efficient Video Understanding via Reinforced Compression of
  Video Cubes
Large Multimodal Models (LMMs) uniformly perceive video frames, creating
computational inefficiency for videos with inherently varying temporal
information density. This paper present Quicksviewer, an LMM with new
perceiving paradigm that partitions a video of nonuniform density into varying
cubes using Gumbel Softmax, followed by a unified resampling for each cube to
achieve efficient video understanding. This simple and intuitive approach
dynamically compress video online based on its temporal density, significantly
reducing spatiotemporal redundancy (overall 45times compression rate), while
enabling efficient training with large receptive field. We train the model from
a language backbone through three progressive stages, each incorporating
lengthy videos on average of 420s/1fps thanks to the perceiving efficiency.
With only 0.8M total video-text samples for training, our model outperforms the
direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in
accuracy, demonstrating the effectiveness in performance. On Video-MME,
Quicksviewer achieves SOTA under modest sequence lengths using just up to 5\%
of tokens per frame required by baselines. With this paradigm, scaling up the
number of input frames reveals a clear power law of the model capabilities. It
is also empirically verified that the segments generated by the cubing network
can help for analyzing continuous events in videos.
68074ea84a1c690663814e56
Ji Qi
claimed_verified
2025-07-02T08:24:34.633Z
6864b9f90a269bace8c92164
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zSeWHZRCZtzZ13CzFrr5u.png
Ji Qi
user
miracle11121
68074ea84a1c690663814e57
Yuan Yao
68074ea84a1c690663814e58
Yushi Bai
admin_assigned
2025-04-22T11:05:19.003Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
68074ea84a1c690663814e59
Bin Xu
68074ea84a1c690663814e5a
Juanzi Li
68074ea84a1c690663814e5b
Zhiyuan Liu
admin_assigned
2025-04-22T11:06:02.057Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
68074ea84a1c690663814e5c
Tat-Seng Chua
admin_assigned
2025-04-22T11:05:41.211Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
An LMM for Efficient Video Understanding via Reinforced Compression of
  Video Cubes
Large Multimodal Models (LMMs) uniformly perceive video frames, creating
computational inefficiency for videos with inherently varying temporal
information density. This paper present Quicksviewer, an LMM with new
perceiving paradigm that partitions a video of nonuniform density into varying
cubes using Gumbel Softmax, followed by a unified resampling for each cube to
achieve efficient video understanding. This simple and intuitive approach
dynamically compress video online based on its temporal density, significantly
reducing spatiotemporal redundancy (overall 45times compression rate), while
enabling efficient training with large receptive field. We train the model from
a language backbone through three progressive stages, each incorporating
lengthy videos on average of 420s/1fps thanks to the perceiving efficiency.
With only 0.8M total video-text samples for training, our model outperforms the
direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in
accuracy, demonstrating the effectiveness in performance. On Video-MME,
Quicksviewer achieves SOTA under modest sequence lengths using just up to 5\%
of tokens per frame required by baselines. With this paradigm, scaling up the
number of input frames reveals a clear power law of the model capabilities. It
is also empirically verified that the segments generated by the cubing network
can help for analyzing continuous events in videos.
Quicksviewer, a Large Multimodal Model using Gumbel Softmax for dynamic video partitioning, achieves efficient video understanding and outperforms fixed partitioning by reducing temporal redundancy.
Large Multimodal Models (LMMs) video frames computational efficiency temporal information density Gumbel Softmax spatiotemporal redundancy receptive field language backbone progressive stages video-MME token efficiency cubing network continuous event analysis