Mavors: Multi-granularity Video Representation for Multimodal Large
  Language Model
Long-context video understanding in multimodal large language models (MLLMs)
faces a critical challenge: balancing computational efficiency with the
retention of fine-grained spatio-temporal patterns. Existing approaches (e.g.,
sparse sampling, dense sampling with low resolution, and token compression)
suffer from significant information loss in temporal dynamics, spatial details,
or subtle interactions, particularly in videos with complex motion or varying
resolutions. To address this, we propose Mavors, a novel framework
that introduces Multi-granularity
video representation for holistic
long-video modeling. Specifically, Mavors directly encodes raw video content
into latent representations through two core components: 1) an Intra-chunk
Vision Encoder (IVE) that preserves high-resolution spatial features via 3D
convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator
(IFA) that establishes temporal coherence across chunks using transformer-based
dependency modeling with chunk-level rotary position encodings. Moreover, the
framework unifies image and video understanding by treating images as
single-frame videos via sub-image decomposition. Experiments across diverse
benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity
and temporal continuity, significantly outperforming existing methods in tasks
requiring fine-grained spatio-temporal reasoning.
67fdd1d7634e600357b5b7ff
Yang Shi
claimed_verified
2025-04-15T07:54:11.086Z
673c7319d11b1c2e246ead9c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
Yang Shi
user
DogNeverSleep
67fdd1d7634e600357b5b800
Jiaheng Liu
admin_assigned
2025-04-15T08:18:07.388Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
67fdd1d7634e600357b5b801
Yushuo Guan
admin_assigned
2025-04-15T08:18:14.282Z
66ac46766c3f950f4f10b9f9
/avatars/027b573bc6e5b18107e762645cec6069.svg
Yushuo Guan
user
UnnamedWatcher
67fdd1d7634e600357b5b802
Zhenhua Wu
admin_assigned
2025-04-15T08:18:32.198Z
646f7b60799a974be3191889
/avatars/8fff4c87a2ea2d12958424074dd8e93d.svg
AaronWuuuuu
user
zhenhuawu
67fdd1d7634e600357b5b803
Yuanxing Zhang
67fdd1d7634e600357b5b804
Zihao Wang
67fdd1d7634e600357b5b805
Weihong Lin
67fdd1d7634e600357b5b806
Jingyun Hua
claimed_verified
2025-07-03T07:42:13.602Z
61540338e5b9ae6774201e58
https://cdn-avatars.huggingface.co/v1/production/uploads/61540338e5b9ae6774201e58/h_159VrXOlIgu0N0pNgXj.png
jingyun
user
hjy
67fdd1d7634e600357b5b807
Zekun Wang
admin_assigned
2025-04-15T08:22:34.448Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
67fdd1d7634e600357b5b808
Xinlong Chen
67fdd1d7634e600357b5b809
Bohan Zeng
claimed_verified
2025-04-15T07:54:08.962Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
user
zbhpku
67fdd1d7634e600357b5b80a
Wentao Zhang
67fdd1d7634e600357b5b80b
Fuzheng Zhang
admin_assigned
2025-04-15T08:21:57.922Z
67c5945da1661d5fa6f29adb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lr9VmRJF0ZFECmNx5hk3x.png
Fuzheng Zhang
user
Edrex
67fdd1d7634e600357b5b80c
Wenjing Yang
67fdd1d7634e600357b5b80d
Di Zhang
admin_assigned
2025-04-15T08:19:13.378Z
644c8324f02250233d0d67d9
/avatars/feb39d281457c1750f3eada3c060a23e.svg
Di Zhang
user
dizhang
Mavors: Multi-granularity Video Representation for Multimodal Large
  Language Model
Long-context video understanding in multimodal large language models (MLLMs)
faces a critical challenge: balancing computational efficiency with the
retention of fine-grained spatio-temporal patterns. Existing approaches (e.g.,
sparse sampling, dense sampling with low resolution, and token compression)
suffer from significant information loss in temporal dynamics, spatial details,
or subtle interactions, particularly in videos with complex motion or varying
resolutions. To address this, we propose Mavors, a novel framework
that introduces Multi-granularity
video representation for holistic
long-video modeling. Specifically, Mavors directly encodes raw video content
into latent representations through two core components: 1) an Intra-chunk
Vision Encoder (IVE) that preserves high-resolution spatial features via 3D
convolutions and Vision Transformers, and 2) an Inter-chunk Feature Aggregator
(IFA) that establishes temporal coherence across chunks using transformer-based
dependency modeling with chunk-level rotary position encodings. Moreover, the
framework unifies image and video understanding by treating images as
single-frame videos via sub-image decomposition. Experiments across diverse
benchmarks demonstrate Mavors' superiority in maintaining both spatial fidelity
and temporal continuity, significantly outperforming existing methods in tasks
requiring fine-grained spatio-temporal reasoning.
Mavors, a novel framework for long-context video understanding, uses a multi-granularity video representation to preserve spatial fidelity and temporal continuity through 3D convolutions, Vision Transformers, and transformer-based dependency modeling.
Intra-chunk Vision Encoder Inter-chunk Feature Aggregator 3D convolutions Vision Transformers transformer-based dependency modeling chunk-level rotary position encodings