Apollo: An Exploration of Video Understanding in Large Multimodal Models
Despite the rapid integration of video perception capabilities into Large
Multimodal Models (LMMs), the underlying mechanisms driving their video
understanding remain poorly understood. Consequently, many design decisions in
this domain are made without proper justification or analysis. The high
computational cost of training and evaluating such models, coupled with limited
open research, hinders the development of video-LMMs. To address this, we
present a comprehensive study that helps uncover what effectively drives video
understanding in LMMs.
  We begin by critically examining the primary contributors to the high
computational requirements associated with video-LMM research and discover
Scaling Consistency, wherein design and training decisions made on smaller
models and datasets (up to a critical size) effectively transfer to larger
models. Leveraging these insights, we explored many video-specific aspects of
video-LMMs, including video sampling, architectures, data composition, training
schedules, and more. For example, we demonstrated that fps sampling during
training is vastly preferable to uniform frame sampling and which vision
encoders are the best for video representation.
  Guided by these findings, we introduce Apollo, a state-of-the-art family of
LMMs that achieve superior performance across different model sizes. Our models
can perceive hour-long videos efficiently, with Apollo-3B outperforming most
existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is
state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on
Video-MME.
675f9d6d2eb87c3a1b120a5f
Orr Zohar
extracted_confirmed
2025-04-08T07:42:49.334Z
648c9605565e3a44f3c9bb7b
https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
Orr Zohar
user
orrzohar
675f9d6d2eb87c3a1b120a60
Xiaohan Wang
claimed_verified
2024-12-16T09:23:12.868Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
675f9d6d2eb87c3a1b120a61
Yann Dubois
admin_assigned
2024-12-16T09:57:20.008Z
641362754e5305c14f233f08
/avatars/bce974ef60c507d22702cc7662033b28.svg
Yann Dubois
user
YannDubs
675f9d6d2eb87c3a1b120a62
Nikhil Mehta
675f9d6d2eb87c3a1b120a63
Tong Xiao
675f9d6d2eb87c3a1b120a64
Philippe Hansen-Estruch
claimed_verified
2024-12-17T08:04:15.986Z
644eb923a8f384abd038a4f7
/avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
Philippe I Hansen-Estruch
user
philippehansen
675f9d6d2eb87c3a1b120a65
Licheng Yu
admin_assigned
2024-12-16T09:58:30.800Z
6372b1f9bd81fae2b3a712e8
/avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
Licheng Yu
user
lichengyu
675f9d6d2eb87c3a1b120a66
Xiaofang Wang
admin_assigned
2024-12-16T09:59:05.541Z
6334c1b3762d2d0e9eb151ea
/avatars/45b3eb63f0515ab417aacd79aa9f8a41.svg
Xiaofang Wang
user
minione
675f9d6d2eb87c3a1b120a67
Felix Juefei-Xu
claimed_verified
2024-12-16T16:55:18.689Z
6417cf37dce1e4c0229f17b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6417cf37dce1e4c0229f17b1/7h-ZCB5f4wif7TsnF-B1M.jpeg
Felix Xu
user
katanaxu
675f9d6d2eb87c3a1b120a68
Ning Zhang
675f9d6d2eb87c3a1b120a69
Serena Yeung-Levy
675f9d6d2eb87c3a1b120a6a
Xide Xia
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Despite the rapid integration of video perception capabilities into Large
Multimodal Models (LMMs), the underlying mechanisms driving their video
understanding remain poorly understood. Consequently, many design decisions in
this domain are made without proper justification or analysis. The high
computational cost of training and evaluating such models, coupled with limited
open research, hinders the development of video-LMMs. To address this, we
present a comprehensive study that helps uncover what effectively drives video
understanding in LMMs.
  We begin by critically examining the primary contributors to the high
computational requirements associated with video-LMM research and discover
Scaling Consistency, wherein design and training decisions made on smaller
models and datasets (up to a critical size) effectively transfer to larger
models. Leveraging these insights, we explored many video-specific aspects of
video-LMMs, including video sampling, architectures, data composition, training
schedules, and more. For example, we demonstrated that fps sampling during
training is vastly preferable to uniform frame sampling and which vision
encoders are the best for video representation.
  Guided by these findings, we introduce Apollo, a state-of-the-art family of
LMMs that achieve superior performance across different model sizes. Our models
can perceive hour-long videos efficiently, with Apollo-3B outperforming most
existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is
state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on
Video-MME.
A comprehensive study reveals effective design principles for video understanding in Large Multimodal Models, introducing Apollo, which achieves superior performance across different model sizes.
LMMs video understanding computational requirements Scaling Consistency video sampling architectures data composition training schedules fps sampling vision encoders Apollo LongVideoBench MLVU Video-MME