Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form
  Video Understanding with Multi-Axis Gradient Checkpointing
With the growing scale and complexity of video data, efficiently processing
long video sequences poses significant challenges due to the quadratic increase
in memory and computational demands associated with existing transformer-based
Large Multi-modal Models (LMMs). To address these issues, we introduce
Video-Ma^2mba, a novel architecture that incorporates State Space Models
(SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This
allows the LMMs to scale linearly in terms of time and memory requirements,
making it feasible to handle long-duration video content. Furthermore, we
enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing
(MA-GC) method, which strategically manages memory by retaining only essential
activations across multiple computational axes. Our approach significantly
reduces the memory footprint compared to standard gradient checkpointing.
Empirical analyses show that Video-Ma^2mba can process extensive video
sequences-equivalent to millions of tokens or over two hours of continuous
sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of
temporal dynamics, our model improves the accuracy and relevance of responses
in long video understanding tasks, demonstrating substantial advantages over
existing frameworks.
674d309e6a8ca5d12c0dac71
Hosu Lee
claimed_verified
2024-12-02T08:49:36.175Z
673060959e631f353ae1b5e0
/avatars/d4b1e23de90ff1d02c38186a259b8d1e.svg
Hosu Lee
user
lakelee
674d309e6a8ca5d12c0dac72
Junho Kim
claimed_verified
2024-12-02T09:46:14.671Z
653238bed0f5a9e537ed966d
/avatars/19de93c8170837d8a8a61473f96617ea.svg
Junho Kim
user
arkimjh
674d309e6a8ca5d12c0dac73
Hyunjun Kim
claimed_verified
2024-12-03T11:16:15.390Z
666419f8d8e1e34bc62166aa
/avatars/9c419003788682800f0dced2f5b52f46.svg
Hyunjun Kim
user
kimhj709
674d309e6a8ca5d12c0dac74
Yong Man Ro
admin_assigned
2024-12-02T09:13:04.450Z
660603529e3555d648e3baf1
/avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
Yong Man Ro
user
dwightro
Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form
  Video Understanding with Multi-Axis Gradient Checkpointing
With the growing scale and complexity of video data, efficiently processing
long video sequences poses significant challenges due to the quadratic increase
in memory and computational demands associated with existing transformer-based
Large Multi-modal Models (LMMs). To address these issues, we introduce
Video-Ma^2mba, a novel architecture that incorporates State Space Models
(SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This
allows the LMMs to scale linearly in terms of time and memory requirements,
making it feasible to handle long-duration video content. Furthermore, we
enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing
(MA-GC) method, which strategically manages memory by retaining only essential
activations across multiple computational axes. Our approach significantly
reduces the memory footprint compared to standard gradient checkpointing.
Empirical analyses show that Video-Ma^2mba can process extensive video
sequences-equivalent to millions of tokens or over two hours of continuous
sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of
temporal dynamics, our model improves the accuracy and relevance of responses
in long video understanding tasks, demonstrating substantial advantages over
existing frameworks.
Video-Ma^2^mba, a novel architecture that replaces attention mechanisms with State Space Models in Mamba-2, efficiently processes long video sequences with linear memory and time scalability, enhancing memory efficiency with Multi-Axis Gradient Checkpointing.
State Space Models Mamba-2 Multi-Axis Gradient Checkpointing gradient checkpointing long video sequences