LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
  Hybrid Architecture
Expanding the long-context capabilities of Multi-modal Large Language
Models~(MLLMs) is crucial for video understanding, high-resolution image
understanding, and multi-modal agents. This involves a series of systematic
optimizations, including model architecture, data construction and training
strategy, particularly addressing challenges such as degraded
performance with more images and high computational costs. In this
paper, we adapt the model architecture to a hybrid of Mamba and Transformer
blocks, approach data construction with both temporal and spatial dependencies
among multiple images and employ a progressive training strategy. The released
model LongLLaVA~(Long-Context Large
Language and Vision Assistant) is the first
hybrid MLLM, which achieved a better balance between efficiency and
effectiveness. LongLLaVA not only achieves competitive results across various
benchmarks, but also maintains high throughput and low memory consumption.
Especially, it could process nearly a thousand images on a single A100 80GB
GPU, showing promising application prospects for a wide range of tasks.
66d917fe8506f9b6cfd098df
Xidong Wang
admin_assigned
2024-09-05T15:31:08.187Z
640ed3e9f2d7c41a1e9a9fde
https://cdn-avatars.huggingface.co/v1/production/uploads/1678693486092-640ed3e9f2d7c41a1e9a9fde.jpeg
Xidong Wang
user
Xidong
66d917fe8506f9b6cfd098e0
Dingjie Song
claimed_verified
2024-09-05T07:23:57.982Z
619f01b8cc04eadf54fa5d5d
/avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
Song Dingjie
user
songdj
66d917fe8506f9b6cfd098e1
Shunian Chen
claimed_verified
2024-09-05T15:31:17.133Z
623be9e1d1eb227788764959
/avatars/b6521b795a59754dbb40123fd4f63b8c.svg
Shunian Chen
user
Shunian
66d917fe8506f9b6cfd098e2
Chen Zhang
66d917fe8506f9b6cfd098e3
Benyou Wang
claimed_verified
2024-09-30T08:07:35.614Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
  Hybrid Architecture
Expanding the long-context capabilities of Multi-modal Large Language
Models~(MLLMs) is crucial for video understanding, high-resolution image
understanding, and multi-modal agents. This involves a series of systematic
optimizations, including model architecture, data construction and training
strategy, particularly addressing challenges such as degraded
performance with more images and high computational costs. In this
paper, we adapt the model architecture to a hybrid of Mamba and Transformer
blocks, approach data construction with both temporal and spatial dependencies
among multiple images and employ a progressive training strategy. The released
model LongLLaVA~(Long-Context Large
Language and Vision Assistant) is the first
hybrid MLLM, which achieved a better balance between efficiency and
effectiveness. LongLLaVA not only achieves competitive results across various
benchmarks, but also maintains high throughput and low memory consumption.
Especially, it could process nearly a thousand images on a single A100 80GB
GPU, showing promising application prospects for a wide range of tasks.
LongLLaVA, a hybrid MLLM combining Mamba and Transformer blocks, enhances video and image understanding with improved efficiency and throughput.
Mamba Transformer blocks temporal dependencies spatial dependencies progressive training strategy LongLLaVA