DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
  Perception
Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
complex understanding of various visual elements, including multiple objects,
text information, and spatial relations. Their development for comprehensive
visual perception hinges on the availability of high-quality image-text
datasets that offer diverse visual elements and throughout image descriptions.
However, the scarcity of such hyper-detailed datasets currently hinders
progress within the MLLM community. The bottleneck stems from the limited
perceptual capabilities of current caption engines, which fall short in
providing complete and accurate annotations. To facilitate the cutting-edge
research of MLLMs on comprehensive vision perception, we thereby propose
Perceptual Fusion, using a low-budget but highly effective caption engine for
complete and accurate image descriptions. Specifically, Perceptual Fusion
integrates diverse perception experts as image priors to provide explicit
information on visual elements and adopts an efficient MLLM as a centric pivot
to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
representative images from uncurated LAION dataset and generate dense
descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
validate that our engine outperforms its counterparts, where the resulting
dataset significantly improves the perception and cognition abilities of
existing MLLMs across diverse vision-language benchmarks, especially with
high-resolution images as inputs. The dataset and code are publicly available
at https://github.com/baaivision/DenseFusion.
66908d786dc63c0346f99abb
Xiaotong Li
66908d786dc63c0346f99abc
Fan Zhang
claimed_verified
2024-10-08T07:22:39.363Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
66908d786dc63c0346f99abd
Haiwen Diao
admin_assigned
2024-07-12T11:48:23.442Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
66908d786dc63c0346f99abe
Yueze Wang
claimed_verified
2024-09-27T07:00:00.388Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
66908d786dc63c0346f99abf
Xinlong Wang
admin_assigned
2024-07-12T11:49:12.383Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
66908d786dc63c0346f99ac0
Ling-Yu Duan
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
  Perception
Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
complex understanding of various visual elements, including multiple objects,
text information, and spatial relations. Their development for comprehensive
visual perception hinges on the availability of high-quality image-text
datasets that offer diverse visual elements and throughout image descriptions.
However, the scarcity of such hyper-detailed datasets currently hinders
progress within the MLLM community. The bottleneck stems from the limited
perceptual capabilities of current caption engines, which fall short in
providing complete and accurate annotations. To facilitate the cutting-edge
research of MLLMs on comprehensive vision perception, we thereby propose
Perceptual Fusion, using a low-budget but highly effective caption engine for
complete and accurate image descriptions. Specifically, Perceptual Fusion
integrates diverse perception experts as image priors to provide explicit
information on visual elements and adopts an efficient MLLM as a centric pivot
to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
representative images from uncurated LAION dataset and generate dense
descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
validate that our engine outperforms its counterparts, where the resulting
dataset significantly improves the perception and cognition abilities of
existing MLLMs across diverse vision-language benchmarks, especially with
high-resolution images as inputs. The dataset and code are publicly available
at https://github.com/baaivision/DenseFusion.
Perceptual Fusion integrates diverse perception experts with an efficient MLLM to generate comprehensive image descriptions, enhancing MLLMs' vision-language capabilities.
Multimodal Large Language Models visual elements image-text datasets caption engines perceptual capabilities image priors vision-language benchmarks DenseFusion-1M