Florence-VL: Enhancing Vision-Language Models with Generative Vision
  Encoder and Depth-Breadth Fusion
We present Florence-VL, a new family of multimodal large language models
(MLLMs) with enriched visual representations produced by Florence-2, a
generative vision foundation model. Unlike the widely used CLIP-style vision
transformer trained by contrastive learning, Florence-2 can capture different
levels and aspects of visual features, which are more versatile to be adapted
to diverse downstream tasks. We propose a novel feature-fusion architecture and
an innovative training recipe that effectively integrates Florence-2's visual
features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
from different depths and under multiple prompts. Our model training is
composed of end-to-end pretraining of the whole model followed by finetuning of
the projection layer and the LLM, on a carefully designed recipe of diverse
open-source datasets that include high-quality image captions and
instruction-tuning pairs. Our quantitative analysis and visualization of
Florence-VL's visual features show its advantages over popular vision encoders
on vision-language alignment, where the enriched depth and breath play
important roles. Florence-VL achieves significant improvements over existing
state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
covering general VQA, perception, hallucination, OCR, Chart,
knowledge-intensive understanding, etc. To facilitate future research, our
models and the complete training recipe are open-sourced.
https://github.com/JiuhaiChen/Florence-VL
67526d9bba72c8bc0779edcf
Jiuhai Chen
admin_assigned
2024-12-06T17:18:45.149Z
6393847e3e30234ae798b7be
/avatars/daeb8c37dff4432d837a69b87c196521.svg
JiuhaiChen
user
jiuhai
67526d9bba72c8bc0779edd0
Jianwei Yang
admin_assigned
2024-12-06T17:18:34.697Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
67526d9bba72c8bc0779edd1
Haiping Wu
admin_assigned
2024-12-06T17:18:17.565Z
6634347b6cab8677737be59b
/avatars/1deebaa2c63309e37df62be4c6521443.svg
Haiping Wu
user
haipingwu
67526d9bba72c8bc0779edd2
Dianqi Li
67526d9bba72c8bc0779edd3
Jianfeng Gao
admin_assigned
2024-12-06T17:17:57.025Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
67526d9bba72c8bc0779edd4
Tianyi Zhou
claimed_verified
2024-12-06T09:09:02.653Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
67526d9bba72c8bc0779edd5
Bin Xiao
Florence-VL: Enhancing Vision-Language Models with Generative Vision
  Encoder and Depth-Breadth Fusion
We present Florence-VL, a new family of multimodal large language models
(MLLMs) with enriched visual representations produced by Florence-2, a
generative vision foundation model. Unlike the widely used CLIP-style vision
transformer trained by contrastive learning, Florence-2 can capture different
levels and aspects of visual features, which are more versatile to be adapted
to diverse downstream tasks. We propose a novel feature-fusion architecture and
an innovative training recipe that effectively integrates Florence-2's visual
features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
from different depths and under multiple prompts. Our model training is
composed of end-to-end pretraining of the whole model followed by finetuning of
the projection layer and the LLM, on a carefully designed recipe of diverse
open-source datasets that include high-quality image captions and
instruction-tuning pairs. Our quantitative analysis and visualization of
Florence-VL's visual features show its advantages over popular vision encoders
on vision-language alignment, where the enriched depth and breath play
important roles. Florence-VL achieves significant improvements over existing
state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
covering general VQA, perception, hallucination, OCR, Chart,
knowledge-intensive understanding, etc. To facilitate future research, our
models and the complete training recipe are open-sourced.
https://github.com/JiuhaiChen/Florence-VL
Florence-VL enhances multimodal language models with improved visual features from Florence-2, achieving state-of-the-art results across various benchmarks through depth-breath fusion and a novel training recipe.
multimodal large language models Florence-2 generative vision foundation model CLIP-style vision transformer contrastive learning feature-fusion architecture depth-breath fusion (DBFusion) projection layer pretraining finetuning high-quality image captions instruction-tuning pairs vision-language alignment general VQA perception hallucination OCR Chart knowledge-intensive understanding