FUSION: Fully Integration of Vision-Language Representations for Deep
  Cross-Modal Understanding
We introduce FUSION, a family of multimodal large language models (MLLMs)
with a fully vision-language alignment and integration paradigm. Unlike
existing methods that primarily rely on late-stage modality interaction during
LLM decoding, our approach achieves deep, dynamic integration throughout the
entire processing pipeline. To this end, we propose Text-Guided Unified Vision
Encoding, incorporating textual information in vision encoding to achieve
pixel-level integration. We further design Context-Aware Recursive Alignment
Decoding that recursively aggregates visual features conditioned on textual
context during decoding, enabling fine-grained, question-level semantic
integration. To guide feature mapping and mitigate modality discrepancies, we
develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
Synthesized Language-Driven Question-Answer (QA) dataset through a new data
synthesis method, prioritizing high-quality QA pairs to optimize text-guided
feature integration. Building on these foundations, we train FUSION at two
scales-3B, 8B-and demonstrate that our full-modality integration approach
significantly outperforms existing methods with only 630 vision tokens.
Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
to 300 vision tokens. Our ablation studies show that FUSION outperforms
LLaVA-NeXT on over half of the benchmarks under same configuration without
dynamic resolution, highlighting the effectiveness of our approach. We release
our code, model weights, and dataset. https://github.com/starriver030515/FUSION
67fdb5f1913c97aa32f130bd
Zheng Liu
claimed_verified
2025-04-15T07:54:57.129Z
6625ef13605f46d05c1d0031
/avatars/22f201dca35e43013cb593884516e96c.svg
Zheng Liu
user
starriver030515
67fdb5f1913c97aa32f130be
Mengjie Liu
admin_assigned
2025-04-15T08:16:15.647Z
672dbf1bec84b9c33412488f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V6o6K28ydpkaS9XFA4Ac3.png
Mengjie Liu
user
Balalauuoo
67fdb5f1913c97aa32f130bf
Jingzhou Chen
67fdb5f1913c97aa32f130c0
Jingwei Xu
admin_assigned
2025-04-15T08:16:53.642Z
672da19bb2f2dc21e176b0de
/avatars/a942ad1c467b865cb9530927fe13f2b7.svg
Jingwei Xu
user
jingwei-xu-00
67fdb5f1913c97aa32f130c1
Bin Cui
67fdb5f1913c97aa32f130c2
Conghui He
admin_assigned
2025-04-15T08:17:01.171Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
67fdb5f1913c97aa32f130c3
Wentao Zhang
FUSION: Fully Integration of Vision-Language Representations for Deep
  Cross-Modal Understanding
We introduce FUSION, a family of multimodal large language models (MLLMs)
with a fully vision-language alignment and integration paradigm. Unlike
existing methods that primarily rely on late-stage modality interaction during
LLM decoding, our approach achieves deep, dynamic integration throughout the
entire processing pipeline. To this end, we propose Text-Guided Unified Vision
Encoding, incorporating textual information in vision encoding to achieve
pixel-level integration. We further design Context-Aware Recursive Alignment
Decoding that recursively aggregates visual features conditioned on textual
context during decoding, enabling fine-grained, question-level semantic
integration. To guide feature mapping and mitigate modality discrepancies, we
develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
Synthesized Language-Driven Question-Answer (QA) dataset through a new data
synthesis method, prioritizing high-quality QA pairs to optimize text-guided
feature integration. Building on these foundations, we train FUSION at two
scales-3B, 8B-and demonstrate that our full-modality integration approach
significantly outperforms existing methods with only 630 vision tokens.
Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
to 300 vision tokens. Our ablation studies show that FUSION outperforms
LLaVA-NeXT on over half of the benchmarks under same configuration without
dynamic resolution, highlighting the effectiveness of our approach. We release
our code, model weights, and dataset. https://github.com/starriver030515/FUSION
FUSION, a multimodal large language model, integrates vision and language through deep, pixel-level and question-level integration, achieving superior performance compared to existing methods.
multimodal large language models MLLMs vision-language alignment Text-Guided Unified Vision Encoding Context-Aware Recursive Alignment Decoding Dual-Supervised Semantic Mapping Loss Synthesized Language-Driven Question-Answer QA dataset full-modality integration feature integration seamless vision-language interaction FUSION 3B FUSION 8B Cambrian-1 8B Florence-VL 8B LLaVA-NeXT dynamic resolution