VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained
  Perception in VLMs
Vision-Language Models (VLMs) excel at high-level scene understanding but
falter on fine-grained perception tasks requiring precise localization. This
failure stems from a fundamental mismatch, as generating exact numerical
coordinates is a challenging task for language-centric architectures. In this
paper, we introduce VLM-FO1, a novel framework that overcomes this limitation
by reframing object-centric perception from a brittle coordinate generation
problem into a robust feature retrieval task. Our method operates as a
plug-and-play module that integrates with any pre-trained VLM. It leverages a
Hybrid Fine-grained Region Encoder (HFRE), featuring a dual vision encoder, to
generate powerful region tokens rich in both semantic and spatial detail. A
token-based referencing system then enables the LLM to seamlessly reason about
and ground language in these specific visual regions. Experiments show that
VLM-FO1 achieves state-of-the-art performance across a diverse suite of
benchmarks, demonstrating exceptional capabilities in object grounding, region
generational understanding, and visual region reasoning. Crucially, our
two-stage training strategy ensures that these perception gains are achieved
without compromising the base model's general visual understanding
capabilities. VLM-FO1 establishes an effective and flexible paradigm for
building perception-aware VLMs, bridging the gap between high-level reasoning
and fine-grained visual grounding.
68de2852a9129e507bf4a41f
Peng Liu
68de2852a9129e507bf4a420
Haozhan Shen
68de2852a9129e507bf4a421
Chunxin Fang
68de2852a9129e507bf4a422
Zhicheng Sun
68de2852a9129e507bf4a423
Jiajia Liao
68de2852a9129e507bf4a424
Tiancheng Zhao
VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained
  Perception in VLMs
Vision-Language Models (VLMs) excel at high-level scene understanding but
falter on fine-grained perception tasks requiring precise localization. This
failure stems from a fundamental mismatch, as generating exact numerical
coordinates is a challenging task for language-centric architectures. In this
paper, we introduce VLM-FO1, a novel framework that overcomes this limitation
by reframing object-centric perception from a brittle coordinate generation
problem into a robust feature retrieval task. Our method operates as a
plug-and-play module that integrates with any pre-trained VLM. It leverages a
Hybrid Fine-grained Region Encoder (HFRE), featuring a dual vision encoder, to
generate powerful region tokens rich in both semantic and spatial detail. A
token-based referencing system then enables the LLM to seamlessly reason about
and ground language in these specific visual regions. Experiments show that
VLM-FO1 achieves state-of-the-art performance across a diverse suite of
benchmarks, demonstrating exceptional capabilities in object grounding, region
generational understanding, and visual region reasoning. Crucially, our
two-stage training strategy ensures that these perception gains are achieved
without compromising the base model's general visual understanding
capabilities. VLM-FO1 establishes an effective and flexible paradigm for
building perception-aware VLMs, bridging the gap between high-level reasoning
and fine-grained visual grounding.
VLM-FO1 enhances vision-language models with a hybrid fine-grained region encoder to improve object localization and region understanding without sacrificing general visual capabilities.
Vision-Language Models VLM-FO1 Hybrid Fine-grained Region Encoder HFRE dual vision encoder region tokens token-based referencing system object grounding region generational understanding visual region reasoning two-stage training strategy