FINECAPTION: Compositional Image Captioning Focusing on Wherever You
  Want at Any Granularity
The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal tasks, enabling more sophisticated and accurate reasoning across
various applications, including image and video captioning, visual question
answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
struggle with fine-grained image regional composition information perception.
Specifically, they have difficulty accurately aligning the segmentation masks
with the corresponding semantics and precisely describing the compositional
aspects of the referred regions.
  However, compositionality - the ability to understand and generate novel
combinations of known visual and textual components - is critical for
facilitating coherent reasoning and understanding across modalities by VLMs. To
address this issue, we propose FINECAPTION, a novel VLM that can recognize
arbitrary masks as referential inputs and process high-resolution images for
compositional image captioning at different granularity levels. To support this
endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
compositional image captioning, which introduces the task of compositional
attribute-aware regional image captioning.
  Empirical results demonstrate the effectiveness of our proposed model
compared to other state-of-the-art VLMs. Additionally, we analyze the
capabilities of current VLMs in recognizing various visual prompts for
compositional region image captioning, highlighting areas for improvement in
VLM design and training.
674689ccfe2f63e2200c520c
Hang Hua
claimed_verified
2025-01-14T08:30:18.387Z
639f8277beb95d698de007dd
/avatars/57f223ccd9d3cb03166ccf0e41361c58.svg
HangHua
user
hhua2
674689ccfe2f63e2200c520d
Qing Liu
674689ccfe2f63e2200c520e
Lingzhi Zhang
674689ccfe2f63e2200c520f
Jing Shi
674689ccfe2f63e2200c5210
Zhifei Zhang
674689ccfe2f63e2200c5211
Yilin Wang
674689ccfe2f63e2200c5212
Jianming Zhang
674689ccfe2f63e2200c5213
Jiebo Luo
FINECAPTION: Compositional Image Captioning Focusing on Wherever You
  Want at Any Granularity
The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal tasks, enabling more sophisticated and accurate reasoning across
various applications, including image and video captioning, visual question
answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
struggle with fine-grained image regional composition information perception.
Specifically, they have difficulty accurately aligning the segmentation masks
with the corresponding semantics and precisely describing the compositional
aspects of the referred regions.
  However, compositionality - the ability to understand and generate novel
combinations of known visual and textual components - is critical for
facilitating coherent reasoning and understanding across modalities by VLMs. To
address this issue, we propose FINECAPTION, a novel VLM that can recognize
arbitrary masks as referential inputs and process high-resolution images for
compositional image captioning at different granularity levels. To support this
endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
compositional image captioning, which introduces the task of compositional
attribute-aware regional image captioning.
  Empirical results demonstrate the effectiveness of our proposed model
compared to other state-of-the-art VLMs. Additionally, we analyze the
capabilities of current VLMs in recognizing various visual prompts for
compositional region image captioning, highlighting areas for improvement in
VLM design and training.
FINECAPTION is a Vision-Language Model that addresses fine-grained image regional information and compositional captioning through a novel dataset and model architecture.
Vision-Language Models image captioning visual question answering cross-modal retrieval segmentation masks compositional image captioning multi-grained region compositional image captioning COMPOSITIONCAP attribute-aware regional image captioning