Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and
  Dual Evaluation Metrics for Factuality and Coverage
Multimodal large language models (MLLMs) excel at generating highly detailed
captions but often produce hallucinations. Our analysis reveals that existing
hallucination detection methods struggle with detailed captions. We attribute
this to the increasing reliance of MLLMs on their generated text, rather than
the input image, as the sequence length grows. To address this issue, we
propose a multiagent approach that leverages LLM-MLLM collaboration to correct
given captions. Additionally, we introduce an evaluation framework and a
benchmark dataset to facilitate the systematic analysis of detailed captions.
Our experiments demonstrate that our proposed evaluation method better aligns
with human judgments of factuality than existing metrics and that existing
approaches to improve the MLLM factuality may fall short in hyper-detailed
image captioning tasks. In contrast, our proposed method significantly enhances
the factual accuracy of captions, even improving those generated by GPT-4V.
Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating
that an MLLM's performance on VQA benchmarks may not correlate with its ability
to generate detailed image captions.
6768c08c75d8e8d042beda80
Saehyung Lee
claimed_verified
2024-12-23T11:10:04.150Z
6768bf111d652acb0c60b938
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/5j0JkrslB8zGf4Wy8tKFc.png
Saehyung Lee
user
saehyungl
6768c08c75d8e8d042beda81
Seunghyun Yoon
6768c08c75d8e8d042beda82
Trung Bui
6768c08c75d8e8d042beda83
Jing Shi
6768c08c75d8e8d042beda84
Sungroh Yoon
Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and
  Dual Evaluation Metrics for Factuality and Coverage
Multimodal large language models (MLLMs) excel at generating highly detailed
captions but often produce hallucinations. Our analysis reveals that existing
hallucination detection methods struggle with detailed captions. We attribute
this to the increasing reliance of MLLMs on their generated text, rather than
the input image, as the sequence length grows. To address this issue, we
propose a multiagent approach that leverages LLM-MLLM collaboration to correct
given captions. Additionally, we introduce an evaluation framework and a
benchmark dataset to facilitate the systematic analysis of detailed captions.
Our experiments demonstrate that our proposed evaluation method better aligns
with human judgments of factuality than existing metrics and that existing
approaches to improve the MLLM factuality may fall short in hyper-detailed
image captioning tasks. In contrast, our proposed method significantly enhances
the factual accuracy of captions, even improving those generated by GPT-4V.
Finally, we highlight a limitation of VQA-centric benchmarking by demonstrating
that an MLLM's performance on VQA benchmarks may not correlate with its ability
to generate detailed image captions.
A multiagent approach using LLM-MLLM collaboration improves the factual accuracy of detailed image captions, surpassing existing methods and highlighting the limitations of VQA benchmarks.
multimodal large language models MLLMs hallucination detection sequence length multiagent approach LLM-MLLM collaboration evaluation framework benchmark dataset factuality GPT-4V VQA benchmarks image captioning