Painting with Words: Elevating Detailed Image Captioning with Benchmark
  and Alignment Learning
Image captioning has long been a pivotal task in visual understanding, with
recent advancements in vision-language models (VLMs) significantly enhancing
the ability to generate detailed image captions. However, the evaluation of
detailed image captioning remains underexplored due to outdated evaluation
metrics and coarse annotations. In this paper, we introduce DeCapBench along
with a novel metric, DCScore, specifically designed for detailed captioning
tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by
deconstructing responses into the smallest self-sufficient units, termed
primitive information units, and assessing them individually. Our evaluation
shows that DCScore aligns more closely with human judgment than other
rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high
correlation with VLM arena results on descriptive tasks, surpassing existing
benchmarks for vision-language models. Additionally, we present an automatic
fine-grained feedback collection method, FeedQuill, for preference optimization
based on our advanced metric, showing robust generalization capabilities across
auto-generated preference data. Extensive experiments on multiple VLMs
demonstrate that our method not only significantly reduces hallucinations but
also enhances performance across various benchmarks, achieving superior detail
captioning performance while surpassing GPT-4o.
67dde94f169ce083a5098930
Qinghao Ye
claimed_verified
2025-05-22T07:18:44.811Z
64530fc01a57e1179c1fe4c0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
QinghaoYe
user
MAGAer13
67dde94f169ce083a5098931
Xianhan Zeng
claimed_verified
2025-05-13T08:13:29.058Z
666d10506ec1340a7421a92d
/avatars/5bff16fca817b35760e0c26d8d7c9424.svg
Xianhan Zeng
user
RitzzZz23
67dde94f169ce083a5098932
Fu Li
67dde94f169ce083a5098933
Chunyuan Li
67dde94f169ce083a5098934
Haoqi Fan
Painting with Words: Elevating Detailed Image Captioning with Benchmark
  and Alignment Learning
Image captioning has long been a pivotal task in visual understanding, with
recent advancements in vision-language models (VLMs) significantly enhancing
the ability to generate detailed image captions. However, the evaluation of
detailed image captioning remains underexplored due to outdated evaluation
metrics and coarse annotations. In this paper, we introduce DeCapBench along
with a novel metric, DCScore, specifically designed for detailed captioning
tasks. DCScore evaluates hallucinations and fine-grained comprehensiveness by
deconstructing responses into the smallest self-sufficient units, termed
primitive information units, and assessing them individually. Our evaluation
shows that DCScore aligns more closely with human judgment than other
rule-based or model-based metrics. Concurrently, DeCapBench exhibits a high
correlation with VLM arena results on descriptive tasks, surpassing existing
benchmarks for vision-language models. Additionally, we present an automatic
fine-grained feedback collection method, FeedQuill, for preference optimization
based on our advanced metric, showing robust generalization capabilities across
auto-generated preference data. Extensive experiments on multiple VLMs
demonstrate that our method not only significantly reduces hallucinations but
also enhances performance across various benchmarks, achieving superior detail
captioning performance while surpassing GPT-4o.
DeCapBench, a new benchmark with DCScore, improves detailed image captioning evaluation and performance by reducing hallucinations and enhancing comprehensiveness.
vision-language models DeCapBench DCScore primitive information units hallucinations fine-grained comprehensiveness FeedQuill preference optimization auto-generated preference data