Guiding Image Captioning Models Toward More Specific Captions
Image captioning is conventionally formulated as the task of generating
captions for images that match the distribution of reference image-caption
pairs. However, reference captions in standard captioning datasets are short
and may not uniquely identify the images they describe. These problems are
further exacerbated when models are trained directly on image-alt text pairs
collected from the internet. In this work, we show that it is possible to
generate more specific captions with minimal changes to the training process.
We implement classifier-free guidance for an autoregressive captioning model by
fine-tuning it to estimate both conditional and unconditional distributions
over captions. The guidance scale applied at decoding controls a trade-off
between maximizing p(caption|image) and
p(image|caption). Compared to standard greedy decoding,
decoding with a guidance scale of 2 substantially improves reference-free
metrics such as CLIPScore (0.808 vs. 0.775) and captiontoimage retrieval
performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens
standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We
further explore the use of language models to guide the decoding process,
obtaining small improvements over the Pareto frontier of reference-free vs.
reference-based captioning metrics that arises from classifier-free guidance,
and substantially improving the quality of captions generated from a model
trained only on minimally curated web data.
64c86d027b4d0d947cd5292e
Simon Kornblith
admin_assigned
2023-08-01T09:26:26.786Z
6443ffc2af034cdfd69c5c00
/avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
Simon Kornblith
user
simonsterk
64c86d027b4d0d947cd5292f
Lala Li
64c86d027b4d0d947cd52930
Zirui Wang
64c86d027b4d0d947cd52931
Thao Nguyen
claimed_verified
2023-08-01T07:44:25.132Z
64ba4923921dbcbc96006fb8
/avatars/4eab3991e8502f72ddf8ca4825c2ad65.svg
Thao Nguyen
user
thaottn
Guiding Image Captioning Models Toward More Specific Captions
Image captioning is conventionally formulated as the task of generating
captions for images that match the distribution of reference image-caption
pairs. However, reference captions in standard captioning datasets are short
and may not uniquely identify the images they describe. These problems are
further exacerbated when models are trained directly on image-alt text pairs
collected from the internet. In this work, we show that it is possible to
generate more specific captions with minimal changes to the training process.
We implement classifier-free guidance for an autoregressive captioning model by
fine-tuning it to estimate both conditional and unconditional distributions
over captions. The guidance scale applied at decoding controls a trade-off
between maximizing p(caption|image) and
p(image|caption). Compared to standard greedy decoding,
decoding with a guidance scale of 2 substantially improves reference-free
metrics such as CLIPScore (0.808 vs. 0.775) and captiontoimage retrieval
performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens
standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We
further explore the use of language models to guide the decoding process,
obtaining small improvements over the Pareto frontier of reference-free vs.
reference-based captioning metrics that arises from classifier-free guidance,
and substantially improving the quality of captions generated from a model
trained only on minimally curated web data.
Classifier-free guidance improves specific caption generation by fine-tuning for both conditional and unconditional distributions, enhancing reference-free metrics and web data quality.
autoregressive captioning model classifier-free guidance conditional distributions unconditional distributions decoding CLIPScore CLIP embedding space recall@1 CIDEr language models Pareto frontier