Understanding Retrieval Robustness for Retrieval-Augmented Image
  Captioning
Recent advances in retrieval-augmented models for image captioning highlight
the benefit of retrieving related captions for efficient, lightweight models
with strong domain-transfer capabilities. While these models demonstrate the
success of retrieval augmentation, retrieval models are still far from perfect
in practice: the retrieved information can sometimes mislead the model,
resulting in incorrect generation and worse performance. In this paper, we
analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
analysis shows that the model is sensitive to tokens that appear in the
majority of the retrieved captions, and the input attribution shows that those
tokens are likely copied into the generated output. Given these findings, we
propose to train the model by sampling retrieved captions from more diverse
sets. This decreases the chance that the model learns to copy majority tokens,
and improves both in-domain and cross-domain performance.
668f02f92eb49d45ac6a4f2a
Wenyan Li
claimed_verified
2024-07-15T09:01:01.526Z
619b506f70d03780cbec5806
/avatars/ea2b0b8f0a3eb16d53ef40da9981c397.svg
wenyan li
user
lyan62
668f02f92eb49d45ac6a4f2b
Jiaang Li
claimed_verified
2024-07-11T08:26:59.087Z
6686658221f096c7fb4e0b24
/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
Jiaang Li
user
jaagli
668f02f92eb49d45ac6a4f2c
Rita Ramos
668f02f92eb49d45ac6a4f2d
Raphael Tang
claimed_verified
2025-09-22T02:29:24.859Z
63250bb8d206fe7b2d2f1b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/1668053024087-63250bb8d206fe7b2d2f1b8b.jpeg
Raphael Tang
user
tetrisd
668f02f92eb49d45ac6a4f2e
Desmond Elliott
Understanding Retrieval Robustness for Retrieval-Augmented Image
  Captioning
Recent advances in retrieval-augmented models for image captioning highlight
the benefit of retrieving related captions for efficient, lightweight models
with strong domain-transfer capabilities. While these models demonstrate the
success of retrieval augmentation, retrieval models are still far from perfect
in practice: the retrieved information can sometimes mislead the model,
resulting in incorrect generation and worse performance. In this paper, we
analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
analysis shows that the model is sensitive to tokens that appear in the
majority of the retrieved captions, and the input attribution shows that those
tokens are likely copied into the generated output. Given these findings, we
propose to train the model by sampling retrieved captions from more diverse
sets. This decreases the chance that the model learns to copy majority tokens,
and improves both in-domain and cross-domain performance.
An analysis of the SmallCap model reveals sensitivity to commonly retrieved tokens and proposes improving diversity in sampled captions to enhance performance.
retrieval-augmented models image captioning retrieval models token sensitivity input attribution majority tokens