Learning the Visualness of Text Using Large Vision-Language Models
Visual text evokes an image in a person's mind, while non-visual text fails
to do so. A method to automatically detect visualness in text will unlock the
ability to augment text with relevant images, as neural text-to-image
generation and retrieval models operate on the implicit assumption that the
input text is visual in nature. We curate a dataset of 3,620 English sentences
and their visualness scores provided by multiple human annotators.
Additionally, we use documents that contain text and visual assets to create a
distantly supervised corpus of document text and associated images. We also
propose a fine-tuning strategy that adapts large vision-language models like
CLIP that assume a one-to-one correspondence between text and image to the task
of scoring text visualness from text input alone. Our strategy involves
modifying the model's contrastive learning objective to map text identified as
non-visual to a common NULL image while matching visual text to their
corresponding images in the document. We evaluate the proposed approach on its
ability to (i) classify visual and non-visual text accurately, and (ii) attend
over words that are identified as visual in psycholinguistic studies. Empirical
evaluation indicates that our approach performs better than several heuristics
and baseline models for the proposed task. Furthermore, to highlight the
importance of modeling the visualness of text, we conduct qualitative analyses
of text-to-image generation systems like DALL-E.
6466fa2aac657f60661ddee6
Gaurav Verma
claimed_verified
2023-05-19T14:33:46.938Z
6359599c6805fab09b0912dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6359599c6805fab09b0912dc/645rCR99A6CyIXSFy-w6_.jpeg
Gaurav Verma
user
gauravverma
6466fa2aac657f60661ddee7
Ryan A. Rossi
6466fa2aac657f60661ddee8
Christopher Tensmeyer
6466fa2aac657f60661ddee9
Jiuxiang Gu
6466fa2aac657f60661ddeea
Ani Nenkova
Learning the Visualness of Text Using Large Vision-Language Models
Visual text evokes an image in a person's mind, while non-visual text fails
to do so. A method to automatically detect visualness in text will unlock the
ability to augment text with relevant images, as neural text-to-image
generation and retrieval models operate on the implicit assumption that the
input text is visual in nature. We curate a dataset of 3,620 English sentences
and their visualness scores provided by multiple human annotators.
Additionally, we use documents that contain text and visual assets to create a
distantly supervised corpus of document text and associated images. We also
propose a fine-tuning strategy that adapts large vision-language models like
CLIP that assume a one-to-one correspondence between text and image to the task
of scoring text visualness from text input alone. Our strategy involves
modifying the model's contrastive learning objective to map text identified as
non-visual to a common NULL image while matching visual text to their
corresponding images in the document. We evaluate the proposed approach on its
ability to (i) classify visual and non-visual text accurately, and (ii) attend
over words that are identified as visual in psycholinguistic studies. Empirical
evaluation indicates that our approach performs better than several heuristics
and baseline models for the proposed task. Furthermore, to highlight the
importance of modeling the visualness of text, we conduct qualitative analyses
of text-to-image generation systems like DALL-E.
A method to detect visualness in text improves neural text-to-image generation by fine-tuning large vision-language models to classify visual and non-visual text accurately.
visualness neural text-to-image generation retrieval models CLIP contrastive learning objective NULL image psycholinguistic studies DALL-E