KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature
  Adaptation of Vision-Language Models
Image ad understanding is a crucial task with wide real-world applications.
Although highly challenging with the involvement of diverse atypical scenes,
real-world entities, and reasoning over scene-texts, how to interpret image ads
is relatively under-explored, especially in the era of foundational
vision-language models (VLMs) featuring impressive generalizability and
adaptability. In this paper, we perform the first empirical study of image ad
understanding through the lens of pre-trained VLMs. We benchmark and reveal
practical challenges in adapting these VLMs to image ad understanding. We
propose a simple feature adaptation strategy to effectively fuse multimodal
information for image ads and further empower it with knowledge of real-world
entities. We hope our study draws more attention to image ad understanding
which is broadly relevant to the advertising industry.
6476bf179db7d9fe2ad83f55
Zhiwei Jia
claimed_verified
2023-05-31T18:31:40.907Z
62d50bf25654195611dba128
/avatars/8f8345ae934406cb3a6ab7d391b4b5e1.svg
Zhiwei Jia
user
SeanJia
6476bf179db7d9fe2ad83f56
Pradyumna Narayana
6476bf179db7d9fe2ad83f57
Arjun R. Akula
6476bf179db7d9fe2ad83f58
Garima Pruthi
6476bf179db7d9fe2ad83f59
Hao Su
6476bf179db7d9fe2ad83f5a
Sugato Basu
6476bf179db7d9fe2ad83f5b
Varun Jampani
admin_assigned
2023-06-02T12:35:33.874Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature
  Adaptation of Vision-Language Models
Image ad understanding is a crucial task with wide real-world applications.
Although highly challenging with the involvement of diverse atypical scenes,
real-world entities, and reasoning over scene-texts, how to interpret image ads
is relatively under-explored, especially in the era of foundational
vision-language models (VLMs) featuring impressive generalizability and
adaptability. In this paper, we perform the first empirical study of image ad
understanding through the lens of pre-trained VLMs. We benchmark and reveal
practical challenges in adapting these VLMs to image ad understanding. We
propose a simple feature adaptation strategy to effectively fuse multimodal
information for image ads and further empower it with knowledge of real-world
entities. We hope our study draws more attention to image ad understanding
which is broadly relevant to the advertising industry.
An empirical study of adapting pre-trained vision-language models to understand image ads reveals practical challenges and proposes a feature adaptation strategy to fuse multimodal information and leverage real-world knowledge.
vision-language models image ad understanding multimodal information