Unified Text-to-Image Generation and Retrieval
How humans can efficiently and effectively acquire images has always been a
perennial question. A typical solution is text-to-image retrieval from an
existing database given the text query; however, the limited database typically
lacks creativity. By contrast, recent breakthroughs in text-to-image generation
have made it possible to produce fancy and diverse visual content, but it faces
challenges in synthesizing knowledge-intensive images. In this work, we rethink
the relationship between text-to-image generation and retrieval and propose a
unified framework in the context of Multimodal Large Language Models (MLLMs).
Specifically, we first explore the intrinsic discriminative abilities of MLLMs
and introduce a generative retrieval method to perform retrieval in a
training-free manner. Subsequently, we unify generation and retrieval in an
autoregressive generation way and propose an autonomous decision module to
choose the best-matched one between generated and retrieved images as the
response to the text query. Additionally, we construct a benchmark called
TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
the evaluation of unified text-to-image generation and retrieval. Extensive
experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
proposed method.
6667c486f3d613f1b1b2ef97
Leigang Qu
admin_assigned
2024-06-11T08:09:40.697Z
641a6e324182690729c860b5
/avatars/4599451d9775bf1aa11ef1e7011cb05a.svg
Leigang Qu
user
leigangqu
6667c486f3d613f1b1b2ef98
Haochuan Li
claimed_verified
2024-06-11T09:21:12.546Z
64723663b628f9ecdf078fee
/avatars/3367ea7edf1355d9c3e42da09f7dd021.svg
Haochuan Li
user
Haochuan
6667c486f3d613f1b1b2ef99
Tan Wang
admin_assigned
2024-06-11T08:10:18.556Z
62a9cb5c0036766ec1b67f09
/avatars/5a9c5b41ef3eebbe7b21184b289ade68.svg
Tan Wang
user
ReaLMe
6667c486f3d613f1b1b2ef9a
Wenjie Wang
6667c486f3d613f1b1b2ef9b
Yongqi Li
admin_assigned
2024-06-11T08:10:50.513Z
639284bce3fa7a0619408feb
/avatars/c5a019abb6858c35524dea5ab2036f26.svg
Yongqi Li
user
ylic
6667c486f3d613f1b1b2ef9c
Liqiang Nie
6667c486f3d613f1b1b2ef9d
Tat-Seng Chua
admin_assigned
2024-06-11T08:11:04.163Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
Unified Text-to-Image Generation and Retrieval
How humans can efficiently and effectively acquire images has always been a
perennial question. A typical solution is text-to-image retrieval from an
existing database given the text query; however, the limited database typically
lacks creativity. By contrast, recent breakthroughs in text-to-image generation
have made it possible to produce fancy and diverse visual content, but it faces
challenges in synthesizing knowledge-intensive images. In this work, we rethink
the relationship between text-to-image generation and retrieval and propose a
unified framework in the context of Multimodal Large Language Models (MLLMs).
Specifically, we first explore the intrinsic discriminative abilities of MLLMs
and introduce a generative retrieval method to perform retrieval in a
training-free manner. Subsequently, we unify generation and retrieval in an
autoregressive generation way and propose an autonomous decision module to
choose the best-matched one between generated and retrieved images as the
response to the text query. Additionally, we construct a benchmark called
TIGeR-Bench, including creative and knowledge-intensive domains, to standardize
the evaluation of unified text-to-image generation and retrieval. Extensive
experimental results on TIGeR-Bench and two retrieval benchmarks, i.e.,
Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our
proposed method.
A unified framework combining text-to-image generation and retrieval using Multimodal Large Language Models improves the quality and diversity of generated images, particularly for knowledge-intensive content.
Multimodal Large Language Models generative retrieval autoregressive generation autonomous decision module TIGeR-Bench Flickr30K MS-COCO