Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular
code generation. In contrast to prior work on modular approaches to VQA, our
approach requires no additional training and relies on pre-trained language
models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
examples used for in-context learning. The generated Python programs invoke and
compose the outputs of the visual models using arithmetic and conditional
logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
the GQA dataset by roughly 2% compared to the few-shot baseline that does not
employ code generation.
64829bf6e4bbb1c2dd35da74
Sanjay Subramanian
admin_assigned
2023-06-12T13:34:58.858Z
62cd947a3a2cecfdabeac91f
/avatars/84393f357c0370edbb90a4188df795d6.svg
Sanjay Subramanian
user
sanjayss34
64829bf6e4bbb1c2dd35da75
Medhini Narasimhan
admin_assigned
2023-06-12T13:35:16.781Z
611ffea4e48a99edad639e8f
/avatars/873ea85dbd5d79509faa3fedfcaa33d8.svg
Medhini Narasimhan
user
medhini
64829bf6e4bbb1c2dd35da76
Kushal Khangaonkar
64829bf6e4bbb1c2dd35da77
Kevin Yang
64829bf6e4bbb1c2dd35da78
Arsha Nagrani
64829bf6e4bbb1c2dd35da79
Cordelia Schmid
64829bf6e4bbb1c2dd35da7a
Andy Zeng
claimed_verified
2023-06-12T09:12:29.862Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
64829bf6e4bbb1c2dd35da7b
Trevor Darrell
64829bf6e4bbb1c2dd35da7c
Dan Klein
admin_assigned
2023-06-12T13:36:14.695Z
632be88b3690fb57e70e0bf1
/avatars/74ff8f30b3662db2602495bdf493d397.svg
Dan Klein
user
danjklein
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular
code generation. In contrast to prior work on modular approaches to VQA, our
approach requires no additional training and relies on pre-trained language
models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA
examples used for in-context learning. The generated Python programs invoke and
compose the outputs of the visual models using arithmetic and conditional
logic. Our approach improves accuracy on the COVR dataset by at least 3% and on
the GQA dataset by roughly 2% compared to the few-shot baseline that does not
employ code generation.
A framework generates modular Python code that combines pre-trained visual and language models for visual question answering, improving accuracy on benchmark datasets.
visual question answering code generation pre-trained language models visual models image-caption pairs in-context learning COVR dataset GQA dataset few-shot baseline