CoVLM: Composing Visual Entities and Relationships in Large Language
  Models Via Communicative Decoding
A remarkable ability of human beings resides in compositional reasoning,
i.e., the capacity to make "infinite use of finite means". However, current
large vision-language foundation models (VLMs) fall short of such compositional
abilities due to their "bag-of-words" behaviors and inability to construct
words that correctly represent visual entities and the relations among the
entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
compose visual entities and relationships among the text and dynamically
communicate with the vision encoder and detection network to achieve
vision-language communicative decoding. Specifically, we first devise a set of
novel communication tokens for the LLM, for dynamic communication between the
visual detection system and the language system. A communication token is
generated by the LLM following a visual entity or a relation, to inform the
detection network to propose regions that are relevant to the sentence
generated so far. The proposed regions-of-interests (ROIs) are then fed back
into the LLM for better language generation contingent on the relevant regions.
The LLM is thus able to compose the visual entities and relationships through
the communication tokens. The vision-to-language and language-to-vision
communication are iteratively performed until the entire sentence is generated.
Our framework seamlessly bridges the gap between visual perception and LLMs and
outperforms previous VLMs by a large margin on compositional reasoning
benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
ARO top-1 accuracy). We also achieve state-of-the-art performances on
traditional vision-language tasks such as referring expression comprehension
and visual question answering.
6549ca71c26b9e805de70681
Junyan Li
claimed_verified
2023-11-07T20:23:33.070Z
62d09eb86a61a88ea0d83918
/avatars/81b511d94cced304ffca058caff662d4.svg
Junyan Li
user
senfu
6549ca71c26b9e805de70682
Delin Chen
admin_assigned
2023-11-07T17:13:21.901Z
62189e34523d13b0992f67ac
/avatars/b5153ff75b20185b195eb38c4786fd34.svg
Delin Chen
user
chendl
6549ca71c26b9e805de70683
Yining Hong
admin_assigned
2023-11-07T17:13:14.644Z
6431b64df76c34519e93d1ba
/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
Yining Hong
user
evelynhong
6549ca71c26b9e805de70684
Zhenfang Chen
admin_assigned
2023-11-07T17:14:51.749Z
6216f8a3a805e5a4d418f4cb
/avatars/8860471b3e397ca61505ee2edf73e4a6.svg
Zhenfang Chen
user
zfchen
6549ca71c26b9e805de70685
Peihao Chen
admin_assigned
2023-11-07T17:15:23.682Z
64bfe895c874dda773303b44
/avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
PeihaoChen
user
PeihaoChen
6549ca71c26b9e805de70686
Yikang Shen
admin_assigned
2023-11-07T17:16:05.485Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
6549ca71c26b9e805de70687
Chuang Gan
CoVLM: Composing Visual Entities and Relationships in Large Language
  Models Via Communicative Decoding
A remarkable ability of human beings resides in compositional reasoning,
i.e., the capacity to make "infinite use of finite means". However, current
large vision-language foundation models (VLMs) fall short of such compositional
abilities due to their "bag-of-words" behaviors and inability to construct
words that correctly represent visual entities and the relations among the
entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
compose visual entities and relationships among the text and dynamically
communicate with the vision encoder and detection network to achieve
vision-language communicative decoding. Specifically, we first devise a set of
novel communication tokens for the LLM, for dynamic communication between the
visual detection system and the language system. A communication token is
generated by the LLM following a visual entity or a relation, to inform the
detection network to propose regions that are relevant to the sentence
generated so far. The proposed regions-of-interests (ROIs) are then fed back
into the LLM for better language generation contingent on the relevant regions.
The LLM is thus able to compose the visual entities and relationships through
the communication tokens. The vision-to-language and language-to-vision
communication are iteratively performed until the entire sentence is generated.
Our framework seamlessly bridges the gap between visual perception and LLMs and
outperforms previous VLMs by a large margin on compositional reasoning
benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
ARO top-1 accuracy). We also achieve state-of-the-art performances on
traditional vision-language tasks such as referring expression comprehension
and visual question answering.
CoVLM enhances compositional reasoning in vision-language models by introducing communication tokens for iterative interaction between vision and language systems.
large vision-language foundation models compositional abilities bag-of-words communication tokens regions-of-interest visual-to-language language-to-vision referring expression comprehension visual question answering