Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
The project is released at: https://github.com/SparksJoe/Prism.
6675136b3ae9e9c659a0266e
Yuxuan Qiao
claimed_verified
2024-06-21T07:05:37.653Z
663cd4e3c1b4f7297c5dd958
https://cdn-avatars.huggingface.co/v1/production/uploads/663cd4e3c1b4f7297c5dd958/XZdsR1DTc9Njh8XnesTg1.jpeg
Yuxuan Qiao
user
Yuxuan-Qiao
6675136b3ae9e9c659a0266f
Haodong Duan
admin_assigned
2024-06-21T07:57:12.383Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
6675136b3ae9e9c659a02670
Xinyu Fang
claimed_verified
2024-06-21T07:00:18.009Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
user
nebulae09
6675136b3ae9e9c659a02671
Junming Yang
claimed_verified
2024-06-21T13:33:23.625Z
653f1d243bd61358055ad51d
/avatars/698c03b9a4bb69659d2ed594626e3895.svg
junmingyang
user
jmyang
6675136b3ae9e9c659a02672
Lin Chen
claimed_verified
2024-06-21T07:05:31.876Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
6675136b3ae9e9c659a02673
Songyang Zhang
admin_assigned
2024-06-21T07:57:59.726Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
6675136b3ae9e9c659a02674
Jiaqi Wang
admin_assigned
2024-06-21T07:58:34.023Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
6675136b3ae9e9c659a02675
Dahua Lin
admin_assigned
2024-06-21T10:19:04.717Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
6675136b3ae9e9c659a02676
Kai Chen
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
The project is released at: https://github.com/SparksJoe/Prism.
Prism is a framework that separates perception and reasoning in Vision Language Models, achieving competitive performance with smaller models and reduced costs.
Vision Language Models Prism Large Language Model multimodal benchmark MMStar LLaVA GPT-3.5