Prismatic VLMs: Investigating the Design Space of Visually-Conditioned
  Language Models
Visually-conditioned language models (VLMs) have seen growing adoption in
applications such as visual dialogue, scene understanding, and robotic task
planning; adoption that has fueled a wealth of new models such as LLaVa,
InstructBLIP, and PaLI-3. Despite the volume of new releases, key design
decisions around image preprocessing, architecture, and optimization are
under-explored, making it challenging to understand what factors account for
model performance - a challenge further complicated by the lack of objective,
consistent evaluations. To address these gaps, we first compile a suite of
standardized evaluations spanning visual question answering, object
localization from language, and targeted challenge sets that probe properties
such as hallucination; evaluations that provide calibrated, fine-grained
insight into a VLM's capabilities. Second, we rigorously investigate VLMs along
key design axes, including pretrained visual representations and quantifying
the tradeoffs of using base vs. instruct-tuned language models, amongst others.
We couple our analysis with three resource contributions: (1) a unified
framework for evaluating VLMs, (2) optimized, flexible code for VLM training,
and (3) checkpoints for all models, including a family of VLMs at the 7-13B
scale that strictly outperform InstructBLIP and LLaVa v1.5, the
state-of-the-art in open-source VLMs.
65cae4c8ab05dc46af7e5bef
Siddharth Karamcheti
admin_assigned
2024-02-13T14:48:39.900Z
6150b090d84cf0532aa1764b
https://cdn-avatars.huggingface.co/v1/production/uploads/1634666324094-6150b090d84cf0532aa1764b.jpeg
Siddharth Karamcheti
user
skaramcheti
65cae4c8ab05dc46af7e5bf0
Suraj Nair
admin_assigned
2024-02-13T14:49:24.667Z
624f0e38c8b211c193d325dd
/avatars/370b7b21bd1bc284d3f48c0c40dfa026.svg
Suraj Nair
user
surajnair
65cae4c8ab05dc46af7e5bf1
Ashwin Balakrishna
admin_assigned
2024-02-13T14:49:43.603Z
653ad464170f0a2a9202dbe3
/avatars/dd48cf05b946663dcb927545cd8edddc.svg
Ashwin Balakrishna
user
abalakrishnaTRI
65cae4c8ab05dc46af7e5bf2
Percy Liang
admin_assigned
2024-02-13T14:49:50.742Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
65cae4c8ab05dc46af7e5bf3
Thomas Kollar
admin_assigned
2024-02-13T14:49:57.889Z
64358e662d0ed796668cb438
/avatars/6f86547d8691e061e289dd906deedfdb.svg
Thomas Kollar
user
thomas-kollar
65cae4c8ab05dc46af7e5bf4
Dorsa Sadigh
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned
  Language Models
Visually-conditioned language models (VLMs) have seen growing adoption in
applications such as visual dialogue, scene understanding, and robotic task
planning; adoption that has fueled a wealth of new models such as LLaVa,
InstructBLIP, and PaLI-3. Despite the volume of new releases, key design
decisions around image preprocessing, architecture, and optimization are
under-explored, making it challenging to understand what factors account for
model performance - a challenge further complicated by the lack of objective,
consistent evaluations. To address these gaps, we first compile a suite of
standardized evaluations spanning visual question answering, object
localization from language, and targeted challenge sets that probe properties
such as hallucination; evaluations that provide calibrated, fine-grained
insight into a VLM's capabilities. Second, we rigorously investigate VLMs along
key design axes, including pretrained visual representations and quantifying
the tradeoffs of using base vs. instruct-tuned language models, amongst others.
We couple our analysis with three resource contributions: (1) a unified
framework for evaluating VLMs, (2) optimized, flexible code for VLM training,
and (3) checkpoints for all models, including a family of VLMs at the 7-13B
scale that strictly outperform InstructBLIP and LLaVa v1.5, the
state-of-the-art in open-source VLMs.
The paper provides standardized evaluations and investigations into visually-conditioned language models (VLMs), exploring factors affecting their performance and offering resources like a unified evaluation framework and optimized training code.
visually-conditioned language models VLMs visual dialogue scene understanding robotic task planning LLaVa InstructBLIP PaLI-3 image preprocessing architecture optimization visual question answering object localization hallucination pretrained visual representations instruct-tuned language models unified framework VLM training checkpoints state-of-the-art