Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
vision-centric approach. While stronger language models can enhance multimodal
capabilities, the design choices for vision components are often insufficiently
explored and disconnected from visual representation learning research. This
gap hinders accurate sensory grounding in real-world scenarios. Our study uses
LLMs and visual instruction tuning as an interface to evaluate various visual
representations, offering new insights into different models and architectures
-- self-supervised, strongly supervised, or combinations thereof -- based on
experiments with over 20 vision encoders. We critically examine existing MLLM
benchmarks, addressing the difficulties involved in consolidating and
interpreting results from various tasks, and introduce a new vision-centric
benchmark, CV-Bench. To further improve visual grounding, we propose the
Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
integrates high-resolution vision features with LLMs while reducing the number
of tokens. Additionally, we discuss the curation of high-quality visual
instruction-tuning data from publicly available sources, emphasizing the
importance of data source balancing and distribution ratio. Collectively,
Cambrian-1 not only achieves state-of-the-art performance but also serves as a
comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
weights, code, supporting tools, datasets, and detailed instruction-tuning and
evaluation recipes. We hope our release will inspire and accelerate
advancements in multimodal systems and visual representation learning.
667a74223463f1209e7ec98f
Shengbang Tong
667a74223463f1209e7ec990
Ellis Brown
claimed_verified
2024-06-25T08:17:45.589Z
626dc5105f7327906f0b2a4e
https://cdn-avatars.huggingface.co/v1/production/uploads/626dc5105f7327906f0b2a4e/QCSzuwYqsv8ozRnusVb-F.jpeg
Ellis Brown
user
ellisbrown
667a74223463f1209e7ec991
Penghao Wu
admin_assigned
2024-06-25T08:36:22.748Z
64101f81b27543634e377fc1
/avatars/557dd9d4707e3b38e0805dfb87c08004.svg
Penghao Wu
user
craigwu
667a74223463f1209e7ec992
Sanghyun Woo
admin_assigned
2024-06-25T08:36:28.657Z
65da437eb62d242ed8f5ffcc
/avatars/94483f3a1fd26ee43b32510538f1cd80.svg
Sanghyun Woo
user
shwoo93
667a74223463f1209e7ec993
Manoj Middepogu
admin_assigned
2024-06-25T08:36:33.894Z
64284d6e6268f155b77ba3c5
/avatars/4b3655264f8119f3a9f22bd547f0100b.svg
Manoj Middepogu
user
manoj1729
667a74223463f1209e7ec994
Sai Charitha Akula
667a74223463f1209e7ec995
Jihan Yang
admin_assigned
2024-06-25T08:37:17.179Z
6304baf041387c7f1177a5d2
/avatars/795c63f2394080eec78ca7981d4a1f78.svg
Jihan Yang
user
jihanyang
667a74223463f1209e7ec996
Shusheng Yang
admin_assigned
2024-06-25T08:37:31.788Z
627ccf058b4e56cfc2716425
https://cdn-avatars.huggingface.co/v1/production/uploads/1652346592327-noauth.jpeg
Shusheng Yang
user
ShushengYang
667a74223463f1209e7ec997
Adithya Iyer
claimed_verified
2024-06-26T08:05:01.569Z
63fe5fb31193ba2c229c73bf
/avatars/b5062ebc056f0659db12c6f6f463984b.svg
Adithya Iyer
user
Adithyaiyer1999
667a74223463f1209e7ec998
Xichen Pan
admin_assigned
2024-06-25T08:38:09.129Z
63172831c92fd6fee3181f50
/avatars/0f57068a138cb181e9451bfc1ed3d1c0.svg
Xichen Pan
user
xcpan
667a74223463f1209e7ec999
Austin Wang
claimed_verified
2024-10-14T07:49:07.327Z
64249f76d476e4ad55665d59
/avatars/a0fec7e423ffae944a874ca267b55c1f.svg
Ziteng Wang
user
AustinWang0330
667a74223463f1209e7ec99a
Rob Fergus
667a74223463f1209e7ec99b
Yann LeCun
admin_assigned
2024-06-25T08:39:07.946Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
667a74223463f1209e7ec99c
Saining Xie
admin_assigned
2024-06-25T08:39:00.996Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
vision-centric approach. While stronger language models can enhance multimodal
capabilities, the design choices for vision components are often insufficiently
explored and disconnected from visual representation learning research. This
gap hinders accurate sensory grounding in real-world scenarios. Our study uses
LLMs and visual instruction tuning as an interface to evaluate various visual
representations, offering new insights into different models and architectures
-- self-supervised, strongly supervised, or combinations thereof -- based on
experiments with over 20 vision encoders. We critically examine existing MLLM
benchmarks, addressing the difficulties involved in consolidating and
interpreting results from various tasks, and introduce a new vision-centric
benchmark, CV-Bench. To further improve visual grounding, we propose the
Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
integrates high-resolution vision features with LLMs while reducing the number
of tokens. Additionally, we discuss the curation of high-quality visual
instruction-tuning data from publicly available sources, emphasizing the
importance of data source balancing and distribution ratio. Collectively,
Cambrian-1 not only achieves state-of-the-art performance but also serves as a
comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
weights, code, supporting tools, datasets, and detailed instruction-tuning and
evaluation recipes. We hope our release will inspire and accelerate
advancements in multimodal systems and visual representation learning.
Cambrian-1, a vision-centric multimodal LLM family, introduces the Spatial Vision Aggregator (SVA) and a new benchmark CV-Bench to enhance visual grounding and representation learning in multimodal systems.
multimodal LLMs vision-centric approach visual representation learning visual instruction tuning vision encoders MLLM benchmarks Spatial Vision Aggregator CV-Bench visual grounding