ColorBench: Can VLMs See and Understand the Colorful World? A
  Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
Color plays an important role in human perception and usually provides
critical clues in visual reasoning. However, it is unclear whether and how
vision-language models (VLMs) can perceive, understand, and leverage color as
humans. This paper introduces ColorBench, an innovative benchmark meticulously
crafted to assess the capabilities of VLMs in color understanding, including
color perception, reasoning, and robustness. By curating a suite of diverse
test scenarios, with grounding in real applications, ColorBench evaluates how
these models perceive colors, infer meanings from color-based cues, and
maintain consistent performance under varying color transformations. Through an
extensive evaluation of 32 VLMs with varying language models and vision
encoders, our paper reveals some undiscovered findings: (i) The scaling law
(larger models are better) still holds on ColorBench, while the language model
plays a more important role than the vision encoder. (ii) However, the
performance gaps across models are relatively small, indicating that color
understanding has been largely neglected by existing VLMs. (iii) CoT reasoning
improves color understanding accuracies and robustness, though they are
vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on
ColorBench but they can also mislead models in some tasks. These findings
highlight the critical limitations of current VLMs and underscore the need to
enhance color comprehension. Our ColorBenchcan serve as a foundational tool for
advancing the study of human-level color understanding of multimodal AI.
67ffedb8b0c26d6ec0b608cf
Yijun Liang
67ffedb8b0c26d6ec0b608d0
Ming Li
67ffedb8b0c26d6ec0b608d1
Chenrui Fan
claimed_verified
2025-04-17T08:05:05.662Z
64a8121e35fab7cd04c30ed0
/avatars/48849b84703158772f1022932331b143.svg
Chenrui Fan
user
Fcr09
67ffedb8b0c26d6ec0b608d2
Ziyue Li
67ffedb8b0c26d6ec0b608d3
Dang Nguyen
67ffedb8b0c26d6ec0b608d4
Kwesi Cobbina
admin_assigned
2025-04-17T08:08:36.225Z
63f546e0fcf95ecac2b0ee3e
/avatars/02a401bcff91cc473d9946bbb771a985.svg
Kwesi Cobbina
user
kweCobi
67ffedb8b0c26d6ec0b608d5
Shweta Bhardwaj
admin_assigned
2025-04-17T08:08:29.680Z
639d4b8d860db464ae35c3ab
/avatars/ec0fa3e91593a03fc9fb611e66b30553.svg
Shweta Bhardwaj
user
shweta12
67ffedb8b0c26d6ec0b608d6
Jiuhai Chen
admin_assigned
2025-04-17T08:08:22.786Z
6393847e3e30234ae798b7be
/avatars/daeb8c37dff4432d837a69b87c196521.svg
JiuhaiChen
user
jiuhai
67ffedb8b0c26d6ec0b608d7
Fuxiao Liu
67ffedb8b0c26d6ec0b608d8
Tianyi Zhou
claimed_verified
2025-04-17T08:05:07.396Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
ColorBench: Can VLMs See and Understand the Colorful World? A
  Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
Color plays an important role in human perception and usually provides
critical clues in visual reasoning. However, it is unclear whether and how
vision-language models (VLMs) can perceive, understand, and leverage color as
humans. This paper introduces ColorBench, an innovative benchmark meticulously
crafted to assess the capabilities of VLMs in color understanding, including
color perception, reasoning, and robustness. By curating a suite of diverse
test scenarios, with grounding in real applications, ColorBench evaluates how
these models perceive colors, infer meanings from color-based cues, and
maintain consistent performance under varying color transformations. Through an
extensive evaluation of 32 VLMs with varying language models and vision
encoders, our paper reveals some undiscovered findings: (i) The scaling law
(larger models are better) still holds on ColorBench, while the language model
plays a more important role than the vision encoder. (ii) However, the
performance gaps across models are relatively small, indicating that color
understanding has been largely neglected by existing VLMs. (iii) CoT reasoning
improves color understanding accuracies and robustness, though they are
vision-centric tasks. (iv) Color clues are indeed leveraged by VLMs on
ColorBench but they can also mislead models in some tasks. These findings
highlight the critical limitations of current VLMs and underscore the need to
enhance color comprehension. Our ColorBenchcan serve as a foundational tool for
advancing the study of human-level color understanding of multimodal AI.
ColorBench evaluates vision-language models' color perception, reasoning, and robustness, revealing limitations and emphasizing the need for enhanced color comprehension in multimodal AI.
vision-language models VLMs color understanding color perception color reasoning color robustness scaling law CoT reasoning