BBA: Bi-Modal Behavioral Alignment for Reasoning with Large
  Vision-Language Models
Multimodal reasoning stands as a pivotal capability for large vision-language
models (LVLMs). The integration with Domain-Specific Languages (DSL), offering
precise visual representations, equips these models with the opportunity to
execute more accurate reasoning in complex and professional domains. However,
the vanilla Chain-of-Thought (CoT) prompting method faces challenges in
effectively leveraging the unique strengths of visual and DSL representations,
primarily due to their differing reasoning mechanisms. Additionally, it often
falls short in addressing critical steps in multi-step reasoning tasks. To
mitigate these challenges, we introduce the Bi-Modal
Behavioral Alignment (BBA) prompting method, designed
to maximize the potential of DSL in augmenting complex multi-modal reasoning
tasks. This method initiates by guiding LVLMs to create separate reasoning
chains for visual and DSL representations. Subsequently, it aligns these chains
by addressing any inconsistencies, thus achieving a cohesive integration of
behaviors from different modalities. Our experiments demonstrate that BBA
substantially improves the performance of GPT-4V(ision) on geometry problem
solving (28.34% to 34.22%), chess positional advantage prediction
(42.08% to 46.99%) and molecular property prediction (77.47% to
83.52%).
65d6c473b20efa5a5f390a90
Xueliang Zhao
65d6c473b20efa5a5f390a91
Xinting Huang
claimed_verified
2024-05-14T13:51:51.164Z
63c227a6c58fcfeac18dec07
/avatars/47df92c937cfaf8d726c7e28cf352eb5.svg
tim huang
user
timmyhhh
65d6c473b20efa5a5f390a92
Tingchen Fu
claimed_verified
2025-05-22T07:20:10.407Z
640ad17a1ee054d66a74783e
https://cdn-avatars.huggingface.co/v1/production/uploads/640ad17a1ee054d66a74783e/u0PjIkyC-9HkGzEyUQ7JN.jpeg
Tingchen Fu
user
TingchenFu
65d6c473b20efa5a5f390a93
Qintong Li
65d6c473b20efa5a5f390a94
Shansan Gong
claimed_verified
2024-10-24T07:16:38.473Z
628c83d186fc004b14e1ed48
/avatars/05ff943a9b89b5f67c5bc254bf45b8f5.svg
Shansan Gong
user
Sansa
65d6c473b20efa5a5f390a95
Lemao Liu
65d6c473b20efa5a5f390a96
Wei Bi
65d6c473b20efa5a5f390a97
Lingpeng Kong
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large
  Vision-Language Models
Multimodal reasoning stands as a pivotal capability for large vision-language
models (LVLMs). The integration with Domain-Specific Languages (DSL), offering
precise visual representations, equips these models with the opportunity to
execute more accurate reasoning in complex and professional domains. However,
the vanilla Chain-of-Thought (CoT) prompting method faces challenges in
effectively leveraging the unique strengths of visual and DSL representations,
primarily due to their differing reasoning mechanisms. Additionally, it often
falls short in addressing critical steps in multi-step reasoning tasks. To
mitigate these challenges, we introduce the Bi-Modal
Behavioral Alignment (BBA) prompting method, designed
to maximize the potential of DSL in augmenting complex multi-modal reasoning
tasks. This method initiates by guiding LVLMs to create separate reasoning
chains for visual and DSL representations. Subsequently, it aligns these chains
by addressing any inconsistencies, thus achieving a cohesive integration of
behaviors from different modalities. Our experiments demonstrate that BBA
substantially improves the performance of GPT-4V(ision) on geometry problem
solving (28.34% to 34.22%), chess positional advantage prediction
(42.08% to 46.99%) and molecular property prediction (77.47% to
83.52%).
The BBA prompting method enhances multimodal reasoning in vision-language models by integrating visual and DSL representations through separate reasoning chains and behavioral alignment, improving performance across various complex tasks.
vision-language models multimodal reasoning Domain-Specific Languages Chain-of-Thought BBA prompting method reasoning chains behavioral alignment GPT-4V geometry problem solving chess positional advantage prediction molecular property prediction