Enhancing Cognition and Explainability of Multimodal Foundation Models
  with Self-Synthesized Data
Large multimodal models (LMMs) have shown impressive capabilities in a wide
range of visual tasks. However, they often struggle with fine-grained visual
reasoning, failing to identify domain-specific objectives and provide
justifiable explanations for their predictions. To address this, we propose a
novel visual rejection sampling framework to improve the cognition and
explainability of LMMs using self-synthesized data. Specifically, visual
fine-tuning requires images, queries, and target answers. Our approach begins
by synthesizing interpretable answers that include human-verifiable visual
features. These features are based on expert-defined concepts, carefully
selected based on their alignment with the image content. After each round of
fine-tuning, we apply a reward model-free filtering mechanism to select the
highest-quality interpretable answers for the next round of tuning. This
iterative process of data synthesis and fine-tuning progressively improves the
model's ability to generate accurate and reasonable explanations. Experimental
results demonstrate the effectiveness of our method in improving both the
accuracy and explainability of specialized visual classification tasks.
67b87e3a346553e4006bf37c
Yucheng Shi
admin_assigned
2025-02-21T15:14:19.237Z
64beb6b6140491ca9f803ebf
/avatars/0daa2e813a13668b8b708cd8c12763d9.svg
Yucheng SHi
user
YuchengShi
67b87e3a346553e4006bf37d
Quanzheng Li
67b87e3a346553e4006bf37e
Jin Sun
admin_assigned
2025-02-21T15:15:13.205Z
6409cde91ee054d66a66c817
/avatars/ed18aa9902760a7ad6e9c5789b26dbe3.svg
jin sun
user
jinsun
67b87e3a346553e4006bf37f
Xiang Li
67b87e3a346553e4006bf380
Ninghao Liu
Enhancing Cognition and Explainability of Multimodal Foundation Models
  with Self-Synthesized Data
Large multimodal models (LMMs) have shown impressive capabilities in a wide
range of visual tasks. However, they often struggle with fine-grained visual
reasoning, failing to identify domain-specific objectives and provide
justifiable explanations for their predictions. To address this, we propose a
novel visual rejection sampling framework to improve the cognition and
explainability of LMMs using self-synthesized data. Specifically, visual
fine-tuning requires images, queries, and target answers. Our approach begins
by synthesizing interpretable answers that include human-verifiable visual
features. These features are based on expert-defined concepts, carefully
selected based on their alignment with the image content. After each round of
fine-tuning, we apply a reward model-free filtering mechanism to select the
highest-quality interpretable answers for the next round of tuning. This
iterative process of data synthesis and fine-tuning progressively improves the
model's ability to generate accurate and reasonable explanations. Experimental
results demonstrate the effectiveness of our method in improving both the
accuracy and explainability of specialized visual classification tasks.
A visual rejection sampling framework enhances the explainability and accuracy of large multimodal models through iterative synthesis and filtering of interpretable answers.
large multimodal models visual rejection sampling fine-grained visual reasoning self-synthesized data visual fine-tuning interpretable answers human-verifiable visual features reward model-free filtering mechanism