Visually Interpretable Subtask Reasoning for Visual Question Answering
Answering complex visual questions like `Which red furniture can be used for
sitting?' requires multi-step reasoning, including object recognition,
attribute filtering, and relational understanding. Recent work improves
interpretability in multimodal large language models (MLLMs) by decomposing
tasks into sub-task programs, but these methods are computationally expensive
and less accurate due to poor adaptation to target data. To address this, we
introduce VISTAR (Visually Interpretable Subtask-Aware Reasoning Model), a
subtask-driven training framework that enhances both interpretability and
reasoning by generating textual and visual explanations within MLLMs. Instead
of relying on external models, VISTAR fine-tunes MLLMs to produce structured
Subtask-of-Thought rationales (step-by-step reasoning sequences). Experiments
on two benchmarks show that VISTAR consistently improves reasoning accuracy
while maintaining interpretability. Our code and dataset will be available at
https://github.com/ChengJade/VISTAR.
6824bf46b4f1e881646d2dda
Yu Cheng
claimed_verified
2025-05-15T10:32:11.536Z
65a84987c5ffe1d019d7ac02
/avatars/1e54a4d99e98da142f0d6441df3d2d4b.svg
Yu Cheng
user
JadeCheng
6824bf46b4f1e881646d2ddb
Arushi Goel
extracted_pending
2025-05-14T16:05:27.452Z
65aa4c2ea92a64ef5b2f2ff9
/avatars/13d894fe51c011620d094c2f1a114aaf.svg
Arushi Goel
user
goarushi
6824bf46b4f1e881646d2ddc
Hakan Bilen
Visually Interpretable Subtask Reasoning for Visual Question Answering
Answering complex visual questions like `Which red furniture can be used for
sitting?' requires multi-step reasoning, including object recognition,
attribute filtering, and relational understanding. Recent work improves
interpretability in multimodal large language models (MLLMs) by decomposing
tasks into sub-task programs, but these methods are computationally expensive
and less accurate due to poor adaptation to target data. To address this, we
introduce VISTAR (Visually Interpretable Subtask-Aware Reasoning Model), a
subtask-driven training framework that enhances both interpretability and
reasoning by generating textual and visual explanations within MLLMs. Instead
of relying on external models, VISTAR fine-tunes MLLMs to produce structured
Subtask-of-Thought rationales (step-by-step reasoning sequences). Experiments
on two benchmarks show that VISTAR consistently improves reasoning accuracy
while maintaining interpretability. Our code and dataset will be available at
https://github.com/ChengJade/VISTAR.
VISTAR enhances multimodal large language models by generating structured reasoning sequences to improve accuracy and interpretability.
multimodal large language models subtask-driven training textual explanations visual explanations Subtask-of-Thought rationales step-by-step reasoning sequences