HoneyBee: Data Recipes for Vision-Language Reasoners
Recent advances in vision-language models (VLMs) have made them highly
effective at reasoning tasks. However, the principles underlying the
construction of performant VL reasoning training datasets remain poorly
understood. In this work, we introduce several data curation approaches and
study their impacts on VL reasoning capabilities by carefully controlling
training and evaluation setups. We analyze the effects of context (image and
question pair) sources, implement targeted data interventions, and explore
scaling up images, questions, and chain-of-thought (CoT) solutions. Our
findings reveal that (a) context source strategies significantly affect VLM
performance, (b) interventions such as auxiliary signals from image captions
and the inclusion of text-only reasoning yield substantial gains, and (c)
scaling all data dimensions (e.g., unique questions per image and unique CoTs
per image-question pair) consistently improves reasoning capability. Motivated
by these insights, we introduce HoneyBee, a large-scale, high-quality CoT
reasoning dataset with 2.5M examples consisting 350K image-question pairs. VLMs
trained with HoneyBee outperform state-of-the-art models across model sizes.
For instance, a HoneyBee-trained VLM with 3B parameters outperforms the SOTA
model and the base model by 7.8% and 24.8%, respectively, on MathVerse.
Furthermore, we propose a test-time scaling strategy that reduces decoding cost
by 73% without sacrificing accuracy. Overall, this work presents improved
strategies for VL reasoning dataset curation research.
68ef03f8486b78128f0e33ee
Hritik Bansal
68ef03f8486b78128f0e33ef
Devandra Singh Sachan
68ef03f8486b78128f0e33f0
Kai-Wei Chang
68ef03f8486b78128f0e33f1
Aditya Grover
68ef03f8486b78128f0e33f2
Gargi Ghosh
68ef03f8486b78128f0e33f3
Wen-tau Yih
68ef03f8486b78128f0e33f4
Ramakanth Pasunuru
HoneyBee: Data Recipes for Vision-Language Reasoners
Recent advances in vision-language models (VLMs) have made them highly
effective at reasoning tasks. However, the principles underlying the
construction of performant VL reasoning training datasets remain poorly
understood. In this work, we introduce several data curation approaches and
study their impacts on VL reasoning capabilities by carefully controlling
training and evaluation setups. We analyze the effects of context (image and
question pair) sources, implement targeted data interventions, and explore
scaling up images, questions, and chain-of-thought (CoT) solutions. Our
findings reveal that (a) context source strategies significantly affect VLM
performance, (b) interventions such as auxiliary signals from image captions
and the inclusion of text-only reasoning yield substantial gains, and (c)
scaling all data dimensions (e.g., unique questions per image and unique CoTs
per image-question pair) consistently improves reasoning capability. Motivated
by these insights, we introduce HoneyBee, a large-scale, high-quality CoT
reasoning dataset with 2.5M examples consisting 350K image-question pairs. VLMs
trained with HoneyBee outperform state-of-the-art models across model sizes.
For instance, a HoneyBee-trained VLM with 3B parameters outperforms the SOTA
model and the base model by 7.8% and 24.8%, respectively, on MathVerse.
Furthermore, we propose a test-time scaling strategy that reduces decoding cost
by 73% without sacrificing accuracy. Overall, this work presents improved
strategies for VL reasoning dataset curation research.