TaiwanVQA: Benchmarking and Enhancing Cultural Understanding in Vision-Language Models
Hsin Yi Hsieh, Shang-Wei Liu, Chang Meng, Chien-Hua Chen, Shuo-Yueh Lin, Hung-Ju Lin, Hen-Hsen Huang, I-Chen Wu
Vision-language models (VLMs) struggle with culturally specific content, an issue often overlooked as existing benchmarks focus on dominant languages and globalized datasets. We introduce TaiwanVQA, a VQA benchmark designed for Taiwanese culture to evaluate recognition and reasoning in regional contexts. TaiwanVQA contains 2,736 images and 5,472 manually designed questions on topics like traditional food, public signs, festivals, and landmarks. The official benchmark set includes 1,000 images and 2,000 questions for systematic assessment, with the remainder of the data used as training material. Tests on state-of-the-art VLMs show strong visual recognition but weaknesses in cultural reasoning. We propose a data augmentation strategy combining human-annotated and synthesized dialogues to improve cultural understanding. Fine-tuning yields significant gains on TaiwanVQA while maintaining stable performance on other multimodal tasks. TaiwanVQA offers a scalable framework for building culturally grounded AI models in low-resource cultures, promoting diversity and fairness in multimodal AI.Our dataset and code are publicly available on [Hugging Face](https://huggingface.co/datasets/hhhuang/TaiwanVQA) and [GitHub](https://github.com/hhhuang/TaiwanVQA).