Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
Yanhao Jia, Ji Xie, S Jivaganesh, Li Hao, Xu Wu, Mengmi Zhang
Imagine hearing a dog bark and instinctively turning toward the sound—only to find a parked car, while a silent dog sits nearby. Such moments of sensory conflict challenge perception, yet humans flexibly resolve these discrepancies, prioritizing auditory cues over misleading visuals to accurately localize sounds.  Despite the rapid advancement of multimodal AI models that integrate vision and sound, little is known about how these systems handle cross-modal conflicts or whether they favor one modality over another. Here, we systematicly and quantitatively investigate modality bias and conflict resolution in AI sound localization. We evaluate a wide range of state-of-the-art multimodal models and compare them against human performance in carefully designed psychophysics experiments spanning six audiovisual conditions, including congruent, conflicting, and absent visual and audio cues. Our results reveal that humans consistently outperform AI in sound localization and exhibit greater robustness to conflicting or absent visual information by effectively prioritizing auditory signals. In contrast, AI shows a pronounced bias toward vision, often failing to suppress irrelevant or conflicting visual input, leading to chance-level performance. To close this gap, we fine-tune a state-of-the-art multimodal model on our carefully curated audio-image dataset synthesized via 3D simulation engines. Despite using fewer training examples, our model outperforms existing AI trained on multiple large-scale audio-visual benchmark datasets. Remarkably, consistent with neuroscience findings, our model exhibits a human-like horizontal localization bias—favoring left-right over up-down precision. This likely stems from training on stereo audio that mirrors the horizontal arrangement of human ears. These results highlight how the quality of sensory inputs and the physical structure of sensory systems jointly shape the precision of multimodal representation. All code, data, and models will be made public.