See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language
  Balance to Mitigate Dominant Modality Bias
Vision-language (VL) models have demonstrated strong performance across
various tasks. However, these models often rely on a specific modality for
predictions, leading to "dominant modality bias.'' This bias significantly
hurts performance, especially when one modality is impaired. In this study, we
analyze model behavior under dominant modality bias and theoretically show that
unaligned gradients or differences in gradient magnitudes prevent balanced
convergence of the loss. Based on these findings, we propose a novel framework,
BalGrad to mitigate dominant modality bias. Our approach includes
inter-modality gradient reweighting, adjusting the gradient of KL divergence
based on each modality's contribution, and inter-task gradient projection to
align task directions in a non-conflicting manner. Experiments on UPMC
Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively
alleviates over-reliance on specific modalities when making predictions.
67dcf6456b575dc3179e05a2
JuneHyoung Kwon
admin_assigned
2025-03-21T13:08:35.264Z
6756f15abba07064c254d8d3
/avatars/0b7067c2475d5cb1950579cbb98b54f3.svg
June Hyoung Kwon
user
herbwood27
67dcf6456b575dc3179e05a3
MiHyeon Kim
67dcf6456b575dc3179e05a4
Eunju Lee
67dcf6456b575dc3179e05a5
Juhwan Choi
admin_assigned
2025-03-21T13:08:21.660Z
65646b22ac9d3c2bd7b14788
/avatars/0bf19dcfa568a694361fb3a63b999997.svg
Juhwan Choi
user
c-juhwan
67dcf6456b575dc3179e05a6
YoungBin Kim
See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language
  Balance to Mitigate Dominant Modality Bias
Vision-language (VL) models have demonstrated strong performance across
various tasks. However, these models often rely on a specific modality for
predictions, leading to "dominant modality bias.'' This bias significantly
hurts performance, especially when one modality is impaired. In this study, we
analyze model behavior under dominant modality bias and theoretically show that
unaligned gradients or differences in gradient magnitudes prevent balanced
convergence of the loss. Based on these findings, we propose a novel framework,
BalGrad to mitigate dominant modality bias. Our approach includes
inter-modality gradient reweighting, adjusting the gradient of KL divergence
based on each modality's contribution, and inter-task gradient projection to
align task directions in a non-conflicting manner. Experiments on UPMC
Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively
alleviates over-reliance on specific modalities when making predictions.
A framework called BalGrad is introduced to mitigate dominant modality bias in vision-language models by reweighting and aligning gradients from different modalities.
modality bias unaligned gradients gradient magnitudes inter-modality gradient reweighting KL divergence inter-task gradient projection UPMC Food-101 Hateful Memes MM-IMDb