Balanced Audiovisual Dataset for Imbalance Analysis
The imbalance problem is widespread in the field of machine learning, which
also exists in multimodal learning areas caused by the intrinsic discrepancy
between modalities of samples. Recent works have attempted to solve the
modality imbalance problem from algorithm perspective, however, they do not
fully analyze the influence of modality bias in datasets. Concretely, existing
multimodal datasets are usually collected under specific tasks, where one
modality tends to perform better than other ones in most conditions. In this
work, to comprehensively explore the influence of modality bias, we first split
existing datasets into different subsets by estimating sample-wise modality
discrepancy. We surprisingly find that: the multimodal models with existing
imbalance algorithms consistently perform worse than the unimodal one on
specific subsets, in accordance with the modality bias. To further explore the
influence of modality bias and analyze the effectiveness of existing imbalance
algorithms, we build a balanced audiovisual dataset, with uniformly distributed
modality discrepancy over the whole dataset. We then conduct extensive
experiments to re-evaluate existing imbalance algorithms and draw some
interesting findings: existing algorithms only provide a compromise between
modalities and suffer from the large modality discrepancy of samples. We hope
that these findings could facilitate future research on the modality imbalance
problem.