Rebalancing Contrastive Alignment with Learnable Semantic Gaps in Text-Video Retrieval
Jian Xiao, Zijie Song, Jialong Hu, Hao Cheng, Jia Li, Zhenzhen Hu, Richang Hong
Recent advances in text-video retrieval have been largely driven by contrastive learning frameworks. However, existing methods overlook a key source of optimization tension: the separation between text and video distributions in the representation space—referred to as the modality gap—and the prevalence of false negatives in batch sampling. These factors lead to conflicting gradients under the InfoNCE loss, impeding stable alignment. To mitigate this, we propose GARE—a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $\Delta_{ij}$ between text $t_i$ and video $v_j$ to offload the tension from the global anchor representation. We first derive the ideal form of $\Delta_{ij}$ via a coupling multivariate first-order Taylor approximation of the InfoNCE loss under a trust-region constraint, revealing it as a key mechanism for resolving gradient conflicts by guiding updates along a locally optimal descent direction in the coupled optimization landscape. Due to the expensive cost of directly approximate $\Delta_{ij}$, we introduce a lightweight neural module conditioned on the semantic gap between each video-text pair, enabling structure-aware correction guided by gradient supervision. To further stabilize learning and promote interpretability, we regularize $\Delta$ via three components: a trust-region constraint regularization to prevent oscillations, a directional diversity term to expand the semantic difference space, and an information bottleneck over $\Delta$ to restrict redundant information. Experiments across four retrieval benchmarks show that GARE consistently improves alignment accuracy and robustness to noisy supervision, confirming the effectiveness of gap-aware tension unloading.