Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Hua Ye, Hang Ding, Siyuan Chen, Yiyang Jiang, changyuan zhang, Xuan Zhang
Most multimodal models treat every negative pair alike, ignoring the ambiguous negatives that differ from the positive by only a small detail.  We propose Boundary-A ware Curriculum with Local Attention(BACL), a lightweight add-on that turns these borderline cases into a curriculum signal.  A Boundary-aware Negative Sampler gradually raises difficulty, while a Contrastive Local Attention loss highlights where the mismatch occurs.  The two modules are fully differentiable and work with any off-the-shelf dual encoder.  Theory predicts a fast $\tilde{\mathcal{O}}(1/n)$ error rate; practice shows up to +32 \% R@1 over CLIP and new SOTA on four large-scale benchmarks, all without extra labels.