Global Minimizers of Sigmoid Contrastive Loss
Kiril Bangachev, Guy Bresler, Iliyas Noman, Yury Polyanskiy
The meta-task of synchronizing representations is steadily gaining importance. Common representations to be synchronized include image and text data, student and teacher models, or multiple augmentations of the same image. In this paper we theoretically explain the advantages of synchronizing with trainable inverse temperature and bias under the sigmoid loss, as implemented in the recent SigLIP models of Google DeepMind. Temperature and bias can drive the loss function to zero for a rich class of configurations that we call $(\mathsf{m}, \mathsf{rb})$-Constellations. $(\mathsf{m}, \mathsf{rb})$-Constellations are a novel combinatorial object related to spherical codes and are parametrized by a margin $\mathsf{m}$ and relative bias $\mathsf{rb}.$ We use our characterization of constellations to theoretically justify the success of SigLIP on retrieval,to explainthe modality gap present in SigLIP,and to identify the necessary dimension for producing high-quality representations. We also propose a re-parameterization of the sigmoid loss with explicit relative bias. This parameterization yields several practical recommendations, including in the contexts of synchronizing with a locked encoder and synchronizing more than two modalities.