Scaling Laws for Comparison of Open Foundation Language-Vision Models and Datasets
Marianna Nezhurina, Tomer Porian, Giovanni Puccetti, Tommie Kerssies, Romain Beaumont, Mehdi Cherti, Jenia Jitsev
In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. We show here how scaling law derivation can also be used for model and dataset comparison, allowing to decide which procedure is to be preferred for pre-training. For the first time, full scaling laws based on dense measurements across wide span of model and samples seen scales are derived for two important language-vision learning procedures, CLIP and MaMMUT, that use either contrastive only or contrastive and captioning text generative loss. Ensuring sufficient prediction accuracy for held out points, we use derived scaling laws to compare both models, obtaining evidence for MaMMUT stronger improvement with scale and better sample efficiency than standard CLIP. To strengthen validity of the comparison, we show scaling laws for various downstream tasks, looking at classification, retrieval, and segmentation, and for two different open datasets, DataComp-1.4B and Re-LAION-1.4B, observing consistently same trends. We show that comparison can be also performed when deriving scaling laws with constant learning rate schedule that reduces compute cost. Accurate derivation of scaling laws provides thus means to perform model and dataset comparison across scale span that avoids misleading conclusions based on measurements from single reference scales only, paving the road for systematic comparison and improvement of open foundation models and datasets for their creation. We release all the pre-trained models with their intermediate checkpoints, including openMammut-L/14, which achieves $80.3$% zero-shot ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code and data to reproduce findings will be released at \url{https://anonymous.4open.science/anonymize/open_clip_scaling-76D2}