Scaling Laws for Robust Comparison of Open Foundation Language-Vision
  Models and Datasets
In studies of transferable learning, scaling laws are obtained for various
important foundation models to predict their properties and performance at
larger scales. We show here how scaling law derivation can also be used for
model and dataset comparison, allowing to decide which procedure is to be
preferred for pre-training. For the first time, full scaling laws based on
dense measurements across a wide span of model and samples seen scales are
derived for two important language-vision learning procedures, CLIP and MaMMUT,
that use either contrastive only or contrastive and captioning text generative
loss. Ensuring sufficient prediction accuracy for held out points, we use
derived scaling laws to compare both models, obtaining evidence for MaMMUT's
stronger improvement with scale and better sample efficiency than standard
CLIP. To strengthen validity of the comparison, we show scaling laws for
various downstream tasks, classification, retrieval, and segmentation, and for
different open datasets, DataComp, DFN and Re-LAION, observing consistently the
same trends. We show that comparison can also be performed when deriving
scaling laws with a constant learning rate schedule, reducing compute cost.
Accurate derivation of scaling laws provides thus means to perform model and
dataset comparison across scale spans, avoiding misleading conclusions based on
measurements from single reference scales only, paving the road for systematic
comparison and improvement of open foundation models and datasets for their
creation. We release all the pre-trained models with their intermediate
checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot
ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for
reproducing experiments in the paper and raw experiments data can be found at
https://github.com/LAION-AI/scaling-laws-for-comparison.
68423f5f0a25ba60aa257550
Marianna Nezhurina
claimed_verified
2025-06-07T05:49:11.910Z
62500f361684e0335e527bc6
/avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
Marianna Nezhurina
user
marianna13
68423f5f0a25ba60aa257551
Tomer Porian
68423f5f0a25ba60aa257552
Giovanni Pucceti
68423f5f0a25ba60aa257553
Tommie Kerssies
68423f5f0a25ba60aa257554
Romain Beaumont
68423f5f0a25ba60aa257555
Mehdi Cherti
claimed_verified
2025-06-09T10:12:34.895Z
61af95fec9ba387cb63f4537
https://cdn-avatars.huggingface.co/v1/production/uploads/61af95fec9ba387cb63f4537/dRl6Orv_MxoUzocEzX2Lc.png
Mehdi Cherti
user
mehdidc
68423f5f0a25ba60aa257556
Jenia Jitsev
claimed_verified
2025-06-16T07:17:49.036Z
6355b485b8b79340d4630dd5
https://cdn-avatars.huggingface.co/v1/production/uploads/6355b485b8b79340d4630dd5/HIZO4ybweRy48VdCtk2MB.jpeg
Jenia Jitsev
user
JJitsev
Scaling Laws for Robust Comparison of Open Foundation Language-Vision
  Models and Datasets
In studies of transferable learning, scaling laws are obtained for various
important foundation models to predict their properties and performance at
larger scales. We show here how scaling law derivation can also be used for
model and dataset comparison, allowing to decide which procedure is to be
preferred for pre-training. For the first time, full scaling laws based on
dense measurements across a wide span of model and samples seen scales are
derived for two important language-vision learning procedures, CLIP and MaMMUT,
that use either contrastive only or contrastive and captioning text generative
loss. Ensuring sufficient prediction accuracy for held out points, we use
derived scaling laws to compare both models, obtaining evidence for MaMMUT's
stronger improvement with scale and better sample efficiency than standard
CLIP. To strengthen validity of the comparison, we show scaling laws for
various downstream tasks, classification, retrieval, and segmentation, and for
different open datasets, DataComp, DFN and Re-LAION, observing consistently the
same trends. We show that comparison can also be performed when deriving
scaling laws with a constant learning rate schedule, reducing compute cost.
Accurate derivation of scaling laws provides thus means to perform model and
dataset comparison across scale spans, avoiding misleading conclusions based on
measurements from single reference scales only, paving the road for systematic
comparison and improvement of open foundation models and datasets for their
creation. We release all the pre-trained models with their intermediate
checkpoints, including openMaMMUT-L/14, which achieves 80.3% zero-shot
ImageNet-1k accuracy, trained on 12.8B samples from DataComp-1.4B. Code for
reproducing experiments in the paper and raw experiments data can be found at
https://github.com/LAION-AI/scaling-laws-for-comparison.
Scaling laws are derived for CLIP and MaMMUT to compare their performance and sample efficiency across different scales and datasets.
scaling laws transferable learning CLIP MaMMUT contrastive loss captioning text generative loss model comparison dataset comparison zero-shot accuracy DataComp DFN Re-LAION learning rate schedule