Does Data Scaling Lead to Visual Compositional Generalization?
Compositional understanding is crucial for human intelligence, yet it remains
unclear whether contemporary vision models exhibit it. The dominant machine
learning paradigm is built on the premise that scaling data and model sizes
will improve out-of-distribution performance, including compositional
generalization. We test this premise through controlled experiments that
systematically vary data scale, concept diversity, and combination coverage. We
find that compositional generalization is driven by data diversity, not mere
data scale. Increased combinatorial coverage forces models to discover a
linearly factored representational structure, where concepts decompose into
additive components. We prove this structure is key to efficiency, enabling
perfect generalization from few observed combinations. Evaluating pretrained
models (DINO, CLIP), we find above-random yet imperfect performance, suggesting
partial presence of this structure. Our work motivates stronger emphasis on
constructing diverse datasets for compositional generalization, and considering
the importance of representational structure that enables efficient
compositional learning. Code available at
https://github.com/oshapio/visual-compositional-generalization.
686f1302d938c25d68441af3
Arnas Uselis
claimed_verified
2025-07-10T07:09:35.377Z
6520898f7bf8cc2dd28b7a9c
/avatars/87a29ba95b71ee2dce18e97aa85e17a1.svg
Arnas Uselis
user
Gigglingface
686f1302d938c25d68441af4
Andrea Dittadi
686f1302d938c25d68441af5
Seong Joon Oh
Does Data Scaling Lead to Visual Compositional Generalization?
Compositional understanding is crucial for human intelligence, yet it remains
unclear whether contemporary vision models exhibit it. The dominant machine
learning paradigm is built on the premise that scaling data and model sizes
will improve out-of-distribution performance, including compositional
generalization. We test this premise through controlled experiments that
systematically vary data scale, concept diversity, and combination coverage. We
find that compositional generalization is driven by data diversity, not mere
data scale. Increased combinatorial coverage forces models to discover a
linearly factored representational structure, where concepts decompose into
additive components. We prove this structure is key to efficiency, enabling
perfect generalization from few observed combinations. Evaluating pretrained
models (DINO, CLIP), we find above-random yet imperfect performance, suggesting
partial presence of this structure. Our work motivates stronger emphasis on
constructing diverse datasets for compositional generalization, and considering
the importance of representational structure that enables efficient
compositional learning. Code available at
https://github.com/oshapio/visual-compositional-generalization.
Compositional generalization in vision models is driven by data diversity rather than scale, and is facilitated by a linearly factored representational structure.
compositional understanding compositional generalization data diversity combinatorial coverage linearly factored representational structure DINO CLIP