Briding the gap to real-world language-grounded visual concept learning
whie jung, Semin Kim, Junee Kim, Seunghoon Hong
Human intelligence effortlessly interprets visual scenes along a rich spectrum of semantic dimensions. However, existing approaches for language-grounded visual concept learning are limited to a few predefined primitive axes such as color and shape, and are explored in synthetic datasets.In this work, we propose a scalable framework that adaptively identifies image-related concept axes and grounds visual concepts along these axes in real-world scenes. Leveraging a pretrained vision-language model with our simple universal prompting strategy, our framework identifies a diverse image-related axes without requiring any prior knowledge. Our universal concept encoder then adaptively binds visual features to the discovered axes without introducing additional model parameters per concept. To ground visual concepts along discovered axes, we maximize the compositional consistency of concept representations, which ensures each axis to be independently manipulated without affecting other axes.We demonstrate the effectiveness of our framework on CelebA-HQ and AFHQ datasets, achieving superior editing capabilities across diverse concepts and strong compositional generalization compared to existing visual concept learning method and text-based editing methods.