I-Con: A Unifying Framework for Representation Learning
As the field of representation learning grows, there has been a proliferation
of different loss functions to solve different classes of problems. We
introduce a single information-theoretic equation that generalizes a large
collection of modern loss functions in machine learning. In particular, we
introduce a framework that shows that several broad classes of machine learning
methods are precisely minimizing an integrated KL divergence between two
conditional distributions: the supervisory and learned representations. This
viewpoint exposes a hidden information geometry underlying clustering, spectral
methods, dimensionality reduction, contrastive learning, and supervised
learning. This framework enables the development of new loss functions by
combining successful techniques from across the literature. We not only present
a wide array of proofs, connecting over 23 different approaches, but we also
leverage these theoretical results to create state-of-the-art unsupervised
image classifiers that achieve a +8% improvement over the prior
state-of-the-art on unsupervised classification on ImageNet-1K. We also
demonstrate that I-Con can be used to derive principled debiasing methods which
improve contrastive representation learners.
6809ba7976a4f4f7268546a7
Shaden Alshammari
6809ba7976a4f4f7268546a8
John Hershey
6809ba7976a4f4f7268546a9
Axel Feldmann
admin_assigned
2025-04-24T11:27:03.416Z
63124ad25e2531edb9edca45
/avatars/9cc2e352ba3f79347e588a3e4a814987.svg
Axel Feldmann
user
axelf
6809ba7976a4f4f7268546aa
William T. Freeman
admin_assigned
2025-04-24T11:27:14.536Z
654e866a85a5e608059a9b4f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
William Freeman
user
mrpuppt
6809ba7976a4f4f7268546ab
Mark Hamilton
admin_assigned
2025-04-24T11:27:36.305Z
62dae3734398e21bf7f53443
https://cdn-avatars.huggingface.co/v1/production/uploads/1658512237806-noauth.jpeg
Mark Hamilton
user
mhamilton723
I-Con: A Unifying Framework for Representation Learning
As the field of representation learning grows, there has been a proliferation
of different loss functions to solve different classes of problems. We
introduce a single information-theoretic equation that generalizes a large
collection of modern loss functions in machine learning. In particular, we
introduce a framework that shows that several broad classes of machine learning
methods are precisely minimizing an integrated KL divergence between two
conditional distributions: the supervisory and learned representations. This
viewpoint exposes a hidden information geometry underlying clustering, spectral
methods, dimensionality reduction, contrastive learning, and supervised
learning. This framework enables the development of new loss functions by
combining successful techniques from across the literature. We not only present
a wide array of proofs, connecting over 23 different approaches, but we also
leverage these theoretical results to create state-of-the-art unsupervised
image classifiers that achieve a +8% improvement over the prior
state-of-the-art on unsupervised classification on ImageNet-1K. We also
demonstrate that I-Con can be used to derive principled debiasing methods which
improve contrastive representation learners.
A unified framework using KL divergence between supervisory and learned representations generalizes multiple machine learning loss functions and improves unsupervised image classification and debiasing.
information-theoretic equation KL divergence conditional distributions information geometry clustering spectral methods dimensionality reduction contrastive learning supervised learning unsupervised image classification ImageNet-1K debiasing methods