How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model
HEE BIN YOO, Sungyoon Lee, Cheongjae Jang, Dong-Sig Han, Jaein Kim, Seunghyeon Lim, Byoung-Tak Zhang
Neural networks learn effective feature representations, enabling feature transfer to new tasks without additional training.While larger datasets are known to improve feature transfer, the theoretical conditions for their success remain unclear.This work investigates feature transfer in classifier-trained networks to identify the conditions that enable effective clustering in unseen classes.We first reveal higher similarity between training and unseen distributions to improve Cohesion and Separability.We then show enhanced feature expressiveness when inputs are similar to the training classes, while the features of irrelevant inputs are indistinguishable.We validate our analysis on synthetic and benchmark datasets, including CAR, CUB, SOP, ISC, and ImageNet.Our analysis highlights the importance of training class similarity to the input distribution for successful feature transfer.