Representational Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One
Itamar Avitan, Tal Golan
Linearly transforming stimulus representations of deep neural networks yieldshigh-performing models of behavioral and neural responses to complex stimuli.But does the test accuracy of such predictions identify genuine representationalalignment? We addressed this question through a large-scale model-recovery study.Twenty diverse vision models were linearly aligned to 4.5 million behavioral judg-ments from the THINGS odd-one-out dataset and calibrated to reproduce humanresponse variability. For each model in turn, we sampled synthetic responses fromits probabilistic predictions, fitted all candidate models to the synthetic data, andtested whether the data-generating model would re-emerge as the best predictorof the simulated data. Model recovery accuracy improved with training-set sizebut plateaued below 80%, even at millions of simulated trials. Regression analyseslinked misidentification primarily to shifts in representational geometry inducedby the linear transformation, as well as to the effective dimensionality of the trans-formed features. These findings demonstrate that, even with massive data, overlyflexible alignment metrics may fail to guide us toward artificial representationsthat are genuinely more human-aligned. Model comparison experiments must bedesigned to balance the trade-off between predictive accuracy and identifiability—ensuring the best-fitting model is also the right one.