Jacobian-Based Interpretation of Nonlinear Neural Encoding Model
Xiaohui Gao, Haoran Yang, Yue Cheng, Mengfei Zuo, Yiheng Liu, Peiyang Li, Xintao Hu
In recent years, the alignment between artificial neural network (ANN) embeddings and voxel-level responses in functional magnetic resonance imaging (fMRI) via neural encoding models has significantly advanced research on neural representation mechanisms and interpretability in the brain. However, these approaches remain limited in their ability to characterize the brain’s inherently nonlinear response properties. To this end, we propose a novel nonlinearity metric—Jacobian-based Nonlinearity Evaluation (JNE)—which quantifies the degree of nonlinearity through interpretability analysis of nonlinear neural encoding model. Simulation experiments demonstrated that JNE consistently achieved high accuracy in measuring nonlinearity across various activation functions and network architectures. Applying this metric to natural image fMRI data reveals that the early visual cortex exhibits low nonlinearity, whereas higher visual areas (e.g., EBA) show significantly enhanced nonlinear responses, which presents a layer-wise increasing trend from lower to higher cortical regions. Moreover, JNE uncovers input-dependent sample selectivity in neural responses, highlighting the critical role of high-level cortex in nonlinear visual processing. As the first interpretability metric for quantifying nonlinear responses, JNE provides a new tool for understanding brain processing mechanisms.