In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
Ethan Hwang, Hossein Adeli, Wenxuan Guo, Andrew Luo, Nikolaus Kriegeskorte
A fine-grained understanding of functional selectivity in human visual cortex is fundamental for elucidating how visual information is processed and represented in the brain. While recent work utilizing controlled stimuli has uncovered categorically selective regions, these approaches apply strong assumptions about the stimulus space and may bias the analysis to preconceived notions of selectivity. Approaches that employ simple encoding models lack the capacity to capture the complexity of neural responses under naturalistic conditions. We propose a novel in silico approach based on an encoder-decoder transformer that leverages a brain-region to image-feature cross-attention mechanism, designed to non-linearly align high-dimensional deep network features with cortical responses. Unlike traditional encoding models that apply linear regression to deep-network activations---which assume cortical activity is a fixed affine transformation of artificial representations---our architecture learns to flexibly bind stimulus features with semantic patterns of fMRI activity in an end-to-end manner. Coupled with diffusion-based image generative models and large image datasets, this method can synthesize and select images that can maximally activate different cortical parcels. We show this approach can uncover regions with complex compositional selectivity involving diverse semantic concepts. Together, these results demonstrate a new data-driven paradigm for discovery of visual selectivity in the brain.