ConnectomeBench: Can LLMs proofread the connectome?
Jeff Brown, Andrew Kirjner, Annika Vivekananthan, Edward Boyden
Connectomics—the mapping of neural connections in an organism's brain—currently requires extraordinary human effort to proofread the data collected from imaging and machine-learning assisted segmentation. With the growing excitement around using AI agents to automate important scientific tasks, we explore whether current AI systems can perform multiple tasks necessary for data proofreading. We introduce ConnectomeBench, a multimodal benchmark evaluating large language model (LLM) capabilities in three critical proofreading tasks: segment type identification, split error correction, and merge error detection. Using two large open-source datasets—a cubic millimeter of mouse visual cortex and the complete Drosophila brain—we systematically evaluate proprietary multimodal LLMs including Claude 3.7 Sonnet, o4-mini, GPT-4.1, GPT-4o, as well as open source models like InternVL-3 and NVLM. Our results demonstrate that while some current models achieve suprisingly high performance in segment identification (67-78\% accuracy) and pairwise split error correction (75-81\% accuracy),  they struggle with single-shot split error correction, as well as single-shot and pairwise merge error identification. These findings suggest that while LLMs show promising capabilities, substantial improvements are needed before they can reliably augment and, eventually, replace human proofreading in connectomics.