Towards Physics-informed Visual-Spatial Intelligence with Human Priors: An Autonomous Driving Study
Guanlin (Frank) Wu, Boyan Su, Yang Zhao, Pu Wang, Yichen Lin, Hao Frank Yang
How to encode visual-spatial intelligence (VSI) into representative and informative features remain an open challenge. Instead of following traditional Visual Question Answering (VQA)-style representation, we introduce spatial intelligence grid (SIG): a structured, grid-based data schema that embeds geometrical spatial relationship among objects along with physical priors in human world. We further derive a set of SIG-based optimal evaluation metrics that rigorously quantify a model’s true VSI capabilities. In few-shot in-context learning experiments on state-of-the-art multimodal LLMs (e.g. GPT-4o, Gemini-2.5-Pro), SIG yields consistently larger, more stable, and more comprehensive improvements across all VSI metrics compared to VQA-style representations, demonstrating its potential as a novel data schema for learning VSI. Based on SIG, we create SIGBench, a benchmark containing 1.4K driving frames annotated with ground-truth SIG labels and human gaze attention, supporting both grid-based machine VSI tasks and human-like attention-driven VSI tasks in autonomous-driving scenarios.