PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?
Atharva Gundawar, Som Sagar, Ransalu Senanayake
Vision-Language Models (VLMs) are increasingly pivotal for generalist robot ma-nipulation, enabling tasks such as physical reasoning, policy generation, and failuredetection. However, their proficiency in these high-level applications often assumesa deep understanding of low-level physical prerequisites, a capability that is largelyunverified. To perform actions reliably, robots must comprehend intrinsic objectproperties (e.g., material, weight), action affordances (e.g., graspable, stackable),and physical constraints (e.g., stability, reachability, or an object’s state like beingclosed). Despite their ubiquitous use in manipulation, we argue that off-the-shelfVLMs may lack this granular, physically-grounded understanding, as these specificprerequisites are often overlooked in their pre-training. Addressing this criticalgap, we introduce PAC Bench, a comprehensive benchmark designed to system-atically evaluate VLM comprehension of these core Properties, Affordances, andConstraints (PAC) from a task executability perspective. PAC Bench features adiverse dataset with over 30,000 annotations, comprising 673 real-world images(115 object classes, 15 property types, 1–3 affordances defined per class), 100real-world humanoid-view scenarios and 120 unique simulated constraint scenariosacross four tasks. Our evaluations reveal significant gaps in the ability of VLMsto grasp fundamental physical concepts, underscoring their current limitations forreliable robot manipulation and pointing to key areas that require targeted research.PAC Bench also serves as a standardized benchmark for rigorously evaluating VLMphysical reasoning and guiding the development of more robust and physicallygrounded models for robotic manipulation.