IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as
  Agentic Inverse Rendering
Vision-language models (VLMs) excel at descriptive tasks, but whether they
truly understand scenes from visual observations remains uncertain. We
introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding
through active creation rather than passive recognition. Grounded in the
analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs)
with actively using programming and rendering tools to recreate the underlying
3D structure of an input image, achieving agentic inverse rendering through
tool use. This "understanding-by-creating" approach probes the tool-using
generative capacity of VLAs, moving beyond the descriptive or conversational
capacity measured by traditional scene understanding benchmarks. We provide a
comprehensive suite of metrics to evaluate geometric accuracy, spatial
relations, appearance attributes, and overall plausibility. Initial experiments
on agentic inverse rendering powered by various state-of-the-art VLMs highlight
current limitations, particularly in visual precision rather than basic tool
usage. IR3D-Bench, including data and evaluation protocols, is released to
facilitate systematic study and development of tool-using VLAs towards genuine
scene understanding by creating.
68653d00d706a6e48024af8f
Parker Liu
claimed_verified
2025-07-02T14:11:22.217Z
647896de5bf35e70ab5da887
/avatars/50a874a0048047e51f25746c5fbe85bb.svg
Liu Hengyu
user
Piang
68653d00d706a6e48024af90
Chenxin Li
68653d00d706a6e48024af91
Zhengxin Li
68653d00d706a6e48024af92
Yipeng Wu
68653d00d706a6e48024af93
Wuyang Li
68653d00d706a6e48024af94
Zhiqin Yang
claimed_verified
2025-09-09T13:51:54.470Z
659cf9791c8b66637e3de72d
/avatars/7d26710f687be9444796980662614f16.svg
zhiqin yang
user
visity
68653d00d706a6e48024af95
Zhenyuan Zhang
68653d00d706a6e48024af96
Yunlong Lin
68653d00d706a6e48024af97
Sirui Han
68653d00d706a6e48024af98
Brandon Y. Feng
IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as
  Agentic Inverse Rendering
Vision-language models (VLMs) excel at descriptive tasks, but whether they
truly understand scenes from visual observations remains uncertain. We
introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding
through active creation rather than passive recognition. Grounded in the
analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs)
with actively using programming and rendering tools to recreate the underlying
3D structure of an input image, achieving agentic inverse rendering through
tool use. This "understanding-by-creating" approach probes the tool-using
generative capacity of VLAs, moving beyond the descriptive or conversational
capacity measured by traditional scene understanding benchmarks. We provide a
comprehensive suite of metrics to evaluate geometric accuracy, spatial
relations, appearance attributes, and overall plausibility. Initial experiments
on agentic inverse rendering powered by various state-of-the-art VLMs highlight
current limitations, particularly in visual precision rather than basic tool
usage. IR3D-Bench, including data and evaluation protocols, is released to
facilitate systematic study and development of tool-using VLAs towards genuine
scene understanding by creating.
IR3D-Bench evaluates vision-language agents' understanding of scenes by requiring them to recreate 3D structures using programming and rendering tools, providing metrics for geometric accuracy, spatial relations, appearance attributes, and plausibility.
vision-language models VLMs Vision-Language Agents VLAs analysis-by-synthesis agentic inverse rendering tool-using generative capacity geometric accuracy spatial relations appearance attributes plausibility