FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D
  Object Placement
Scene generation with 3D assets presents a complex challenge, requiring both
high-level semantic understanding and low-level geometric reasoning. While
Multimodal Large Language Models (MLLMs) excel at semantic tasks, their
application to 3D scene generation is hindered by their limited grounding on 3D
geometry. In this paper, we investigate how to best work with MLLMs in an
object placement task. Towards this goal, we introduce a novel framework,
FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the
extraction of relevant geometric details from the 3D scene, (2) constructing
and solving geometric constraints on the extracted low-level geometry, and (3)
pruning for final placements that conform to common sense. By combining
geometric reasoning with real-world understanding of MLLMs, our method can
propose object placements that satisfy both geometric constraints as well as
high-level semantic common-sense considerations. Our experiments show that
these capabilities allow our method to place objects more effectively in
complex scenes with intricate geometry, surpassing the quality of prior work.
67e480296d47bc4578afc3a4
Ian Huang
claimed_verified
2025-03-27T09:03:43.079Z
639ce51c66106be14370f510
/avatars/74f68f5951fff9c02c7607ff197812aa.svg
Ian Huang
user
ianhuang
67e480296d47bc4578afc3a5
Yanan Bao
67e480296d47bc4578afc3a6
Karen Truong
67e480296d47bc4578afc3a7
Howard Zhou
67e480296d47bc4578afc3a8
Cordelia Schmid
67e480296d47bc4578afc3a9
Leonidas Guibas
67e480296d47bc4578afc3aa
Alireza Fathi
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D
  Object Placement
Scene generation with 3D assets presents a complex challenge, requiring both
high-level semantic understanding and low-level geometric reasoning. While
Multimodal Large Language Models (MLLMs) excel at semantic tasks, their
application to 3D scene generation is hindered by their limited grounding on 3D
geometry. In this paper, we investigate how to best work with MLLMs in an
object placement task. Towards this goal, we introduce a novel framework,
FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the
extraction of relevant geometric details from the 3D scene, (2) constructing
and solving geometric constraints on the extracted low-level geometry, and (3)
pruning for final placements that conform to common sense. By combining
geometric reasoning with real-world understanding of MLLMs, our method can
propose object placements that satisfy both geometric constraints as well as
high-level semantic common-sense considerations. Our experiments show that
these capabilities allow our method to place objects more effectively in
complex scenes with intricate geometry, surpassing the quality of prior work.
FirePlace combines Multimodal Large Language Models with geometric reasoning to effectively place objects in complex 3D scenes, outperforming existing methods.
Multimodal Large Language Models 3D geometric reasoning object placement geometric constraints common sense scene generation