FactoredScenes: Real-World Scene Generation via Library Learning of Layout and Pose Prediction
Joy Hsu, Emily Jin, Jiajun Wu, Niloy Mitra
Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes, by leveraging the underlying structure of rooms, while learning the variation of object poses from lived-in scenes. We propose a factored room representation that decomposes scenes into hierarchically organized concepts of programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.