Holodeck: Language Guided Generation of 3D Embodied AI Environments
3D simulated environments play a critical role in Embodied AI, but their
creation requires expertise and extensive manual effort, restricting their
diversity and scope. To mitigate this limitation, we present Holodeck, a system
that generates 3D environments to match a user-supplied prompt fully
automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
museums, adjust the designs for styles, and can capture the semantics of
complex queries such as "apartment for a researcher with a cat" and "office of
a professor who is a fan of Star Wars". Holodeck leverages a large language
model (GPT-4) for common sense knowledge about what the scene might look like
and uses a large collection of 3D assets from Objaverse to populate the scene
with diverse objects. To address the challenge of positioning objects
correctly, we prompt GPT-4 to generate spatial relational constraints between
objects and then optimize the layout to satisfy those constraints. Our
large-scale human evaluation shows that annotators prefer Holodeck over
manually designed procedural baselines in residential scenes and that Holodeck
can produce high-quality outputs for diverse scene types. We also demonstrate
an exciting application of Holodeck in Embodied AI, training agents to navigate
in novel scenes like music rooms and daycares without human-constructed data,
which is a significant step forward in developing general-purpose embodied
agents.
657bb7555c6f0b1f36f47613
Yue Yang
claimed_verified
2023-12-27T11:06:07.639Z
62f6c68904e5e02f82b04690
https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
Yue Yang
user
yyupenn
657bb7555c6f0b1f36f47614
Fan-Yun Sun
claimed_verified
2024-06-19T07:40:26.886Z
66287db3fe7ace12c40026b6
/avatars/ee86cddd6c7f6f456f7643ee0ddb084a.svg
Fan-Yun Sun
user
sunfanyun
657bb7555c6f0b1f36f47615
Luca Weihs
admin_assigned
2023-12-15T13:09:33.767Z
620acbdb3c0931626a7c9297
/avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
Luca Weihs
user
lucaweihs
657bb7555c6f0b1f36f47616
Eli VanderBilt
657bb7555c6f0b1f36f47617
Alvaro Herrasti
657bb7555c6f0b1f36f47618
Winson Han
657bb7555c6f0b1f36f47619
Jiajun Wu
admin_assigned
2023-12-15T13:09:17.412Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
657bb7555c6f0b1f36f4761a
Nick Haber
claimed_verified
2023-12-18T08:48:19.263Z
657d3b7d69cfe9b4cd0bfa04
/avatars/a97b0df7187d08ccca8a6a597cba7f67.svg
Nick Haber
user
nhaber
657bb7555c6f0b1f36f4761b
Ranjay Krishna
657bb7555c6f0b1f36f4761c
Lingjie Liu
657bb7555c6f0b1f36f4761d
Chris Callison-Burch
admin_assigned
2023-12-15T13:08:47.889Z
6303ce25fc783bfc744216af
/avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
Chris Callison-Burch
user
CCB
657bb7555c6f0b1f36f4761e
Mark Yatskar
admin_assigned
2023-12-15T13:08:41.546Z
631f42e1b6628770f6efd87a
/avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
Mark Yatskar
user
myatskar
657bb7555c6f0b1f36f4761f
Aniruddha Kembhavi
657bb7555c6f0b1f36f47620
Christopher Clark
Holodeck: Language Guided Generation of 3D Embodied AI Environments
3D simulated environments play a critical role in Embodied AI, but their
creation requires expertise and extensive manual effort, restricting their
diversity and scope. To mitigate this limitation, we present Holodeck, a system
that generates 3D environments to match a user-supplied prompt fully
automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
museums, adjust the designs for styles, and can capture the semantics of
complex queries such as "apartment for a researcher with a cat" and "office of
a professor who is a fan of Star Wars". Holodeck leverages a large language
model (GPT-4) for common sense knowledge about what the scene might look like
and uses a large collection of 3D assets from Objaverse to populate the scene
with diverse objects. To address the challenge of positioning objects
correctly, we prompt GPT-4 to generate spatial relational constraints between
objects and then optimize the layout to satisfy those constraints. Our
large-scale human evaluation shows that annotators prefer Holodeck over
manually designed procedural baselines in residential scenes and that Holodeck
can produce high-quality outputs for diverse scene types. We also demonstrate
an exciting application of Holodeck in Embodied AI, training agents to navigate
in novel scenes like music rooms and daycares without human-constructed data,
which is a significant step forward in developing general-purpose embodied
agents.
Holodeck automatically generates diverse and semantically detailed 3D environments using a large language model and 3D asset optimization, advancing Embodied AI training.
Embodied AI 3D environments Holodeck GPT-4 Objaverse spatial relational constraints human evaluation procedural baselines novel scenes navigation training