Scenethesis: A Language and Vision Agentic Framework for 3D Scene
  Generation
Synthesizing interactive 3D scenes from text is essential for gaming, virtual
reality, and embodied AI. However, existing methods face several challenges.
Learning-based approaches depend on small-scale indoor datasets, limiting the
scene diversity and layout complexity. While large language models (LLMs) can
leverage diverse text-domain knowledge, they struggle with spatial realism,
often producing unnatural object placements that fail to respect common sense.
Our key insight is that vision perception can bridge this gap by providing
realistic spatial guidance that LLMs lack. To this end, we introduce
Scenethesis, a training-free agentic framework that integrates LLM-based scene
planning with vision-guided layout refinement. Given a text prompt, Scenethesis
first employs an LLM to draft a coarse layout. A vision module then refines it
by generating an image guidance and extracting scene structure to capture
inter-object relations. Next, an optimization module iteratively enforces
accurate pose alignment and physical plausibility, preventing artifacts like
object penetration and instability. Finally, a judge module verifies spatial
coherence. Comprehensive experiments show that Scenethesis generates diverse,
realistic, and physically plausible 3D interactive scenes, making it valuable
for virtual content creation, simulation environments, and embodied AI
research.
6819a5106d73cfa9574a900a
Lu Ling
claimed_verified
2025-05-06T08:32:27.485Z
6347935d79e42766e212fa53
https://cdn-avatars.huggingface.co/v1/production/uploads/6347935d79e42766e212fa53/9BBADaR0yV1wjC91pGYmZ.png
Lu Ling
user
LuLing
6819a5106d73cfa9574a900b
Chen-Hsuan Lin
6819a5106d73cfa9574a900c
Tsung-Yi Lin
6819a5106d73cfa9574a900d
Yifan Ding
6819a5106d73cfa9574a900e
Yu Zeng
6819a5106d73cfa9574a900f
Yichen Sheng
6819a5106d73cfa9574a9010
Yunhao Ge
6819a5106d73cfa9574a9011
Ming-Yu Liu
6819a5106d73cfa9574a9012
Aniket Bera
6819a5106d73cfa9574a9013
Zhaoshuo Li
Scenethesis: A Language and Vision Agentic Framework for 3D Scene
  Generation
Synthesizing interactive 3D scenes from text is essential for gaming, virtual
reality, and embodied AI. However, existing methods face several challenges.
Learning-based approaches depend on small-scale indoor datasets, limiting the
scene diversity and layout complexity. While large language models (LLMs) can
leverage diverse text-domain knowledge, they struggle with spatial realism,
often producing unnatural object placements that fail to respect common sense.
Our key insight is that vision perception can bridge this gap by providing
realistic spatial guidance that LLMs lack. To this end, we introduce
Scenethesis, a training-free agentic framework that integrates LLM-based scene
planning with vision-guided layout refinement. Given a text prompt, Scenethesis
first employs an LLM to draft a coarse layout. A vision module then refines it
by generating an image guidance and extracting scene structure to capture
inter-object relations. Next, an optimization module iteratively enforces
accurate pose alignment and physical plausibility, preventing artifacts like
object penetration and instability. Finally, a judge module verifies spatial
coherence. Comprehensive experiments show that Scenethesis generates diverse,
realistic, and physically plausible 3D interactive scenes, making it valuable
for virtual content creation, simulation environments, and embodied AI
research.
Scenethesis combines language models with vision-guided refinement to create diverse, realistic, and physically plausible 3D scenes from text.
LLMs vision perception Scenethesis agentic framework scene planning vision module image guidance scene structure inter-object relations optimization module pose alignment physical plausibility judge module spatial coherence 3D scenes virtual content creation simulation environments embodied AI