ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling
Shuyuan Zhang, ChenHan Jiang, Zuoou Li, Jiankang Deng
Constructing structured 3D shapes is essential for gaming, virtual reality, and embodied AI. However, this process typically demands significant expertise and manual effort using conventional 3D modeling software. To make 3D content creation more accessible, we present ShapeCraft, a novel system that leverages large language model (LLM) agents to autonomously generate structured 3D shapes from natural language instructions. At the core of ShapeCraft is a graph-based procedural shape (GPS) representation, whose nodes store corresponding program snippets and layout information, facilitating efficient programmatic updates and flexible structure manipulation. This representation allows for post-modeling execution as structured geometry while preserving editability for artists. In the ShapeCraft workflow, LLM agents first hierarchically parses user's input and initialize GPS representation, then iteratively refines the procedural modeling and texturing steps to produce structured, textured, and interactive 3D assets. Our experiments show that ShapeCraft significantly outperforms existing LLM-based agents in generating geometrically accurate and semantically rich 3D models. Moreover, it achieves higher shape fidelity compared to optimization-based text-to-3D generation methods. We further demonstrate the versatility of ShapeCraft through examples of animated and user-customized editing, highlighting its potential for broader interactive applications.