LatticeWorld: A Multimodal Large Language Model-Empowered Framework for
  Interactive Complex World Generation
Recent research has been increasingly focusing on developing 3D world models
that simulate complex real-world scenarios. World models have found broad
applications across various domains, including embodied AI, autonomous driving,
entertainment, etc. A more realistic simulation with accurate physics will
effectively narrow the sim-to-real gap and allow us to gather rich information
about the real world conveniently. While traditional manual modeling has
enabled the creation of virtual 3D scenes, modern approaches have leveraged
advanced machine learning algorithms for 3D world generation, with most recent
advances focusing on generative methods that can create virtual worlds based on
user instructions. This work explores such a research direction by proposing
LatticeWorld, a simple yet effective 3D world generation framework that
streamlines the industrial production pipeline of 3D environments. LatticeWorld
leverages lightweight LLMs (LLaMA-2-7B) alongside the industry-grade rendering
engine (e.g., Unreal Engine 5) to generate a dynamic environment. Our proposed
framework accepts textual descriptions and visual instructions as multimodal
inputs and creates large-scale 3D interactive worlds with dynamic agents,
featuring competitive multi-agent interaction, high-fidelity physics
simulation, and real-time rendering. We conduct comprehensive experiments to
evaluate LatticeWorld, showing that it achieves superior accuracy in scene
layout generation and visual fidelity. Moreover, LatticeWorld achieves over a
90times increase in industrial production efficiency while maintaining high
creative quality compared with traditional manual production methods. Our demo
video is available at https://youtu.be/8VWZXpERR18
68be4a9dc123124955ef5fd9
Yinglin Duan
68be4a9dc123124955ef5fda
Zhengxia Zou
68be4a9dc123124955ef5fdb
Tongwei Gu
68be4a9dc123124955ef5fdc
Wei Jia
68be4a9dc123124955ef5fdd
Zhan Zhao
68be4a9dc123124955ef5fde
Luyi Xu
68be4a9dc123124955ef5fdf
Xinzhu Liu
68be4a9dc123124955ef5fe0
Hao Jiang
68be4a9dc123124955ef5fe1
Kang Chen
68be4a9dc123124955ef5fe2
Shuang Qiu
LatticeWorld: A Multimodal Large Language Model-Empowered Framework for
  Interactive Complex World Generation
Recent research has been increasingly focusing on developing 3D world models
that simulate complex real-world scenarios. World models have found broad
applications across various domains, including embodied AI, autonomous driving,
entertainment, etc. A more realistic simulation with accurate physics will
effectively narrow the sim-to-real gap and allow us to gather rich information
about the real world conveniently. While traditional manual modeling has
enabled the creation of virtual 3D scenes, modern approaches have leveraged
advanced machine learning algorithms for 3D world generation, with most recent
advances focusing on generative methods that can create virtual worlds based on
user instructions. This work explores such a research direction by proposing
LatticeWorld, a simple yet effective 3D world generation framework that
streamlines the industrial production pipeline of 3D environments. LatticeWorld
leverages lightweight LLMs (LLaMA-2-7B) alongside the industry-grade rendering
engine (e.g., Unreal Engine 5) to generate a dynamic environment. Our proposed
framework accepts textual descriptions and visual instructions as multimodal
inputs and creates large-scale 3D interactive worlds with dynamic agents,
featuring competitive multi-agent interaction, high-fidelity physics
simulation, and real-time rendering. We conduct comprehensive experiments to
evaluate LatticeWorld, showing that it achieves superior accuracy in scene
layout generation and visual fidelity. Moreover, LatticeWorld achieves over a
90times increase in industrial production efficiency while maintaining high
creative quality compared with traditional manual production methods. Our demo
video is available at https://youtu.be/8VWZXpERR18
LatticeWorld, a 3D world generation framework using lightweight LLMs and Unreal Engine 5, creates dynamic, interactive environments from textual and visual inputs, achieving high accuracy and efficiency.
LLMs LLaMA-2-7B Unreal Engine 5 multimodal inputs dynamic agents multi-agent interaction high-fidelity physics simulation real-time rendering scene layout generation visual fidelity