ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured
  Proxies
Automatic creation of 3D scenes for immersive VR presence has been a
significant research focus for decades. However, existing methods often rely on
either high-poly mesh modeling with post-hoc simplification or massive 3D
Gaussians, resulting in a complex pipeline or limited visual realism. In this
paper, we demonstrate that such exhaustive modeling is unnecessary for
achieving compelling immersive experience. We introduce ImmerseGen, a novel
agent-guided framework for compact and photorealistic world modeling.
ImmerseGen represents scenes as hierarchical compositions of lightweight
geometric proxies, i.e., simplified terrain and billboard meshes, and generates
photorealistic appearance by synthesizing RGBA textures onto these proxies.
Specifically, we propose terrain-conditioned texturing for user-centric base
world synthesis, and RGBA asset texturing for midground and foreground scenery.
This reformulation offers several advantages: (i) it simplifies modeling by
enabling agents to guide generative models in producing coherent textures that
integrate seamlessly with the scene; (ii) it bypasses complex geometry creation
and decimation by directly synthesizing photorealistic textures on proxies,
preserving visual quality without degradation; (iii) it enables compact
representations suitable for real-time rendering on mobile VR headsets. To
automate scene creation from text prompts, we introduce VLM-based modeling
agents enhanced with semantic grid-based analysis for improved spatial
reasoning and accurate asset placement. ImmerseGen further enriches scenes with
dynamic effects and ambient audio to support multisensory immersion.
Experiments on scene generation and live VR showcases demonstrate that
ImmerseGen achieves superior photorealism, spatial coherence and rendering
efficiency compared to prior methods. Project webpage:
https://immersegen.github.io.
6853c08e99bf39f9665c7a71
Jinyan Yuan
claimed_verified
2025-06-19T09:10:07.549Z
662e1323a314b134a229cc2f
/avatars/1eea9ab5edeb3d0e2db33dbfa15bfbd1.svg
Jinyan Yuan
user
yjyyy
6853c08e99bf39f9665c7a72
Bangbang Yang
claimed_verified
2025-06-19T15:11:28.147Z
63d748ff6f49aa82306b7e48
/avatars/9f0b8b8a09b14d76e52ed1bd312e6b63.svg
BB Yang
user
ybbbbt
6853c08e99bf39f9665c7a73
Keke Wang
6853c08e99bf39f9665c7a74
Panwang Pan
claimed_verified
2025-06-19T15:11:25.970Z
654866e8cd0a5621395f8287
https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
Panwang Pan
user
paulpanwang
6853c08e99bf39f9665c7a75
Lin Ma
6853c08e99bf39f9665c7a76
Xuehai Zhang
6853c08e99bf39f9665c7a77
Xiao Liu
6853c08e99bf39f9665c7a78
Zhaopeng Cui
6853c08e99bf39f9665c7a79
Yuewen Ma
ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured
  Proxies
Automatic creation of 3D scenes for immersive VR presence has been a
significant research focus for decades. However, existing methods often rely on
either high-poly mesh modeling with post-hoc simplification or massive 3D
Gaussians, resulting in a complex pipeline or limited visual realism. In this
paper, we demonstrate that such exhaustive modeling is unnecessary for
achieving compelling immersive experience. We introduce ImmerseGen, a novel
agent-guided framework for compact and photorealistic world modeling.
ImmerseGen represents scenes as hierarchical compositions of lightweight
geometric proxies, i.e., simplified terrain and billboard meshes, and generates
photorealistic appearance by synthesizing RGBA textures onto these proxies.
Specifically, we propose terrain-conditioned texturing for user-centric base
world synthesis, and RGBA asset texturing for midground and foreground scenery.
This reformulation offers several advantages: (i) it simplifies modeling by
enabling agents to guide generative models in producing coherent textures that
integrate seamlessly with the scene; (ii) it bypasses complex geometry creation
and decimation by directly synthesizing photorealistic textures on proxies,
preserving visual quality without degradation; (iii) it enables compact
representations suitable for real-time rendering on mobile VR headsets. To
automate scene creation from text prompts, we introduce VLM-based modeling
agents enhanced with semantic grid-based analysis for improved spatial
reasoning and accurate asset placement. ImmerseGen further enriches scenes with
dynamic effects and ambient audio to support multisensory immersion.
Experiments on scene generation and live VR showcases demonstrate that
ImmerseGen achieves superior photorealism, spatial coherence and rendering
efficiency compared to prior methods. Project webpage:
https://immersegen.github.io.
An agent-guided framework generates photorealistic 3D scenes for VR by synthesizing textures onto lightweight geometric proxies, enabling real-time rendering and superior visual quality.
agent-guided framework photorealistic world modeling hierarchical compositions lightweight geometric proxies simplified terrain billboard meshes terrain-conditioned texturing RGBA asset texturing generative models coherent textures semantic grid-based analysis VLM-based modeling agents dynamic effects ambient audio spatial reasoning asset placement photorealism spatial coherence rendering efficiency