DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
LiDAR scene generation has been developing rapidly recently. However,
existing methods primarily focus on generating static and single-frame scenes,
overlooking the inherently dynamic nature of real-world driving environments.
In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework
capable of generating large-scale, high-quality LiDAR scenes that capture the
temporal evolution of dynamic environments. DynamicCity mainly consists of two
key models. 1) A VAE model for learning HexPlane as the compact 4D
representation. Instead of using naive averaging operations, DynamicCity
employs a novel Projection Module to effectively compress 4D LiDAR features
into six 2D feature maps for HexPlane construction, which significantly
enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we
utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in
parallel, which improves both network training efficiency and reconstruction
accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x
training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model
for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded
Rollout Operation is proposed to reorganize all six feature planes of the
HexPlane as a squared 2D feature map. In particular, various conditions could
be introduced in the diffusion or sampling process, supporting versatile 4D
generation applications, such as trajectory- and command-driven generation,
inpainting, and layout-conditioned generation. Extensive experiments on the
CarlaSC and Waymo datasets demonstrate that DynamicCity significantly
outperforms existing state-of-the-art 4D LiDAR generation methods across
multiple metrics. The code will be released to facilitate future research.
671a297304f808324dfa8237
Hengwei Bian
admin_assigned
2024-10-24T14:25:40.458Z
65990b48ac728bc303537234
/avatars/717f599381ae845721c6daccb3bb1f76.svg
Hengwei Bian
user
hengwei
671a297304f808324dfa8238
Lingdong Kong
admin_assigned
2024-10-24T14:25:25.234Z
62df78222d89ce551ce0f71d
/avatars/89fba294cff2d2f941d121c1923e4c76.svg
Lingdong Kong
user
ldkong
671a297304f808324dfa8239
Haozhe Xie
claimed_verified
2024-10-24T12:35:00.035Z
63f47b5321eb234ab739e91a
https://cdn-avatars.huggingface.co/v1/production/uploads/63f47b5321eb234ab739e91a/vWfFNVtMkHl8gieha5PPd.jpeg
Haozhe Xie
user
hzxie
671a297304f808324dfa823a
Liang Pan
claimed_verified
2024-11-05T16:09:29.882Z
64ef30fb0de94d31b7921175
/avatars/d4f4c7be460befc1cedec13bbf9db972.svg
pan
user
pldeqiushui
671a297304f808324dfa823b
Yu Qiao
671a297304f808324dfa823c
Ziwei Liu
admin_assigned
2024-10-24T14:25:47.367Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
LiDAR scene generation has been developing rapidly recently. However,
existing methods primarily focus on generating static and single-frame scenes,
overlooking the inherently dynamic nature of real-world driving environments.
In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework
capable of generating large-scale, high-quality LiDAR scenes that capture the
temporal evolution of dynamic environments. DynamicCity mainly consists of two
key models. 1) A VAE model for learning HexPlane as the compact 4D
representation. Instead of using naive averaging operations, DynamicCity
employs a novel Projection Module to effectively compress 4D LiDAR features
into six 2D feature maps for HexPlane construction, which significantly
enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we
utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in
parallel, which improves both network training efficiency and reconstruction
accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x
training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model
for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded
Rollout Operation is proposed to reorganize all six feature planes of the
HexPlane as a squared 2D feature map. In particular, various conditions could
be introduced in the diffusion or sampling process, supporting versatile 4D
generation applications, such as trajectory- and command-driven generation,
inpainting, and layout-conditioned generation. Extensive experiments on the
CarlaSC and Waymo datasets demonstrate that DynamicCity significantly
outperforms existing state-of-the-art 4D LiDAR generation methods across
multiple metrics. The code will be released to facilitate future research.
DynamicCity generates large-scale, high-quality 4D LiDAR scenes by integrating a VAE model with a DiT-based diffusion model, effectively capturing dynamic environments.
VAE HexPlane Projection Module Expansion & Squeeze Strategy DiT-based diffusion model Padded Rollout Operation 4D LiDAR generation trajectory-driven generation command-driven generation inpainting layout-conditioned generation CarlaSC Waymo datasets