EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
  Self-Supervision
We present EmerNeRF, a simple yet powerful approach for learning
spatial-temporal representations of dynamic driving scenes. Grounded in neural
fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
First, it stratifies scenes into static and dynamic fields. This decomposition
emerges purely from self-supervision, enabling our model to learn from general,
in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
from the dynamic field and uses this flow field to further aggregate
multi-frame features, amplifying the rendering precision of dynamic objects.
Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
represent highly-dynamic scenes self-sufficiently, without relying on ground
truth object annotations or pre-trained models for dynamic object segmentation
or optical flow estimation. Our method achieves state-of-the-art performance in
sensor simulation, significantly outperforming previous methods when
reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
foundation model features into 4D space-time and address a general positional
bias in modern Transformers, significantly boosting 3D perception performance
(e.g., 37.50% relative improvement in occupancy prediction accuracy on
average). Finally, we construct a diverse and challenging 120-sequence dataset
to benchmark neural fields under extreme and highly-dynamic settings.
65487e36af5d7944326e34ca
Jiawei Yang
65487e36af5d7944326e34cb
Boris Ivanovic
65487e36af5d7944326e34cc
Or Litany
admin_assigned
2023-11-06T13:53:21.563Z
630509558d7f39c0ecf29dc3
/avatars/6b225b95f625152dc69fedb5f0b348d6.svg
Or Litany
user
orlitany
65487e36af5d7944326e34cd
Xinshuo Weng
65487e36af5d7944326e34ce
Seung Wook Kim
admin_assigned
2023-11-06T13:53:36.472Z
64fb5827d82fc6977d70582d
/avatars/54180874d2f162572c414970b8750759.svg
Seung Wook Kim
user
swkimbd
65487e36af5d7944326e34cf
Boyi Li
65487e36af5d7944326e34d0
Tong Che
65487e36af5d7944326e34d1
Danfei Xu
admin_assigned
2023-11-06T13:56:25.046Z
636eb10afd9751c3d08279e9
/avatars/8e9833b2766c76e92be4dba2ceadd61f.svg
Xu
user
Danfei
65487e36af5d7944326e34d2
Sanja Fidler
65487e36af5d7944326e34d3
Marco Pavone
65487e36af5d7944326e34d4
Yue Wang
EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via
  Self-Supervision
We present EmerNeRF, a simple yet powerful approach for learning
spatial-temporal representations of dynamic driving scenes. Grounded in neural
fields, EmerNeRF simultaneously captures scene geometry, appearance, motion,
and semantics via self-bootstrapping. EmerNeRF hinges upon two core components:
First, it stratifies scenes into static and dynamic fields. This decomposition
emerges purely from self-supervision, enabling our model to learn from general,
in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field
from the dynamic field and uses this flow field to further aggregate
multi-frame features, amplifying the rendering precision of dynamic objects.
Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to
represent highly-dynamic scenes self-sufficiently, without relying on ground
truth object annotations or pre-trained models for dynamic object segmentation
or optical flow estimation. Our method achieves state-of-the-art performance in
sensor simulation, significantly outperforming previous methods when
reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In
addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual
foundation model features into 4D space-time and address a general positional
bias in modern Transformers, significantly boosting 3D perception performance
(e.g., 37.50% relative improvement in occupancy prediction accuracy on
average). Finally, we construct a diverse and challenging 120-sequence dataset
to benchmark neural fields under extreme and highly-dynamic settings.
EmerNeRF uses neural fields with self-bootstrapping to learn and represent static and dynamic components of driving scenes efficiently, achieving state-of-the-art performance in sensor simulation and semantic generalization.
neural fields EmerNeRF self-bootstrapping static fields dynamic fields flow field multi-frame features rendering precision 3D perception occupancy prediction 4D space-time semantic generalization