Taming Latent Diffusion Model for Neural Radiance Field Inpainting
Neural Radiance Field (NeRF) is a representation for 3D reconstruction from
multi-view images. Despite some recent work showing preliminary success in
editing a reconstructed NeRF with diffusion prior, they remain struggling to
synthesize reasonable geometry in completely uncovered regions. One major
reason is the high diversity of synthetic contents from the diffusion model,
which hinders the radiance field from converging to a crisp and deterministic
geometry. Moreover, applying latent diffusion models on real data often yields
a textural shift incoherent to the image condition due to auto-encoding errors.
These two problems are further reinforced with the use of pixel-distance
losses. To address these issues, we propose tempering the diffusion model's
stochasticity with per-scene customization and mitigating the textural shift
with masked adversarial training. During the analyses, we also found the
commonly used pixel and perceptual losses are harmful in the NeRF inpainting
task. Through rigorous experiments, our framework yields state-of-the-art NeRF
inpainting results on various real-world scenes. Project page:
https://hubert0527.github.io/MALD-NeRF
661dec6bbca1a6038b587775
Chieh Hubert Lin
admin_assigned
2024-04-18T11:11:30.943Z
62cb9c172e4fbd838f6b4f37
/avatars/ea0f7d9e746bb8a6e574854deea4859a.svg
Hubert Lin
user
hubert0527
661dec6bbca1a6038b587776
Changil Kim
661dec6bbca1a6038b587777
Jia-Bin Huang
admin_assigned
2024-04-18T11:12:39.974Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
661dec6bbca1a6038b587778
Qinbo Li
661dec6bbca1a6038b587779
Chih-Yao Ma
admin_assigned
2024-04-18T11:12:57.881Z
6515bf00c2aedaa7e692a1ec
https://cdn-avatars.huggingface.co/v1/production/uploads/6515bf00c2aedaa7e692a1ec/PYpS1QwTPdQYW5_BrvwzP.png
Chih-Yao Ma
user
chihyaoma
661dec6bbca1a6038b58777a
Johannes Kopf
661dec6bbca1a6038b58777b
Ming-Hsuan Yang
661dec6bbca1a6038b58777c
Hung-Yu Tseng
Taming Latent Diffusion Model for Neural Radiance Field Inpainting
Neural Radiance Field (NeRF) is a representation for 3D reconstruction from
multi-view images. Despite some recent work showing preliminary success in
editing a reconstructed NeRF with diffusion prior, they remain struggling to
synthesize reasonable geometry in completely uncovered regions. One major
reason is the high diversity of synthetic contents from the diffusion model,
which hinders the radiance field from converging to a crisp and deterministic
geometry. Moreover, applying latent diffusion models on real data often yields
a textural shift incoherent to the image condition due to auto-encoding errors.
These two problems are further reinforced with the use of pixel-distance
losses. To address these issues, we propose tempering the diffusion model's
stochasticity with per-scene customization and mitigating the textural shift
with masked adversarial training. During the analyses, we also found the
commonly used pixel and perceptual losses are harmful in the NeRF inpainting
task. Through rigorous experiments, our framework yields state-of-the-art NeRF
inpainting results on various real-world scenes. Project page:
https://hubert0527.github.io/MALD-NeRF
The proposed framework addresses limitations in NeRF editing by tempering diffusion model stochasticity and mitigating textural shifts, leading to state-of-the-art NeRF inpainting results.
Neural Radiance Field NeRF diffusion prior latent diffusion models radiance field textural shift auto-encoding errors pixel-distance losses perceptual losses masked adversarial training NeRF inpainting