Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering
  for HDR View Synthesis
Volumetric rendering based methods, like NeRF, excel in HDR view synthesis
from RAWimages, especially for nighttime scenes. While, they suffer from long
training times and cannot perform real-time rendering due to dense sampling
requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time
rendering and faster training. However, implementing RAW image-based view
synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1)
in nighttime scenes, extremely low SNR leads to poor structure-from-motion
(SfM) estimation in distant views; 2) the limited representation capacity of
spherical harmonics (SH) function is unsuitable for RAW linear color space; and
3) inaccurate scene structure hampers downstream tasks such as refocusing. To
address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our
method proposes Cone Scatter Initialization to enrich the estimation of SfM,
and replaces SH with a Color MLP to represent the RAW linear color space.
Additionally, we introduce depth distortion and near-far regularizations to
improve the accuracy of scene structure for downstream tasks. These designs
enable LE3D to perform real-time novel view synthesis, HDR rendering,
refocusing, and tone-mapping changes. Compared to previous volumetric rendering
based methods, LE3D reduces training time to 1% and improves rendering speed by
up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can
be found in https://github.com/Srameo/LE3D .
6667d23d7dbe2bcda2997580
Xin Jin
claimed_verified
2024-06-11T06:58:58.651Z
6537e8eab01250d1d6efed3a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/gMx73gwdfEhcCFioStGCE.jpeg
Xin
user
Srameo
6667d23d7dbe2bcda2997581
Pengyi Jiao
6667d23d7dbe2bcda2997582
Zheng-Peng Duan
6667d23d7dbe2bcda2997583
Xingchao Yang
6667d23d7dbe2bcda2997584
Chun-Le Guo
claimed_verified
2024-06-11T09:21:14.284Z
66680e227e6e0fee406f57e4
/avatars/2050193d83949a829afc351d996c737f.svg
Guo
user
Chun-Le
6667d23d7dbe2bcda2997585
Bo Ren
6667d23d7dbe2bcda2997586
Chongyi Li
Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering
  for HDR View Synthesis
Volumetric rendering based methods, like NeRF, excel in HDR view synthesis
from RAWimages, especially for nighttime scenes. While, they suffer from long
training times and cannot perform real-time rendering due to dense sampling
requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time
rendering and faster training. However, implementing RAW image-based view
synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1)
in nighttime scenes, extremely low SNR leads to poor structure-from-motion
(SfM) estimation in distant views; 2) the limited representation capacity of
spherical harmonics (SH) function is unsuitable for RAW linear color space; and
3) inaccurate scene structure hampers downstream tasks such as refocusing. To
address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our
method proposes Cone Scatter Initialization to enrich the estimation of SfM,
and replaces SH with a Color MLP to represent the RAW linear color space.
Additionally, we introduce depth distortion and near-far regularizations to
improve the accuracy of scene structure for downstream tasks. These designs
enable LE3D to perform real-time novel view synthesis, HDR rendering,
refocusing, and tone-mapping changes. Compared to previous volumetric rendering
based methods, LE3D reduces training time to 1% and improves rendering speed by
up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can
be found in https://github.com/Srameo/LE3D .
LE3D enhances 3D Gaussian Splatting for real-time HDR rendering from RAW images, particularly in low light, by improving structure-from-motion and using a Color MLP for color representation.
NeRF HDR view synthesis RAW images nighttime scenes training times real-time rendering 3D Gaussian Splatting Cone Scatter Initialization spherical harmonics Color MLP depth distortion near-far regularizations novel view synthesis refocusing tone-mapping