GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video
  Diffusion Priors
Reconstructing 3D scenes using 3D Gaussian Splatting (3DGS) from sparse views
is an ill-posed problem due to insufficient information, often resulting in
noticeable artifacts. While recent approaches have sought to leverage
generative priors to complete information for under-constrained regions, they
struggle to generate content that remains consistent with input observations.
To address this challenge, we propose GSFixer, a novel framework designed to
improve the quality of 3DGS representations reconstructed from sparse inputs.
The core of our approach is the reference-guided video restoration model, built
upon a DiT-based video diffusion model trained on paired artifact 3DGS renders
and clean frames with additional reference-based conditions. Considering the
input sparse views as references, our model integrates both 2D semantic
features and 3D geometric features of reference views extracted from the visual
geometry foundation model, enhancing the semantic coherence and 3D consistency
when fixing artifact novel views. Furthermore, considering the lack of suitable
benchmarks for 3DGS artifact restoration evaluation, we present DL3DV-Res which
contains artifact frames rendered using low-quality 3DGS. Extensive experiments
demonstrate our GSFixer outperforms current state-of-the-art methods in 3DGS
artifact restoration and sparse-view 3D reconstruction. Project page:
https://github.com/GVCLab/GSFixer.
689dd697b083e610d741eb4d
Xingyilang Yin
689dd697b083e610d741eb4e
Qi Zhang
689dd697b083e610d741eb4f
Jiahao Chang
689dd697b083e610d741eb50
Ying Feng
689dd697b083e610d741eb51
Qingnan Fan
689dd697b083e610d741eb52
Xi Yang
689dd697b083e610d741eb53
Chi-Man Pun
689dd697b083e610d741eb54
Huaqi Zhang
689dd697b083e610d741eb55
Xiaodong Cun
GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video
  Diffusion Priors
Reconstructing 3D scenes using 3D Gaussian Splatting (3DGS) from sparse views
is an ill-posed problem due to insufficient information, often resulting in
noticeable artifacts. While recent approaches have sought to leverage
generative priors to complete information for under-constrained regions, they
struggle to generate content that remains consistent with input observations.
To address this challenge, we propose GSFixer, a novel framework designed to
improve the quality of 3DGS representations reconstructed from sparse inputs.
The core of our approach is the reference-guided video restoration model, built
upon a DiT-based video diffusion model trained on paired artifact 3DGS renders
and clean frames with additional reference-based conditions. Considering the
input sparse views as references, our model integrates both 2D semantic
features and 3D geometric features of reference views extracted from the visual
geometry foundation model, enhancing the semantic coherence and 3D consistency
when fixing artifact novel views. Furthermore, considering the lack of suitable
benchmarks for 3DGS artifact restoration evaluation, we present DL3DV-Res which
contains artifact frames rendered using low-quality 3DGS. Extensive experiments
demonstrate our GSFixer outperforms current state-of-the-art methods in 3DGS
artifact restoration and sparse-view 3D reconstruction. Project page:
https://github.com/GVCLab/GSFixer.
GSFixer enhances 3D Gaussian Splatting reconstructions from sparse views using a DiT-based video diffusion model with reference-guided conditions, improving artifact restoration and 3D consistency.
3D Gaussian Splatting DiT-based video diffusion model reference-guided video restoration 2D semantic features 3D geometric features visual geometry foundation model DL3DV-Res artifact restoration sparse-view 3D reconstruction