VGGT-X: When VGGT Meets Dense Novel View Synthesis
We study the problem of applying 3D Foundation Models (3DFMs) to dense Novel
View Synthesis (NVS). Despite significant progress in Novel View Synthesis
powered by NeRF and 3DGS, current approaches remain reliant on accurate 3D
attributes (e.g., camera poses and point clouds) acquired from
Structure-from-Motion (SfM), which is often slow and fragile in low-texture or
low-overlap captures. Recent 3DFMs showcase orders of magnitude speedup over
the traditional pipeline and great potential for online NVS. But most of the
validation and conclusions are confined to sparse-view settings. Our study
reveals that naively scaling 3DFMs to dense views encounters two fundamental
barriers: dramatically increasing VRAM burden and imperfect outputs that
degrade initialization-sensitive 3D training. To address these barriers, we
introduce VGGT-X, incorporating a memory-efficient VGGT implementation that
scales to 1,000+ images, an adaptive global alignment for VGGT output
enhancement, and robust 3DGS training practices. Extensive experiments show
that these measures substantially close the fidelity gap with
COLMAP-initialized pipelines, achieving state-of-the-art results in dense
COLMAP-free NVS and pose estimation. Additionally, we analyze the causes of
remaining gaps with COLMAP-initialized rendering, providing insights for the
future development of 3D foundation models and dense NVS. Our project page is
available at https://dekuliutesla.github.io/vggt-x.github.io/
68db55d7d2bf1f4b15ec751e
Yang Liu
68db55d7d2bf1f4b15ec751f
Chuanchen Luo
68db55d7d2bf1f4b15ec7520
Zimo Tang
68db55d7d2bf1f4b15ec7521
Junran Peng
68db55d7d2bf1f4b15ec7522
Zhaoxiang Zhang
VGGT-X: When VGGT Meets Dense Novel View Synthesis
We study the problem of applying 3D Foundation Models (3DFMs) to dense Novel
View Synthesis (NVS). Despite significant progress in Novel View Synthesis
powered by NeRF and 3DGS, current approaches remain reliant on accurate 3D
attributes (e.g., camera poses and point clouds) acquired from
Structure-from-Motion (SfM), which is often slow and fragile in low-texture or
low-overlap captures. Recent 3DFMs showcase orders of magnitude speedup over
the traditional pipeline and great potential for online NVS. But most of the
validation and conclusions are confined to sparse-view settings. Our study
reveals that naively scaling 3DFMs to dense views encounters two fundamental
barriers: dramatically increasing VRAM burden and imperfect outputs that
degrade initialization-sensitive 3D training. To address these barriers, we
introduce VGGT-X, incorporating a memory-efficient VGGT implementation that
scales to 1,000+ images, an adaptive global alignment for VGGT output
enhancement, and robust 3DGS training practices. Extensive experiments show
that these measures substantially close the fidelity gap with
COLMAP-initialized pipelines, achieving state-of-the-art results in dense
COLMAP-free NVS and pose estimation. Additionally, we analyze the causes of
remaining gaps with COLMAP-initialized rendering, providing insights for the
future development of 3D foundation models and dense NVS. Our project page is
available at https://dekuliutesla.github.io/vggt-x.github.io/
VGGT-X addresses VRAM and output quality issues in scaling 3D Foundation Models for dense Novel View Synthesis without relying on COLMAP.
3D Foundation Models Novel View Synthesis NeRF 3DGS Structure-from-Motion VGGT-X memory-efficient adaptive global alignment robust 3DGS training COLMAP-free NVS pose estimation