Spectrally Pruned Gaussian Fields with Neural Compensation
Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered
attention for its fast rendering speed and high rendering quality. However,
this comes with high memory consumption, e.g., a well-trained Gaussian field
may utilize three million Gaussian primitives and over 700 MB of memory. We
credit this high memory footprint to the lack of consideration for the
relationship between primitives. In this paper, we propose a memory-efficient
Gaussian field named SUNDAE with spectral pruning and neural compensation. On
one hand, we construct a graph on the set of Gaussian primitives to model their
relationship and design a spectral down-sampling module to prune out primitives
while preserving desired signals. On the other hand, to compensate for the
quality loss of pruning Gaussians, we exploit a lightweight neural network head
to mix splatted features, which effectively compensates for quality losses
while capturing the relationship between primitives in its weights. We
demonstrate the performance of SUNDAE with extensive results. For example,
SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla
Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB
memory, on the Mip-NeRF360 dataset. Codes are publicly available at
https://runyiyang.github.io/projects/SUNDAE/.
6632f7041fd3ec937a476f03
Runyi Yang
admin_assigned
2024-05-02T07:29:13.791Z
646b15873e2a7b065945679b
/avatars/d3ffef9ea73dfa7cf37e9a06c41df2dc.svg
Runyi Yang
user
Runyi-Y
6632f7041fd3ec937a476f04
Zhenxin Zhu
6632f7041fd3ec937a476f05
Zhou Jiang
6632f7041fd3ec937a476f06
Baijun Ye
6632f7041fd3ec937a476f07
Xiaoxue Chen
admin_assigned
2024-05-02T07:29:45.894Z
650c1106b92f35bd37dbcdfa
/avatars/d41088e6c5cca83a567fdf9c48160eb7.svg
xiaoxue chen
user
xxuechen
6632f7041fd3ec937a476f08
Yifei Zhang
6632f7041fd3ec937a476f09
Yuantao Chen
6632f7041fd3ec937a476f0a
Jian Zhao
6632f7041fd3ec937a476f0b
Hao Zhao
Spectrally Pruned Gaussian Fields with Neural Compensation
Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered
attention for its fast rendering speed and high rendering quality. However,
this comes with high memory consumption, e.g., a well-trained Gaussian field
may utilize three million Gaussian primitives and over 700 MB of memory. We
credit this high memory footprint to the lack of consideration for the
relationship between primitives. In this paper, we propose a memory-efficient
Gaussian field named SUNDAE with spectral pruning and neural compensation. On
one hand, we construct a graph on the set of Gaussian primitives to model their
relationship and design a spectral down-sampling module to prune out primitives
while preserving desired signals. On the other hand, to compensate for the
quality loss of pruning Gaussians, we exploit a lightweight neural network head
to mix splatted features, which effectively compensates for quality losses
while capturing the relationship between primitives in its weights. We
demonstrate the performance of SUNDAE with extensive results. For example,
SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla
Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB
memory, on the Mip-NeRF360 dataset. Codes are publicly available at
https://runyiyang.github.io/projects/SUNDAE/.
SUNDAE, a memory-efficient 3D Gaussian field, reduces memory usage by pruning primitives using spectral down-sampling and compensates for quality loss with a neural network head, achieving comparable or better performance than standard Gaussian splatting.
3D Gaussian Splatting Gaussian field memory-efficient spectral pruning neural compensation graph spectral down-sampling module PSNR FPS Mip-NeRF360 dataset