SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via
  Stein Identity
Score distillation has emerged as one of the most prevalent approaches for
text-to-3D asset synthesis. Essentially, score distillation updates 3D
parameters by lifting and back-propagating scores averaged over different
views. In this paper, we reveal that the gradient estimation in score
distillation is inherent to high variance. Through the lens of variance
reduction, the effectiveness of SDS and VSD can be interpreted as applications
of various control variates to the Monte Carlo estimator of the distilled
score. Motivated by this rethinking and based on Stein's identity, we propose a
more general solution to reduce variance for score distillation, termed Stein
Score Distillation (SSD). SSD incorporates control variates constructed by
Stein identity, allowing for arbitrary baseline functions. This enables us to
include flexible guidance priors and network architectures to explicitly
optimize for variance reduction. In our experiments, the overall pipeline,
dubbed SteinDreamer, is implemented by instantiating the control variate with a
monocular depth estimator. The results suggest that SSD can effectively reduce
the distillation variance and consistently improve visual quality for both
object- and scene-level generation. Moreover, we demonstrate that SteinDreamer
achieves faster convergence than existing methods due to more stable gradient
updates.
65939dbb1adf6d577e153d16
Peihao Wang
claimed_verified
2024-02-09T09:28:34.015Z
62c8f1773e28b8ee4cddca09
/avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
Peihao Wang
user
peihaowang
65939dbb1adf6d577e153d17
Zhiwen Fan
65939dbb1adf6d577e153d18
Dejia Xu
claimed_verified
2024-06-03T07:06:12.137Z
62f687a0c58915315c4ff75d
/avatars/b657180c7666735062782edd4f6a69c9.svg
Dejia Xu
user
ir1d
65939dbb1adf6d577e153d19
Dilin Wang
65939dbb1adf6d577e153d1a
Sreyas Mohan
65939dbb1adf6d577e153d1b
Forrest Iandola
65939dbb1adf6d577e153d1c
Rakesh Ranjan
65939dbb1adf6d577e153d1d
Yilei Li
65939dbb1adf6d577e153d1e
Qiang Liu
65939dbb1adf6d577e153d1f
Zhangyang Wang
65939dbb1adf6d577e153d20
Vikas Chandra
SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via
  Stein Identity
Score distillation has emerged as one of the most prevalent approaches for
text-to-3D asset synthesis. Essentially, score distillation updates 3D
parameters by lifting and back-propagating scores averaged over different
views. In this paper, we reveal that the gradient estimation in score
distillation is inherent to high variance. Through the lens of variance
reduction, the effectiveness of SDS and VSD can be interpreted as applications
of various control variates to the Monte Carlo estimator of the distilled
score. Motivated by this rethinking and based on Stein's identity, we propose a
more general solution to reduce variance for score distillation, termed Stein
Score Distillation (SSD). SSD incorporates control variates constructed by
Stein identity, allowing for arbitrary baseline functions. This enables us to
include flexible guidance priors and network architectures to explicitly
optimize for variance reduction. In our experiments, the overall pipeline,
dubbed SteinDreamer, is implemented by instantiating the control variate with a
monocular depth estimator. The results suggest that SSD can effectively reduce
the distillation variance and consistently improve visual quality for both
object- and scene-level generation. Moreover, we demonstrate that SteinDreamer
achieves faster convergence than existing methods due to more stable gradient
updates.
Stein Score Distillation (SSD) reduces variance in score distillation for text-to-3D asset synthesis, improving visual quality and convergence speed.
score distillation gradient estimation variance reduction control variates Monte Carlo estimator Stein's identity Stein Score Distillation SteinDreamer monocular depth estimator visual quality convergence speed