Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
  Prediction
Leveraging the visual priors of pre-trained text-to-image diffusion models
offers a promising solution to enhance zero-shot generalization in dense
prediction tasks. However, existing methods often uncritically use the original
diffusion formulation, which may not be optimal due to the fundamental
differences between dense prediction and image generation. In this paper, we
provide a systemic analysis of the diffusion formulation for the dense
prediction, focusing on both quality and efficiency. And we find that the
original parameterization type for image generation, which learns to predict
noise, is harmful for dense prediction; the multi-step noising/denoising
diffusion process is also unnecessary and challenging to optimize. Based on
these insights, we introduce Lotus, a diffusion-based visual foundation model
with a simple yet effective adaptation protocol for dense prediction.
Specifically, Lotus is trained to directly predict annotations instead of
noise, thereby avoiding harmful variance. We also reformulate the diffusion
process into a single-step procedure, simplifying optimization and
significantly boosting inference speed. Additionally, we introduce a novel
tuning strategy called detail preserver, which achieves more accurate and
fine-grained predictions. Without scaling up the training data or model
capacity, Lotus achieves SoTA performance in zero-shot depth and normal
estimation across various datasets. It also significantly enhances efficiency,
being hundreds of times faster than most existing diffusion-based methods.
66f62b506b0e782fa32a02db
Jing He
claimed_verified
2024-10-03T08:31:50.539Z
6548fae806396ffce13092f4
/avatars/aaee62e508ed96790cab1e3f0158fafd.svg
Jing He
user
jingheya
66f62b506b0e782fa32a02dc
Haodong Li
claimed_verified
2024-09-27T06:58:00.560Z
641d211e353524fe41f16387
/avatars/c6e72c82c029b415a035beebee50b52c.svg
Haodong Li
user
haodongli
66f62b506b0e782fa32a02dd
Wei Yin
claimed_verified
2025-03-17T08:46:11.070Z
654a2b1a83e7bfc4313a5cc7
/avatars/dc3dfc3fcd26bb7350a9db0d075c5ea0.svg
Wei Yin
user
WonderingWorld
66f62b506b0e782fa32a02de
Yixun Liang
66f62b506b0e782fa32a02df
Leheng Li
claimed_verified
2024-10-02T07:41:30.485Z
6343e37f73b4f9cedab1c846
/avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
Leheng Li
user
lilelife
66f62b506b0e782fa32a02e0
Kaiqiang Zhou
66f62b506b0e782fa32a02e1
Hongbo Liu
66f62b506b0e782fa32a02e2
Bingbing Liu
66f62b506b0e782fa32a02e3
Ying-Cong Chen
Lotus: Diffusion-based Visual Foundation Model for High-quality Dense
  Prediction
Leveraging the visual priors of pre-trained text-to-image diffusion models
offers a promising solution to enhance zero-shot generalization in dense
prediction tasks. However, existing methods often uncritically use the original
diffusion formulation, which may not be optimal due to the fundamental
differences between dense prediction and image generation. In this paper, we
provide a systemic analysis of the diffusion formulation for the dense
prediction, focusing on both quality and efficiency. And we find that the
original parameterization type for image generation, which learns to predict
noise, is harmful for dense prediction; the multi-step noising/denoising
diffusion process is also unnecessary and challenging to optimize. Based on
these insights, we introduce Lotus, a diffusion-based visual foundation model
with a simple yet effective adaptation protocol for dense prediction.
Specifically, Lotus is trained to directly predict annotations instead of
noise, thereby avoiding harmful variance. We also reformulate the diffusion
process into a single-step procedure, simplifying optimization and
significantly boosting inference speed. Additionally, we introduce a novel
tuning strategy called detail preserver, which achieves more accurate and
fine-grained predictions. Without scaling up the training data or model
capacity, Lotus achieves SoTA performance in zero-shot depth and normal
estimation across various datasets. It also significantly enhances efficiency,
being hundreds of times faster than most existing diffusion-based methods.
A diffusion-based visual foundation model, Lotus, is introduced for dense prediction tasks, achieving state-of-the-art performance in zero-shot depth and normal estimation while improving efficiency.
text-to-image diffusion models zero-shot generalization dense prediction diffusion formulation noise prediction multi-step noising/denoising Lotus adaptation protocol direct annotation prediction single-step diffusion process detail preserver SoTA performance