Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
Recent work showed that large diffusion models can be reused as highly
precise monocular depth estimators by casting depth estimation as an
image-conditional image generation task. While the proposed model achieved
state-of-the-art results, high computational demands due to multi-step
inference limited its use in many scenarios. In this paper, we show that the
perceived inefficiency was caused by a flaw in the inference pipeline that has
so far gone unnoticed. The fixed model performs comparably to the best
previously reported configuration while being more than 200times faster. To
optimize for downstream task performance, we perform end-to-end fine-tuning on
top of the single-step model with task-specific losses and get a deterministic
model that outperforms all other diffusion-based depth and normal estimation
models on common zero-shot benchmarks. We surprisingly find that this
fine-tuning protocol also works directly on Stable Diffusion and achieves
comparable performance to current state-of-the-art diffusion-based depth and
normal estimation models, calling into question some of the conclusions drawn
from prior works.
66ea40c200ac2066be49bf21
Gonzalo Martin Garcia
admin_assigned
2024-09-18T08:02:39.128Z
65c78db0de6bfaeded594924
https://cdn-avatars.huggingface.co/v1/production/uploads/65c78db0de6bfaeded594924/Nngmla4CS7twp54uCOcA-.png
Gonzalo Martin Garcia
user
GonzaloMG
66ea40c200ac2066be49bf22
Karim Abou Zeid
claimed_verified
2024-09-18T07:01:25.614Z
63d50ffa695c73ecef30cc34
https://cdn-avatars.huggingface.co/v1/production/uploads/63d50ffa695c73ecef30cc34/ocahS59a6EDsQSM6uHZ5l.jpeg
Karim Abou Zeid
user
kabouzeid
66ea40c200ac2066be49bf23
Christian Schmidt
66ea40c200ac2066be49bf24
Daan de Geus
admin_assigned
2024-09-18T08:02:49.437Z
6392408ae836c1426b776472
/avatars/380b66f142197572e4d8516046c82508.svg
Daan de Geus
user
ddegeus
66ea40c200ac2066be49bf25
Alexander Hermans
claimed_verified
2024-09-19T07:07:00.765Z
631b65d65b628ccf3ce5fef6
https://cdn-avatars.huggingface.co/v1/production/uploads/631b65d65b628ccf3ce5fef6/7zFpgePJX5gOlRqPRO6sr.png
Alexander Hermans
user
Pandoro
66ea40c200ac2066be49bf26
Bastian Leibe
Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
Recent work showed that large diffusion models can be reused as highly
precise monocular depth estimators by casting depth estimation as an
image-conditional image generation task. While the proposed model achieved
state-of-the-art results, high computational demands due to multi-step
inference limited its use in many scenarios. In this paper, we show that the
perceived inefficiency was caused by a flaw in the inference pipeline that has
so far gone unnoticed. The fixed model performs comparably to the best
previously reported configuration while being more than 200times faster. To
optimize for downstream task performance, we perform end-to-end fine-tuning on
top of the single-step model with task-specific losses and get a deterministic
model that outperforms all other diffusion-based depth and normal estimation
models on common zero-shot benchmarks. We surprisingly find that this
fine-tuning protocol also works directly on Stable Diffusion and achieves
comparable performance to current state-of-the-art diffusion-based depth and
normal estimation models, calling into question some of the conclusions drawn
from prior works.
Efficient depth estimation is achieved by optimizing the inference pipeline of diffusion models and fine-tuning them for specific tasks, outperforming existing methods.
diffusion models image-conditional image generation monocular depth estimation inference pipeline end-to-end fine-tuning task-specific losses zero-shot benchmarks Stable Diffusion