DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
  Videos
Despite significant advancements in monocular depth estimation for static
images, estimating video depth in the open world remains challenging, since
open-world videos are extremely diverse in content, motion, camera movement,
and length. We present DepthCrafter, an innovative method for generating
temporally consistent long depth sequences with intricate details for
open-world videos, without requiring any supplementary information such as
camera poses or optical flow. DepthCrafter achieves generalization ability to
open-world videos by training a video-to-depth model from a pre-trained
image-to-video diffusion model, through our meticulously designed three-stage
training strategy with the compiled paired video-depth datasets. Our training
approach enables the model to generate depth sequences with variable lengths at
one time, up to 110 frames, and harvest both precise depth details and rich
content diversity from realistic and synthetic datasets. We also propose an
inference strategy that processes extremely long videos through segment-wise
estimation and seamless stitching. Comprehensive evaluations on multiple
datasets reveal that DepthCrafter achieves state-of-the-art performance in
open-world video depth estimation under zero-shot settings. Furthermore,
DepthCrafter facilitates various downstream applications, including depth-based
visual effects and conditional video generation.
66d7f2770b84833536d80d7d
Wenbo Hu
claimed_verified
2024-09-04T12:00:48.078Z
657a7458afbb0117ba15c59f
https://cdn-avatars.huggingface.co/v1/production/uploads/657a7458afbb0117ba15c59f/8_iwTS1UG_mKnfylFbLsY.jpeg
Wenbo Hu
user
wbhu-tc
66d7f2770b84833536d80d7e
Xiangjun Gao
admin_assigned
2024-09-04T08:45:38.421Z
64c0953a8137192a1e2474dc
/avatars/546405a7eaf2f60ad108ceaa0dda7d08.svg
xiangjun gao
user
xiangjun0211
66d7f2770b84833536d80d7f
Xiaoyu Li
claimed_verified
2024-09-04T09:30:49.559Z
66d8284b6bddfb32e77ddafb
/avatars/edd371ecef6e7d58b945a30bbc6095ee.svg
Xiaoyu Li
user
Xiaoyu521
66d7f2770b84833536d80d80
Sijie Zhao
admin_assigned
2024-09-04T08:45:13.134Z
652cf6bad78452c474434fa6
/avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
Sijie Zhao
user
walkingshadow
66d7f2770b84833536d80d81
Xiaodong Cun
admin_assigned
2024-09-04T08:43:56.736Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
66d7f2770b84833536d80d82
Yong Zhang
66d7f2770b84833536d80d83
Long Quan
66d7f2770b84833536d80d84
Ying Shan
admin_assigned
2024-09-04T08:43:36.253Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
  Videos
Despite significant advancements in monocular depth estimation for static
images, estimating video depth in the open world remains challenging, since
open-world videos are extremely diverse in content, motion, camera movement,
and length. We present DepthCrafter, an innovative method for generating
temporally consistent long depth sequences with intricate details for
open-world videos, without requiring any supplementary information such as
camera poses or optical flow. DepthCrafter achieves generalization ability to
open-world videos by training a video-to-depth model from a pre-trained
image-to-video diffusion model, through our meticulously designed three-stage
training strategy with the compiled paired video-depth datasets. Our training
approach enables the model to generate depth sequences with variable lengths at
one time, up to 110 frames, and harvest both precise depth details and rich
content diversity from realistic and synthetic datasets. We also propose an
inference strategy that processes extremely long videos through segment-wise
estimation and seamless stitching. Comprehensive evaluations on multiple
datasets reveal that DepthCrafter achieves state-of-the-art performance in
open-world video depth estimation under zero-shot settings. Furthermore,
DepthCrafter facilitates various downstream applications, including depth-based
visual effects and conditional video generation.
DepthCrafter generates temporally consistent depth sequences for open-world videos using a video-to-depth model derived from a pre-trained image-to-video diffusion model.
depth estimation open-world videos video-to-depth model image-to-video diffusion model three-stage training strategy paired video-depth datasets segment-wise estimation seamless stitching depth-based visual effects conditional video generation