World-consistent Video Diffusion with Explicit 3D Modeling
Recent advancements in diffusion models have set new benchmarks in image and
video generation, enabling realistic visual synthesis across single- and
multi-frame contexts. However, these models still struggle with efficiently and
explicitly generating 3D-consistent content. To address this, we propose
World-consistent Video Diffusion (WVD), a novel framework that incorporates
explicit 3D supervision using XYZ images, which encode global 3D coordinates
for each image pixel. More specifically, we train a diffusion transformer to
learn the joint distribution of RGB and XYZ frames. This approach supports
multi-task adaptability via a flexible inpainting strategy. For example, WVD
can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
using XYZ projections along a specified camera trajectory. In doing so, WVD
unifies tasks like single-image-to-3D generation, multi-view stereo, and
camera-controlled video generation. Our approach demonstrates competitive
performance across multiple benchmarks, providing a scalable solution for
3D-consistent video and image generation with a single pretrained model.
674f26532836633b518dd279
Qihang Zhang
admin_assigned
2024-12-03T16:05:47.586Z
64da979f3a7ab21ea7d499c3
/avatars/84fd959092e5a5d3b29720f6f7464b25.svg
Zhang
user
QihangZhang
674f26532836633b518dd27a
Shuangfei Zhai
claimed_verified
2024-12-13T15:23:17.582Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
674f26532836633b518dd27b
Miguel Angel Bautista
admin_assigned
2024-12-03T16:05:28.650Z
65dd58aa29b761fff4bd3a4c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SD7Vh0i5HwJJsQxXpSTFa.jpeg
Miguel Angel Bautista
user
FlippyCode
674f26532836633b518dd27c
Kevin Miao
674f26532836633b518dd27d
Alexander Toshev
admin_assigned
2024-12-03T16:04:58.423Z
6541636c4f939214d3a0c742
/avatars/c9507d6ec2b81681461837c8261703c7.svg
Alexander Toshev
user
toshev
674f26532836633b518dd27e
Joshua Susskind
674f26532836633b518dd27f
Jiatao Gu
admin_assigned
2024-12-03T16:04:43.809Z
6164e72d73996c363c52e66d
https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
Jiatao Gu
user
thomagram
World-consistent Video Diffusion with Explicit 3D Modeling
Recent advancements in diffusion models have set new benchmarks in image and
video generation, enabling realistic visual synthesis across single- and
multi-frame contexts. However, these models still struggle with efficiently and
explicitly generating 3D-consistent content. To address this, we propose
World-consistent Video Diffusion (WVD), a novel framework that incorporates
explicit 3D supervision using XYZ images, which encode global 3D coordinates
for each image pixel. More specifically, we train a diffusion transformer to
learn the joint distribution of RGB and XYZ frames. This approach supports
multi-task adaptability via a flexible inpainting strategy. For example, WVD
can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
using XYZ projections along a specified camera trajectory. In doing so, WVD
unifies tasks like single-image-to-3D generation, multi-view stereo, and
camera-controlled video generation. Our approach demonstrates competitive
performance across multiple benchmarks, providing a scalable solution for
3D-consistent video and image generation with a single pretrained model.
A new framework, World-consistent Video Diffusion (WVD), uses XYZ images for 3D supervision to improve the generation of 3D-consistent content in videos and images.
diffusion models World-consistent Video Diffusion (WVD) XYZ images global 3D coordinates diffusion transformer joint distribution RGB flexible inpainting strategy single-image-to-3D generation multi-view stereo camera-controlled video generation