One Diffusion to Generate Them All
We introduce OneDiffusion, a versatile, large-scale diffusion model that
seamlessly supports bidirectional image synthesis and understanding across
diverse tasks. It enables conditional generation from inputs such as text,
depth, pose, layout, and semantic maps, while also handling tasks like image
deblurring, upscaling, and reverse processes such as depth estimation and
segmentation. Additionally, OneDiffusion allows for multi-view generation,
camera pose estimation, and instant personalization using sequential image
inputs. Our model takes a straightforward yet effective approach by treating
all tasks as frame sequences with varying noise scales during training,
allowing any frame to act as a conditioning image at inference time. Our
unified training framework removes the need for specialized architectures,
supports scalable multi-task training, and adapts smoothly to any resolution,
enhancing both generalization and scalability. Experimental results demonstrate
competitive performance across tasks in both generation and prediction such as
text-to-image, multiview generation, ID preservation, depth estimation and
camera pose estimation despite relatively small training dataset. Our code and
checkpoint are freely available at https://github.com/lehduong/OneDiffusion
674569ce9930041fff3f7bf0
Duong H. Le
674569ce9930041fff3f7bf1
Tuan Pham
admin_assigned
2024-11-26T22:47:18.582Z
64d3d401c1f4edb3faeae442
/avatars/fc814ea60df1b7c552230a1de03fdd11.svg
Tuan Pham
user
tuanpham
674569ce9930041fff3f7bf2
Sangho Lee
claimed_verified
2024-12-04T09:46:45.705Z
664fc02d6f16bbd9a168654d
/avatars/9f726ede9973706bc198845464238452.svg
Sangho Lee
user
sanghol
674569ce9930041fff3f7bf3
Christopher Clark
admin_assigned
2024-11-26T22:47:00.768Z
6523efa8db79e53948543f79
/avatars/11a9e82b1d6a578526e261c33ae097e3.svg
Christopher Clark
user
ChristopherClark123
674569ce9930041fff3f7bf4
Aniruddha Kembhavi
674569ce9930041fff3f7bf5
Stephan Mandt
admin_assigned
2024-11-26T22:46:44.367Z
6645f8a8a781869d13c40624
/avatars/726e618765daa5e4987cc32e91994aaa.svg
Stephan Mandt
user
smandt
674569ce9930041fff3f7bf6
Ranjay Krishna
admin_assigned
2024-11-26T22:46:39.290Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
674569ce9930041fff3f7bf7
Jiasen Lu
admin_assigned
2024-11-26T22:46:33.522Z
62b6b0397523238923221df9
/avatars/77068771dd51df7519516cd502a88789.svg
Jiasenlu
user
Jiasenlu
One Diffusion to Generate Them All
We introduce OneDiffusion, a versatile, large-scale diffusion model that
seamlessly supports bidirectional image synthesis and understanding across
diverse tasks. It enables conditional generation from inputs such as text,
depth, pose, layout, and semantic maps, while also handling tasks like image
deblurring, upscaling, and reverse processes such as depth estimation and
segmentation. Additionally, OneDiffusion allows for multi-view generation,
camera pose estimation, and instant personalization using sequential image
inputs. Our model takes a straightforward yet effective approach by treating
all tasks as frame sequences with varying noise scales during training,
allowing any frame to act as a conditioning image at inference time. Our
unified training framework removes the need for specialized architectures,
supports scalable multi-task training, and adapts smoothly to any resolution,
enhancing both generalization and scalability. Experimental results demonstrate
competitive performance across tasks in both generation and prediction such as
text-to-image, multiview generation, ID preservation, depth estimation and
camera pose estimation despite relatively small training dataset. Our code and
checkpoint are freely available at https://github.com/lehduong/OneDiffusion
OneDiffusion is a versatile diffusion model that supports bidirectional image synthesis and understanding, enabling tasks like conditional generation, image deblurring, and camera pose estimation through unified frame sequence training and conditioning.
diffusion model bidirectional image synthesis understanding conditional generation text depth pose layout semantic maps image deblurring upscaling depth estimation segmentation multi-view generation camera pose estimation instant personalization frame sequences noise scales unified training framework scalable multi-task training resolution adaptability generalization scalability text-to-image multiview generation ID preservation