Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Encouraged by the growing availability of pre-trained 2D diffusion models,
image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
making remarkable progress. Most existing methods combine novel-view lifting
from 2D diffusion models which usually take the reference image as a condition
while applying hard L2 image supervision at the reference view. Yet heavily
adhering to the image is prone to corrupting the inductive knowledge of the 2D
diffusion model leading to flat or distorted 3D generation frequently. In this
work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
an image-to-3D generation pipeline that takes only an image CLIP embedding as
input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
angle by solely resting on the SDS loss. The core of our framework lies in a
two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
diffusion model by substituting its text encoder with an image encoder, by
which the model preliminarily acquires image-to-image capabilities. Secondly,
we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
combines noisy multi-view images with the noise-free reference image as an
explicit condition. CLIP embedding is sent to the diffusion model throughout
the whole process while reference images are discarded once after fine-tuning.
As a result, with a single image CLIP embedding, Isotropic3D is capable of
generating multi-view mutually consistent images and also a 3D model with more
symmetrical and neat content, well-proportioned geometry, rich colored texture,
and less distortion compared with existing image-to-3D methods while still
preserving the similarity to the reference image to a large extent. The project
page is available at https://isotropic3d.github.io/. The code and models are
available at https://github.com/pkunliu/Isotropic3D.
65f79f593a8814cf2f4fe9c8
Pengkun Liu
admin_assigned
2024-03-18T09:11:57.663Z
63daf1700cc3bc12bc09de74
/avatars/773a14ead497813c9d102bea1222d681.svg
Pengkun Liu
user
pengkunliu
65f79f593a8814cf2f4fe9c9
Yikai Wang
65f79f593a8814cf2f4fe9ca
Fuchun Sun
65f79f593a8814cf2f4fe9cb
Jiafang Li
admin_assigned
2024-03-18T09:12:44.232Z
645633adc8f569b995d7628b
/avatars/5ffd94289a53b6690f9f7db37557497d.svg
jiafang.li
user
jiafang
65f79f593a8814cf2f4fe9cc
Hang Xiao
65f79f593a8814cf2f4fe9cd
Hongxiang Xue
65f79f593a8814cf2f4fe9ce
Xinzhou Wang
admin_assigned
2024-03-18T09:13:16.794Z
6459c45b5a263a4c87b7c639
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t3iK1wPBOP-h_kLBzZA_h.png
Xinzhou Wang
user
zz7379
Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Encouraged by the growing availability of pre-trained 2D diffusion models,
image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
making remarkable progress. Most existing methods combine novel-view lifting
from 2D diffusion models which usually take the reference image as a condition
while applying hard L2 image supervision at the reference view. Yet heavily
adhering to the image is prone to corrupting the inductive knowledge of the 2D
diffusion model leading to flat or distorted 3D generation frequently. In this
work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
an image-to-3D generation pipeline that takes only an image CLIP embedding as
input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
angle by solely resting on the SDS loss. The core of our framework lies in a
two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
diffusion model by substituting its text encoder with an image encoder, by
which the model preliminarily acquires image-to-image capabilities. Secondly,
we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
combines noisy multi-view images with the noise-free reference image as an
explicit condition. CLIP embedding is sent to the diffusion model throughout
the whole process while reference images are discarded once after fine-tuning.
As a result, with a single image CLIP embedding, Isotropic3D is capable of
generating multi-view mutually consistent images and also a 3D model with more
symmetrical and neat content, well-proportioned geometry, rich colored texture,
and less distortion compared with existing image-to-3D methods while still
preserving the similarity to the reference image to a large extent. The project
page is available at https://isotropic3d.github.io/. The code and models are
available at https://github.com/pkunliu/Isotropic3D.
Isotropic3D generates 3D models and multi-view images from a single image CLIP embedding using two-stage diffusion model fine-tuning with Score Distillation Sampling and Explicit Multi-view Attention.
Score Distillation Sampling SDS image-to-3D generation novel-view lifting latent space isotropic optimization azimuth angle text-to-3D diffusion model image encoder Explicit Multi-view Attention multi-view images 3D model symmetrical neat content well-proportioned geometry rich colored texture distortion