One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape
  Optimization
Single image 3D reconstruction is an important but challenging task that
requires extensive knowledge of our natural world. Many existing methods solve
this problem by optimizing a neural radiance field under the guidance of 2D
diffusion models but suffer from lengthy optimization time, 3D inconsistency
results, and poor geometry. In this work, we propose a novel method that takes
a single image of any object as input and generates a full 360-degree 3D
textured mesh in a single feed-forward pass. Given a single image, we first use
a view-conditioned 2D diffusion model, Zero123, to generate multi-view images
for the input view, and then aim to lift them up to 3D space. Since traditional
reconstruction methods struggle with inconsistent multi-view predictions, we
build our 3D reconstruction module upon an SDF-based generalizable neural
surface reconstruction method and propose several critical training strategies
to enable the reconstruction of 360-degree meshes. Without costly
optimizations, our method reconstructs 3D shapes in significantly less time
than existing methods. Moreover, our method favors better geometry, generates
more 3D consistent results, and adheres more closely to the input image. We
evaluate our approach on both synthetic data and in-the-wild images and
demonstrate its superiority in terms of both mesh quality and runtime. In
addition, our approach can seamlessly support the text-to-3D task by
integrating with off-the-shelf text-to-image diffusion models.
649e46c9ab373ff26ac4d9d8
Minghua Liu
admin_assigned
2023-06-30T09:40:23.246Z
63b70a696fca9d2a1904f606
/avatars/a11741787b884cc865c1b7e9e6a7ec24.svg
Minghua Liu
user
minghua
649e46c9ab373ff26ac4d9d9
Chao Xu
admin_assigned
2023-06-30T09:46:17.347Z
6468a48cff18750165a5aad2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
Chao Xu
user
chaoxu
649e46c9ab373ff26ac4d9da
Haian Jin
claimed_verified
2023-07-03T07:27:28.138Z
64a04ad150251e408af12e01
/avatars/e4f630616a16a7fc691b5275bb62f2aa.svg
Haian Jin
user
coast01
649e46c9ab373ff26ac4d9db
Linghao Chen
649e46c9ab373ff26ac4d9dc
Mukund Varma T
admin_assigned
2023-06-30T09:48:30.611Z
61b9b452d9b770fe69ff439e
/avatars/69f49cc9b5a472393905f6b0b223a273.svg
Mukund Varma T
user
sneezygiraffe
649e46c9ab373ff26ac4d9dd
Zexiang Xu
claimed_verified
2024-10-21T07:54:58.221Z
65565b1b8616d8d740abb976
/avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
Zexiang Xu
user
zexiangxu
649e46c9ab373ff26ac4d9de
Hao Su
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape
  Optimization
Single image 3D reconstruction is an important but challenging task that
requires extensive knowledge of our natural world. Many existing methods solve
this problem by optimizing a neural radiance field under the guidance of 2D
diffusion models but suffer from lengthy optimization time, 3D inconsistency
results, and poor geometry. In this work, we propose a novel method that takes
a single image of any object as input and generates a full 360-degree 3D
textured mesh in a single feed-forward pass. Given a single image, we first use
a view-conditioned 2D diffusion model, Zero123, to generate multi-view images
for the input view, and then aim to lift them up to 3D space. Since traditional
reconstruction methods struggle with inconsistent multi-view predictions, we
build our 3D reconstruction module upon an SDF-based generalizable neural
surface reconstruction method and propose several critical training strategies
to enable the reconstruction of 360-degree meshes. Without costly
optimizations, our method reconstructs 3D shapes in significantly less time
than existing methods. Moreover, our method favors better geometry, generates
more 3D consistent results, and adheres more closely to the input image. We
evaluate our approach on both synthetic data and in-the-wild images and
demonstrate its superiority in terms of both mesh quality and runtime. In
addition, our approach can seamlessly support the text-to-3D task by
integrating with off-the-shelf text-to-image diffusion models.
A novel feed-forward method generates high-quality 360-degree 3D meshes from single images using a view-conditioned 2D diffusion model and an SDF-based neural surface reconstruction module.
neural radiance field 2D diffusion models Zero123 3D reconstruction feed-forward pass multi-view images SDF-based neural surface reconstruction 360-degree meshes mesh quality text-to-3D text-to-image diffusion models