MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
  Model
Open-world 3D reconstruction models have recently garnered significant
attention. However, without sufficient 3D inductive bias, existing methods
typically entail expensive training costs and struggle to extract high-quality
3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
model that explicitly leverages 3D native structure, input guidance, and
training supervision. Specifically, instead of using a triplane representation,
we store features in 3D sparse voxels and combine transformers with 3D
convolutions to leverage an explicit 3D structure and projective bias. In
addition to sparse-view RGB input, we require the network to take input and
generate corresponding normal maps. The input normal maps can be predicted by
2D diffusion models, significantly aiding in the guidance and refinement of the
geometry's learning. Moreover, by combining Signed Distance Function (SDF)
supervision with surface rendering, we directly learn to generate high-quality
meshes without the need for complex multi-stage training processes. By
incorporating these explicit 3D biases, MeshFormer can be trained efficiently
and deliver high-quality textured meshes with fine-grained geometric details.
It can also be integrated with 2D diffusion models to enable fast
single-image-to-3D and text-to-3D tasks. Project page:
https://meshformer3d.github.io
66c3f562d9a7974122b19504
Minghua Liu
66c3f562d9a7974122b19505
Chong Zeng
claimed_verified
2024-08-22T19:51:49.317Z
60747cbf3ea03830676542b5
https://cdn-avatars.huggingface.co/v1/production/uploads/60747cbf3ea03830676542b5/wGr1Jzz520JM9nZ-UcLyb.png
Chong Zeng
user
NCJ
66c3f562d9a7974122b19506
Xinyue Wei
66c3f562d9a7974122b19507
Ruoxi Shi
66c3f562d9a7974122b19508
Linghao Chen
66c3f562d9a7974122b19509
Chao Xu
claimed_verified
2024-08-22T19:51:46.522Z
6468a48cff18750165a5aad2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
Chao Xu
user
chaoxu
66c3f562d9a7974122b1950a
Mengqi Zhang
66c3f562d9a7974122b1950b
Zhaoning Wang
66c3f562d9a7974122b1950c
Xiaoshuai Zhang
66c3f562d9a7974122b1950d
Isabella Liu
66c3f562d9a7974122b1950e
Hongzhi Wu
66c3f562d9a7974122b1950f
Hao Su
MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
  Model
Open-world 3D reconstruction models have recently garnered significant
attention. However, without sufficient 3D inductive bias, existing methods
typically entail expensive training costs and struggle to extract high-quality
3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
model that explicitly leverages 3D native structure, input guidance, and
training supervision. Specifically, instead of using a triplane representation,
we store features in 3D sparse voxels and combine transformers with 3D
convolutions to leverage an explicit 3D structure and projective bias. In
addition to sparse-view RGB input, we require the network to take input and
generate corresponding normal maps. The input normal maps can be predicted by
2D diffusion models, significantly aiding in the guidance and refinement of the
geometry's learning. Moreover, by combining Signed Distance Function (SDF)
supervision with surface rendering, we directly learn to generate high-quality
meshes without the need for complex multi-stage training processes. By
incorporating these explicit 3D biases, MeshFormer can be trained efficiently
and deliver high-quality textured meshes with fine-grained geometric details.
It can also be integrated with 2D diffusion models to enable fast
single-image-to-3D and text-to-3D tasks. Project page:
https://meshformer3d.github.io
MeshFormer, a sparse-view 3D reconstruction model, enhances geometry learning and mesh generation by integrating 3D sparse voxels, transformers, 3D convolutions, 2D diffusion models, SDF supervision, and surface rendering.
sparse-view reconstruction 3D sparse voxels transformers 3D convolutions 2D diffusion models Signed Distance Function SDF supervision surface rendering