Pushing Auto-regressive Models for 3D Shape Generation at Capacity and
  Scalability
Auto-regressive models have achieved impressive results in 2D image
generation by modeling joint distributions in grid space. In this paper, we
extend auto-regressive models to 3D domains, and seek a stronger ability of 3D
shape generation by improving auto-regressive models at capacity and
scalability simultaneously. Firstly, we leverage an ensemble of publicly
available 3D datasets to facilitate the training of large-scale models. It
consists of a comprehensive collection of approximately 900,000 objects, with
multiple properties of meshes, points, voxels, rendered images, and text
captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our
model to learn from a wide range of object variations. However, directly
applying 3D auto-regression encounters critical challenges of high
computational demands on volumetric grids and ambiguous auto-regressive order
along grid dimensions, resulting in inferior quality of 3D shapes. To this end,
we then present a novel framework Argus3D in terms of capacity. Concretely, our
approach introduces discrete representation learning based on a latent vector
instead of volumetric grids, which not only reduces computational costs but
also preserves essential geometric details by learning the joint distributions
in a more tractable order. The capacity of conditional generation can thus be
realized by simply concatenating various conditioning inputs to the latent
vector, such as point clouds, categories, images, and texts. In addition,
thanks to the simplicity of our model architecture, we naturally scale up our
approach to a larger model with an impressive 3.6 billion parameters, further
enhancing the quality of versatile 3D generation. Extensive experiments on four
generation tasks demonstrate that Argus3D can synthesize diverse and faithful
shapes across multiple categories, achieving remarkable performance.
65d424d0c3df2fd80eebd14b
Xuelin Qian
65d424d0c3df2fd80eebd14c
Yu Wang
65d424d0c3df2fd80eebd14d
Simian Luo
admin_assigned
2024-02-20T10:59:54.705Z
652a4f174d0eef4a2bf85096
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ofYN04WWtD60oeDP45pQN.jpeg
Simian Luo
user
SimianLuo
65d424d0c3df2fd80eebd14e
Yinda Zhang
65d424d0c3df2fd80eebd14f
Ying Tai
admin_assigned
2024-02-20T11:00:21.165Z
65734004769f3ee9bde1af10
/avatars/d6310ed861972fd691687d8f47413f33.svg
Ying Tai
user
yingtai
65d424d0c3df2fd80eebd150
Zhenyu Zhang
admin_assigned
2024-02-20T11:01:42.415Z
642e6cc9be01b88c9446fe3d
/avatars/43a8f9bf35bea54bcda547174d921aa4.svg
Zhenyu Zhang
user
JesseZhang
65d424d0c3df2fd80eebd151
Chengjie Wang
admin_assigned
2024-02-20T11:01:49.947Z
65729087379284b904a6d81d
/avatars/0e7968b1d9221d523a54e3ac787e449d.svg
Chengjie Wang
user
chengjie-wang
65d424d0c3df2fd80eebd152
Xiangyang Xue
65d424d0c3df2fd80eebd153
Bo Zhao
claimed_verified
2024-02-20T08:38:49.386Z
64acafac1aee69ece03af05a
/avatars/6380777852546dbd432fed5544d5db0b.svg
Bo Zhao
user
Z-LIRA
65d424d0c3df2fd80eebd154
Tiejun Huang
65d424d0c3df2fd80eebd155
Yunsheng Wu
65d424d0c3df2fd80eebd156
Yanwei Fu
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and
  Scalability
Auto-regressive models have achieved impressive results in 2D image
generation by modeling joint distributions in grid space. In this paper, we
extend auto-regressive models to 3D domains, and seek a stronger ability of 3D
shape generation by improving auto-regressive models at capacity and
scalability simultaneously. Firstly, we leverage an ensemble of publicly
available 3D datasets to facilitate the training of large-scale models. It
consists of a comprehensive collection of approximately 900,000 objects, with
multiple properties of meshes, points, voxels, rendered images, and text
captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our
model to learn from a wide range of object variations. However, directly
applying 3D auto-regression encounters critical challenges of high
computational demands on volumetric grids and ambiguous auto-regressive order
along grid dimensions, resulting in inferior quality of 3D shapes. To this end,
we then present a novel framework Argus3D in terms of capacity. Concretely, our
approach introduces discrete representation learning based on a latent vector
instead of volumetric grids, which not only reduces computational costs but
also preserves essential geometric details by learning the joint distributions
in a more tractable order. The capacity of conditional generation can thus be
realized by simply concatenating various conditioning inputs to the latent
vector, such as point clouds, categories, images, and texts. In addition,
thanks to the simplicity of our model architecture, we naturally scale up our
approach to a larger model with an impressive 3.6 billion parameters, further
enhancing the quality of versatile 3D generation. Extensive experiments on four
generation tasks demonstrate that Argus3D can synthesize diverse and faithful
shapes across multiple categories, achieving remarkable performance.
Argus3D framework uses discrete representation learning with a latent vector to enhance 3D shape generation capacity and scalability, significantly improving performance across various generation tasks.
auto-regressive models volumetric grids latent vector discrete representation learning conditional generation point clouds categories images texts Argus3D