Auto-Regressively Generating Multi-View Consistent Images
Generating multi-view images from human instructions is crucial for 3D
content creation. The primary challenges involve maintaining consistency across
multiple views and effectively synthesizing shapes and textures under diverse
conditions. In this paper, we propose the Multi-View Auto-Regressive (MV-AR)
method, which leverages an auto-regressive model to progressively generate
consistent multi-view images from arbitrary prompts. Firstly, the
next-token-prediction capability of the AR model significantly enhances its
effectiveness in facilitating progressive multi-view synthesis. When generating
widely-separated views, MV-AR can utilize all its preceding views to extract
effective reference information. Subsequently, we propose a unified model that
accommodates various prompts via architecture designing and training
strategies. To address multiple conditions, we introduce condition injection
modules for text, camera pose, image, and shape. To manage multi-modal
conditions simultaneously, a progressive training strategy is employed. This
strategy initially adopts the text-to-multi-view (t2mv) model as a baseline to
enhance the development of a comprehensive X-to-multi-view (X2mv) model through
the randomly dropping and combining conditions. Finally, to alleviate the
overfitting problem caused by limited high-quality data, we propose the
"Shuffle View" data augmentation technique, thus significantly expanding the
training data by several magnitudes. Experiments demonstrate the performance
and versatility of our MV-AR, which consistently generates consistent
multi-view images across a range of conditions and performs on par with leading
diffusion-based multi-view image generation models. Code and models will be
released at https://github.com/MILab-PKU/MVAR.
685a0cba0e4ad7e21975850c
JiaKui Hu
admin_assigned
2025-06-24T13:35:44.052Z
64ccd5cc4726a3f833831087
/avatars/6364b1ebb1d06c68245f4c786fb07ee9.svg
Hu
user
Jiakui
685a0cba0e4ad7e21975850d
Yuxiao Yang
685a0cba0e4ad7e21975850e
Jialun Liu
admin_assigned
2025-06-24T13:35:50.679Z
64df25e49bc64767a6411aa5
/avatars/301fb787186717f8a7654d82b6906ad9.svg
Jialun Liu
user
JialunLiu
685a0cba0e4ad7e21975850f
Jinbo Wu
685a0cba0e4ad7e219758510
Chen Zhao
685a0cba0e4ad7e219758511
Yanye Lu
Auto-Regressively Generating Multi-View Consistent Images
Generating multi-view images from human instructions is crucial for 3D
content creation. The primary challenges involve maintaining consistency across
multiple views and effectively synthesizing shapes and textures under diverse
conditions. In this paper, we propose the Multi-View Auto-Regressive (MV-AR)
method, which leverages an auto-regressive model to progressively generate
consistent multi-view images from arbitrary prompts. Firstly, the
next-token-prediction capability of the AR model significantly enhances its
effectiveness in facilitating progressive multi-view synthesis. When generating
widely-separated views, MV-AR can utilize all its preceding views to extract
effective reference information. Subsequently, we propose a unified model that
accommodates various prompts via architecture designing and training
strategies. To address multiple conditions, we introduce condition injection
modules for text, camera pose, image, and shape. To manage multi-modal
conditions simultaneously, a progressive training strategy is employed. This
strategy initially adopts the text-to-multi-view (t2mv) model as a baseline to
enhance the development of a comprehensive X-to-multi-view (X2mv) model through
the randomly dropping and combining conditions. Finally, to alleviate the
overfitting problem caused by limited high-quality data, we propose the
"Shuffle View" data augmentation technique, thus significantly expanding the
training data by several magnitudes. Experiments demonstrate the performance
and versatility of our MV-AR, which consistently generates consistent
multi-view images across a range of conditions and performs on par with leading
diffusion-based multi-view image generation models. Code and models will be
released at https://github.com/MILab-PKU/MVAR.
The Multi-View Auto-Regressive (MV-AR) method uses an auto-regressive model to generate consistent multi-view images from prompts, addressing challenges in shape and texture synthesis across diverse conditions.
Multi-View Auto-Regressive MV-AR auto-regressive model next-token-prediction condition injection modules text-to-multi-view X-to-multi-view progressive training strategy Shuffle View data augmentation