MV-RAG: Retrieval Augmented Multiview Diffusion
Text-to-3D generation approaches have advanced significantly by leveraging
pretrained 2D diffusion priors, producing high-quality and 3D-consistent
outputs. However, they often fail to produce out-of-domain (OOD) or rare
concepts, yielding inconsistent or inaccurate results. To this end, we propose
MV-RAG, a novel text-to-3D pipeline that first retrieves relevant 2D images
from a large in-the-wild 2D database and then conditions a multiview diffusion
model on these images to synthesize consistent and accurate multiview outputs.
Training such a retrieval-conditioned model is achieved via a novel hybrid
strategy bridging structured multiview data and diverse 2D image collections.
This involves training on multiview data using augmented conditioning views
that simulate retrieval variance for view-specific reconstruction, alongside
training on sets of retrieved real-world 2D images using a distinctive held-out
view prediction objective: the model predicts the held-out view from the other
views to infer 3D consistency from 2D data. To facilitate a rigorous OOD
evaluation, we introduce a new collection of challenging OOD prompts.
Experiments against state-of-the-art text-to-3D, image-to-3D, and
personalization baselines show that our approach significantly improves 3D
consistency, photorealism, and text adherence for OOD/rare concepts, while
maintaining competitive performance on standard benchmarks.
68ac574486b21a0e2e358c17
Yosef Dayani
claimed_verified
2025-08-25T12:36:57.099Z
680645323889e86c69a3daf6
/avatars/3a361a01ee36965b523d73ff292ce1ef.svg
Yosef Dayani
user
yosepyossi
68ac574486b21a0e2e358c18
Omer Benishu
claimed_verified
2025-08-25T19:31:52.191Z
64543a1ccd09ceba0e14ecfd
/avatars/d4f3aca9aa8bb4188f68ffd9e0d1f881.svg
Omer Benishu
user
omerbenishu
68ac574486b21a0e2e358c19
Sagie Benaim
claimed_verified
2025-08-25T19:31:54.300Z
6345a9b9a8c2ff9f1377faab
/avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
Sagie Benaim
user
sagiebenaim
MV-RAG: Retrieval Augmented Multiview Diffusion
Text-to-3D generation approaches have advanced significantly by leveraging
pretrained 2D diffusion priors, producing high-quality and 3D-consistent
outputs. However, they often fail to produce out-of-domain (OOD) or rare
concepts, yielding inconsistent or inaccurate results. To this end, we propose
MV-RAG, a novel text-to-3D pipeline that first retrieves relevant 2D images
from a large in-the-wild 2D database and then conditions a multiview diffusion
model on these images to synthesize consistent and accurate multiview outputs.
Training such a retrieval-conditioned model is achieved via a novel hybrid
strategy bridging structured multiview data and diverse 2D image collections.
This involves training on multiview data using augmented conditioning views
that simulate retrieval variance for view-specific reconstruction, alongside
training on sets of retrieved real-world 2D images using a distinctive held-out
view prediction objective: the model predicts the held-out view from the other
views to infer 3D consistency from 2D data. To facilitate a rigorous OOD
evaluation, we introduce a new collection of challenging OOD prompts.
Experiments against state-of-the-art text-to-3D, image-to-3D, and
personalization baselines show that our approach significantly improves 3D
consistency, photorealism, and text adherence for OOD/rare concepts, while
maintaining competitive performance on standard benchmarks.
MV-RAG enhances text-to-3D generation by retrieving 2D images and conditioning a multiview diffusion model to improve consistency and accuracy, especially for out-of-domain concepts.
pretrained 2D diffusion priors MV-RAG multiview diffusion model retrieval-conditioned model hybrid strategy augmented conditioning views held-out view prediction objective OOD prompts text-to-3D image-to-3D personalization baselines 3D consistency photorealism text adherence