Reconstruction Alignment Improves Unified Multimodal Models
Unified multimodal models (UMMs) unify visual understanding and generation
within a single architecture. However, conventional training relies on
image-text pairs (or sequences) whose captions are typically sparse and miss
fine-grained visual details--even when they use hundreds of words to describe a
simple image. We introduce Reconstruction Alignment (RecA), a
resource-efficient post-training method that leverages visual understanding
encoder embeddings as dense "text prompts," providing rich supervision without
captions. Concretely, RecA conditions a UMM on its own visual understanding
embeddings and optimizes it to reconstruct the input image with a
self-supervised reconstruction loss, thereby realigning understanding and
generation. Despite its simplicity, RecA is broadly applicable: across
autoregressive, masked-autoregressive, and diffusion-based UMMs, it
consistently improves generation and editing fidelity. With only 27 GPU-hours,
post-training with RecA substantially improves image generation performance on
GenEval (0.73rightarrow0.90) and DPGBench (80.93rightarrow88.15), while
also boosting editing benchmarks (ImgEdit 3.38rightarrow3.75, GEdit
6.94rightarrow7.25). Notably, RecA surpasses much larger open-source models
and applies broadly across diverse UMM architectures, establishing it as an
efficient and general post-training alignment strategy for UMMs
68c0e2033912ed54cf543276
Ji Xie
claimed_verified
2025-09-12T16:17:47.815Z
64e99fc07e2ec711a7138262
https://cdn-avatars.huggingface.co/v1/production/uploads/64e99fc07e2ec711a7138262/FmP3F8_UXgh9K-0gwS99A.jpeg
Ji Xie
user
sanaka87
68c0e2033912ed54cf543277
Trevor Darrell
68c0e2033912ed54cf543278
Luke Zettlemoyer
68c0e2033912ed54cf543279
XuDong Wang
Reconstruction Alignment Improves Unified Multimodal Models
Unified multimodal models (UMMs) unify visual understanding and generation
within a single architecture. However, conventional training relies on
image-text pairs (or sequences) whose captions are typically sparse and miss
fine-grained visual details--even when they use hundreds of words to describe a
simple image. We introduce Reconstruction Alignment (RecA), a
resource-efficient post-training method that leverages visual understanding
encoder embeddings as dense "text prompts," providing rich supervision without
captions. Concretely, RecA conditions a UMM on its own visual understanding
embeddings and optimizes it to reconstruct the input image with a
self-supervised reconstruction loss, thereby realigning understanding and
generation. Despite its simplicity, RecA is broadly applicable: across
autoregressive, masked-autoregressive, and diffusion-based UMMs, it
consistently improves generation and editing fidelity. With only 27 GPU-hours,
post-training with RecA substantially improves image generation performance on
GenEval (0.73rightarrow0.90) and DPGBench (80.93rightarrow88.15), while
also boosting editing benchmarks (ImgEdit 3.38rightarrow3.75, GEdit
6.94rightarrow7.25). Notably, RecA surpasses much larger open-source models
and applies broadly across diverse UMM architectures, establishing it as an
efficient and general post-training alignment strategy for UMMs
Reconstruction Alignment (RecA) is a post-training method that enhances multimodal models by using visual embeddings as dense prompts, improving image generation and editing fidelity.
Unified multimodal models Reconstruction Alignment visual understanding encoder embeddings self-supervised reconstruction loss autoregressive masked-autoregressive diffusion-based GenEval DPGBench ImgEdit GEdit