Can Understanding and Generation Truly Benefit Together -- or Just
  Coexist?
In this paper, we introduce an insightful paradigm through the Auto-Encoder
lens-understanding as the encoder (I2T) that compresses images into text, and
generation as the decoder (T2I) that reconstructs images from that text. Using
reconstruction fidelity as the unified training objective, we enforce the
coherent bidirectional information flow between the understanding and
generation processes, bringing mutual gains. To implement this, we propose UAE,
a novel framework for unified multimodal learning. We begin by pre-training the
decoder with large-scale long-context image captions to capture fine-grained
semantic and complex spatial relationships. We then propose Unified-GRPO via
reinforcement learning (RL), which covers three stages: (1) A cold-start phase
to gently initialize both encoder and decoder with a semantic reconstruction
loss; (2) Generation for Understanding, where the encoder is trained to
generate informative captions that maximize the decoder's reconstruction
quality, enhancing its visual understanding; (3) Understanding for Generation,
where the decoder is refined to reconstruct from these captions, forcing it to
leverage every detail and improving its long-context instruction following and
generation fidelity. For evaluation, we introduce Unified-Bench, the first
benchmark tailored to assess the degree of unification of the UMMs. A
surprising "aha moment" arises within the multimodal learning domain: as RL
progresses, the encoder autonomously produces more descriptive captions, while
the decoder simultaneously demonstrates a profound ability to understand these
intricate descriptions, resulting in reconstructions of striking fidelity.
68c3b899fc1747b912403acb
Zhiyuan Yan
68c3b899fc1747b912403acc
Kaiqing Lin
68c3b899fc1747b912403acd
Zongjian Li
68c3b899fc1747b912403ace
Junyan Ye
68c3b899fc1747b912403acf
Hui Han
68c3b899fc1747b912403ad0
Zhendong Wang
68c3b899fc1747b912403ad1
Hao Liu
68c3b899fc1747b912403ad2
Bin Lin
68c3b899fc1747b912403ad3
Hao Li
claimed_verified
2025-09-12T16:07:31.281Z
6678e670a2873f979b492c5b
https://cdn-avatars.huggingface.co/v1/production/uploads/6678e670a2873f979b492c5b/qlhNpwbbfL00SdpnFhUnn.png
HaoLi
user
OzymandisLi
68c3b899fc1747b912403ad4
Xue Xu
68c3b899fc1747b912403ad5
Xinyan Xiao
68c3b899fc1747b912403ad6
Jingdong Wang
68c3b899fc1747b912403ad7
Haifeng Wang
68c3b899fc1747b912403ad8
Li Yuan
Can Understanding and Generation Truly Benefit Together -- or Just
  Coexist?
In this paper, we introduce an insightful paradigm through the Auto-Encoder
lens-understanding as the encoder (I2T) that compresses images into text, and
generation as the decoder (T2I) that reconstructs images from that text. Using
reconstruction fidelity as the unified training objective, we enforce the
coherent bidirectional information flow between the understanding and
generation processes, bringing mutual gains. To implement this, we propose UAE,
a novel framework for unified multimodal learning. We begin by pre-training the
decoder with large-scale long-context image captions to capture fine-grained
semantic and complex spatial relationships. We then propose Unified-GRPO via
reinforcement learning (RL), which covers three stages: (1) A cold-start phase
to gently initialize both encoder and decoder with a semantic reconstruction
loss; (2) Generation for Understanding, where the encoder is trained to
generate informative captions that maximize the decoder's reconstruction
quality, enhancing its visual understanding; (3) Understanding for Generation,
where the decoder is refined to reconstruct from these captions, forcing it to
leverage every detail and improving its long-context instruction following and
generation fidelity. For evaluation, we introduce Unified-Bench, the first
benchmark tailored to assess the degree of unification of the UMMs. A
surprising "aha moment" arises within the multimodal learning domain: as RL
progresses, the encoder autonomously produces more descriptive captions, while
the decoder simultaneously demonstrates a profound ability to understand these
intricate descriptions, resulting in reconstructions of striking fidelity.
A novel framework UAE uses reinforcement learning to unify image-to-text and text-to-image processes, enhancing mutual understanding and generation fidelity.
Auto-Encoder encoder decoder reconstruction fidelity unified multimodal learning UAE Unified-GRPO reinforcement learning semantic reconstruction loss Unified-Bench multimodal learning