SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Recently, remarkable progress has been made in Unified Multimodal Models
(UMMs), which integrate vision-language generation and understanding
capabilities within a single framework. However, a significant gap exists where
a model's strong visual understanding often fails to transfer to its visual
generation. A model might correctly understand an image based on user
instructions, yet be unable to generate a faithful image from text prompts.
This phenomenon directly raises a compelling question: Can a model achieve
self-improvement by using its understanding module to reward its generation
module? To bridge this gap and achieve self-improvement, we introduce SRUM, a
self-rewarding post-training framework that can be directly applied to existing
UMMs of various designs. SRUM creates a feedback loop where the model's own
understanding module acts as an internal ``evaluator'', providing corrective
signals to improve its generation module, without requiring additional
human-labeled data. To ensure this feedback is comprehensive, we designed a
global-local dual reward system. To tackle the inherent structural complexity
of images, this system offers multi-scale guidance: a global reward
ensures the correctness of the overall visual semantics and layout, while a
local reward refines fine-grained, object-level fidelity. SRUM leads
to powerful capabilities and shows strong generalization, boosting performance
on T2I-CompBench from 82.18 to 88.37 and on T2I-ReasonBench from 43.82
to 46.75. Overall, our work establishes a powerful new paradigm for
enabling a UMMs' understanding module to guide and enhance its own generation
via self-rewarding.
68ef011a486b78128f0e33ce
Weiyang Jin
claimed_verified
2025-10-15T15:24:54.354Z
66608add236f958513d21d2e
/avatars/53eca0891c98cbb93be899885160a983.svg
Weiyang Jin
user
Wayne-King
68ef011a486b78128f0e33cf
Yuwei Niu
68ef011a486b78128f0e33d0
Jiaqi Liao
68ef011a486b78128f0e33d1
Chengqi Duan
68ef011a486b78128f0e33d2
Aoxue Li
68ef011a486b78128f0e33d3
Shenghua Gao
68ef011a486b78128f0e33d4
Xihui Liu
SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Recently, remarkable progress has been made in Unified Multimodal Models
(UMMs), which integrate vision-language generation and understanding
capabilities within a single framework. However, a significant gap exists where
a model's strong visual understanding often fails to transfer to its visual
generation. A model might correctly understand an image based on user
instructions, yet be unable to generate a faithful image from text prompts.
This phenomenon directly raises a compelling question: Can a model achieve
self-improvement by using its understanding module to reward its generation
module? To bridge this gap and achieve self-improvement, we introduce SRUM, a
self-rewarding post-training framework that can be directly applied to existing
UMMs of various designs. SRUM creates a feedback loop where the model's own
understanding module acts as an internal ``evaluator'', providing corrective
signals to improve its generation module, without requiring additional
human-labeled data. To ensure this feedback is comprehensive, we designed a
global-local dual reward system. To tackle the inherent structural complexity
of images, this system offers multi-scale guidance: a global reward
ensures the correctness of the overall visual semantics and layout, while a
local reward refines fine-grained, object-level fidelity. SRUM leads
to powerful capabilities and shows strong generalization, boosting performance
on T2I-CompBench from 82.18 to 88.37 and on T2I-ReasonBench from 43.82
to 46.75. Overall, our work establishes a powerful new paradigm for
enabling a UMMs' understanding module to guide and enhance its own generation
via self-rewarding.