Consolidating Reinforcement Learning for Multimodal Discrete Diffusion
  Models
Optimizing discrete diffusion model (DDM) with rewards remains a challenge:
the non-autoregressive paradigm makes importance sampling intractable and
rollout complex, puzzling reinforcement learning methods such as Group Relative
Policy Optimization (GRPO). In this study, we introduce MaskGRPO, the first
viable approach to enable scalable multimodal reinforcement learning in
discrete diffusion with effective importance sampling and modality-specific
adaptations. To this end, we first clarify the theoretical foundation for DDMs,
which facilitates building an importance estimator that captures valuable token
fluctuation for gradient updates. We then delicately tailored the rollout
method for visual sequences, which yields diverse completions and reliable
optimization gradients. Upon math reasoning, coding, and visual generation
benchmarks, MaskGRPO brings more stable and efficient updates, leading to
stronger reasoning performance and better generation quality. This study
establishes MaskGRPO as a systematic policy optimization approach and the first
practical way for discretized visual diffusion.
68e3752773e20ab5778420f1
Tianren Ma
claimed_verified
2025-10-06T12:48:37.179Z
65ab7ed7d6b10af911dc8085
/avatars/0b033760f3a785ac82a1e913b28e9dec.svg
Tianren Ma
user
Vivre
68e3752773e20ab5778420f2
Mu Zhang
68e3752773e20ab5778420f3
Yibing Wang
68e3752773e20ab5778420f4
Qixiang Ye
Consolidating Reinforcement Learning for Multimodal Discrete Diffusion
  Models
Optimizing discrete diffusion model (DDM) with rewards remains a challenge:
the non-autoregressive paradigm makes importance sampling intractable and
rollout complex, puzzling reinforcement learning methods such as Group Relative
Policy Optimization (GRPO). In this study, we introduce MaskGRPO, the first
viable approach to enable scalable multimodal reinforcement learning in
discrete diffusion with effective importance sampling and modality-specific
adaptations. To this end, we first clarify the theoretical foundation for DDMs,
which facilitates building an importance estimator that captures valuable token
fluctuation for gradient updates. We then delicately tailored the rollout
method for visual sequences, which yields diverse completions and reliable
optimization gradients. Upon math reasoning, coding, and visual generation
benchmarks, MaskGRPO brings more stable and efficient updates, leading to
stronger reasoning performance and better generation quality. This study
establishes MaskGRPO as a systematic policy optimization approach and the first
practical way for discretized visual diffusion.
MaskGRPO addresses challenges in optimizing discrete diffusion models with rewards through effective importance sampling and modality-specific adaptations, improving reasoning and generation quality.
discrete diffusion model DDM rewards non-autoregressive paradigm importance sampling rollout Group Relative Policy Optimization GRPO MaskGRPO multimodal reinforcement learning importance estimator token fluctuation gradient updates visual sequences diverse completions optimization gradients math reasoning visual generation reasoning performance generation quality