Taming Masked Diffusion Language Models via Consistency Trajectory
  Reinforcement Learning with Fewer Decoding Step
Masked diffusion language models (MDLMs) have recently emerged as a promising
alternative to autoregressive (AR) language models, offering properties such as
parallel decoding, flexible generation orders, and the potential for fewer
inference steps. Despite these advantages, decoding strategies and
reinforcement learning (RL) algorithms tailored for MDLMs remain underexplored.
A naive approach is to directly transfer techniques well-established for AR
models to MDLMs. However, this raises an immediate question: Is such a naive
transfer truly optimal? For example, 1) Block-wise and semi-AR decoding
strategies are not employed during the training of MDLMs, so why do they
outperform full diffusion-style decoding during inference? 2) Applying RL
algorithms designed for AR models directly to MDLMs exhibits a
training-inference inconsistency, since MDLM decoding are non-causal
(parallel). This results in inconsistencies between the rollout trajectory and
the optimization trajectory. To address these challenges, we propose EOS Early
Rejection (EOSER) and Ascending Step-Size (ASS) decoding scheduler, which
unlock the potential of MDLMs to perform full diffusion-style decoding,
achieving competitive performance with fewer decoding steps. Additionally, we
introduce Consistency Trajectory Group Relative Policy Optimization (CJ-GRPO)
for taming MDLMs, which emphasizes the consistency between rollout trajectory
and optimization trajectory, and reduces the optimization errors caused by
skip-step optimization. We conduct extensive experiments on reasoning tasks,
such as mathematical and planning benchmarks, using LLaDA-8B-Instruct. The
results demonstrate that the proposed EOSER and ASS mechanisms, together with
CJ-GRPO, hold significant promise for effectively and efficiently taming MDLMs.
Code: https://github.com/yjyddq/EOSER-ASS-RL.
68db800bd2bf1f4b15ec77d0
Jingyi Yang
claimed_verified
2025-09-30T07:12:40.232Z
64f73a44102fbfb26410962e
/avatars/328302a495de6a4418be835456d1d3c6.svg
jingyi Yang
user
JY-Young
68db800bd2bf1f4b15ec77d1
Guanxu Chen
claimed_verified
2025-10-13T16:09:36.714Z
6763f633e90f0e38c79a69cf
/avatars/125c3f93229a3a794863723ac0fa9088.svg
Frank  Chen
user
quantumfr
68db800bd2bf1f4b15ec77d2
Xuhao Hu
claimed_verified
2025-10-10T09:49:50.608Z
6372813520a58a5e14c596a3
/avatars/9135151259db3e5b9c8969e1d00c949d.svg
XuHao Hu
user
Foreshhh
68db800bd2bf1f4b15ec77d3
Jing Shao
Taming Masked Diffusion Language Models via Consistency Trajectory
  Reinforcement Learning with Fewer Decoding Step
Masked diffusion language models (MDLMs) have recently emerged as a promising
alternative to autoregressive (AR) language models, offering properties such as
parallel decoding, flexible generation orders, and the potential for fewer
inference steps. Despite these advantages, decoding strategies and
reinforcement learning (RL) algorithms tailored for MDLMs remain underexplored.
A naive approach is to directly transfer techniques well-established for AR
models to MDLMs. However, this raises an immediate question: Is such a naive
transfer truly optimal? For example, 1) Block-wise and semi-AR decoding
strategies are not employed during the training of MDLMs, so why do they
outperform full diffusion-style decoding during inference? 2) Applying RL
algorithms designed for AR models directly to MDLMs exhibits a
training-inference inconsistency, since MDLM decoding are non-causal
(parallel). This results in inconsistencies between the rollout trajectory and
the optimization trajectory. To address these challenges, we propose EOS Early
Rejection (EOSER) and Ascending Step-Size (ASS) decoding scheduler, which
unlock the potential of MDLMs to perform full diffusion-style decoding,
achieving competitive performance with fewer decoding steps. Additionally, we
introduce Consistency Trajectory Group Relative Policy Optimization (CJ-GRPO)
for taming MDLMs, which emphasizes the consistency between rollout trajectory
and optimization trajectory, and reduces the optimization errors caused by
skip-step optimization. We conduct extensive experiments on reasoning tasks,
such as mathematical and planning benchmarks, using LLaDA-8B-Instruct. The
results demonstrate that the proposed EOSER and ASS mechanisms, together with
CJ-GRPO, hold significant promise for effectively and efficiently taming MDLMs.
Code: https://github.com/yjyddq/EOSER-ASS-RL.
Proposed decoding strategies and reinforcement learning algorithms improve the performance and efficiency of masked diffusion language models during inference.
Masked diffusion language models autoregressive models parallel decoding flexible generation orders block-wise decoding semi-AR decoding full diffusion-style decoding reinforcement learning non-causal decoding rollout trajectory optimization trajectory EOS Early Rejection Ascending Step-Size Consistency Trajectory Group Relative Policy Optimization LLaDA-8B-Instruct