Inpainting-Guided Policy Optimization for Diffusion Large Language
  Models
Masked diffusion large language models (dLLMs) are emerging as promising
alternatives to autoregressive LLMs, offering competitive performance while
supporting unique generation capabilities such as inpainting. We explore how
inpainting can inform RL algorithm design for dLLMs. Aligning LLMs with
reinforcement learning faces an exploration challenge: sparse reward signals
and sample waste when models fail to discover correct solutions. While this
inefficiency affects LLMs broadly, dLLMs offer a distinctive opportunity--their
inpainting ability can guide exploration. We introduce IGPO (Inpainting Guided
Policy Optimization), an RL framework that strategically inserts partial
ground-truth reasoning traces during online sampling. Unlike providing full
solutions, inpainting steers exploration toward promising trajectory spaces
while preserving self-generated reasoning, bridging supervised fine-tuning and
reinforcement learning. We apply IGPO to group-based optimization methods such
as GRPO, where exploration failures cause zero advantages and gradients. IGPO
restores meaningful gradients while improving sample efficiency. We also
propose supervised fine-tuning on synthetically rewritten concise traces that
better align with dLLM generation patterns. With additional techniques
including entropy-based filtering, our training recipe yields substantial gains
across three mathematical benchmarks--GSM8K, Math500, and AMC--achieving new
state-of-the-art results for full-attention masked dLLMs.
68c79831ee0eed1697d6b760
Siyan Zhao
claimed_verified
2025-09-15T15:06:27.936Z
64a7411223622f7f188e30de
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lvH-Sg8V8pTEer91kwIIV.jpeg
siyan zhao
user
siyanzhao
68c79831ee0eed1697d6b761
Mengchen Liu
68c79831ee0eed1697d6b762
Jing Huang
68c79831ee0eed1697d6b763
Miao Liu
68c79831ee0eed1697d6b764
Chenyu Wang
68c79831ee0eed1697d6b765
Bo Liu
68c79831ee0eed1697d6b766
Yuandong Tian
68c79831ee0eed1697d6b767
Guan Pang
68c79831ee0eed1697d6b768
Sean Bell
68c79831ee0eed1697d6b769
Aditya Grover
68c79831ee0eed1697d6b76a
Feiyu Chen
claimed_verified
2025-09-16T09:43:12.195Z
68816709eebc721047b0743b
/avatars/fcd38117651a1136f2beb9abf20b0365.svg
Feiyu Chen
user
feiyuc
Inpainting-Guided Policy Optimization for Diffusion Large Language
  Models
Masked diffusion large language models (dLLMs) are emerging as promising
alternatives to autoregressive LLMs, offering competitive performance while
supporting unique generation capabilities such as inpainting. We explore how
inpainting can inform RL algorithm design for dLLMs. Aligning LLMs with
reinforcement learning faces an exploration challenge: sparse reward signals
and sample waste when models fail to discover correct solutions. While this
inefficiency affects LLMs broadly, dLLMs offer a distinctive opportunity--their
inpainting ability can guide exploration. We introduce IGPO (Inpainting Guided
Policy Optimization), an RL framework that strategically inserts partial
ground-truth reasoning traces during online sampling. Unlike providing full
solutions, inpainting steers exploration toward promising trajectory spaces
while preserving self-generated reasoning, bridging supervised fine-tuning and
reinforcement learning. We apply IGPO to group-based optimization methods such
as GRPO, where exploration failures cause zero advantages and gradients. IGPO
restores meaningful gradients while improving sample efficiency. We also
propose supervised fine-tuning on synthetically rewritten concise traces that
better align with dLLM generation patterns. With additional techniques
including entropy-based filtering, our training recipe yields substantial gains
across three mathematical benchmarks--GSM8K, Math500, and AMC--achieving new
state-of-the-art results for full-attention masked dLLMs.
IGPO, an RL framework utilizing inpainting in masked diffusion large language models, enhances sample efficiency and achieves state-of-the-art results in mathematical benchmarks.
masked diffusion large language models dLLMs autoregressive LLMs inpainting RL algorithm design reinforcement learning sparse reward signals sample waste IGPO Inpainting Guided Policy Optimization GRPO group-based optimization methods entropy-based filtering GSM8K Math500 AMC full-attention masked dLLMs