Revolutionizing Reinforcement Learning Framework for Diffusion Large
  Language Models
We propose TraceRL, a trajectory-aware reinforcement learning framework for
diffusion language models (DLMs) that incorporates preferred inference
trajectory into post-training, and is applicable across different
architectures. Equipped with a diffusion-based value model that enhances
training stability, we demonstrate improved reasoning performance on complex
math and coding tasks. Besides, it can also be applied to adapt block-specific
models to larger blocks, which improves sampling flexibility. Employing
TraceRL, we derive a series of state-of-the-art diffusion language models,
namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still
consistently outperforms them across complex math reasoning tasks.
TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over
Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical
reasoning benchmarks. Through curriculum learning, we also derive the first
long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1%
relative accuracy gain. To facilitate reproducible research and practical
applications, we release a comprehensive open-source framework for building,
training, and deploying diffusion LLMs across diverse architectures. The
framework integrates accelerated KV-cache techniques and inference engines for
both inference and reinforcement learning, and includes implementations of
various supervised fine-tuning and RL methods for mathematics, coding, and
general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL
68bf87ff207285de11b07b39
Yinjie Wang
68bf87ff207285de11b07b3a
Ling Yang
claimed_verified
2025-09-09T13:48:22.774Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
68bf87ff207285de11b07b3b
Bowen Li
68bf87ff207285de11b07b3c
Ye Tian
claimed_verified
2025-09-12T16:20:25.210Z
64e357dd825f4133e7427bf8
https://cdn-avatars.huggingface.co/v1/production/uploads/64e357dd825f4133e7427bf8/HwaWhINrzkbXG6SHG2oyf.jpeg
tyfeld
user
tyfeld
68bf87ff207285de11b07b3d
Ke Shen
68bf87ff207285de11b07b3e
Mengdi Wang
Revolutionizing Reinforcement Learning Framework for Diffusion Large
  Language Models
We propose TraceRL, a trajectory-aware reinforcement learning framework for
diffusion language models (DLMs) that incorporates preferred inference
trajectory into post-training, and is applicable across different
architectures. Equipped with a diffusion-based value model that enhances
training stability, we demonstrate improved reasoning performance on complex
math and coding tasks. Besides, it can also be applied to adapt block-specific
models to larger blocks, which improves sampling flexibility. Employing
TraceRL, we derive a series of state-of-the-art diffusion language models,
namely TraDo. Although smaller than 7B-scale AR models, TraDo-4B-Instruct still
consistently outperforms them across complex math reasoning tasks.
TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over
Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical
reasoning benchmarks. Through curriculum learning, we also derive the first
long-CoT DLM, outperforming Qwen2.5-7B-Instruct on MATH500 with an 18.1%
relative accuracy gain. To facilitate reproducible research and practical
applications, we release a comprehensive open-source framework for building,
training, and deploying diffusion LLMs across diverse architectures. The
framework integrates accelerated KV-cache techniques and inference engines for
both inference and reinforcement learning, and includes implementations of
various supervised fine-tuning and RL methods for mathematics, coding, and
general tasks. Code and Models: https://github.com/Gen-Verse/dLLM-RL
TraceRL enhances diffusion language models with trajectory-aware reinforcement learning, improving reasoning performance on complex tasks and enabling flexible sampling.
trajectory-aware reinforcement learning diffusion language models diffusion-based value model sampling flexibility curriculum learning long-CoT DLM KV-cache techniques inference engines supervised fine-tuning RL methods