DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per
  Token via Reinforcement Learning
Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve
strong performance via extended chains of thought but often generate
unnecessarily long outputs. Maximizing intelligence per token--accuracy
relative to response length--remains an open problem. We revisit reinforcement
learning (RL) with the simplest length penalty--truncation--and show that
accuracy degradation arises not from the lack of sophisticated penalties but
from inadequate RL optimization. We identify three key challenges: (i) large
bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward
signal. We address them with Doing Length pEnalty Right (DLER), a training
recipe combining batch-wise reward normalization, higher clipping, dynamic
sampling, and a simple truncation length penalty. DLER achieves
state-of-the-art accuracy--efficiency trade-offs, cutting output length by over
70 percent while surpassing all previous baseline accuracy. It also improves
test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple
concise responses in parallel with 28 percent higher accuracy and lower
latency. We further introduce Difficulty-Aware DLER, which adaptively tightens
truncation on easier questions for additional efficiency gains. We also propose
an update-selective merging method that preserves baseline accuracy while
retaining the concise reasoning ability of the DLER model, which is useful for
scenarios where RL training data is scarce.
68f5a8b68589920bf4d32145
Shih-Yang Liu
claimed_verified
2025-10-23T15:01:32.094Z
62b58c68a1bae3c711c41321
https://cdn-avatars.huggingface.co/v1/production/uploads/62b58c68a1bae3c711c41321/FGQ1ifsPpmRi8P0ElxV5J.png
LIU Shih-yang
user
sliuau
68f5a8b68589920bf4d32146
Xin Dong
68f5a8b68589920bf4d32147
Ximing Lu
68f5a8b68589920bf4d32148
Shizhe Diao
68f5a8b68589920bf4d32149
Mingjie Liu
68f5a8b68589920bf4d3214a
Min-Hung Chen
claimed_verified
2025-10-20T12:10:37.892Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
68f5a8b68589920bf4d3214b
Hongxu Yin
68f5a8b68589920bf4d3214c
Yu-Chiang Frank Wang
68f5a8b68589920bf4d3214d
Kwang-Ting Cheng
68f5a8b68589920bf4d3214e
Yejin Choi
68f5a8b68589920bf4d3214f
Jan Kautz
68f5a8b68589920bf4d32150
Pavlo Molchanov
DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per
  Token via Reinforcement Learning
Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve
strong performance via extended chains of thought but often generate
unnecessarily long outputs. Maximizing intelligence per token--accuracy
relative to response length--remains an open problem. We revisit reinforcement
learning (RL) with the simplest length penalty--truncation--and show that
accuracy degradation arises not from the lack of sophisticated penalties but
from inadequate RL optimization. We identify three key challenges: (i) large
bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward
signal. We address them with Doing Length pEnalty Right (DLER), a training
recipe combining batch-wise reward normalization, higher clipping, dynamic
sampling, and a simple truncation length penalty. DLER achieves
state-of-the-art accuracy--efficiency trade-offs, cutting output length by over
70 percent while surpassing all previous baseline accuracy. It also improves
test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple
concise responses in parallel with 28 percent higher accuracy and lower
latency. We further introduce Difficulty-Aware DLER, which adaptively tightens
truncation on easier questions for additional efficiency gains. We also propose
an update-selective merging method that preserves baseline accuracy while
retaining the concise reasoning ability of the DLER model, which is useful for
scenarios where RL training data is scarce.
DLER, a reinforcement learning training recipe, improves the accuracy-efficiency trade-off in reasoning language models by addressing challenges in advantage estimation, entropy collapse, and sparse reward signals, leading to shorter outputs and better test-time scaling.
reinforcement learning advantage estimation entropy collapse sparse reward signal Doing Length pEnalty Right batch-wise reward normalization dynamic sampling Difficulty-Aware DLER update-selective merging