LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM
  Reasoning
Since the release of Deepseek-R1, reinforcement learning with verifiable
rewards (RLVR) has become a central approach for training large language models
(LLMs) on reasoning tasks. Recent work has largely focused on modifying loss
functions to make RLVR more efficient and effective. In this paper, motivated
by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy
Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects
training data at each step based on the average response length. We evaluate
LSPO across multiple base models and datasets, demonstrating that it
consistently improves learning effectiveness. In addition, we conduct a
detailed ablation study to examine alternative ways of incorporating length
signals into dynamic sampling, offering further insights and highlighting
promising directions for future research.
68e3449a73e20ab57784206e
Weizhe Chen
68e3449a73e20ab57784206f
Sven Koenig
68e3449a73e20ab577842070
Bistra Dilkina
LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM
  Reasoning
Since the release of Deepseek-R1, reinforcement learning with verifiable
rewards (RLVR) has become a central approach for training large language models
(LLMs) on reasoning tasks. Recent work has largely focused on modifying loss
functions to make RLVR more efficient and effective. In this paper, motivated
by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy
Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects
training data at each step based on the average response length. We evaluate
LSPO across multiple base models and datasets, demonstrating that it
consistently improves learning effectiveness. In addition, we conduct a
detailed ablation study to examine alternative ways of incorporating length
signals into dynamic sampling, offering further insights and highlighting
promising directions for future research.
Length-aware Sampling for Policy Optimization (LSPO) is a meta-RLVR algorithm that dynamically selects training data based on response length, improving learning effectiveness in large language models.
reinforcement learning with verifiable rewards RLVR large language models LLMs meta-RLVR Length-aware Sampling for Policy Optimization LSPO dynamic sampling length signals ablation study