Sample More to Think Less: Group Filtered Policy Optimization for
  Concise Reasoning
Large language models trained with reinforcement learning with verifiable
rewards tend to trade accuracy for length--inflating response lengths to
achieve gains in accuracy. While longer answers may be warranted for harder
problems, many tokens are merely "filler": repetitive, verbose text that makes
no real progress. We introduce GFPO (Group Filtered Policy Optimization), which
curbs this length explosion by sampling larger groups per problem during
training and filtering responses to train on based on two key metrics: (1)
response length and (2) token efficiency: reward per token ratio. By sampling
more at training time, we teach models to think less at inference time. On the
Phi-4-reasoning model, GFPO cuts GRPO's length inflation by 46-71% across
challenging STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH,
LiveCodeBench) while maintaining accuracy. Optimizing for reward per token
further increases reductions in length inflation to 71-85%. We also propose
Adaptive Difficulty GFPO, which dynamically allocates more training resources
to harder problems based on real-time difficulty estimates, improving the
balance between computational efficiency and accuracy especially on difficult
questions. GFPO demonstrates that increased training-time compute directly
translates to reduced test-time compute--a simple yet effective trade-off for
efficient reasoning.
689d9af7b083e610d741eadd
Vaishnavi Shrivastava
689d9af7b083e610d741eade
Ahmed Awadallah
689d9af7b083e610d741eadf
Vidhisha Balachandran
689d9af7b083e610d741eae0
Shivam Garg
689d9af7b083e610d741eae1
Harkirat Behl
689d9af7b083e610d741eae2
Dimitris Papailiopoulos
Sample More to Think Less: Group Filtered Policy Optimization for
  Concise Reasoning
Large language models trained with reinforcement learning with verifiable
rewards tend to trade accuracy for length--inflating response lengths to
achieve gains in accuracy. While longer answers may be warranted for harder
problems, many tokens are merely "filler": repetitive, verbose text that makes
no real progress. We introduce GFPO (Group Filtered Policy Optimization), which
curbs this length explosion by sampling larger groups per problem during
training and filtering responses to train on based on two key metrics: (1)
response length and (2) token efficiency: reward per token ratio. By sampling
more at training time, we teach models to think less at inference time. On the
Phi-4-reasoning model, GFPO cuts GRPO's length inflation by 46-71% across
challenging STEM and coding benchmarks (AIME 24/25, GPQA, Omni-MATH,
LiveCodeBench) while maintaining accuracy. Optimizing for reward per token
further increases reductions in length inflation to 71-85%. We also propose
Adaptive Difficulty GFPO, which dynamically allocates more training resources
to harder problems based on real-time difficulty estimates, improving the
balance between computational efficiency and accuracy especially on difficult
questions. GFPO demonstrates that increased training-time compute directly
translates to reduced test-time compute--a simple yet effective trade-off for
efficient reasoning.
GFPO reduces length inflation in large language models by sampling larger groups and filtering responses based on length and token efficiency, maintaining accuracy and improving computational efficiency.
reinforcement learning verifiable rewards length explosion token efficiency Group Filtered Policy Optimization GFPO GRPO Phi-4-reasoning model AIME GPQA Omni-MATH LiveCodeBench Adaptive Difficulty GFPO