Evolving Language Models without Labels: Majority Drives Selection,
  Novelty Promotes Variation
Large language models (LLMs) are increasingly trained with reinforcement
learning from verifiable rewards (RLVR), yet real-world deployment demands
models that can self-improve without labels or external judges. Existing
label-free methods, confidence minimization, self-consistency, or majority-vote
objectives, stabilize learning but steadily shrink exploration, causing an
entropy collapse: generations become shorter, less diverse, and brittle. Unlike
prior approaches such as Test-Time Reinforcement Learning (TTRL), which
primarily adapt models to the immediate unlabeled dataset at hand, our goal is
broader: to enable general improvements without sacrificing the model's
inherent exploration capacity and generalization ability, i.e., evolving. We
formalize this issue and propose EVolution-Oriented and Label-free
Reinforcement Learning (EVOL-RL), a simple rule that couples stability with
variation under a label-free setting. EVOL-RL keeps the majority-voted answer
as a stable anchor (selection) while adding a novelty-aware reward that favors
responses whose reasoning differs from what has already been produced
(variation), measured in semantic space. Implemented with GRPO, EVOL-RL also
uses asymmetric clipping to preserve strong signals and an entropy regularizer
to sustain search. This majority-for-selection + novelty-for-variation design
prevents collapse, maintains longer and more informative chains of thought, and
improves both pass@1 and pass@n. EVOL-RL consistently outperforms the
majority-only TTRL baseline; e.g., training on label-free AIME24 lifts
Qwen3-4B-Base AIME25 pass@1 from TTRL's 4.6% to 16.4%, and pass@16 from 18.5%
to 37.9%. EVOL-RL not only prevents diversity collapse but also unlocks
stronger generalization across domains (e.g., GPQA). Furthermore, we
demonstrate that EVOL-RL also boosts performance in the RLVR setting,
highlighting its broad applicability.
68ccb4383df9ac65e93dc5d0
Yujun Zhou
claimed_verified
2025-09-19T06:48:53.689Z
65a05abf07184d32fa002d41
/avatars/3a23e7e568d2024381ed31b56c1c461a.svg
Yujun Zhou
user
yujunzhou
68ccb4383df9ac65e93dc5d1
Zhenwen Liang
claimed_verified
2025-09-19T06:48:56.281Z
62ffa3f8311cad266f9af236
/avatars/203dac40bc546ee25a01d8715a4b3049.svg
Zhenwen Liang
user
invokerliang
68ccb4383df9ac65e93dc5d2
Haolin Liu
68ccb4383df9ac65e93dc5d3
Wenhao Yu
68ccb4383df9ac65e93dc5d4
Kishan Panaganti
claimed_verified
2025-09-21T13:13:06.636Z
68266b5261ed4d89177c3612
/avatars/e41ef277047334174eca408aba2a63db.svg
Kishan
user
kishanpb
68ccb4383df9ac65e93dc5d5
Linfeng Song
68ccb4383df9ac65e93dc5d6
Dian Yu
68ccb4383df9ac65e93dc5d7
Xiangliang Zhang
68ccb4383df9ac65e93dc5d8
Haitao Mi
68ccb4383df9ac65e93dc5d9
Dong Yu
Evolving Language Models without Labels: Majority Drives Selection,
  Novelty Promotes Variation
Large language models (LLMs) are increasingly trained with reinforcement
learning from verifiable rewards (RLVR), yet real-world deployment demands
models that can self-improve without labels or external judges. Existing
label-free methods, confidence minimization, self-consistency, or majority-vote
objectives, stabilize learning but steadily shrink exploration, causing an
entropy collapse: generations become shorter, less diverse, and brittle. Unlike
prior approaches such as Test-Time Reinforcement Learning (TTRL), which
primarily adapt models to the immediate unlabeled dataset at hand, our goal is
broader: to enable general improvements without sacrificing the model's
inherent exploration capacity and generalization ability, i.e., evolving. We
formalize this issue and propose EVolution-Oriented and Label-free
Reinforcement Learning (EVOL-RL), a simple rule that couples stability with
variation under a label-free setting. EVOL-RL keeps the majority-voted answer
as a stable anchor (selection) while adding a novelty-aware reward that favors
responses whose reasoning differs from what has already been produced
(variation), measured in semantic space. Implemented with GRPO, EVOL-RL also
uses asymmetric clipping to preserve strong signals and an entropy regularizer
to sustain search. This majority-for-selection + novelty-for-variation design
prevents collapse, maintains longer and more informative chains of thought, and
improves both pass@1 and pass@n. EVOL-RL consistently outperforms the
majority-only TTRL baseline; e.g., training on label-free AIME24 lifts
Qwen3-4B-Base AIME25 pass@1 from TTRL's 4.6% to 16.4%, and pass@16 from 18.5%
to 37.9%. EVOL-RL not only prevents diversity collapse but also unlocks
stronger generalization across domains (e.g., GPQA). Furthermore, we
demonstrate that EVOL-RL also boosts performance in the RLVR setting,
highlighting its broad applicability.
EVOL-RL, a label-free reinforcement learning method, enhances large language models by balancing stability and variation, preventing entropy collapse and improving generalization.
reinforcement learning from verifiable rewards RLVR label-free methods confidence minimization self-consistency majority-vote objectives entropy collapse Test-Time Reinforcement Learning TTRL EVolution-Oriented and Label-free Reinforcement Learning EVOL-RL GRPO asymmetric clipping entropy regularizer pass@1 pass@n AIME24 Qwen3-4B-Base AIME25 GPQA