The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Entropy minimization (EM) trains the model to concentrate even more
probability mass on its most confident outputs. We show that this simple
objective alone, without any labeled data, can substantially improve large
language models' (LLMs) performance on challenging math, physics, and coding
tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy
similarly to instruction finetuning, but on unlabeled outputs drawn from the
model; (2) EM-RL: reinforcement learning with negative entropy as the only
reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce
entropy without any training data or parameter updates. On Qwen-7B, EM-RL,
without any labeled data, achieves comparable or better performance than strong
RL baselines such as GRPO and RLOO that are trained on 60K labeled examples.
Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of
proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the
challenging SciCode benchmark, while being 3x more efficient than
self-consistency and sequential refinement. Our findings reveal that many
pretrained LLMs possess previously underappreciated reasoning capabilities that
can be effectively elicited through entropy minimization alone, without any
labeled data or even any parameter updates.
682f8212bcb3f2f6c7bfe588
Shivam Agarwal
682f8212bcb3f2f6c7bfe589
Zimin Zhang
682f8212bcb3f2f6c7bfe58a
Lifan Yuan
682f8212bcb3f2f6c7bfe58b
Jiawei Han
682f8212bcb3f2f6c7bfe58c
Hao Peng
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Entropy minimization (EM) trains the model to concentrate even more
probability mass on its most confident outputs. We show that this simple
objective alone, without any labeled data, can substantially improve large
language models' (LLMs) performance on challenging math, physics, and coding
tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy
similarly to instruction finetuning, but on unlabeled outputs drawn from the
model; (2) EM-RL: reinforcement learning with negative entropy as the only
reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce
entropy without any training data or parameter updates. On Qwen-7B, EM-RL,
without any labeled data, achieves comparable or better performance than strong
RL baselines such as GRPO and RLOO that are trained on 60K labeled examples.
Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of
proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the
challenging SciCode benchmark, while being 3x more efficient than
self-consistency and sequential refinement. Our findings reveal that many
pretrained LLMs possess previously underappreciated reasoning capabilities that
can be effectively elicited through entropy minimization alone, without any
labeled data or even any parameter updates.
Entropy minimization improves large language models' performance on math, physics, and coding tasks without labeled data through various methods like fine-tuning, reinforcement learning, and inference-time adjustments.
entropy minimization EM-FT EM-RL EM-INF token-level entropy instruction finetuning reinforcement learning logit adjustment Qwen-7B Qwen-32B SciCode benchmark self-consistency sequential refinement large language models