Reinforcement Learning Finetunes Small Subnetworks in Large Language
  Models
Reinforcement learning (RL) yields substantial improvements in large language
models (LLMs) downstream task performance and alignment with human values.
Surprisingly, such large gains result from updating only a small subnetwork
comprising just 5 percent to 30 percent of the parameters, with the rest
effectively unchanged. We refer to this phenomenon as parameter update sparsity
induced by RL. It is observed across all 7 widely used RL algorithms (e.g.,
PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
This sparsity is intrinsic and occurs without any explicit sparsity promoting
regularizations or architectural constraints. Finetuning the subnetwork alone
recovers the test accuracy, and, remarkably, produces a model nearly identical
to the one obtained via full finetuning. The subnetworks from different random
seeds, training data, and even RL algorithms show substantially greater overlap
than expected by chance. Our analysis suggests that this sparsity is not due to
updating only a subset of layers, instead, nearly all parameter matrices
receive similarly sparse updates. Moreover, the updates to almost all parameter
matrices are nearly full-rank, suggesting RL updates a small subset of
parameters that nevertheless span almost the full subspaces that the parameter
matrices can represent. We conjecture that the this update sparsity can be
primarily attributed to training on data that is near the policy distribution,
techniques that encourage the policy to remain close to the pretrained model,
such as the KL regularization and gradient clipping, have limited impact.
682e0d9540c6417d9962227a
Sagnik Mukherjee
claimed_verified
2025-05-22T07:17:26.467Z
6255a34d7dacca56ac2b04e4
/avatars/3e7751aa6ef7c880e3e36ac995c9a191.svg
sagnik mukherjee
user
sagnikM
682e0d9540c6417d9962227b
Lifan Yuan
682e0d9540c6417d9962227c
Dilek Hakkani-Tur
682e0d9540c6417d9962227d
Hao Peng
Reinforcement Learning Finetunes Small Subnetworks in Large Language
  Models
Reinforcement learning (RL) yields substantial improvements in large language
models (LLMs) downstream task performance and alignment with human values.
Surprisingly, such large gains result from updating only a small subnetwork
comprising just 5 percent to 30 percent of the parameters, with the rest
effectively unchanged. We refer to this phenomenon as parameter update sparsity
induced by RL. It is observed across all 7 widely used RL algorithms (e.g.,
PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
This sparsity is intrinsic and occurs without any explicit sparsity promoting
regularizations or architectural constraints. Finetuning the subnetwork alone
recovers the test accuracy, and, remarkably, produces a model nearly identical
to the one obtained via full finetuning. The subnetworks from different random
seeds, training data, and even RL algorithms show substantially greater overlap
than expected by chance. Our analysis suggests that this sparsity is not due to
updating only a subset of layers, instead, nearly all parameter matrices
receive similarly sparse updates. Moreover, the updates to almost all parameter
matrices are nearly full-rank, suggesting RL updates a small subset of
parameters that nevertheless span almost the full subspaces that the parameter
matrices can represent. We conjecture that the this update sparsity can be
primarily attributed to training on data that is near the policy distribution,
techniques that encourage the policy to remain close to the pretrained model,
such as the KL regularization and gradient clipping, have limited impact.
Reinforcement learning improves large language models with minimal parameter updates, affecting only a small subnetwork without explicit sparsity techniques.
reinforcement learning large language models parameter update sparsity PPO GRPO DPO policy distribution KL regularization gradient clipping