Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement
  Learning
Fine-tuning pre-trained large language models (LLMs) for down-stream tasks is
a critical step in the AI deployment pipeline. Reinforcement learning (RL) is
arguably the most prominent fine-tuning method, contributing to the birth of
many state-of-the-art LLMs. In contrast, evolution strategies (ES), which once
showed comparable performance to RL on models with a few million parameters,
was neglected due to the pessimistic perception of its scalability to larger
models. In this work, we report the first successful attempt to scale up ES for
fine-tuning the full parameters of LLMs, showing the surprising fact that ES
can search efficiently over billions of parameters and outperform existing RL
fine-tuning methods in multiple respects, including sample efficiency,
tolerance to long-horizon rewards, robustness to different base LLMs, less
tendency to reward hacking, and more stable performance across runs. It
therefore serves as a basis to unlock a new direction in LLM fine-tuning beyond
what current RL techniques provide. The source codes are provided at:
https://github.com/VsonicV/es-fine-tuning-paper.
68db43d5d2bf1f4b15ec7345
Xin Qiu
claimed_verified
2025-10-01T10:25:01.278Z
64c083fbe9263c783d46fb21
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/99WqMzp_LbrxjJKnTjrzk.png
Xin Qiu
user
VsonicV
68db43d5d2bf1f4b15ec7346
Yulu Gan
claimed_verified
2025-10-01T10:25:03.163Z
63fde8116315a264aba6d48f
/avatars/01be674381a52187a5fe2e92f1f81890.svg
Yulu Gan
user
yulu2
68db43d5d2bf1f4b15ec7347
Conor F. Hayes
68db43d5d2bf1f4b15ec7348
Qiyao Liang
68db43d5d2bf1f4b15ec7349
Elliot Meyerson
68db43d5d2bf1f4b15ec734a
Babak Hodjat
68db43d5d2bf1f4b15ec734b
Risto Miikkulainen
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement
  Learning
Fine-tuning pre-trained large language models (LLMs) for down-stream tasks is
a critical step in the AI deployment pipeline. Reinforcement learning (RL) is
arguably the most prominent fine-tuning method, contributing to the birth of
many state-of-the-art LLMs. In contrast, evolution strategies (ES), which once
showed comparable performance to RL on models with a few million parameters,
was neglected due to the pessimistic perception of its scalability to larger
models. In this work, we report the first successful attempt to scale up ES for
fine-tuning the full parameters of LLMs, showing the surprising fact that ES
can search efficiently over billions of parameters and outperform existing RL
fine-tuning methods in multiple respects, including sample efficiency,
tolerance to long-horizon rewards, robustness to different base LLMs, less
tendency to reward hacking, and more stable performance across runs. It
therefore serves as a basis to unlock a new direction in LLM fine-tuning beyond
what current RL techniques provide. The source codes are provided at:
https://github.com/VsonicV/es-fine-tuning-paper.
Evolution strategies successfully scale to fine-tune large language models, outperforming reinforcement learning in sample efficiency and robustness.
reinforcement learning evolution strategies large language models fine-tuning sample efficiency long-horizon rewards reward hacking stable performance