Language Self-Play For Data-Free Training
Large language models (LLMs) have advanced rapidly in recent years, driven by
scale, abundant high-quality training data, and reinforcement learning. Yet
this progress faces a fundamental bottleneck: the need for ever more data from
which models can continue to learn. In this work, we propose a reinforcement
learning approach that removes this dependency by enabling models to improve
without additional data. Our method leverages a game-theoretic framework of
self-play, where a model's capabilities are cast as performance in a
competitive game and stronger policies emerge by having the model play against
itself - a process we call Language Self-Play (LSP). Experiments with
Llama-3.2-3B-Instruct on instruction-following benchmarks show that pretrained
models can not only enhance their performance on challenging tasks through
self-play alone, but can also do so more effectively than data-driven
baselines.
68c0dee53912ed54cf543256
Jakub Grudzien Kuba
68c0dee53912ed54cf543257
Mengting Gu
68c0dee53912ed54cf543258
Qi Ma
68c0dee53912ed54cf543259
Yuandong Tian
68c0dee53912ed54cf54325a
Vijai Mohan
Language Self-Play For Data-Free Training
Large language models (LLMs) have advanced rapidly in recent years, driven by
scale, abundant high-quality training data, and reinforcement learning. Yet
this progress faces a fundamental bottleneck: the need for ever more data from
which models can continue to learn. In this work, we propose a reinforcement
learning approach that removes this dependency by enabling models to improve
without additional data. Our method leverages a game-theoretic framework of
self-play, where a model's capabilities are cast as performance in a
competitive game and stronger policies emerge by having the model play against
itself - a process we call Language Self-Play (LSP). Experiments with
Llama-3.2-3B-Instruct on instruction-following benchmarks show that pretrained
models can not only enhance their performance on challenging tasks through
self-play alone, but can also do so more effectively than data-driven
baselines.
Language Self-Play (LSP) enhances large language models' performance on instruction-following tasks through self-play, surpassing data-driven methods.
large language models reinforcement learning self-play Language Self-Play (LSP) instruction-following benchmarks pretrained models