From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by
  Composing Old Ones
Does RL teach LLMs genuinely new skills, or does it merely activate existing
ones? This question lies at the core of ongoing debates about the role of RL in
LLM post-training. On one side, strong empirical results can be achieved with
RL even without preceding supervised finetuning; on the other, critics argue
that RL contributes little beyond reweighting existing reasoning strategies.
This work provides concrete evidence that LLMs can acquire genuinely new skills
during RL by composing existing ones, mirroring one of the central mechanisms
by which humans acquire new cognitive skills. To mitigate data contamination
and other confounding factors, and to allow precise control over task
complexity, we develop a synthetic framework for our investigation.
Specifically, we define a skill as the ability to infer the output of a string
transformation function f(x) given x. When an LLM has already learned f and g
prior to RL, our experiments reveal that RL enables it to learn unseen
compositions of them h(x)=g(f(x)). Further, this compositional ability
generalizes to more difficult problems such as compositions of >2 functions
unseen during RL training. Surprisingly, our experiments show that
compositional skill acquired on a source task transfers to a different target
task. This transfer happens even without compositional training on the target,
requiring only prior knowledge of the target's atomic skills. Our qualitative
analysis shows that RL fundamentally changes the reasoning behaviors of the
models. In contrast, next-token training with the same data yields none of
these findings. Our systematic experiments provide fresh insights into LLM
learning, suggesting the value of first building base models with basic skills,
then using RL to incentivize advanced, generalizable skills for complex
problems.
68db6278d2bf1f4b15ec75e3
Lifan Yuan
68db6278d2bf1f4b15ec75e4
Weize Chen
68db6278d2bf1f4b15ec75e5
Yuchen Zhang
68db6278d2bf1f4b15ec75e6
Ganqu Cui
68db6278d2bf1f4b15ec75e7
Hanbin Wang
68db6278d2bf1f4b15ec75e8
Ziming You
68db6278d2bf1f4b15ec75e9
Ning Ding
68db6278d2bf1f4b15ec75ea
Zhiyuan Liu
68db6278d2bf1f4b15ec75eb
Maosong Sun
68db6278d2bf1f4b15ec75ec
Hao Peng
From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by
  Composing Old Ones
Does RL teach LLMs genuinely new skills, or does it merely activate existing
ones? This question lies at the core of ongoing debates about the role of RL in
LLM post-training. On one side, strong empirical results can be achieved with
RL even without preceding supervised finetuning; on the other, critics argue
that RL contributes little beyond reweighting existing reasoning strategies.
This work provides concrete evidence that LLMs can acquire genuinely new skills
during RL by composing existing ones, mirroring one of the central mechanisms
by which humans acquire new cognitive skills. To mitigate data contamination
and other confounding factors, and to allow precise control over task
complexity, we develop a synthetic framework for our investigation.
Specifically, we define a skill as the ability to infer the output of a string
transformation function f(x) given x. When an LLM has already learned f and g
prior to RL, our experiments reveal that RL enables it to learn unseen
compositions of them h(x)=g(f(x)). Further, this compositional ability
generalizes to more difficult problems such as compositions of >2 functions
unseen during RL training. Surprisingly, our experiments show that
compositional skill acquired on a source task transfers to a different target
task. This transfer happens even without compositional training on the target,
requiring only prior knowledge of the target's atomic skills. Our qualitative
analysis shows that RL fundamentally changes the reasoning behaviors of the
models. In contrast, next-token training with the same data yields none of
these findings. Our systematic experiments provide fresh insights into LLM
learning, suggesting the value of first building base models with basic skills,
then using RL to incentivize advanced, generalizable skills for complex
problems.
Reinforcement learning enables large language models to acquire new compositional skills by combining existing ones, which transfer to different tasks and improve reasoning behaviors.
reinforcement learning large language models compositional skills reasoning behaviors next-token training