RLAD: Training LLMs to Discover Abstractions for Solving Reasoning
  Problems
Reasoning requires going beyond pattern matching or memorization of solutions
to identify and implement "algorithmic procedures" that can be used to deduce
answers to hard problems. Doing so requires realizing the most relevant
primitives, intermediate results, or shared procedures, and building upon them.
While RL post-training on long chains of thought ultimately aims to uncover
this kind of algorithmic behavior, most reasoning traces learned by large
models fail to consistently capture or reuse procedures, instead drifting into
verbose and degenerate exploration. To address more effective reasoning, we
introduce reasoning abstractions: concise natural language descriptions of
procedural and factual knowledge that guide the model toward learning
successful reasoning. We train models to be capable of proposing multiple
abstractions given a problem, followed by RL that incentivizes building a
solution while using the information provided by these abstractions. This
results in a two-player RL training paradigm, abbreviated as RLAD, that jointly
trains an abstraction generator and a solution generator. This setup
effectively enables structured exploration, decouples learning signals of
abstraction proposal and solution generation, and improves generalization to
harder problems. We also show that allocating more test-time compute to
generating abstractions is more beneficial for performance than generating more
solutions at large test budgets, illustrating the role of abstractions in
guiding meaningful exploration.
68df8203df49fb0df1e03e0f
Yuxiao Qu
68df8203df49fb0df1e03e10
Anikait Singh
claimed_verified
2025-10-05T12:46:54.793Z
6511ee845b7e52b0251fdee9
https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
Anikait Singh
user
Asap7772
68df8203df49fb0df1e03e11
Yoonho Lee
68df8203df49fb0df1e03e12
Amrith Setlur
68df8203df49fb0df1e03e13
Ruslan Salakhutdinov
68df8203df49fb0df1e03e14
Chelsea Finn
68df8203df49fb0df1e03e15
Aviral Kumar
RLAD: Training LLMs to Discover Abstractions for Solving Reasoning
  Problems
Reasoning requires going beyond pattern matching or memorization of solutions
to identify and implement "algorithmic procedures" that can be used to deduce
answers to hard problems. Doing so requires realizing the most relevant
primitives, intermediate results, or shared procedures, and building upon them.
While RL post-training on long chains of thought ultimately aims to uncover
this kind of algorithmic behavior, most reasoning traces learned by large
models fail to consistently capture or reuse procedures, instead drifting into
verbose and degenerate exploration. To address more effective reasoning, we
introduce reasoning abstractions: concise natural language descriptions of
procedural and factual knowledge that guide the model toward learning
successful reasoning. We train models to be capable of proposing multiple
abstractions given a problem, followed by RL that incentivizes building a
solution while using the information provided by these abstractions. This
results in a two-player RL training paradigm, abbreviated as RLAD, that jointly
trains an abstraction generator and a solution generator. This setup
effectively enables structured exploration, decouples learning signals of
abstraction proposal and solution generation, and improves generalization to
harder problems. We also show that allocating more test-time compute to
generating abstractions is more beneficial for performance than generating more
solutions at large test budgets, illustrating the role of abstractions in
guiding meaningful exploration.
Introducing reasoning abstractions in reinforcement learning improves structured exploration and generalization for complex problem-solving.
reinforcement learning reasoning abstractions procedural knowledge factual knowledge abstraction generator solution generator RLAD structured exploration generalization