Bootstrapping Task Spaces for Self-Improvement
Progress in many task domains emerges from repeated revisions to previous
solution attempts. Training agents that can reliably self-improve over such
sequences at inference-time is a natural target for reinforcement learning
(RL), yet the naive approach assumes a fixed maximum iteration depth, which can
be both costly and arbitrary. We present Exploratory Iteration (ExIt), a family
of autocurriculum RL methods that directly exploits the recurrent structure of
self-improvement tasks to train LLMs to perform multi-step self-improvement at
inference-time while only training on the most informative single-step
iterations. ExIt grows a task space by selectively sampling the most
informative intermediate, partial histories encountered during an episode for
continued iteration, treating these starting points as new self-iteration task
instances to train a self-improvement policy. ExIt can further pair with
explicit exploration mechanisms to sustain greater task diversity. Across
several domains, encompassing competition math, multi-turn tool-use, and
machine learning engineering, we demonstrate that ExIt strategies, starting
from either a single or many task instances, can produce policies exhibiting
strong inference-time self-improvement on held-out task instances, and the
ability to iterate towards higher performance over a step budget extending
beyond the average iteration depth encountered during training.
68bea66dc123124955ef61aa
Minqi Jiang
claimed_verified
2025-09-09T13:49:37.483Z
630368320547362a22a21798
/avatars/035b468584e43b4fbbf4c4aa1d2bbac7.svg
Minqi Jiang
user
minqi
68bea66dc123124955ef61ab
Andrei Lupu
68bea66dc123124955ef61ac
Yoram Bachrach
Bootstrapping Task Spaces for Self-Improvement
Progress in many task domains emerges from repeated revisions to previous
solution attempts. Training agents that can reliably self-improve over such
sequences at inference-time is a natural target for reinforcement learning
(RL), yet the naive approach assumes a fixed maximum iteration depth, which can
be both costly and arbitrary. We present Exploratory Iteration (ExIt), a family
of autocurriculum RL methods that directly exploits the recurrent structure of
self-improvement tasks to train LLMs to perform multi-step self-improvement at
inference-time while only training on the most informative single-step
iterations. ExIt grows a task space by selectively sampling the most
informative intermediate, partial histories encountered during an episode for
continued iteration, treating these starting points as new self-iteration task
instances to train a self-improvement policy. ExIt can further pair with
explicit exploration mechanisms to sustain greater task diversity. Across
several domains, encompassing competition math, multi-turn tool-use, and
machine learning engineering, we demonstrate that ExIt strategies, starting
from either a single or many task instances, can produce policies exhibiting
strong inference-time self-improvement on held-out task instances, and the
ability to iterate towards higher performance over a step budget extending
beyond the average iteration depth encountered during training.
Exploratory Iteration (ExIt) is an autocurriculum RL method that trains LLMs to perform multi-step self-improvement at inference-time by selectively sampling informative intermediate histories, enabling strong self-improvement on unseen tasks.
reinforcement learning RL autocurriculum RL LLMs self-improvement inference-time multi-step self-improvement task space intermediate histories explicit exploration mechanisms competition math multi-turn tool-use machine learning engineering