Language Models can Self-Improve at State-Value Estimation for Better
  Search
Collecting ground truth task completion rewards or human demonstrations for
multi-step reasoning tasks is often cost-prohibitive and time-consuming,
especially in interactive domains like web tasks. To address this bottleneck,
we present self-taught lookahead, a self-supervised method that leverages
state-transition dynamics to train a value model capable of effectively guiding
language model-controlled search. We find that moderately sized (8 billion
parameters) open-weight value models improved with self-taught lookahead can
match the performance of using a frontier LLM such as gpt-4o as the value
model. Furthermore, we find that self-taught lookahead improves performance by
20% while reducing costs 37x compared to previous LLM-based tree search,
without relying on ground truth rewards.
67c7bf7c40de8b1b534d23fa
Ethan Mendes
extracted_confirmed
2025-03-05T03:06:00.588Z
635d76ce94e5b275ca74b967
/avatars/5d9a389e5fd558c0b8f0724bf0838a3e.svg
Ethan Mendes
user
emendes3
67c7bf7c40de8b1b534d23fb
Alan Ritter
admin_assigned
2025-03-05T11:19:56.686Z
6377da9065ce6b04bd493cf8
/avatars/6be2632d6da49857928f8bcce8278f49.svg
Alan Ritter
user
rittera
Language Models can Self-Improve at State-Value Estimation for Better
  Search
Collecting ground truth task completion rewards or human demonstrations for
multi-step reasoning tasks is often cost-prohibitive and time-consuming,
especially in interactive domains like web tasks. To address this bottleneck,
we present self-taught lookahead, a self-supervised method that leverages
state-transition dynamics to train a value model capable of effectively guiding
language model-controlled search. We find that moderately sized (8 billion
parameters) open-weight value models improved with self-taught lookahead can
match the performance of using a frontier LLM such as gpt-4o as the value
model. Furthermore, we find that self-taught lookahead improves performance by
20% while reducing costs 37x compared to previous LLM-based tree search,
without relying on ground truth rewards.
A self-supervised method called self-taught lookahead improves multi-step reasoning task performance by leveraging state-transition dynamics, matching advanced LLMs while significantly reducing costs.
self-taught lookahead state-transition dynamics value model language model-controlled search open-weight value models frontier LLM task performance multi-step reasoning tasks ground truth rewards