Stream of Search (SoS): Learning to Search in Language
Language models are rarely shown fruitful mistakes while training. They then
struggle to look beyond the next token, suffering from a snowballing of errors
and struggling to predict the consequence of their actions several steps ahead.
In this paper, we show how language models can be taught to search by
representing the process of search in language, as a flattened string -- a
stream of search (SoS). We propose a unified language for search that captures
an array of different symbolic search strategies. We demonstrate our approach
using the simple yet difficult game of Countdown, where the goal is to combine
input numbers with arithmetic operations to reach a target number. We pretrain
a transformer-based language model from scratch on a dataset of streams of
search generated by heuristic solvers. We find that SoS pretraining increases
search accuracy by 25% over models trained to predict only the optimal search
trajectory. We further finetune this model with two policy improvement methods:
Advantage-Induced Policy Alignment (APA) and Self-Taught Reasoner (STaR). The
finetuned SoS models solve 36% of previously unsolved problems, including
problems that cannot be solved by any of the heuristic solvers. Our results
indicate that language models can learn to solve problems via search,
self-improve to flexibly use different search strategies, and potentially
discover new ones.
6613588b188ff298b0521791
Kanishk Gandhi
extracted_confirmed
2024-09-19T01:34:54.793Z
63e6a880f2e9a8f22c5a1630
/avatars/53b57690fe052ce6882bbfc87b11567c.svg
Kanishk Gandhi
user
obiwan96
6613588b188ff298b0521792
Denise Lee
6613588b188ff298b0521793
Gabriel Grand
6613588b188ff298b0521794
Muxin Liu
6613588b188ff298b0521795
Winson Cheng
6613588b188ff298b0521796
Archit Sharma
6613588b188ff298b0521797
Noah D. Goodman
claimed_verified
2024-11-15T09:47:53.860Z
67321274c1f20c742bcf7a8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
Noah D. Goodman
user
ngoodman
Stream of Search (SoS): Learning to Search in Language
Language models are rarely shown fruitful mistakes while training. They then
struggle to look beyond the next token, suffering from a snowballing of errors
and struggling to predict the consequence of their actions several steps ahead.
In this paper, we show how language models can be taught to search by
representing the process of search in language, as a flattened string -- a
stream of search (SoS). We propose a unified language for search that captures
an array of different symbolic search strategies. We demonstrate our approach
using the simple yet difficult game of Countdown, where the goal is to combine
input numbers with arithmetic operations to reach a target number. We pretrain
a transformer-based language model from scratch on a dataset of streams of
search generated by heuristic solvers. We find that SoS pretraining increases
search accuracy by 25% over models trained to predict only the optimal search
trajectory. We further finetune this model with two policy improvement methods:
Advantage-Induced Policy Alignment (APA) and Self-Taught Reasoner (STaR). The
finetuned SoS models solve 36% of previously unsolved problems, including
problems that cannot be solved by any of the heuristic solvers. Our results
indicate that language models can learn to solve problems via search,
self-improve to flexibly use different search strategies, and potentially
discover new ones.
Language models can learn complex problem-solving strategies via search by pretraining on streams of search generated by heuristic solvers and further improving with policy-based methods, solving previously unsolvable problems.
transformer-based language model pretrain Advantage-Induced Policy Alignment Self-Taught Reasoner streams of search Countdown heuristic solvers