BOW: Bottlenecked Next Word Exploration
Large language models (LLMs) are typically trained via next-word prediction
(NWP), which provides strong surface-level fluency but often lacks support for
robust reasoning. We propose BOttlenecked next Word exploration (BOW), a novel
RL framework that rethinks NWP by introducing a reasoning bottleneck where a
policy model first generates a reasoning path rather than predicting the next
token directly, after which a frozen judge model predicts the next token
distribution based solely on this reasoning path. We train the policy model
using GRPO with rewards that quantify how effectively the reasoning path
facilitates next-word recovery. Compared with other continual pretraining
baselines, we show that BOW improves both the general and next-word reasoning
capabilities of the base model, evaluated on various benchmarks. Our findings
show that BOW can serve as an effective and scalable alternative to vanilla
NWP.
6851ece80164cd13167103f1
Ming Shen
6851ece80164cd13167103f2
Zhikun Xu
claimed_verified
2025-06-18T12:16:40.629Z
65685a95677a71b8ab1406cf
https://cdn-avatars.huggingface.co/v1/production/uploads/65685a95677a71b8ab1406cf/6tl-GiLHlYk7M17dPZjTr.jpeg
Zhikun Xu
user
JerrrrryKun
6851ece80164cd13167103f3
Xiao Ye
6851ece80164cd13167103f4
Jacob Dineen
6851ece80164cd13167103f5
Ben Zhou
BOW: Bottlenecked Next Word Exploration
Large language models (LLMs) are typically trained via next-word prediction
(NWP), which provides strong surface-level fluency but often lacks support for
robust reasoning. We propose BOttlenecked next Word exploration (BOW), a novel
RL framework that rethinks NWP by introducing a reasoning bottleneck where a
policy model first generates a reasoning path rather than predicting the next
token directly, after which a frozen judge model predicts the next token
distribution based solely on this reasoning path. We train the policy model
using GRPO with rewards that quantify how effectively the reasoning path
facilitates next-word recovery. Compared with other continual pretraining
baselines, we show that BOW improves both the general and next-word reasoning
capabilities of the base model, evaluated on various benchmarks. Our findings
show that BOW can serve as an effective and scalable alternative to vanilla
NWP.
BOW, a reinforcement learning framework, enhances language model reasoning by introducing a reasoning bottleneck between the policy and judge models, improving general and next-word reasoning.
LLMs next-word prediction NWP BOttlenecked next Word exploration BOW RL policy model frozen judge model reasoning path GRPO continual pretraining general reasoning next-word reasoning