Soft Tokens, Hard Truths
The use of continuous instead of discrete tokens during the Chain-of-Thought
(CoT) phase of reasoning LLMs has garnered attention recently, based on the
intuition that a continuous mixture of discrete tokens could simulate a
superposition of several reasoning paths simultaneously. Theoretical results
have formally proven that continuous tokens have much greater expressivity and
can solve specific problems more efficiently. However, practical use of
continuous tokens has been limited by strong training difficulties: previous
works either just use continuous tokens at inference time on a pre-trained
discrete-token model, or must distill the continuous CoT from ground-truth
discrete CoTs and face computational costs that limit the CoT to very few
tokens.
  This is the first work introducing a scalable method to learn continuous CoTs
via reinforcement learning (RL), without distilling from reference discrete
CoTs. We use "soft" tokens: mixtures of tokens together with noise on the input
embedding to provide RL exploration. Computational overhead is minimal,
enabling us to learn continuous CoTs with hundreds of tokens. On math reasoning
benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs
match discrete-token CoTs for pass@1 and surpass them for pass@32, showing
greater CoT diversity. In systematic comparisons, the best-performing scenario
is to train with continuous CoT tokens then use discrete tokens for inference,
meaning the "soft" models can be deployed in a standard way. Finally, we show
continuous CoT RL training better preserves the predictions of the base model
on out-of-domain tasks, thus providing a softer touch to the base model.
68d378190e215259d193b34d
Natasha Butt
claimed_verified
2025-09-25T07:11:33.208Z
67235358177d9eae488ff01b
/avatars/b94e228ceae5155898af4a6d1bb06ed3.svg
Natasha Butt
user
NatashaEB
68d378190e215259d193b34e
Ariel Kwiatkowski
68d378190e215259d193b34f
Ismail Labiad
68d378190e215259d193b350
Julia Kempe
68d378190e215259d193b351
Yann Ollivier
Soft Tokens, Hard Truths
The use of continuous instead of discrete tokens during the Chain-of-Thought
(CoT) phase of reasoning LLMs has garnered attention recently, based on the
intuition that a continuous mixture of discrete tokens could simulate a
superposition of several reasoning paths simultaneously. Theoretical results
have formally proven that continuous tokens have much greater expressivity and
can solve specific problems more efficiently. However, practical use of
continuous tokens has been limited by strong training difficulties: previous
works either just use continuous tokens at inference time on a pre-trained
discrete-token model, or must distill the continuous CoT from ground-truth
discrete CoTs and face computational costs that limit the CoT to very few
tokens.
  This is the first work introducing a scalable method to learn continuous CoTs
via reinforcement learning (RL), without distilling from reference discrete
CoTs. We use "soft" tokens: mixtures of tokens together with noise on the input
embedding to provide RL exploration. Computational overhead is minimal,
enabling us to learn continuous CoTs with hundreds of tokens. On math reasoning
benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs
match discrete-token CoTs for pass@1 and surpass them for pass@32, showing
greater CoT diversity. In systematic comparisons, the best-performing scenario
is to train with continuous CoT tokens then use discrete tokens for inference,
meaning the "soft" models can be deployed in a standard way. Finally, we show
continuous CoT RL training better preserves the predictions of the base model
on out-of-domain tasks, thus providing a softer touch to the base model.
A scalable reinforcement learning method for learning continuous chain-of-thought tokens in large language models improves performance and diversity over discrete tokens while preserving out-of-domain predictions.
Chain-of-Thought (CoT) continuous tokens discrete tokens reinforcement learning (RL) soft tokens noise on input embedding math reasoning benchmarks pass@1 pass@32 out-of-domain tasks