Learning a Continue-Thinking Token for Enhanced Test-Time Scaling
Test-time scaling has emerged as an effective approach for improving language
model performance by utilizing additional compute at inference time. Recent
studies have shown that overriding end-of-thinking tokens (e.g., replacing
"</think>" with "Wait") can extend reasoning steps and improve accuracy. In
this work, we explore whether a dedicated continue-thinking token can be
learned to trigger extended reasoning. We augment a distilled version of
DeepSeek-R1 with a single learned "<|continue-thinking|>" token, training only
its embedding via reinforcement learning while keeping the model weights
frozen. Our experiments show that this learned token achieves improved accuracy
on standard math benchmarks compared to both the baseline model and a test-time
scaling approach that uses a fixed token (e.g., "Wait") for budget forcing. In
particular, we observe that in cases where the fixed-token approach enhances
the base model's accuracy, our method achieves a markedly greater improvement.
For example, on the GSM8K benchmark, the fixed-token approach yields a 1.3%
absolute improvement in accuracy, whereas our learned-token method achieves a
4.2% improvement over the base model that does not use budget forcing.
68500fd9a81b87ad9c3862a2
Liran Ringel
claimed_verified
2025-06-16T12:56:01.319Z
63a4a8f0769ff94bc94b343f
/avatars/422545ce2b822389316340c8f0ef8cc4.svg
Liran Ringel
user
liranringel
68500fd9a81b87ad9c3862a3
Elad Tolochinsky
extracted_confirmed
2025-06-16T13:02:52.713Z
651bcd5d5c81ea4d1b721b94
/avatars/0a1a5f33e81204eeada03875713ca76c.svg
Elad Tolochinsky
user
eladto
68500fd9a81b87ad9c3862a4
Yaniv Romano
Learning a Continue-Thinking Token for Enhanced Test-Time Scaling
Test-time scaling has emerged as an effective approach for improving language
model performance by utilizing additional compute at inference time. Recent
studies have shown that overriding end-of-thinking tokens (e.g., replacing
"</think>" with "Wait") can extend reasoning steps and improve accuracy. In
this work, we explore whether a dedicated continue-thinking token can be
learned to trigger extended reasoning. We augment a distilled version of
DeepSeek-R1 with a single learned "<|continue-thinking|>" token, training only
its embedding via reinforcement learning while keeping the model weights
frozen. Our experiments show that this learned token achieves improved accuracy
on standard math benchmarks compared to both the baseline model and a test-time
scaling approach that uses a fixed token (e.g., "Wait") for budget forcing. In
particular, we observe that in cases where the fixed-token approach enhances
the base model's accuracy, our method achieves a markedly greater improvement.
For example, on the GSM8K benchmark, the fixed-token approach yields a 1.3%
absolute improvement in accuracy, whereas our learned-token method achieves a
4.2% improvement over the base model that does not use budget forcing.
A continuous thinking token learned via reinforcement learning improves language model accuracy more effectively than a fixed token during inference.
test-time scaling language model end-of-thinking tokens continue-thinking token reinforcement learning distilled version GSM8K benchmark