Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
  Approach
We study a novel language model architecture that is capable of scaling
test-time computation by implicitly reasoning in latent space. Our model works
by iterating a recurrent block, thereby unrolling to arbitrary depth at
test-time. This stands in contrast to mainstream reasoning models that scale up
compute by producing more tokens. Unlike approaches based on chain-of-thought,
our approach does not require any specialized training data, can work with
small context windows, and can capture types of reasoning that are not easily
represented in words. We scale a proof-of-concept model to 3.5 billion
parameters and 800 billion tokens. We show that the resulting model can improve
its performance on reasoning benchmarks, sometimes dramatically, up to a
computation load equivalent to 50 billion parameters.
67a97e27495b23306cd5ea56
Jonas Geiping
admin_assigned
2025-02-10T16:10:13.470Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
67a97e27495b23306cd5ea57
Sean McLeish
claimed_verified
2025-02-10T13:11:57.835Z
65255f1073a043e50d043641
/avatars/257085f01c439d7c84787a4e6d085b3d.svg
Sean McLeish
user
smcleish
67a97e27495b23306cd5ea58
Neel Jain
admin_assigned
2025-02-10T16:10:34.976Z
63e2b1ec282ee5f9624cfbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
Neel Jain
user
nsjain
67a97e27495b23306cd5ea59
John Kirchenbauer
admin_assigned
2025-02-10T16:10:40.754Z
63d98af1897746d6496177df
/avatars/c5d0031c796a3c11bcb0d01b959168dc.svg
John Kirchenbauer
user
jwkirchenbauer
67a97e27495b23306cd5ea5a
Siddharth Singh
67a97e27495b23306cd5ea5b
Brian R. Bartoldson
67a97e27495b23306cd5ea5c
Bhavya Kailkhura
admin_assigned
2025-02-10T16:11:04.907Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
67a97e27495b23306cd5ea5d
Abhinav Bhatele
admin_assigned
2025-02-10T16:11:12.069Z
6361d9ce6bd72c97d005b4db
https://cdn-avatars.huggingface.co/v1/production/uploads/6361d9ce6bd72c97d005b4db/vjKaW2JFavVffoRxwgFwn.jpeg
Abhinav Bhatele
user
bhatele
67a97e27495b23306cd5ea5e
Tom Goldstein
admin_assigned
2025-02-10T16:11:20.220Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
  Approach
We study a novel language model architecture that is capable of scaling
test-time computation by implicitly reasoning in latent space. Our model works
by iterating a recurrent block, thereby unrolling to arbitrary depth at
test-time. This stands in contrast to mainstream reasoning models that scale up
compute by producing more tokens. Unlike approaches based on chain-of-thought,
our approach does not require any specialized training data, can work with
small context windows, and can capture types of reasoning that are not easily
represented in words. We scale a proof-of-concept model to 3.5 billion
parameters and 800 billion tokens. We show that the resulting model can improve
its performance on reasoning benchmarks, sometimes dramatically, up to a
computation load equivalent to 50 billion parameters.
A novel language model scales test-time computation by reasoning in latent space, outperforming token-based models on reasoning benchmarks.
recurrent block latent space chain-of-thought reasoning benchmarks