Inverse Scaling in Test-Time Compute
We construct evaluation tasks where extending the reasoning length of Large
Reasoning Models (LRMs) deteriorates performance, exhibiting an inverse scaling
relationship between test-time compute and accuracy. Our evaluation tasks span
four categories: simple counting tasks with distractors, regression tasks with
spurious features, deduction tasks with constraint tracking, and advanced AI
risks. We identify five distinct failure modes when models reason for longer:
1) Claude models become increasingly distracted by irrelevant information; 2)
OpenAI o-series models resist distractors but overfit to problem framings; 3)
models shift from reasonable priors to spurious correlations; 4) all models
show difficulties in maintaining focus on complex deductive tasks; and 5)
extended reasoning may amplify concerning behaviors, with Claude Sonnet 4
showing increased expressions of self-preservation. These findings suggest that
while test-time compute scaling remains promising for improving model
capabilities, it may inadvertently reinforce problematic reasoning patterns.
Our results demonstrate the importance of evaluating models across diverse
reasoning lengths to identify and address these failure modes in LRMs.
687f5f9233947f780d9b4c14
Aryo Pradipta Gema
claimed_verified
2025-07-22T13:05:22.839Z
644f895e23d7eb05ca695054
/avatars/3fb04dd8544b403262bf98507de05453.svg
Aryo Pradipta Gema
user
aryopg
687f5f9233947f780d9b4c15
Alexander Hägele
claimed_verified
2025-07-22T13:05:19.119Z
65785d22dddc2360b01702e1
/avatars/8e3ddf25b9c423f57484fddef4f0aafd.svg
Alexander Hägele
user
haeggee
687f5f9233947f780d9b4c16
Runjin Chen
admin_assigned
2025-07-22T14:20:53.155Z
6504cfc81dab48600892b2db
/avatars/90f372c251d02733c0c5f10e3c0e0026.svg
Runjin Chen
user
Runjin
687f5f9233947f780d9b4c17
Andy Arditi
admin_assigned
2025-07-22T14:20:59.380Z
63c5d13ae2804cb2407e2bba
https://cdn-avatars.huggingface.co/v1/production/uploads/63c5d13ae2804cb2407e2bba/xZbVXx7yoZFWqNTEuZxKX.jpeg
Andy Arditi
user
andyrdt
687f5f9233947f780d9b4c18
Jacob Goldman-Wetzler
687f5f9233947f780d9b4c19
Kit Fraser-Taliente
admin_assigned
2025-07-22T14:21:07.942Z
67f640d457b84b9f4966c8fb
/avatars/e3641b78f3355ed9f6823d88fb9e42b1.svg
Kit Fraser-Taliente
user
kitft
687f5f9233947f780d9b4c1a
Henry Sleight
687f5f9233947f780d9b4c1b
Linda Petrini
admin_assigned
2025-07-22T14:21:16.868Z
686bc6580e2c7804bfe9219a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/gSA7VYoQUoZS4EQNF5-IK.png
Linda Petrini
user
lindapetrini
687f5f9233947f780d9b4c1c
Julian Michael
admin_assigned
2025-07-22T14:21:24.013Z
6745164c55df0b402fcb7b73
/avatars/7a93b3a62d59ae9b9a1500561904c2d7.svg
Julian Michael
user
julianmichael
687f5f9233947f780d9b4c1d
Beatrice Alex
687f5f9233947f780d9b4c1e
Pasquale Minervini
claimed_verified
2025-07-22T13:05:21.109Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
687f5f9233947f780d9b4c1f
Yanda Chen
687f5f9233947f780d9b4c20
Joe Benton
687f5f9233947f780d9b4c21
Ethan Perez
Inverse Scaling in Test-Time Compute
We construct evaluation tasks where extending the reasoning length of Large
Reasoning Models (LRMs) deteriorates performance, exhibiting an inverse scaling
relationship between test-time compute and accuracy. Our evaluation tasks span
four categories: simple counting tasks with distractors, regression tasks with
spurious features, deduction tasks with constraint tracking, and advanced AI
risks. We identify five distinct failure modes when models reason for longer:
1) Claude models become increasingly distracted by irrelevant information; 2)
OpenAI o-series models resist distractors but overfit to problem framings; 3)
models shift from reasonable priors to spurious correlations; 4) all models
show difficulties in maintaining focus on complex deductive tasks; and 5)
extended reasoning may amplify concerning behaviors, with Claude Sonnet 4
showing increased expressions of self-preservation. These findings suggest that
while test-time compute scaling remains promising for improving model
capabilities, it may inadvertently reinforce problematic reasoning patterns.
Our results demonstrate the importance of evaluating models across diverse
reasoning lengths to identify and address these failure modes in LRMs.
Evaluation of Large Reasoning Models across different reasoning lengths reveals that increased test-time compute can lead to performance degradation and amplify problematic reasoning patterns.
Large Reasoning Models reasoning length test-time compute accuracy inverse scaling relationship simple counting tasks regression tasks deduction tasks advanced AI risks failure modes irrelevant information distractors overfitting spurious correlations complex deductive tasks concerning behaviors self-preservation