Comment on The Illusion of Thinking: Understanding the Strengths and
  Limitations of Reasoning Models via the Lens of Problem Complexity
Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit
"accuracy collapse" on planning puzzles beyond certain complexity thresholds.
We demonstrate that their findings primarily reflect experimental design
limitations rather than fundamental reasoning failures. Our analysis reveals
three critical issues: (1) Tower of Hanoi experiments systematically exceed
model output token limits at reported failure points, with models explicitly
acknowledging these constraints in their outputs; (2) The authors' automated
evaluation framework fails to distinguish between reasoning failures and
practical constraints, leading to misclassification of model capabilities; (3)
Most concerningly, their River Crossing benchmarks include mathematically
impossible instances for N > 5 due to insufficient boat capacity, yet models
are scored as failures for not solving these unsolvable problems. When we
control for these experimental artifacts, by requesting generating functions
instead of exhaustive move lists, preliminary experiments across multiple
models indicate high accuracy on Tower of Hanoi instances previously reported
as complete failures. These findings highlight the importance of careful
experimental design when evaluating AI reasoning capabilities.
684c94ae3b733ba3336872f1
C. Opus
684c94ae3b733ba3336872f2
A. Lawsen
Comment on The Illusion of Thinking: Understanding the Strengths and
  Limitations of Reasoning Models via the Lens of Problem Complexity
Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit
"accuracy collapse" on planning puzzles beyond certain complexity thresholds.
We demonstrate that their findings primarily reflect experimental design
limitations rather than fundamental reasoning failures. Our analysis reveals
three critical issues: (1) Tower of Hanoi experiments systematically exceed
model output token limits at reported failure points, with models explicitly
acknowledging these constraints in their outputs; (2) The authors' automated
evaluation framework fails to distinguish between reasoning failures and
practical constraints, leading to misclassification of model capabilities; (3)
Most concerningly, their River Crossing benchmarks include mathematically
impossible instances for N > 5 due to insufficient boat capacity, yet models
are scored as failures for not solving these unsolvable problems. When we
control for these experimental artifacts, by requesting generating functions
instead of exhaustive move lists, preliminary experiments across multiple
models indicate high accuracy on Tower of Hanoi instances previously reported
as complete failures. These findings highlight the importance of careful
experimental design when evaluating AI reasoning capabilities.
Evaluation artifacts, particularly token limits and impractical instances in benchmarks, lead to misreported failures in Large Reasoning Models on planning puzzles.
Large Reasoning Models accuracy collapse Tower of Hanoi token limits automated evaluation framework River Crossing benchmarks generating functions exhaustive move lists