ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
We investigate the logical reasoning capabilities of large language models
(LLMs) and their scalability in complex non-monotonic reasoning. To this end,
we introduce ZebraLogic, a comprehensive evaluation framework for assessing LLM
reasoning performance on logic grid puzzles derived from constraint
satisfaction problems (CSPs). ZebraLogic enables the generation of puzzles with
controllable and quantifiable complexity, facilitating a systematic study of
the scaling limits of models such as Llama, o1 models, and DeepSeek-R1. By
encompassing a broad range of search space complexities and diverse logical
constraints, ZebraLogic provides a structured environment to evaluate reasoning
under increasing difficulty.
  Our results reveal a significant decline in accuracy as problem complexity
grows -- a phenomenon we term the curse of complexity. This limitation persists
even with larger models and increased inference-time computation, suggesting
inherent constraints in current LLM reasoning capabilities. Additionally, we
explore strategies to enhance logical reasoning, including Best-of-N sampling,
backtracking mechanisms, and self-verification prompts. Our findings offer
critical insights into the scalability of LLM reasoning, highlight fundamental
limitations, and outline potential directions for improvement.
67a1a649f4aecd0dfc96ebf4
Bill Yuchen Lin
claimed_verified
2025-02-04T09:39:17.972Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
67a1a649f4aecd0dfc96ebf5
Ronan Le Bras
extracted_pending
2025-02-04T05:31:56.722Z
635049104e753c9940fefd71
https://cdn-avatars.huggingface.co/v1/production/uploads/635049104e753c9940fefd71/HgR43XIFw3dneY5ufrAE8.jpeg
Ronan Le Bras
user
ronanlb
67a1a649f4aecd0dfc96ebf6
Kyle Richardson
claimed_verified
2025-02-27T09:18:11.166Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
67a1a649f4aecd0dfc96ebf7
Ashish Sabharwal
67a1a649f4aecd0dfc96ebf8
Radha Poovendran
67a1a649f4aecd0dfc96ebf9
Peter Clark
admin_assigned
2025-02-04T17:38:40.973Z
64d265cfbe712cda5ab7cc3f
/avatars/caab6fa5764a0271552ae589d352b592.svg
Peter Clarke
user
PeterClarke
67a1a649f4aecd0dfc96ebfa
Yejin Choi
admin_assigned
2025-02-04T17:38:53.950Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
We investigate the logical reasoning capabilities of large language models
(LLMs) and their scalability in complex non-monotonic reasoning. To this end,
we introduce ZebraLogic, a comprehensive evaluation framework for assessing LLM
reasoning performance on logic grid puzzles derived from constraint
satisfaction problems (CSPs). ZebraLogic enables the generation of puzzles with
controllable and quantifiable complexity, facilitating a systematic study of
the scaling limits of models such as Llama, o1 models, and DeepSeek-R1. By
encompassing a broad range of search space complexities and diverse logical
constraints, ZebraLogic provides a structured environment to evaluate reasoning
under increasing difficulty.
  Our results reveal a significant decline in accuracy as problem complexity
grows -- a phenomenon we term the curse of complexity. This limitation persists
even with larger models and increased inference-time computation, suggesting
inherent constraints in current LLM reasoning capabilities. Additionally, we
explore strategies to enhance logical reasoning, including Best-of-N sampling,
backtracking mechanisms, and self-verification prompts. Our findings offer
critical insights into the scalability of LLM reasoning, highlight fundamental
limitations, and outline potential directions for improvement.
logical reasoning capabilities large language models (LLMs) scalability non-monotonic reasoning ZebraLogic evaluation framework logic grid puzzles constraint satisfaction problems (CSPs) controllable complexity quantifiable complexity systematic study scaling limits search space complexities logical constraints structured environment curse of complexity Best-of-N sampling backtracking mechanisms self-verification prompts