MathOOD: Exploring the Generalization Boundaries of Mathematical Reasoning in Large Language Models
Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hanna Hajishirzi, Nouha Dziri, Dawn Song
Recent large-scale language models (LLMs) with long Chain-of-Thought reasoning—such as DeepSeek-R1—have achieved impressive results on Olympiad-level mathematics benchmarks. However, they often rely on a narrow set of strategies and struggle with problems that require a novel way of thinking. To systematically probe these limitations, we introduce MathOOD, a controlled yet diverse benchmark designed to evaluate three axes of out-of-distribution generalization inspired by Boden’s typology of creativity: (1) Exploratory—applying known problem-solving skills to more complex instances within the same problem domain; (2) Compositional— combining distinct reasoning skills, previously learned in isolation, to solve novel problems that require integrating these skills in new and coherent ways; and (3) Transformative—adopting novel, often unconventional strategies by moving beyond familiar approaches to solve problems more effectively.  MathOOD consists of programmatically generated training–test pairs derived from templated problem generators across geometry, number theory, algebra, combinatorics, logic, and puzzles, with solutions verified using symbolic, numerical, or graphical methods. We evaluate top-tiered LLMs and observe sharp performance degradation as problem complexity increases. Moreover, we fine-tune the Qwen-series models across all generalization settings and observe notable improvements in exploratory generalization, while compositional generalization remains limited and transformative reasoning shows 0 improvement. By isolating and quantifying these fine-grained failures, MathOOD lays the groundwork for advancing LLMs toward genuine mathematical creativity beyond mechanical proficiency.