MathBode: Frequency-Domain Fingerprints of LLM Mathematical Reasoning
This paper presents MathBode, a dynamic diagnostic for mathematical reasoning
in large language models (LLMs). Instead of one-shot accuracy, MathBode treats
each parametric problem as a system: we drive a single parameter sinusoidally
and fit first-harmonic responses of model outputs and exact solutions. This
yields interpretable, frequency-resolved metrics -- gain (amplitude tracking)
and phase (lag) -- that form Bode-style fingerprints. Across five closed-form
families (linear solve, ratio/saturation, compound interest, 2x2 linear
systems, similar triangles), the diagnostic surfaces systematic low-pass
behavior and growing phase lag that accuracy alone obscures. We compare several
models against a symbolic baseline that calibrates the instrument (G approx
1, phi approx 0). Results separate frontier from mid-tier models on
dynamics, providing a compact, reproducible protocol that complements standard
benchmarks with actionable measurements of reasoning fidelity and consistency.
We open-source the dataset and code to enable further research and adoption.
68db2802d2bf1f4b15ec7233
Charles L. Wang
claimed_verified
2025-09-30T06:35:31.804Z
67c604ff3f8c7b027d479671
https://cdn-avatars.huggingface.co/v1/production/uploads/67c604ff3f8c7b027d479671/IhfN3eWRQ8iNfcsDKA1Ep.png
Charles Wang
user
charleslwang
MathBode: Frequency-Domain Fingerprints of LLM Mathematical Reasoning
This paper presents MathBode, a dynamic diagnostic for mathematical reasoning
in large language models (LLMs). Instead of one-shot accuracy, MathBode treats
each parametric problem as a system: we drive a single parameter sinusoidally
and fit first-harmonic responses of model outputs and exact solutions. This
yields interpretable, frequency-resolved metrics -- gain (amplitude tracking)
and phase (lag) -- that form Bode-style fingerprints. Across five closed-form
families (linear solve, ratio/saturation, compound interest, 2x2 linear
systems, similar triangles), the diagnostic surfaces systematic low-pass
behavior and growing phase lag that accuracy alone obscures. We compare several
models against a symbolic baseline that calibrates the instrument (G approx
1, phi approx 0). Results separate frontier from mid-tier models on
dynamics, providing a compact, reproducible protocol that complements standard
benchmarks with actionable measurements of reasoning fidelity and consistency.
We open-source the dataset and code to enable further research and adoption.
MathBode provides a diagnostic for mathematical reasoning in LLMs by analyzing frequency-resolved metrics of model outputs compared to exact solutions, revealing systematic low-pass behavior and phase lag.
MathBode dynamic diagnostic mathematical reasoning large language models parametric problem sinusoidal drive first-harmonic responses gain phase Bode-style fingerprints closed-form families linear solve ratio/saturation compound interest 2x2 linear systems similar triangles low-pass behavior phase lag symbolic baseline reasoning fidelity consistency