Evaluating LLM Code Reasoning Robustness under Structural and Textual Distractions
Man Ho Lam, Chaozheng Wang, Jen-Tse Huang, Michael R Lyu
Large Language Models (LLMs) have demonstrated remarkable performance on various code-related tasks, but their reliability in code comprehension under perturbations remains insufficiently studied.We present CodeCrash, a comprehensive stress-testing benchmark with 1,279 questions collected from two established datasets, CruxEval and LiveCodeBench, to evaluate 17 LLMs across input and output prediction tasks under structural and textual perturbations.Our evaluation reveals that LLMs heavily rely on embedded natural language cues for reasoning and understanding, resulting in over 13 percentage point degradation in output prediction tasks and leading to superficial reasoning shortcuts in input prediction tasks.Furthermore, we identify a progressive relationship between reasoning depth and model reliability, where explicit step-by-step execution improves robustness, and internal reasoning models achieve even greater robustness but expose critical vulnerabilities.The self-reflection mechanism increases token usage by 2-3 times, reduces confidence, and even leads to severe reasoning collapse under reasoning-level textual perturbations.CodeCrash provides a rigorous benchmark for evaluating robustness in code understanding, guiding future research toward more reliable and resilient LLMs in code reasoning.