On Code-Induced Reasoning in LLMs
Code data has been shown to enhance the reasoning capabilities of large
language models (LLMs), but it remains unclear which aspects of code are most
responsible. We investigate this question with a systematic, data-centric
framework. We construct parallel instruction datasets in ten programming
languages and apply controlled perturbations that selectively disrupt
structural or semantic properties of code. We then finetune LLMs from five
model families and eight scales on each variant and evaluate their performance
on natural language, math, and code tasks. Across 3,331 experiments, our
results show that LLMs are more vulnerable to structural perturbations than
semantic ones, particularly on math and code tasks. Appropriate abstractions
like pseudocode and flowcharts can be as effective as code, while encoding the
same information with fewer tokens without adhering to original syntax can
often retain or even improve performance. Remarkably, even corrupted code with
misleading signals remains competitive when surface-level regularities persist.
Finally, syntactic styles also shape task-specific gains with Python favoring
natural language reasoning and lower-level languages such as Java and Rust
favoring math. Through our systematic framework, we aim to provide insight into
how different properties of code influence reasoning and inform the design of
training data for enhancing LLM reasoning capabilities.
68dee350df49fb0df1e03acb
Abdul Waheed
68dee350df49fb0df1e03acc
Zhen Wu
claimed_verified
2025-10-08T13:49:48.950Z
65e13a4b06c9c86f2515afdf
https://cdn-avatars.huggingface.co/v1/production/uploads/65e13a4b06c9c86f2515afdf/-ozW4YbRrzsBInoQU-4pP.jpeg
Zhen Wu
user
swzwan
68dee350df49fb0df1e03acd
Carolyn Rosé
68dee350df49fb0df1e03ace
Daphne Ippolito
On Code-Induced Reasoning in LLMs
Code data has been shown to enhance the reasoning capabilities of large
language models (LLMs), but it remains unclear which aspects of code are most
responsible. We investigate this question with a systematic, data-centric
framework. We construct parallel instruction datasets in ten programming
languages and apply controlled perturbations that selectively disrupt
structural or semantic properties of code. We then finetune LLMs from five
model families and eight scales on each variant and evaluate their performance
on natural language, math, and code tasks. Across 3,331 experiments, our
results show that LLMs are more vulnerable to structural perturbations than
semantic ones, particularly on math and code tasks. Appropriate abstractions
like pseudocode and flowcharts can be as effective as code, while encoding the
same information with fewer tokens without adhering to original syntax can
often retain or even improve performance. Remarkably, even corrupted code with
misleading signals remains competitive when surface-level regularities persist.
Finally, syntactic styles also shape task-specific gains with Python favoring
natural language reasoning and lower-level languages such as Java and Rust
favoring math. Through our systematic framework, we aim to provide insight into
how different properties of code influence reasoning and inform the design of
training data for enhancing LLM reasoning capabilities.
Systematic investigation reveals that large language models are more sensitive to structural than semantic code perturbations, with implications for training data design.
large language models LLMs instruction datasets programming languages controlled perturbations structural perturbations semantic perturbations pseudocode flowcharts token encoding syntactic styles natural language reasoning math reasoning