How Much Backtracking is Enough? Exploring the Interplay of SFT and RL
  in Enhancing LLM Reasoning
Recent breakthroughs in large language models (LLMs) have effectively
improved their reasoning abilities, particularly on mathematical and logical
problems that have verifiable answers, through techniques such as supervised
finetuning (SFT) and reinforcement learning (RL). Prior research indicates that
RL effectively internalizes search strategies, enabling long chain-of-thought
(CoT) reasoning, with backtracking emerging naturally as a learned capability.
However, the precise benefits of backtracking, specifically, how significantly
it contributes to reasoning improvements and the optimal extent of its use,
remain poorly understood. In this work, we systematically investigate the
dynamics between SFT and RL on eight reasoning tasks: Countdown, Sudoku, Arc
1D, Geometry, Color Cube Rotation, List Functions, Zebra Puzzles, and Self
Reference. Our findings highlight that short CoT sequences used in SFT as a
warm-up do have moderate contribution to RL training, compared with cold-start
RL; however such contribution diminishes when tasks become increasingly
difficult. Motivated by this observation, we construct synthetic datasets
varying systematically in the number of backtracking steps and conduct
controlled experiments to isolate the influence of either the correctness
(content) or the structure (i.e., backtrack frequency). We find that (1) longer
CoT with backtracks generally induce better and more stable RL training, (2)
more challenging problems with larger search space tend to need higher numbers
of backtracks during the SFT stage. Additionally, we demonstrate through
experiments on distilled data that RL training is largely unaffected by the
correctness of long CoT sequences, suggesting that RL prioritizes structural
patterns over content correctness. Collectively, our results offer practical
insights into designing optimal training strategies to effectively scale
reasoning in LLMs.
683f21ed6ba11d78e3e383f6
Hongyi James Cai
claimed_verified
2025-06-04T09:01:32.571Z
65f7c56fc6356b5cc5ab8245
/avatars/ff8d3f3526e915f269274cc0dc5ca8ef.svg
James Cai
user
jamescai20
683f21ed6ba11d78e3e383f7
Junlin Wang
683f21ed6ba11d78e3e383f8
Xiaoyin Chen
extracted_confirmed
2025-06-18T01:30:57.960Z
65d66cb2b06abf924b07ff76
/avatars/de94e2fe07040b7dc3053bcaafa64ffe.svg
Xiaoyin Chen
user
chenyn66
683f21ed6ba11d78e3e383f9
Bhuwan Dhingra
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL
  in Enhancing LLM Reasoning
Recent breakthroughs in large language models (LLMs) have effectively
improved their reasoning abilities, particularly on mathematical and logical
problems that have verifiable answers, through techniques such as supervised
finetuning (SFT) and reinforcement learning (RL). Prior research indicates that
RL effectively internalizes search strategies, enabling long chain-of-thought
(CoT) reasoning, with backtracking emerging naturally as a learned capability.
However, the precise benefits of backtracking, specifically, how significantly
it contributes to reasoning improvements and the optimal extent of its use,
remain poorly understood. In this work, we systematically investigate the
dynamics between SFT and RL on eight reasoning tasks: Countdown, Sudoku, Arc
1D, Geometry, Color Cube Rotation, List Functions, Zebra Puzzles, and Self
Reference. Our findings highlight that short CoT sequences used in SFT as a
warm-up do have moderate contribution to RL training, compared with cold-start
RL; however such contribution diminishes when tasks become increasingly
difficult. Motivated by this observation, we construct synthetic datasets
varying systematically in the number of backtracking steps and conduct
controlled experiments to isolate the influence of either the correctness
(content) or the structure (i.e., backtrack frequency). We find that (1) longer
CoT with backtracks generally induce better and more stable RL training, (2)
more challenging problems with larger search space tend to need higher numbers
of backtracks during the SFT stage. Additionally, we demonstrate through
experiments on distilled data that RL training is largely unaffected by the
correctness of long CoT sequences, suggesting that RL prioritizes structural
patterns over content correctness. Collectively, our results offer practical
insights into designing optimal training strategies to effectively scale
reasoning in LLMs.
This study investigates the interplay between supervised fine-tuning and reinforcement learning in large language models, focusing on the role of backtracking in enhancing reasoning capabilities across various tasks.
supervised finetuning reinforcement learning chain-of-thought backtracking countdown sudoku arc 1d geometry color cube rotation list functions zebra puzzles self reference synthetic datasets distilled data