The Valley of Code Reasoning: Scaling Knowledge Distillation of Large
  Language Models
Distilling the thinking traces of a Large Language Model (LLM) with reasoning
capabilities into a smaller model has been proven effective. Yet, there is a
scarcity of work done on how model performances scale with the quantity of
distillation data. In this work, we study the scaling trend of distilling
competitive coding skills on two small non-reasoning LLMs. We validate the
hypothesis that there is a valley of code reasoning: downstream
performance on competitive coding first drops as data quantity increases, then
it steadily increases in a sharper-than-log-linear fashion. Having identified
the trend, we further fine-tune the models at two different distillation stages
on the same data to ground conclusions on their respective learning phases. We
learn that across stages in the low and medium-low data regimes, small models
benefit significantly from easier coding questions than from harder ones. We
also find that, surprisingly, the correctness of outputs in training data makes
no difference to distillation outcomes. Our work represents a step forward in
understanding the training dynamics of code reasoning distillation outside
intuition
68e66698975ac4c405ef2294
Muyu He
68e66698975ac4c405ef2295
Muhammad Ali Shafique
68e66698975ac4c405ef2296
Anand Kumar
68e66698975ac4c405ef2297
Tsach Mackey
68e66698975ac4c405ef2298
Nazneen Rajani
The Valley of Code Reasoning: Scaling Knowledge Distillation of Large
  Language Models
Distilling the thinking traces of a Large Language Model (LLM) with reasoning
capabilities into a smaller model has been proven effective. Yet, there is a
scarcity of work done on how model performances scale with the quantity of
distillation data. In this work, we study the scaling trend of distilling
competitive coding skills on two small non-reasoning LLMs. We validate the
hypothesis that there is a valley of code reasoning: downstream
performance on competitive coding first drops as data quantity increases, then
it steadily increases in a sharper-than-log-linear fashion. Having identified
the trend, we further fine-tune the models at two different distillation stages
on the same data to ground conclusions on their respective learning phases. We
learn that across stages in the low and medium-low data regimes, small models
benefit significantly from easier coding questions than from harder ones. We
also find that, surprisingly, the correctness of outputs in training data makes
no difference to distillation outcomes. Our work represents a step forward in
understanding the training dynamics of code reasoning distillation outside
intuition
Research on distilling coding skills from large language models to smaller ones reveals a "valley of code reasoning" where performance initially decreases with more data before improving sharply, and that small models benefit more from easier questions during distillation.
Large Language Model (LLM) distillation coding skills valley of code reasoning learning phases data regimes coding questions