Iterative Self-Training for Code Generation via Reinforced Re-Ranking
Generating high-quality code that solves complex programming tasks is
challenging, especially with current decoder-based models that produce highly
stochastic outputs. In code generation, even minor errors can easily break the
entire solution. Leveraging multiple sampled solutions can significantly
improve the overall output quality.
  One effective way to enhance code generation is by pairing a code generation
model with a reranker model, which selects the best solution from the generated
samples. We propose a novel iterative self-training approach for self-training
reranker models using Proximal Policy Optimization (PPO), aimed at improving
both reranking accuracy and the overall code generation process. Unlike
traditional PPO approaches, where the focus is on optimizing a generative model
with a reward model, our approach emphasizes the development of a robust
reward/reranking model. This model improves the quality of generated code
through reranking and addresses problems and errors that the reward model might
overlook during PPO alignment with the reranker. Our method iteratively refines
the training dataset by re-evaluating outputs, identifying high-scoring
negative examples, and incorporating them into the training loop, that boosting
model performance.
  Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter
model outperforms a 33B model in code generation quality while being three
times faster. Moreover, it achieves performance comparable to GPT-4 and
surpasses it in one programming language.
67fe391f684ac7f04ce21df7
Nikita Sorokin
67fe391f684ac7f04ce21df8
Ivan Sedykh
claimed_verified
2025-04-16T09:15:30.948Z
610e8c12119bebecb4d807b6
/avatars/7230b1584ec45585c12eb5703fd80ff3.svg
Ivan Sedykh
user
idsedykh
67fe391f684ac7f04ce21df9
Valentin Malykh
Iterative Self-Training for Code Generation via Reinforced Re-Ranking
Generating high-quality code that solves complex programming tasks is
challenging, especially with current decoder-based models that produce highly
stochastic outputs. In code generation, even minor errors can easily break the
entire solution. Leveraging multiple sampled solutions can significantly
improve the overall output quality.
  One effective way to enhance code generation is by pairing a code generation
model with a reranker model, which selects the best solution from the generated
samples. We propose a novel iterative self-training approach for self-training
reranker models using Proximal Policy Optimization (PPO), aimed at improving
both reranking accuracy and the overall code generation process. Unlike
traditional PPO approaches, where the focus is on optimizing a generative model
with a reward model, our approach emphasizes the development of a robust
reward/reranking model. This model improves the quality of generated code
through reranking and addresses problems and errors that the reward model might
overlook during PPO alignment with the reranker. Our method iteratively refines
the training dataset by re-evaluating outputs, identifying high-scoring
negative examples, and incorporating them into the training loop, that boosting
model performance.
  Our evaluation on the MultiPL-E dataset demonstrates that our 13.4B parameter
model outperforms a 33B model in code generation quality while being three
times faster. Moreover, it achieves performance comparable to GPT-4 and
surpasses it in one programming language.
An iterative self-training approach using Proximal Policy Optimization enhances reranker models for code generation, improving accuracy and quality compared to larger models.
decoder-based models code generation reranker model Proximal Policy Optimization reranking accuracy MultiPL-E dataset GPT-4