Multi-Turn Code Generation Through Single-Step Rewards
We address the problem of code generation from multi-turn execution feedback.
Existing methods either generate code without feedback or use complex,
hierarchical reinforcement learning to optimize multi-turn rewards. We propose
a simple yet scalable approach, muCode, that solves multi-turn code
generation using only single-step rewards. Our key insight is that code
generation is a one-step recoverable MDP, where the correct code can be
recovered from any intermediate code state in a single turn. muCode
iteratively trains both a generator to provide code solutions conditioned on
multi-turn execution feedback and a verifier to score the newly generated code.
Experimental evaluations show that our approach achieves significant
improvements over the state-of-the-art baselines. We provide analysis of the
design choices of the reward models and policy, and show the efficacy of
muCode at utilizing the execution feedback. Our code is available at
https://github.com/portal-cornell/muCode.
67c34e3beae05d8f94f800b4
Arnav Kumar Jain
67c34e3beae05d8f94f800b5
Gonzalo Gonzalez-Pumariega
claimed_verified
2025-03-02T20:15:14.593Z
6421d2972143035270db37b9
/avatars/4fadeafc273d32cf72fe2f12d444c5e8.svg
Gonzalo Gonzalez
user
chalo2000
67c34e3beae05d8f94f800b6
Wayne Chen
67c34e3beae05d8f94f800b7
Alexander M Rush
67c34e3beae05d8f94f800b8
Wenting Zhao
67c34e3beae05d8f94f800b9
Sanjiban Choudhury
Multi-Turn Code Generation Through Single-Step Rewards
We address the problem of code generation from multi-turn execution feedback.
Existing methods either generate code without feedback or use complex,
hierarchical reinforcement learning to optimize multi-turn rewards. We propose
a simple yet scalable approach, muCode, that solves multi-turn code
generation using only single-step rewards. Our key insight is that code
generation is a one-step recoverable MDP, where the correct code can be
recovered from any intermediate code state in a single turn. muCode
iteratively trains both a generator to provide code solutions conditioned on
multi-turn execution feedback and a verifier to score the newly generated code.
Experimental evaluations show that our approach achieves significant
improvements over the state-of-the-art baselines. We provide analysis of the
design choices of the reward models and policy, and show the efficacy of
muCode at utilizing the execution feedback. Our code is available at
https://github.com/portal-cornell/muCode.
MDP $\mu$Code generator verifier reward models policy