Code Execution with Pre-trained Language Models
Code execution is a fundamental aspect of programming language semantics that
reflects the exact behavior of the code. However, most pre-trained models for
code intelligence ignore the execution trace and only rely on source code and
syntactic structures. In this paper, we investigate how well pre-trained models
can understand and perform code execution. We develop a mutation-based data
augmentation technique to create a large-scale and realistic Python dataset and
task for code execution, which challenges existing models such as Codex. We
then present CodeExecutor, a Transformer model that leverages code execution
pre-training and curriculum learning to enhance its semantic comprehension. We
evaluate CodeExecutor on code execution and show its promising performance and
limitations. We also demonstrate its potential benefits for code intelligence
tasks such as zero-shot code-to-code search and text-to-code generation. Our
analysis provides insights into the learning and generalization abilities of
pre-trained models for code execution.
645af8c4e7e8a4b57bdc4f50
Chenxiao Liu
645af8c4e7e8a4b57bdc4f51
Shuai Lu
admin_assigned
2023-05-24T09:24:17.666Z
617a3066f5910ea887210d91
/avatars/5193c5f523459f3d977defb257ef6f0d.svg
Shuai Lu
user
shuailu
645af8c4e7e8a4b57bdc4f52
Weizhu Chen
645af8c4e7e8a4b57bdc4f53
Daxin Jiang
admin_assigned
2023-05-24T09:25:24.554Z
60d4440fe648443279aaffd8
/avatars/bf7209c1f14ae120f5bfda5fda1301b7.svg
Daxin Jiang
user
djiang
645af8c4e7e8a4b57bdc4f54
Alexey Svyatkovskiy
645af8c4e7e8a4b57bdc4f55
Shengyu Fu
645af8c4e7e8a4b57bdc4f56
Neel Sundaresan
645af8c4e7e8a4b57bdc4f57
Nan Duan
Code Execution with Pre-trained Language Models
Code execution is a fundamental aspect of programming language semantics that
reflects the exact behavior of the code. However, most pre-trained models for
code intelligence ignore the execution trace and only rely on source code and
syntactic structures. In this paper, we investigate how well pre-trained models
can understand and perform code execution. We develop a mutation-based data
augmentation technique to create a large-scale and realistic Python dataset and
task for code execution, which challenges existing models such as Codex. We
then present CodeExecutor, a Transformer model that leverages code execution
pre-training and curriculum learning to enhance its semantic comprehension. We
evaluate CodeExecutor on code execution and show its promising performance and
limitations. We also demonstrate its potential benefits for code intelligence
tasks such as zero-shot code-to-code search and text-to-code generation. Our
analysis provides insights into the learning and generalization abilities of
pre-trained models for code execution.
CodeExecutor, a Transformer model enhanced with code execution pre-training and curriculum learning, demonstrates improved performance in code execution tasks and benefits for code intelligence tasks like zero-shot search and text-to-code generation.
Transformer model code execution pre-training curriculum learning semantic comprehension code execution tasks zero-shot code-to-code search text-to-code generation