AST-T5: Structure-Aware Pretraining for Code Generation and
  Understanding
Large language models (LLMs) have made significant advancements in
code-related tasks, yet many LLMs treat code as simple sequences, neglecting
its structured nature. We introduce AST-T5, a novel pretraining paradigm that
leverages the Abstract Syntax Tree (AST) for enhanced code generation,
transpilation, and understanding. Using dynamic programming, our AST-Aware
Segmentation retains code structure, while our AST-Aware Span Corruption
objective equips the model to reconstruct various code structures. Unlike other
models, AST-T5 avoids intricate program analyses or architectural changes, so
it integrates seamlessly with any encoder-decoder Transformer. Evaluations show
that AST-T5 consistently outperforms similar-sized LMs across various
code-related tasks. Structure-awareness makes AST-T5 particularly powerful in
code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the
Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in
CodeXGLUE. Our code and model are publicly available at
https://github.com/gonglinyuan/ast_t5.
659cbf6f0030e8faffb5e657
Linyuan Gong
admin_assigned
2024-01-09T08:53:07.555Z
645b61c21c24cd669dd64080
https://cdn-avatars.huggingface.co/v1/production/uploads/645b61c21c24cd669dd64080/o-uAnxZ56kwFkD_wirf7-.jpeg
Linyuan Gong
user
gonglinyuan
659cbf6f0030e8faffb5e658
Mostafa Elhoushi
extracted_confirmed
2024-01-09T12:57:02.649Z
63c9725ebedad7e2bf160bdc
https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
Mostafa Elhoushi
user
melhoushi
659cbf6f0030e8faffb5e659
Alvin Cheung
AST-T5: Structure-Aware Pretraining for Code Generation and
  Understanding
Large language models (LLMs) have made significant advancements in
code-related tasks, yet many LLMs treat code as simple sequences, neglecting
its structured nature. We introduce AST-T5, a novel pretraining paradigm that
leverages the Abstract Syntax Tree (AST) for enhanced code generation,
transpilation, and understanding. Using dynamic programming, our AST-Aware
Segmentation retains code structure, while our AST-Aware Span Corruption
objective equips the model to reconstruct various code structures. Unlike other
models, AST-T5 avoids intricate program analyses or architectural changes, so
it integrates seamlessly with any encoder-decoder Transformer. Evaluations show
that AST-T5 consistently outperforms similar-sized LMs across various
code-related tasks. Structure-awareness makes AST-T5 particularly powerful in
code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the
Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in
CodeXGLUE. Our code and model are publicly available at
https://github.com/gonglinyuan/ast_t5.
AST-T5, a pretraining paradigm leveraging Abstract Syntax Trees, enhances code generation, transpilation, and understanding, particularly in code-to-code tasks, outperforming similar models like CodeT5.
Abstract Syntax Tree (AST) AST-Aware Segmentation dynamic programming AST-Aware Span Corruption encoder-decoder Transformer Bugs2Fix Java-C# Transpilation CodeXGLUE