A Survey on Language Models for Code
In this work we systematically review the recent advancements in code
processing with language models, covering 50+ models, 30+ evaluation tasks, and
500 related works. We break down code processing models into general language
models represented by the GPT family and specialized models that are
specifically pretrained on code, often with tailored objectives. We discuss the
relations and differences between these models, and highlight the historical
transition of code modeling from statistical models and RNNs to pretrained
Transformers and LLMs, which is exactly the same course that had been taken by
NLP. We also discuss code-specific features such as AST, CFG, and unit tests,
along with their application in training code language models, and identify key
challenges and potential future directions in this domain. We keep the survey
open and updated on github repository at
https://github.com/codefuse-ai/Awesome-Code-LLM.
65543b3143cc10c9d094f5c1
Ziyin Zhang
admin_assigned
2024-09-13T18:03:46.110Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
65543b3143cc10c9d094f5c2
Chaoyu Chen
admin_assigned
2023-11-15T09:54:44.160Z
63f4860dcfd7ba6e26f3c694
/avatars/de1ac926d4395688e1da75568ac50c76.svg
Chaoyu Chen
user
chencyudel
65543b3143cc10c9d094f5c3
Bingchang Liu
claimed_verified
2025-04-08T08:47:04.451Z
640ec9fc8512ec51d7f64409
https://cdn-avatars.huggingface.co/v1/production/uploads/640ec9fc8512ec51d7f64409/LSq6prKfiWCsvAPcBI4Rm.jpeg
L BC
user
twelveand0
65543b3143cc10c9d094f5c4
Cong Liao
65543b3143cc10c9d094f5c5
Zi Gong
65543b3143cc10c9d094f5c6
Hang Yu
65543b3143cc10c9d094f5c7
Jianguo Li
admin_assigned
2023-11-15T09:55:37.006Z
6539fc3fe279d2fc80c88902
/avatars/432ac2287a7f7d04c2c7e505070a3331.svg
Jianguo Li
user
jglee2046
65543b3143cc10c9d094f5c8
Rui Wang
A Survey on Language Models for Code
In this work we systematically review the recent advancements in code
processing with language models, covering 50+ models, 30+ evaluation tasks, and
500 related works. We break down code processing models into general language
models represented by the GPT family and specialized models that are
specifically pretrained on code, often with tailored objectives. We discuss the
relations and differences between these models, and highlight the historical
transition of code modeling from statistical models and RNNs to pretrained
Transformers and LLMs, which is exactly the same course that had been taken by
NLP. We also discuss code-specific features such as AST, CFG, and unit tests,
along with their application in training code language models, and identify key
challenges and potential future directions in this domain. We keep the survey
open and updated on github repository at
https://github.com/codefuse-ai/Awesome-Code-LLM.
GPT family pretrained Transformers LLMs Statistical models RNNs code-specific features AST (Abstract Syntax Tree) CFG (Control Flow Graph) unit tests