A.S.E: A Repository-Level Benchmark for Evaluating Security in
  AI-Generated Code
The increasing adoption of large language models (LLMs) in software
engineering necessitates rigorous security evaluation of their generated code.
However, existing benchmarks often lack relevance to real-world AI programming
scenarios, making them inadequate for assessing the practical security risks
associated with AI-generated code in production environments. To address this
gap, we introduce A.S.E (AI Code Generation Security Evaluation), a
repository-level evaluation benchmark designed to closely mirror real-world AI
programming tasks, offering a comprehensive and reliable framework for
assessing the security of AI-generated code. Our evaluation of leading LLMs on
A.S.E reveals several key findings. In particular, current LLMs still struggle
with secure coding. The complexity in repository-level scenarios presents
challenges for LLMs that typically perform well on snippet-level tasks.
Morever, a larger reasoning budget does not necessarily lead to better code
generation. These observations offer valuable insights into the current state
of AI code generation, assisting developers in selecting the most appropriate
models for practical tasks, while laying the foundation for refining LLMs to
generate secure and efficient code in real-world applications.
68b50b5f851c6e7b001eca21
Keke Lian
claimed_verified
2025-09-01T07:50:36.564Z
67ca63c1e10c4e1bf8262576
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/e9MvyBRetgZXk73pQcsOX.png
Keke Lian
user
KekeLian
68b50b5f851c6e7b001eca22
Bin Wang
68b50b5f851c6e7b001eca23
Lei Zhang
68b50b5f851c6e7b001eca24
Libo Chen
68b50b5f851c6e7b001eca25
Junjie Wang
claimed_verified
2025-09-01T07:50:42.819Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
68b50b5f851c6e7b001eca26
Ziming Zhao
68b50b5f851c6e7b001eca27
Yujiu Yang
68b50b5f851c6e7b001eca28
Haotong Duan
claimed_verified
2025-09-09T13:50:08.391Z
68be8c0f612f38f15ca656dc
/avatars/79357b76acd073d888e625c21eca3b83.svg
Haotong Duan
user
Bestats
68b50b5f851c6e7b001eca29
Haoran Zhao
68b50b5f851c6e7b001eca2a
Shuang Liao
68b50b5f851c6e7b001eca2b
Mingda Guo
68b50b5f851c6e7b001eca2c
Jiazheng Quan
claimed_verified
2025-09-01T13:51:34.413Z
68b51853a494ea3733e4e851
https://cdn-avatars.huggingface.co/v1/production/uploads/68b51853a494ea3733e4e851/-j-nJWeZZpmKzqHsIl7UN.png
Jiazheng Quan
user
jzquan
68b50b5f851c6e7b001eca2d
Yilu Zhong
claimed_verified
2025-09-12T16:21:37.506Z
66ae660bf0bb529189e89c3a
https://cdn-avatars.huggingface.co/v1/production/uploads/66ae660bf0bb529189e89c3a/7mpvVzbZvRGmdPYjjw8xU.png
Yilu Zhong
user
tangaaang
68b50b5f851c6e7b001eca2e
Chenhao He
68b50b5f851c6e7b001eca2f
Zichuan Chen
68b50b5f851c6e7b001eca30
Jie Wu
68b50b5f851c6e7b001eca31
Haoling Li
claimed_verified
2025-09-02T07:57:06.228Z
650be23ec4e52db6a4db63ef
/avatars/03af548029b38bee49ec295fefe74f9a.svg
Haoling Li
user
Ringo1110
68b50b5f851c6e7b001eca32
Zhaoxuan Li
68b50b5f851c6e7b001eca33
Jiongchi Yu
68b50b5f851c6e7b001eca34
Hui Li
68b50b5f851c6e7b001eca35
Dong Zhang
claimed_verified
2025-09-03T08:29:27.965Z
68b5182cb710fd5fa8af991d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NZtlwYoqS9tg3l7G43_S9.png
dong zhang
user
zalezhang
A.S.E: A Repository-Level Benchmark for Evaluating Security in
  AI-Generated Code
The increasing adoption of large language models (LLMs) in software
engineering necessitates rigorous security evaluation of their generated code.
However, existing benchmarks often lack relevance to real-world AI programming
scenarios, making them inadequate for assessing the practical security risks
associated with AI-generated code in production environments. To address this
gap, we introduce A.S.E (AI Code Generation Security Evaluation), a
repository-level evaluation benchmark designed to closely mirror real-world AI
programming tasks, offering a comprehensive and reliable framework for
assessing the security of AI-generated code. Our evaluation of leading LLMs on
A.S.E reveals several key findings. In particular, current LLMs still struggle
with secure coding. The complexity in repository-level scenarios presents
challenges for LLMs that typically perform well on snippet-level tasks.
Morever, a larger reasoning budget does not necessarily lead to better code
generation. These observations offer valuable insights into the current state
of AI code generation, assisting developers in selecting the most appropriate
models for practical tasks, while laying the foundation for refining LLMs to
generate secure and efficient code in real-world applications.
A.S.E is a repository-level benchmark for evaluating the security of AI-generated code, highlighting challenges in secure coding and the limitations of LLMs in real-world scenarios.
large language models LLMs AI code generation security evaluation repository-level evaluation secure coding reasoning budget