A.S.E: A Repository-Level Benchmark for Evaluating Security in
  AI-Generated Code
The increasing adoption of large language models (LLMs) in software
engineering necessitates rigorous security evaluation of their generated code.
However, existing benchmarks are inadequate, as they focus on isolated code
snippets, employ unstable evaluation methods that lack reproducibility, and
fail to connect the quality of input context with the security of the output.
To address these gaps, we introduce A.S.E (AI Code Generation Security
Evaluation), a benchmark for repository-level secure code generation. A.S.E
constructs tasks from real-world repositories with documented CVEs, preserving
full repository context like build systems and cross-file dependencies. Its
reproducible, containerized evaluation framework uses expert-defined rules to
provide stable, auditable assessments of security, build quality, and
generation stability. Our evaluation of leading LLMs on A.S.E reveals three key
findings: (1) Claude-3.7-Sonnet achieves the best overall performance. (2) The
security gap between proprietary and open-source models is narrow;
Qwen3-235B-A22B-Instruct attains the top security score. (3) Concise,
``fast-thinking'' decoding strategies consistently outperform complex,
``slow-thinking'' reasoning for security patching.
68b50b5f851c6e7b001eca21
Keke Lian
claimed_verified
2025-09-01T07:50:36.564Z
67ca63c1e10c4e1bf8262576
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/e9MvyBRetgZXk73pQcsOX.png
Keke Lian
user
KekeLian
68b50b5f851c6e7b001eca22
Bin Wang
68b50b5f851c6e7b001eca23
Lei Zhang
68b50b5f851c6e7b001eca24
Libo Chen
68b50b5f851c6e7b001eca25
Junjie Wang
claimed_verified
2025-09-01T07:50:42.819Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
68b50b5f851c6e7b001eca26
Ziming Zhao
68b50b5f851c6e7b001eca27
Yujiu Yang
68b50b5f851c6e7b001eca28
Haotong Duan
68b50b5f851c6e7b001eca29
Haoran Zhao
68b50b5f851c6e7b001eca2a
Shuang Liao
68b50b5f851c6e7b001eca2b
Mingda Guo
68b50b5f851c6e7b001eca2c
Jiazheng Quan
claimed_verified
2025-09-01T13:51:34.413Z
68b51853a494ea3733e4e851
https://cdn-avatars.huggingface.co/v1/production/uploads/68b51853a494ea3733e4e851/-j-nJWeZZpmKzqHsIl7UN.png
Jiazheng Quan
user
jzquan
68b50b5f851c6e7b001eca2d
Yilu Zhong
68b50b5f851c6e7b001eca2e
Chenhao He
68b50b5f851c6e7b001eca2f
Zichuan Chen
68b50b5f851c6e7b001eca30
Jie Wu
68b50b5f851c6e7b001eca31
Haoling Li
claimed_verified
2025-09-02T07:57:06.228Z
650be23ec4e52db6a4db63ef
/avatars/03af548029b38bee49ec295fefe74f9a.svg
Haoling Li
user
Ringo1110
68b50b5f851c6e7b001eca32
Zhaoxuan Li
68b50b5f851c6e7b001eca33
Jiongchi Yu
68b50b5f851c6e7b001eca34
Hui Li
68b50b5f851c6e7b001eca35
Dong Zhang
claimed_verified
2025-09-03T08:29:27.965Z
68b5182cb710fd5fa8af991d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NZtlwYoqS9tg3l7G43_S9.png
dong zhang
user
zalezhang
A.S.E: A Repository-Level Benchmark for Evaluating Security in
  AI-Generated Code
The increasing adoption of large language models (LLMs) in software
engineering necessitates rigorous security evaluation of their generated code.
However, existing benchmarks are inadequate, as they focus on isolated code
snippets, employ unstable evaluation methods that lack reproducibility, and
fail to connect the quality of input context with the security of the output.
To address these gaps, we introduce A.S.E (AI Code Generation Security
Evaluation), a benchmark for repository-level secure code generation. A.S.E
constructs tasks from real-world repositories with documented CVEs, preserving
full repository context like build systems and cross-file dependencies. Its
reproducible, containerized evaluation framework uses expert-defined rules to
provide stable, auditable assessments of security, build quality, and
generation stability. Our evaluation of leading LLMs on A.S.E reveals three key
findings: (1) Claude-3.7-Sonnet achieves the best overall performance. (2) The
security gap between proprietary and open-source models is narrow;
Qwen3-235B-A22B-Instruct attains the top security score. (3) Concise,
``fast-thinking'' decoding strategies consistently outperform complex,
``slow-thinking'' reasoning for security patching.
A.S.E is a benchmark for evaluating the security of code generated by large language models using real-world repositories and expert-defined rules, revealing insights into model performance and decoding strategies.
large language models LLMs software engineering security evaluation generated code benchmarks code snippets evaluation methods reproducibility input context repository-level secure code generation CVEs build systems cross-file dependencies containerized evaluation expert-defined rules security build quality generation stability Claude-3.7-Sonnet Qwen3-235B-A22B-Instruct decoding strategies security patching