Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on
  Open & Industry Data
Deep learning solutions for vulnerability detection proposed in academic
research are not always accessible to developers, and their applicability in
industrial settings is rarely addressed. Transferring such technologies from
academia to industry presents challenges related to trustworthiness, legacy
systems, limited digital literacy, and the gap between academic and industrial
expertise. For deep learning in particular, performance and integration into
existing workflows are additional concerns. In this work, we first evaluate the
performance of CodeBERT for detecting vulnerable functions in industrial and
open-source software. We analyse its cross-domain generalisation when
fine-tuned on open-source data and tested on industrial data, and vice versa,
also exploring strategies for handling class imbalance. Based on these results,
we develop AI-DO(Automating vulnerability detection Integration for Developers'
Operations), a Continuous Integration-Continuous Deployment (CI/CD)-integrated
recommender system that uses fine-tuned CodeBERT to detect and localise
vulnerabilities during code review without disrupting workflows. Finally, we
assess the tool's perceived usefulness through a survey with the company's IT
professionals. Our results show that models trained on industrial data detect
vulnerabilities accurately within the same domain but lose performance on
open-source code, while a deep learner fine-tuned on open data, with
appropriate undersampling techniques, improves the detection of
vulnerabilities.
68c3ba4dfc1747b912403ae3
Moritz Mock
claimed_verified
2025-09-12T16:07:26.778Z
65d6202775878874dabd2d16
https://cdn-avatars.huggingface.co/v1/production/uploads/65d6202775878874dabd2d16/ZLmk_pMXTCMsk_GapUHbl.jpeg
Moritz Mock
user
mmock
68c3ba4dfc1747b912403ae4
Thomas Forrer
68c3ba4dfc1747b912403ae5
Barbara Russo
Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on
  Open & Industry Data
Deep learning solutions for vulnerability detection proposed in academic
research are not always accessible to developers, and their applicability in
industrial settings is rarely addressed. Transferring such technologies from
academia to industry presents challenges related to trustworthiness, legacy
systems, limited digital literacy, and the gap between academic and industrial
expertise. For deep learning in particular, performance and integration into
existing workflows are additional concerns. In this work, we first evaluate the
performance of CodeBERT for detecting vulnerable functions in industrial and
open-source software. We analyse its cross-domain generalisation when
fine-tuned on open-source data and tested on industrial data, and vice versa,
also exploring strategies for handling class imbalance. Based on these results,
we develop AI-DO(Automating vulnerability detection Integration for Developers'
Operations), a Continuous Integration-Continuous Deployment (CI/CD)-integrated
recommender system that uses fine-tuned CodeBERT to detect and localise
vulnerabilities during code review without disrupting workflows. Finally, we
assess the tool's perceived usefulness through a survey with the company's IT
professionals. Our results show that models trained on industrial data detect
vulnerabilities accurately within the same domain but lose performance on
open-source code, while a deep learner fine-tuned on open data, with
appropriate undersampling techniques, improves the detection of
vulnerabilities.
A recommender system integrated into CI/CD pipelines uses fine-tuned CodeBERT to detect and localize vulnerabilities in code without disrupting workflows, showing improved performance with appropriate undersampling techniques.
CodeBERT vulnerability detection cross-domain generalisation class imbalance Continuous Integration-Continuous Deployment CI/CD recommender system code review