Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Large language models (LLMs) have demonstrated exceptional capabilities when
trained within executable runtime environments, notably excelling at software
engineering tasks through verified feedback loops. Yet, scalable and
generalizable execution-grounded environments remain scarce, limiting progress
in training more capable ML agents. We introduce CTF-Dojo, the first
large-scale executable runtime tailored for training LLMs with verifiable
feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style
challenges containerized in Docker with guaranteed reproducibility. To enable
rapid scaling without manual intervention, we develop CTF-Forge, an automated
pipeline that transforms publicly available artifacts into ready-to-use
execution environments in minutes, eliminating weeks of expert configuration
traditionally required. We trained LLM-based agents on just 486 high-quality,
execution-verified trajectories from CTF-Dojo, achieving up to 11.6% absolute
gains over strong baselines across three competitive benchmarks: InterCode-CTF,
NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1,
establishing a new open-weight state-of-the-art that rivals frontier models
like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a
benchmark for executable-agent learning, CTF-Dojo demonstrates that
execution-grounded training signals are not only effective but pivotal in
advancing high-performance ML agents without dependence on costly proprietary
systems.
68ae7956364411bea07df78a
Terry Yue Zhuo
claimed_verified
2025-08-27T07:10:42.523Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
68ae7956364411bea07df78b
Dingmin Wang
68ae7956364411bea07df78c
Hantian Ding
68ae7956364411bea07df78d
Varun Kumar
68ae7956364411bea07df78e
Zijian Wang
Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Large language models (LLMs) have demonstrated exceptional capabilities when
trained within executable runtime environments, notably excelling at software
engineering tasks through verified feedback loops. Yet, scalable and
generalizable execution-grounded environments remain scarce, limiting progress
in training more capable ML agents. We introduce CTF-Dojo, the first
large-scale executable runtime tailored for training LLMs with verifiable
feedback, featuring 658 fully functional Capture-The-Flag (CTF)-style
challenges containerized in Docker with guaranteed reproducibility. To enable
rapid scaling without manual intervention, we develop CTF-Forge, an automated
pipeline that transforms publicly available artifacts into ready-to-use
execution environments in minutes, eliminating weeks of expert configuration
traditionally required. We trained LLM-based agents on just 486 high-quality,
execution-verified trajectories from CTF-Dojo, achieving up to 11.6% absolute
gains over strong baselines across three competitive benchmarks: InterCode-CTF,
NYU CTF Bench, and Cybench. Our best-performing 32B model reaches 31.9% Pass@1,
establishing a new open-weight state-of-the-art that rivals frontier models
like DeepSeek-V3-0324 and Gemini-2.5-Flash. By framing CTF-style tasks as a
benchmark for executable-agent learning, CTF-Dojo demonstrates that
execution-grounded training signals are not only effective but pivotal in
advancing high-performance ML agents without dependence on costly proprietary
systems.
CTF-Dojo, a large-scale executable runtime with 658 CTF challenges, enables rapid training of LLM-based agents with verifiable feedback, achieving state-of-the-art performance in competitive benchmarks.
LLMs software engineering tasks verified feedback loops CTF-Dojo Capture-The-Flag Docker CTF-Forge automated pipeline execution-verified trajectories InterCode-CTF NYU CTF Bench Cybench Pass@1 DeepSeek-V3-0324 Gemini-2.5-Flash executable-agent learning