Local Success Does Not Compose: Benchmarking Large Language Models for
  Compositional Formal Verification
We introduce DafnyCOMP, a benchmark for evaluating large language models
(LLMs) on compositional specification generation in Dafny. Unlike prior
benchmarks that focus on single-function tasks, DafnyCOMP targets programs
composed of multiple interacting functions with data dependencies, requiring
reasoning across component boundaries. The benchmark consists of 300
automatically synthesized multi-function programs. We evaluate several
state-of-the-art LLM families and find that, while they perform well on
single-function verification, their performance drops sharply on compositional
tasks. Analysis reveals systematic failures in cross-functional reasoning,
including fragile specifications, misalignment between implementations and
proofs, and unstable reasoning. DafnyCOMP thus provides a diagnostic tool for
measuring progress toward reliable, verifiable, and compositional code
generation with LLMs.
68db8c9fd2bf1f4b15ec7869
Xu Xu
68db8c9fd2bf1f4b15ec786a
Xin Li
claimed_verified
2025-09-30T10:17:52.923Z
64a948d9723beceb2f13f5eb
https://cdn-avatars.huggingface.co/v1/production/uploads/64a948d9723beceb2f13f5eb/UQMO7LczJpabtbcoNt1zQ.jpeg
XinLi
user
XINLI1997
68db8c9fd2bf1f4b15ec786b
Xingwei Qu
68db8c9fd2bf1f4b15ec786c
Jie Fu
68db8c9fd2bf1f4b15ec786d
Binhang Yuan
Local Success Does Not Compose: Benchmarking Large Language Models for
  Compositional Formal Verification
We introduce DafnyCOMP, a benchmark for evaluating large language models
(LLMs) on compositional specification generation in Dafny. Unlike prior
benchmarks that focus on single-function tasks, DafnyCOMP targets programs
composed of multiple interacting functions with data dependencies, requiring
reasoning across component boundaries. The benchmark consists of 300
automatically synthesized multi-function programs. We evaluate several
state-of-the-art LLM families and find that, while they perform well on
single-function verification, their performance drops sharply on compositional
tasks. Analysis reveals systematic failures in cross-functional reasoning,
including fragile specifications, misalignment between implementations and
proofs, and unstable reasoning. DafnyCOMP thus provides a diagnostic tool for
measuring progress toward reliable, verifiable, and compositional code
generation with LLMs.
DafnyCOMP evaluates large language models on generating compositional specifications in Dafny, highlighting their weaknesses in cross-functional reasoning.
large language models compositional specification generation Dafny multi-function programs cross-functional reasoning fragile specifications misalignment unstable reasoning