NetPress: Dynamically Generated LLM Benchmarks for Network Applications
Despite growing interest in domain-specific benchmarking of large language
models (LLMs) and agents, current evaluations remain limited to static,
small-scale datasets, especially in high-stakes tasks like network operations
that demand reliability for deployments. We present NetPress, an automated
benchmark generation framework for evaluating LLM agents in network
applications. NetPress introduces a unified abstraction with state and action,
enabling dynamic generation of diverse query sets along with corresponding
ground truths. At runtime, users can specify benchmark configurations to
generate millions of queries on the fly. In addition to dynamic benchmark
construction, NetPress integrates with network emulators to provide realistic
environment feedback, supporting comprehensive evaluation across correctness,
safety, and latency. We instantiate NetPress on three representative
applications, revealing interesting fine-grained differences in agent behavior
that static, correctness-only benchmarks often miss. NetPress moves LLM
evaluation toward realistic, scalable testing in infrastructure-centric
domains, helping close the gap between benchmark performance and real-world
deployment readiness. Code is available at
https://github.com/Froot-NetSys/NetPress.
684891e442e4f9106973f1de
Yajie Zhou
claimed_verified
2025-06-11T08:35:21.667Z
64825efa2e73ce18ad32d007
/avatars/f39772eb5960d3faf556d16a87940316.svg
Yajie Zhou
user
lesleychou
684891e442e4f9106973f1df
Jiajun Ruan
684891e442e4f9106973f1e0
Eric S. Wang
684891e442e4f9106973f1e1
Sadjad Fouladi
684891e442e4f9106973f1e2
Francis Y. Yan
684891e442e4f9106973f1e3
Kevin Hsieh
684891e442e4f9106973f1e4
Zaoxing Liu
NetPress: Dynamically Generated LLM Benchmarks for Network Applications
Despite growing interest in domain-specific benchmarking of large language
models (LLMs) and agents, current evaluations remain limited to static,
small-scale datasets, especially in high-stakes tasks like network operations
that demand reliability for deployments. We present NetPress, an automated
benchmark generation framework for evaluating LLM agents in network
applications. NetPress introduces a unified abstraction with state and action,
enabling dynamic generation of diverse query sets along with corresponding
ground truths. At runtime, users can specify benchmark configurations to
generate millions of queries on the fly. In addition to dynamic benchmark
construction, NetPress integrates with network emulators to provide realistic
environment feedback, supporting comprehensive evaluation across correctness,
safety, and latency. We instantiate NetPress on three representative
applications, revealing interesting fine-grained differences in agent behavior
that static, correctness-only benchmarks often miss. NetPress moves LLM
evaluation toward realistic, scalable testing in infrastructure-centric
domains, helping close the gap between benchmark performance and real-world
deployment readiness. Code is available at
https://github.com/Froot-NetSys/NetPress.
NetPress generates dynamic benchmarks for evaluating large language model agents in network operations, providing realistic tests across correctness, safety, and latency.
large language models agents benchmark generation state and action dynamic benchmark construction network emulators query sets ground truths