An Empirical Study of Testing Practices in Open Source AI Agent
  Frameworks and Agentic Applications
Foundation model (FM)-based AI agents are rapidly gaining adoption across
diverse domains, but their inherent non-determinism and non-reproducibility
pose testing and quality assurance challenges. While recent benchmarks provide
task-level evaluations, there is limited understanding of how developers verify
the internal correctness of these agents during development.
  To address this gap, we conduct the first large-scale empirical study of
testing practices in the AI agent ecosystem, analyzing 39 open-source agent
frameworks and 439 agentic applications. We identify ten distinct testing
patterns and find that novel, agent-specific methods like DeepEval are seldom
used (around 1%), while traditional patterns like negative and membership
testing are widely adapted to manage FM uncertainty. By mapping these patterns
to canonical architectural components of agent frameworks and agentic
applications, we uncover a fundamental inversion of testing effort:
deterministic components like Resource Artifacts (tools) and Coordination
Artifacts (workflows) consume over 70% of testing effort, while the FM-based
Plan Body receives less than 5%. Crucially, this reveals a critical blind spot,
as the Trigger component (prompts) remains neglected, appearing in around 1% of
all tests.
  Our findings offer the first empirical testing baseline in FM-based agent
frameworks and agentic applications, revealing a rational but incomplete
adaptation to non-determinism. To address it, framework developers should
improve support for novel testing methods, application developers must adopt
prompt regression testing, and researchers should explore barriers to adoption.
Strengthening these practices is vital for building more robust and dependable
AI agents.
68d4a1a136950a9dff156889
Mohammed Mehedi Hasan
68d4a1a136950a9dff15688a
Hao Li
claimed_verified
2025-09-25T07:11:05.515Z
62b4f3b7464e664268bf4e85
/avatars/16e79e11a8734b1d241e0f0c55a54045.svg
Leo
user
hao-li
68d4a1a136950a9dff15688b
Emad Fallahzadeh
68d4a1a136950a9dff15688c
Gopi Krishnan Rajbahadur
68d4a1a136950a9dff15688d
Bram Adams
68d4a1a136950a9dff15688e
Ahmed E. Hassan
An Empirical Study of Testing Practices in Open Source AI Agent
  Frameworks and Agentic Applications
Foundation model (FM)-based AI agents are rapidly gaining adoption across
diverse domains, but their inherent non-determinism and non-reproducibility
pose testing and quality assurance challenges. While recent benchmarks provide
task-level evaluations, there is limited understanding of how developers verify
the internal correctness of these agents during development.
  To address this gap, we conduct the first large-scale empirical study of
testing practices in the AI agent ecosystem, analyzing 39 open-source agent
frameworks and 439 agentic applications. We identify ten distinct testing
patterns and find that novel, agent-specific methods like DeepEval are seldom
used (around 1%), while traditional patterns like negative and membership
testing are widely adapted to manage FM uncertainty. By mapping these patterns
to canonical architectural components of agent frameworks and agentic
applications, we uncover a fundamental inversion of testing effort:
deterministic components like Resource Artifacts (tools) and Coordination
Artifacts (workflows) consume over 70% of testing effort, while the FM-based
Plan Body receives less than 5%. Crucially, this reveals a critical blind spot,
as the Trigger component (prompts) remains neglected, appearing in around 1% of
all tests.
  Our findings offer the first empirical testing baseline in FM-based agent
frameworks and agentic applications, revealing a rational but incomplete
adaptation to non-determinism. To address it, framework developers should
improve support for novel testing methods, application developers must adopt
prompt regression testing, and researchers should explore barriers to adoption.
Strengthening these practices is vital for building more robust and dependable
AI agents.
The study identifies testing practices in AI agent frameworks and applications, highlighting a focus on deterministic components and a neglect of the Trigger component, suggesting improvements for robustness.
Foundation model AI agents non-determinism non-reproducibility testing practices DeepEval negative testing membership testing Resource Artifacts Coordination Artifacts Plan Body Trigger component prompt regression testing