Workflow & Agent Testing
Document ID: TEST-003
File Path: docs/15-testing/workflow-tests.md
Version: 1.0.0
Status: Draft
Owner: Quality Engineering Team
Last Updated: 2026-06-27
1. Purpose
Section titled “1. Purpose”This document defines end-to-end behavioral testing of workflows and agents — verifying that authored definitions produce correct outcomes across the full runtime, including durability, retries, compensation, and agent reasoning.
2. Workflow Testing
Section titled “2. Workflow Testing”2.1 Levels
Section titled “2.1 Levels”| Level | Verifies |
|---|---|
| Definition | DSL validates and compiles (validation rules) |
| Execution | Correct path, outputs, and state transitions |
| Durability | Survives restart via checkpointing |
| Resilience | Retry and compensation behave |
2.2 Deterministic Replay
Section titled “2.2 Deterministic Replay”Because execution is deterministic, tests replay an execution from its event history and assert identical state — catching non-determinism and validating checkpoint/resume.
2.3 Time & Events
Section titled “2.3 Time & Events”Tests use a fake clock to fast-forward timers and inject events/signals to drive waits, rather than waiting in real time.
3. Agent Testing
Section titled “3. Agent Testing”3.1 Behavioral Tests
Section titled “3.1 Behavioral Tests”Run an agent against fixed inputs with a fake provider returning scripted model responses, asserting the agent plans, calls the right tools, and produces expected output.
3.2 Trace Assertions
Section titled “3.2 Trace Assertions”Tests assert on the run trace: planner steps, tool calls and arguments, and memory reads — verifying how the agent reached its answer, not just the final text.
3.3 Tool & Memory Stubs
Section titled “3.3 Tool & Memory Stubs”- Tools can run as deterministic stubs or in the real Tool Runtime sandbox.
- Memory is seeded with fixtures so retrieval is reproducible.
4. Evaluation (Quality) Testing
Section titled “4. Evaluation (Quality) Testing”Beyond pass/fail, agents are evaluated for quality:
- Test cases with golden outputs or rubric-based grading.
- Version-over-version comparison (quality, cost, latency) in Agent Studio.
- Regression gates: a new agent/prompt version must not regress key metrics.
LLM-as-judge grading uses a pinned judge model via the LLM Gateway for repeatability.
5. Golden / Snapshot Suites
Section titled “5. Golden / Snapshot Suites”Curated suites of representative workflows and agent scenarios run in CI as snapshots; intentional output changes update the snapshot under review.
6. Human Tasks & Approvals
Section titled “6. Human Tasks & Approvals”Tests drive human tasks programmatically (approve/reject) to cover suspended/resumed paths and separation-of-duties rules.
7. CI Integration
Section titled “7. CI Integration”A core set runs per PR; broader evaluation suites run nightly (they can be slower and cost provider tokens where live models are used).
8. Dependencies
Section titled “8. Dependencies”03-workflow-engine/execution-model.md04-agent-framework/agent-runtime-protocol.md15-testing/integration-tests.md
9. Related Documents
Section titled “9. Related Documents”10. Revision History
Section titled “10. Revision History”| Version | Date | Description |
|---|---|---|
| 1.0.0 | 2026-06-27 | Initial Workflow & Agent Testing specification |