An illustrative scenario
Does the intent survive the handoff?
A research agent gathers supplier options, a planning agent chooses a purchase, and an execution agent prepares the order. A delivery constraint appears in the first conversation but is missing from the handoff. Each agent can complete its local task while the overall order still fails.
What to vary
Keep the comparison focused. Change the conditions that can alter the right next action.
- Shared context
- Incomplete summaries, stale records, conflicting assumptions, and corrections after a handoff.
- Responsibility
- Ambiguous ownership, unavailable agents, approval boundaries, and competing task priorities.
- Execution state
- Duplicate actions, partial completion, tool timeouts, and recovery after a failed step.
What to measure
Agree the checks before the evaluation. Inspect outcomes and failures alongside the summary.
- End-to-end outcome
- Check the final result against the original intent and constraints, as well as each local step.
- Context preservation
- Trace where evidence or permissions were lost, altered, or used after becoming stale.
- Recovery and ownership
- Inspect who detects a failure, who may act, and whether retries create conflicting changes.
Define your engagement
Start with a decision you can evaluate.
- You bring
- A workflow graph, agent and tool interfaces, shared-state definitions, and an accountable owner for the final outcome.
- We scope
- Handoff scenarios, trace-level error categories, intervention points, and a baseline comparison across the full workflow.
Define observability across all participating systems. Without shared identifiers and final-state checks, a clean local trace may conceal an unresolved end-to-end task.
Plan a scoped pilotQuestions about this use case
Why evaluate the full workflow if each agent passes its tests?
Local tests may not exercise how one agent’s output becomes another agent’s input. The evaluation checks whether the original task, permissions, and state remain intact across those transitions.
Does every handoff need human review?
The pilot should identify which handoffs need review and which can proceed within explicit rules. Review burden and missed issues are measured together; universal approval is not the goal.