How to evaluate an AI agent and its harness

1. Write the decision the evaluation should support

Choose the actual question: whether to change a model, adopt a new harness, reduce a review step, or expand into another task family. Avoid mixing these into a single claim of overall agent quality.

Define the primary outcome and the important failure categories. For an account update, the check might require the correct record to change once with a valid approval. A fluent final message alone does not establish that result.

2. Freeze the configuration and the comparison

Record model and prompt versions, tool definitions, memory access, retry limits, stopping rules, and environment state. Compare candidate configurations on matched tasks and use repeated runs where stochastic behavior affects the result.

If several components change together, report that the whole configuration changed. Separate controlled comparisons are needed to attribute an improvement to one component. Keep evaluation tasks out of tuning.

3. Inspect outcomes, cost, and human involvement

Report failures by task family and consequence, alongside completion, latency, tool calls, and measurement costs. Review the traces behind the summary: retries can mask fragile behavior, and aggregate scores can hide a weak subgroup.

Human review is part of the protocol. Check whether automated graders agree with independent experts, preserve uncertainty, and record both missed issues and unnecessary interventions.

4. Decide what the evidence supports

State the environments, tasks, and conditions that were actually tested. Compare the new configuration against the current baseline using acceptance criteria agreed before the runs. A useful result may identify a narrow improvement or a reason to keep the current system.

Package the configuration, scenario versions, outcome checks, traces, and limitations together. A deployment decision needs integration and operating evidence in addition to a good offline result.

Make the distinction explicit.