An illustrative scenario
Which change actually improved the agent?
A team sees a higher completion rate after changing both the model and retry policy. The new configuration also makes more tool calls. A useful comparison separates the model effect, harness effect, and extra resource budget before selecting a deployment candidate.
What to vary
Keep the comparison focused. Change the conditions that can alter the right next action.
- Model configuration
- Candidate model, prompt version, sampling settings, and context window policy.
- Harness configuration
- Planning, tool definitions, memory, retries, stopping rules, and human escalation.
- Environment conditions
- User behavior, task difficulty, tool failures, time limits, and available observations.
What to measure
Agree the checks before the evaluation. Inspect outcomes and failures alongside the summary.
- Task outcomes
- Report completion and failure categories on held-out task families, not only an aggregate score.
- Resource use
- Compare latency, tool calls, and cost under an agreed budget and measurement method.
- Review and uncertainty
- Inspect repeated runs, expert disagreement, unnecessary interventions, and high-impact misses.
Define your engagement
Start with a decision you can evaluate.
- You bring
- Candidate configurations, a baseline agent, target task families, tool adapters, and the evaluation budget.
- We scope
- A versioned comparison protocol, paired task runs, reproducible traces, and a readout of model-versus-harness tradeoffs.
Use the same task distribution and record all configuration changes. Automated graders need human checks; a higher score on one environment does not establish general agent capability.
Plan a scoped pilotQuestions about this use case
What is an agent harness?
An agent harness is the surrounding software that manages observations, tools, memory, execution, retries, and stopping rules. Helm scopes evaluations of that complete configuration alongside the underlying model.
Can this compare models and agents from different providers?
A comparison can be scoped when the interfaces expose the required observations, actions, and outcomes. Adapter work and the fairness of the comparison are agreed before the engagement.