An illustrative scenario
Does the right click lead to the right outcome?
An operations agent is asked to update a supplier record. Two suppliers have similar names, a modal obscures a warning, and the workflow requires a preview before committing. Success means the correct record changed once, with the required approval.
What to vary
Keep the comparison focused. Change the conditions that can alter the right next action.
- Interface state
- Reordered controls, loading delays, stale pages, dialogs, and interrupted sessions.
- Task context
- Similar records, incomplete requests, changing instructions, and ambiguous field names.
- Commit boundaries
- Preview requirements, reversible edits, final confirmations, and failed save operations.
What to measure
Agree the checks before the evaluation. Inspect outcomes and failures alongside the summary.
- Final-state correctness
- Compare the intended record change with the application state after the run.
- Authority adherence
- Check whether the agent paused before consequential actions outside its permission.
- Recovery quality
- Inspect retries, duplicate changes, abandoned edits, and useful requests for clarification.
Define your engagement
Start with a decision you can evaluate.
- You bring
- A sandbox application, representative tasks, screen or tool access, reset procedures, and the rules for committed actions.
- We scope
- A versioned screen-task suite, action traces, outcome checks, and failure categories for comparing models or harnesses.
A screen recording alone may not prove that a database update succeeded. Scope observable final-state checks and application reset access; keep production credentials outside the test environment.
Plan a scoped pilotQuestions about this use case
Can the same model behave differently with another harness?
Yes. In a scoped comparison, tool access, observations, memory, retry logic, and stopping rules are separate configuration choices. Record and control them so the comparison answers a specific question.
Does Helm replace the browser or application?
The engagement defines adapters and an evaluation environment around the application. Compatibility, reset access, and final-state verification are established during scoping.