An illustrative scenario
Does your agent remember what still matters?
A travel assistant learns that a person prefers morning flights. On a later trip, the person asks for an evening departure because of a meeting. The agent should preserve the general preference while applying the exception to the current trip.
What to vary
Keep the comparison focused. Change the conditions that can alter the right next action.
- Time and context
- Long gaps, a new task, incomplete history, and preferences relevant to only one situation.
- Corrections
- Changed priorities, explicit reversals, conflicting memories, and ambiguous updates.
- Access boundaries
- Separate users, shared workspaces, restricted records, and requests to forget information.
What to measure
Agree the checks before the evaluation. Inspect outcomes and failures alongside the summary.
- Relevant recall
- Was the useful preference retrieved without bringing unrelated personal context into the task?
- Update fidelity
- Did the correction change the right memory at the right scope?
- Boundary handling
- Were access restrictions, user separation, and deletion expectations respected?
Define your engagement
Start with a decision you can evaluate.
- You bring
- The agent’s memory interface, a target task family, context-access rules, and a definition of persistence and deletion behavior.
- We scope
- Multi-session scenarios, controlled memory variants, outcome checks, and traces of retrieval and correction failures.
Simulated recall success does not establish population-wide personalization quality. Validate with people and tasks outside the scenario design, and separate interaction realism from data-access enforcement.
Plan a scoped pilotQuestions about this use case
How is this different from a one-turn preference test?
The scenarios separate when a preference is learned, when it is useful, and when it changes. Evaluation follows the memory state and the task outcome over more than one session.
Can evaluation data be used for training?
Evaluation access and training rights are agreed separately. Keep independent tasks and human evidence outside the training process so the comparison remains meaningful.