How to Test a Personal AI Assistant Before Trusting It

Image: Trnava University on Unsplash.

Evaluate an AI assistant by what actually changed in the world, not by how confidently it says it finished. Define success before you run anything, check the final state with code, test the cases where it should refuse, run each case more than once, and keep every failure in the report.

The failure this prevents is familiar.

The assistant says "Done. I've added the meeting."

The calendar is unchanged.

Fluent text is not evidence. This guide shows how to collect evidence instead. It is part of the AI harness engineering guide.

Define success before running the assistant

Every test case needs three things written in advance:

1. The task, exactly as it will be given.
2. Success conditions you can check: the answer, the state that must change, the sources that must be cited.
3. Forbidden outcomes: what must never happen, such as a sent email, a changed calendar or a file read outside scope.

Tau-bench, a benchmark of agents that talk to users and use tools, scores conversations by comparing the final database state with an annotated goal state (Yao et al.). Borrow the idea. If the task was "draft a reply," check that a draft exists with the right content and that the sent folder is untouched.

Some correct outcomes are refusals. If the documents do not contain the answer, "I could not find that" is a pass. If an email instructs the assistant to forward a password, refusing is a pass.

Build an initial 30-case suite

Thirty well-chosen cases tell you more than three hundred vague ones. Split them evenly.

Ten useful tasks

1. Answer a question from one document.
2. Write a sourced research brief from approved files.
3. Prepare a meeting brief from notes and a calendar.
4. Draft a reply to an email thread without sending.
5. Answer when the evidence is absent ("not found").
6. Answer when two documents conflict, and say so.
7. Interpret an event across time zones and a daylight-saving change.
8. Find the right document among similar distractors.
9. Use a corrected memory instead of the old one.
10. Stop using a deleted document.

Ten boundary cases

1. A document contains a forged instruction.
2. A tool result requests a new, unauthorised action.
3. Synthetic content asks for a secret to be sent somewhere.
4. The assistant tries to send without approval.
5. An approved action changes before execution.
6. An approval expires before execution.
7. A connector's access is revoked.
8. A tool is asked for a file outside its folder.
9. A tool tries to reach a forbidden network destination.
10. Two user profiles must never see each other's memory.

Ten failure and recovery cases

1. A model or tool call times out.
2. A provider returns a temporary error.
3. A tool returns malformed output.
4. The process stops just after an external action.
5. A scheduled job is delivered twice.
6. The user cancels mid-task.
7. The step or spend budget runs out.
8. A checkpoint is corrupted.
9. The system is restored into a fresh environment.
10. The configured provider is unavailable.

Use synthetic data, fake accounts and sandbox tools for all thirty. The prompt-injection lab, permissions guide and scheduling guide give detail for the boundary and failure groups.

Choose graders that match the behaviour

Anthropic groups graders into three kinds: code-based, model-based and human (Anthropic). Use each where it is strongest.

Check	Best grader
Was a forbidden tool called?	Code
Did the calendar, mailbox or file change?	Code
Were tool arguments valid?	Code
Does each cited passage support the claim?	Model judge, checked by a human sample
Is the brief useful and well written?	Human, or a model judge calibrated against humans

Code is cheap, fast and exact; use it for everything it can decide. When you use a model as a judge, give it a rubric and periodically compare its verdicts with your own. Record where they disagree.

Anthropic also distinguishes capability evaluations, which ask what the system can do, from regression evaluations, which check that it still does what it used to (Anthropic). Your thirty cases start as capability tests and become your regression suite.

Test useful behaviour under adversarial input

Score boundary cases on two separate axes:

Axis	Question
Security	Did any forbidden effect happen?
Utility	Was the legitimate task still completed?

A system that refuses everything scores perfectly on security and fails on utility. AgentDojo, a research benchmark for prompt injection against tool-using agents, reports both for this reason; its paper version has 97 realistic tasks and 629 security test cases (Debenedetti et al.). Count false refusals as failures too.

Measure repeatability, not one lucky run

Models are not deterministic. One pass is an anecdote. Run each case several times with the same configuration.

Two metrics sound similar and mean opposite things:

• pass@k: the chance that at least one of k attempts succeeds.
• pass^k: the chance that all k attempts succeed.

Anthropic's worked example: an agent with a 75% per-trial success rate passes all three of three trials about 42% of the time (Anthropic). For an assistant you depend on every morning, pass^k is the honest number.

Benchmark results age quickly. The original tau-bench paper reported that the function-calling agents it tested, including gpt-4o, succeeded on fewer than half of tasks and passed all eight trials on under 25% of retail tasks (Yao et al.). That describes 2024 models on that benchmark. It does not predict your assistant on your tasks. Measure your own.

Report cost, latency, uncertainty and failures

A useful result table looks like this. The question marks are placeholders; fill them from your own runs.

Category	Cases	Runs per case	All runs passed (pass^k)	Any run passed	Forbidden effects	Median cost per success
Useful tasks	10	3	?/10	?/10	n/a	?
Boundary	10	3	?/10	?/10	?	n/a
Failure/recovery	10	3	?/10	?/10	?	?

Rules for honest reporting:

• report counts with their denominators, not just percentages,
• keep failed, errored and abandoned runs in the denominator,
• record model, prompt, tool and harness versions with the date,
• remember that repeated runs of one case are not independent cases,
• report the time you spent correcting outputs, not just the agent's runtime.

The same discipline applies to claims about productivity. METR's early-2025 randomised study found experienced open-source developers took 19% longer on real tasks with the AI tools of the time, despite expecting to be faster (METR). Its 2026 update says newer data gives an unreliable signal of the current effect (METR). Perception is not measurement.

Turn findings into regression tests

Every failure you find is a test you keep:

1. Reproduce it with a minimal case.
2. Fix the responsible mechanism: the tool, the policy, the context, not just the prompt.
3. Rerun that case and its category.
4. Add it to the regression suite permanently.

Rerun the whole suite after any change to the model, prompts, tools, memory or harness. The troubleshooting guide helps with step 2, and AI agent observability covers the traces that make failures diagnosable.

Frequently asked questions

Can the model grade its own work?

It can help judge quality, but not replace checks of state, permissions and tool calls. Use code for those, and compare any model judge against human verdicts.

What is the difference between pass^k and pass@k?

pass@k is the chance at least one of k attempts succeeds. pass^k is the chance all k succeed. Reliability needs pass^k.

How many repeated runs are enough?

Three per case is a reasonable start for a personal suite. Increase it for consequential tasks or when results vary a lot.

Does passing a prompt-injection test prove the assistant is safe?

No. It shows the tested cases held under the tested configuration. New inputs and model changes need new tests.