O&A Consulting · Answer

What's the difference between testing an AI agent and trusting it?

A test asks whether an agent behaved the way you expected on cases you thought of in advance. Trust asks whether a person can rely on it — including on the cases nobody wrote down — and be willing to say so in front of colleagues. These come apart in both directions: an agent can pass every test and still invent a citation on an unanticipated input, and it can fail a test because it improved past a stale expected answer. Testing is a property of your suite; trust is a property of the evidence you can show someone.

Agent EvaluationAI GovernanceRiskLLM Agents
01 Answer

Both failure directions are real

Passing but untrustworthy. Your suite covers the inputs someone imagined on a Tuesday. Production supplies the ones they didn't. An agent that never hallucinated across forty test cases can still fabricate a source on the forty-first, and nothing in a green dashboard warns you.

Failing but improved. A golden answer key freezes a fact. The world changes, the agent now returns the correct current answer, and the suite marks it a regression. The test punishes the right output, and somebody spends an afternoon "fixing" an improvement.

A suite that produces both errors isn't badly written. It's answering a narrower question than the one being asked of it.

02 Answer

What trust requires that a test doesn't

  • An agreed definition of success, written before the run. Otherwise you are scoring against whatever the evaluator assumed, and nobody consented to that.
  • Evidence behind every score. A number with no path back to the trace asks for exactly the faith you were trying to replace.
  • Repetition. Agents are stochastic. One green run is close to no information.
  • Something that checks the checker. If nothing evaluates the evaluator, the trust problem didn't disappear — it moved up a level and got quieter.
  • Stated scope. What wasn't checked, said out loud. This is what makes the artifact hold up when someone pushes on it.
03 Answer

Keep the tests

None of this argues for deleting your suite. Regression tests and drift harnesses are good engineering and they catch real breakage cheaply. Sometimes catching breakage is the entire job.

The argument is about what the suite entitles you to claim. Green means the agent behaved on the cases you encoded. If someone is being asked to accept accountability for the system, they need a different artifact, and no amount of additional test coverage converts one into the other.

04 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it: