O&A Consulting · Answer

How do you know an AI agent is ready for production?

An AI agent is ready for production when three things are true: someone has written down what success means in terms specific enough to check, there is evidence each of those conditions was met on real runs, and a named person is willing to sign off on that evidence. A passing test suite is not the same thing. Tests tell you the agent behaved on the cases you thought of; readiness is a claim about whether you can rely on it, including on the cases you didn't. Most teams have the first kind of evidence and are being asked for the second.

Agent EvaluationProduction ReadinessLLM AgentsAI Governance
01 Answer

Why a green test suite isn't the answer

Drift harnesses and regression suites are good engineering, and worth building. They answer a specific question well: did the agent behave the way it behaved last week? That catches real problems.

But the question a stakeholder has to put their name on is different: can we rely on this? Those come apart in ways that matter. An agent can pass every test and still fabricate a citation on an input nobody wrote a test for. An agent can fail a test because it got better — a golden answer key goes stale, the agent returns the now-correct answer, and the suite marks it a regression.

The gap isn't a gap in test coverage. It's that a test compares output to an expectation, and readiness is a question about properties: did it stay inside policy, was every claim grounded in something real, did it miss anything that mattered, what did it cost.

02 Answer

What readiness evidence actually looks like

Concretely, before an agent ships you want to be able to hand someone:

  • A written definition of done. The behaviours that must hold, with thresholds a machine can check — not a paragraph of intent. Someone has to have agreed to it before the run, or you're scoring against whatever the evaluator happened to assume.
  • Evidence tied to the trace. Every finding pointing at the step and the quoted text it came from. A score with no evidence path behind it asks for the same faith you were trying to stop giving.
  • Results across repeated runs. Agents are stochastic. A single passing run tells you almost nothing; the same task across trials tells you whether it passes reliably or sometimes.
  • A record of what was checked and what wasn't. Scope honesty is what makes the artifact defensible when someone pushes on it.
03 Answer

The question that settles it

If you want a single diagnostic, ask the team this: if it passed every test you have, would you ship it tomorrow?

Hesitation is the answer. It means the tests aren't measuring the thing that would let someone sign off, and everyone in the room already knows it. That gap is what readiness work is actually for.

04 Answer

A caution on asking a model to grade it

The obvious shortcut is to have a strong model read the transcript and judge whether things went well. It works better than people expect on coarse questions and worse than people expect on precise ones, and the difference is not about model strength.

A judge is reliable in proportion to how well-posed the question is. Ask "which of these agents failed" and a frontier model does fine. Ask "which single step was the decisive error" — a question competent human reviewers answer differently from each other — and accuracy collapses, and a better model doesn't rescue it. The fix is making the question determinate before asking, which is what a written contract does.

05 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it: