O&A
O&A Consulting · Answer

How do you know your evaluation dataset is labelled correctly?

Check the labels against each other before using them to score anything. A label set can contradict itself in ways no accuracy number reveals — and the cheapest check needs no second annotator, just code that looks for entries whose own fields disagree. Running that across a hand-labelled public benchmark of 184 agent-failure traces surfaced 6 internally inconsistent entries. An evaluation is capped by its answer key, and most answer keys have never been audited.

EvaluationGround TruthBenchmarksData QualityAnnotation
01 Answer

Start with the check that needs no second opinion

Inter-annotator agreement is the textbook answer and it's expensive — you need a second qualified person to re-label a sample. Before paying for that, run the check that costs nothing: does each record contradict itself?

Structured annotations usually carry several fields that constrain one another, and nothing enforces the constraint at labelling time. Examples that are pure code to detect:

  • A named culprit step that falls outside the step range the same record marks as the failure window.
  • A stated root cause the referenced evidence doesn't contain.
  • A category assignment that the record's own free-text explanation contradicts.
  • Index references pointing past the end of the trace they belong to.

We ran this over a hand-labelled public benchmark of agent failure traces: 6 of 184 entries were internally inconsistent. Found in code, in one pass, without re-reading a single trace by hand.

02 Answer

Why this matters more than the percentage suggests

Roughly 3% sounds tolerable, and it isn't, for two reasons.

First, those are only the errors a machine could see. Self-contradiction is the most detectable class of labelling error. Entries that are simply wrong but internally coherent are invisible to this check, and there is no reason to think they're rarer.

Second, error in the answer key doesn't average out — it sets a ceiling. A model scored against a partly wrong key is penalised for being right, and two systems can only be distinguished within the noise the labels introduce. If competing methods are separated by a few points and the key has a few points of error, the comparison has no resolving power, whatever the leaderboard says.

03 Answer

Then the three that cost something

  • Measure agreement, don't assume it. Have a second qualified annotator re-label a sample and compute chance-corrected agreement. Low agreement is not a people problem — it means the label is not well-posed. "Which step was the decisive error?" often isn't, because several steps are jointly responsible and the format forces a single pick.
  • Re-derive a sample independently. Have someone produce labels from scratch without seeing the originals, then diff. Reviewing an existing label invites agreement; producing one doesn't.
  • Treat disputed failures as candidate label bugs. When a model fails a case and an engineer says "that answer looks right to me," that is evidence about the label, not just an excuse. Track how often it resolves in favour of the key. If it's often, audit the key.
04 Answer

What to do with a key you can't fully trust

You usually can't rebuild a benchmark, so bound the claim instead of pretending the problem away:

  • Report the audit alongside the score. "94% on a key with a measured 3% internal inconsistency rate" is an honest number. "94%" is not.
  • Exclude, don't silently fix. Drop entries that fail the consistency check and say how many you dropped, so the result stays reproducible.
  • Don't claim differences inside the noise floor. If label error is 3%, a 2-point gap between two methods is not a finding.
  • Prefer checks that need no key at all. Schema conformance, forbidden actions, budgets, and citations resolving to retrieved sources are all exactly checkable without an answer key — and they can't be contradicted by one.
05 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it:

Goldens aren't ground truth — the argument, interactive ↗Why isn't output accuracy enough for evaluating AI agents? →The full guide to AI agent evaluation ↗