O&A Consulting · Answer

Should you ask an LLM how similar two things are?

Q: Should you ask an LLM how similar two things are?

Usually not, if you can compute it instead. Similarity between two structured things — two deals, two incidents, two counterparties — has a definition you can write down: which attributes count, how each is compared, and how they are weighted. Computing it that way gives you three properties an LLM judgement cannot: it is reproducible, so the same pair scores the same every time; it is decomposable, so you can say which attributes drove the match rather than only that a match occurred; and it is auditable, so a disagreement is a conversation about weights rather than about what a model felt. Ask a model and you get a number that changes between runs, cannot be explained beyond its own summary, and quietly encodes whatever notion of similarity the model happened to bring. The model's real job in this problem is the parts that genuinely need language: normalising messy text into the structured attributes, and explaining a computed result once the computation has produced it. Compute the comparison, then let the model narrate it.

Usually not, if you can compute it instead. Similarity between structured things has a definition you can write down — which attributes count, how each is compared, how they're weighted — and computing it buys three things a model judgement can't: it's reproducible, it's decomposable (you can say which attributes drove the match), and it's auditable. Let the model do the parts that need language: normalising inputs, and explaining the result afterwards.

SimilarityEmbeddingsLLM JudgesRetrievalDeterminism

01 Answer

What you give up by asking

"How similar are these two deals, on a scale of 1 to 10?" gets an answer immediately, and the answer has three defects that only surface later:

It moves. Same pair, same prompt, different run, different number. Any threshold built on it is a threshold on a moving quantity, which means your alerting is partly noise.
It doesn't decompose. The useful output is almost never the score — it's which attributes made these comparable. A single number can't be broken back down, so the model is asked to explain itself, and that explanation is generated after the fact rather than derived from the computation.
It imports a definition you didn't choose. Similar in what sense? Structure, size, sector, timing, counterparty? Unless you said, the model picked — differently for each pair, invisibly.

02 Answer

Write the definition down instead

Nearly always, "similar" is a weighted comparison over attributes you can enumerate. Making that explicit turns a judgement into a calculation:

Which attributes count. Sector, size band, structure, timing, counterparty overlap — whatever the domain expert says matters.
How each is compared. Exact match, numeric distance, set overlap, embedding distance for genuinely free text.
How they're weighted. The part worth arguing about, and the part that should live in a versioned file rather than in a prompt.

Now a match explains itself: these two scored high on structure and counterparty overlap, low on size. That is a sentence an analyst can disagree with productively — and disagreement produces a weight change, which is a durable improvement, rather than a prompt tweak nobody can evaluate.

03 Answer

Structural similarity when attributes aren't enough

Some resemblance isn't in the attributes at all — it's in how a thing sits in a network. Two counterparties with nothing in common on paper can occupy the same structural position: same kind of neighbours, same role in the flow.

That's computable too, with graph measures and structural embeddings, and it's still deterministic and still decomposable. When you can corroborate it — for example two independent algorithms drawing similar community boundaries on the same graph — you have convergent evidence rather than one method's opinion. State the caveat honestly when you do: methods reading the same input aren't fully independent, and some agreement is guaranteed by construction.

04 Answer

Where the model genuinely earns its place

Not judging the comparison — feeding it and explaining it:

Normalisation. Turning messy prose into the structured attributes the computation needs. This is real language work and models are good at it.
Free-text fields. Where an attribute really is unstructured, an embedding distance is the right comparator — a computation, not a judgement.
Explanation. Once the computation has produced a decomposed result, having a model narrate it in plain language is genuinely useful, and it's grounded in numbers that exist independently of the narration.
Cases the definition doesn't cover. Route them to a person. An honest "this doesn't fit the model" beats a confident number.

05 Answer

The general rule

This is one instance of a rule worth applying across an agent system: never ask a model what code can answer. Deterministic checks are reproducible, cheap, and explainable; model judgements are none of those and should be spent only where language really is the problem.

The same logic decides where an LLM belongs in evaluation — mechanical checks first, a model only where meaning is genuinely at stake.

06 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it:

Should we build one knowledge graph for the whole firm? →How do you measure whether your retrieval pipeline is any good? →Don't hand the agent a query language ↗