O&A
O&A Consulting · Answer

Why doesn't anyone use the knowledge base we built?

Usually because the first thing it asks of a person is the one thing they can't do: state precisely what they want from a corpus they've never seen. Retrieval quality is rarely why adoption fails. A chat box is a blank sheet of paper — it works beautifully for the engineer who knows what's in there, and strands everyone else. The fix isn't better retrieval. It's picking one user, one expensive question, and opening with something on the screen instead of a cursor.

Knowledge BasesRAGAdoptionUser ResearchProduct
01 Answer

The hallway test

Before O&A, our principal built a chat-based retrieval demo over an internal corpus and put it in front of a few coworkers with no briefing, on purpose. The interface was a greeting and a cursor. The response, close to verbatim: "what do I ask, what do I type, what do I do?"

The retrieval was fine. Ask it a well-formed question and it returned good passages with citations. Nobody got that far. The greeting was doing real work — polite, attentive, and completely uninformative at the moment that decided whether anyone came back.

Call this what it was: a hallway test with a handful of people, not a study. The result wasn't marginal, which is the only reason it's worth reporting.

02 Answer

The blank page problem

A text box is a request for a well-posed query. Posing one requires knowing roughly what's in the corpus, what vocabulary it uses, and what kinds of question it can answer at all. The person who has all three is the person who built it.

Everyone else is being asked to guess, and the cost of a wrong guess is silent. A bad query doesn't return an error; it returns something plausible and mediocre. The user can't tell whether the system is weak or their question was, so they conclude the system is weak — and that conclusion is very hard to reverse on a second visit that never happens.

03 Answer

One box, three users, three different failures

The second reason is that "the firm's knowledge base" is usually several products wearing one interface:

  • The analyst. Retrieval already works for them. The win isn't answering well — it's surfacing a connection they wouldn't have thought to look for. A box that waits to be asked can't do that by construction.
  • The portfolio manager. Prose gets them most of the way and then leaves them digging for the one number that actually drives the decision. Close enough to be frustrating.
  • The quant. The same-looking box is categorically the wrong deliverable. They need structured, time-aware, machine-readable output — signals per issuer per day — not prose, however good. Serving them means an API, stable entity IDs that don't churn between runs, and point-in-time correctness. Miss that and every backtest built on it is quietly worthless.

That last one is invisible to anyone designing for the analyst, and it's the first thing a quant asks about.

04 Answer

The OpenDoc problem

There's a well-known 1997 exchange where a developer asks Steve Jobs why Apple killed OpenDoc, and Jobs concedes the technology is genuinely excellent before asking the question that actually decided it: what customer experience does this fit, and will anyone pay for it?

The money part doesn't transfer — nobody counts revenue on an internal knowledge base. What transfers is the shape of the failure: build the excellent thing first, look for who wanted it afterwards. And infrastructure nobody adopts is more expensive than a product nobody buys, because it looks like success on every internal metric right up until someone checks who's actually using it.

05 Answer

What to do instead

  • Open with content, not a cursor. Show what's in the corpus and what's moved recently. Give people something to react to — reacting is easy, originating is hard.
  • Pick one user and one expensive question. Not a platform. One question that costs somebody real time today and that they've already told you they need answered.
  • Let the system start work unprompted. The most valuable mode has no query in it at all — a report that runs on a schedule and proposes something. There, precision beats recall: forty suggestions with thirty wrong gets ignored within two weeks.
  • Watch someone use it, early and without briefing. People can't reliably describe what they want from a system they haven't used. They can tell you instantly when they're stuck.
  • Set the bar at behaviour, not benchmarks. Did this change what a specific person does at a specific moment? A knowledge base every engineer is proud of, that no analyst opens, has accomplished nothing.
06 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it:

Should we build one knowledge graph for the whole firm? →Don't hand the agent a query language ↗How do you evaluate a RAG system? →