O&A Consulting
Field notes on retrieval · 04

Don't hand the agent a query language.

A graph and a Cypher prompt turns retrieval into a guessing game. Named traversal tools fix it, and the part everyone gets wrong is the escape hatch. Two of these slides you can click.

O&A Consulting, LLC oandaconsult.com
02 · The thing everyone builds first

“Here's the schema. Write Cypher.”

It demos beautifully. The model knows the syntax, writes a valid query on the first try, and returns rows. Then you put a real question to it and watch what actually happens.

Failure 1

Valid, and empty

A syntactically perfect query against a label that doesn't exist, or a relationship pointing the wrong way. Zero rows, no error, no signal about which.

Failure 2

Schema by guesswork

It can't know that MENTIONS is dense and CO_OCCURS is the one worth walking. That's operator knowledge, and it isn't in the schema.

Failure 3

No sense of “enough”

An empty result and a wrong question look identical from inside the loop, so it retries the same shape with different names until the budget runs out.

Failure 4

Retrieval as a fishing trip

When something finally returns rows, it takes them. Not because they answer the question, but because they're the first thing that wasn't empty.

None of this is a reasoning failure. It's a navigation failure. ↓

03 · The same question, two ways · click it

“Which companies sit between the banking and energy stories?”

with a query language

Press run to watch it work the problem.

  1. MATCH (c:Company)-[:IN_SECTOR]->(s) WHERE s.name IN ['Banking','Energy'] RETURN c
    → 0 rows
    no IN_SECTOR relationship in this graph
  2. CALL db.schema.visualization()
    → returns 14 labels, 31 relationship types
    now it has to pick from 31
  3. MATCH (e:Entity)-[:MENTIONS]-(t:Topic) RETURN e LIMIT 100
    → 100 rows
    MENTIONS is the dense edge — this is everything, ranked by nothing
  4. MATCH (e:Entity)-[r:CO_OCCURS]-(e2) WHERE r.weight > 0.5 RETURN e, e2
    → 0 rows
    weights are z-scored here; 0.5 is far above anything real
  5. … retries with weight > 0.1
    → 4,200 rows
    takes the first 20 and calls it an answer
with traversal tools

Same question. Same model.

  1. find_bridging_entities(cluster_a='banking-risk', cluster_b='energy-supply')
    → 6 entities, ranked by bridge score
    the tool knows CO_OCCURS is the edge and how to normalize it
  2. get_source_documents(entity='…', edge='bridge')
    → 18 source chunks, with ids
    real rows, ready to cite
5 queries · 4 dead ends 2 calls · answered

The model is identical in both columns. Only what it was handed changed.

04 · What replaces it

Tools that encode how to look.

01Signal tools. The questions worth asking, named. find_bridging_entities, find_emerging_cluster. Each one carries the operator knowledge about which edges to walk and what counts as a strong result.
02Navigation tools. Move around without knowing the schema. Pull a cluster's neighbourhood, expand an entity, get the documents behind an edge.
03A generic query tool, kept deliberately. The escape hatch. Skip this and you've built a menu.

The agent still decides where to go. The tools decide what a step means, so a step always lands somewhere real.

This is a smaller change than it sounds. The same traversals happen; the difference is that they're written once by someone who knows the graph, rather than re-derived from a schema dump on every run by something that has never seen the data.

05 · The fair objection · toggle it

“So you've put the agent in a cage.”

This is the right worry, and it's why the generic tool stays in the set. Watch what a curated toolbelt can and can't answer with the escape hatch removed.

  • Which entities bridge two clusters?covered by a signal tool
  • What's the source text behind this edge?covered by a navigation tool
  • Which cluster grew fastest this week?covered by a signal tool
  • Which entities appear in exactly three clusters, none of them energy?nobody wrote a tool for this
  • Are there entities with no edges at all?nobody thought to ask this

With the escape hatch in place, every question is reachable. Two of them only through it.

06 · Where else this shows up

It was never about graphs.

The pattern generalizes to anything an agent has to search that's larger than its context: give it a closed set of retrieval verbs written by someone who knows the terrain, log every call, and keep one way out.

In a knowledge graph

Clusters, entities, edges

Signal tools carry which relationships are worth walking. Source documents come back through the same interface, so a citation resolves to a real row instead of a plausible string.

In agent-trustkit

A failed agent trace

The investigator gets read-only navigation tools over one trace and pulls only the slice each claim needs. Every retrieval is logged, so the evidence path is replayable rather than asserted.

Two different problems. The same answer, arrived at twice. ↓

07 · O&A Consulting

We build retrieval you can follow.

Knowledge-graph ETL shaped for agentic retrieval, and trust evaluation for the agents that use it. Both start from the same idea: a step should land somewhere real, and you should be able to see where it landed.

See Aurum → [email protected]

The short version: How do you evaluate an agent's tool calling?

Don't hand the agent a query language Charlotte, NC · NAICS 541511