Case 04 · Natural-language layer over governed data
A governed model will answer almost any question you can express in it. Expressing it requires knowing the model — which puts a person in the loop for every ad-hoc question. During close, that person was me. This is the agent that removed that dependency, first as a prototype and now in production, without giving up auditability.
The reporting estate answered the questions it had been built to answer. Anything outside that — a one-off cut, an unusual comparison, a question raised live in a review — required someone who knew the semantic model to translate it.
The load was concentrated exactly where it was least affordable. Ad-hoc requests peak during close, which is the week the same person is running reconciliation and consolidation.
A language model that computes a financial figure is a liability. A language model that decides which governed measure answers your question is an interface.
The failure mode of this class of tool is well known and severe: a model that does arithmetic will sometimes do it wrong, and will present the wrong result with exactly the same confidence as a right one. In a finance context that is worse than having no tool, because the output looks like the output of the certified system.
The constraint I set was that no figure reaching a user may have been computed by the model. Every number had to come from the same governed engine that produces the board pack, by the same path, or the agent had failed regardless of how well it demonstrated.
The architecture separates routing from computation. The model's only job is to resolve a plain language question into three things: which governed measure, which dimension, which filter. Its vocabulary is the existing semantic layer — it cannot invent a measure that does not exist, because it is choosing from a list rather than writing arithmetic.
That resolution is handed to the governed query engine, which computes the figure against certified data exactly as any dashboard would.
The answer is returned with the resolution plan attached — the measure, the dimension, the filter, stated plainly. The user does not have to trust the routing, because they can read it. And where the question cannot be resolved unambiguously, the system asks a clarifying question rather than picking the most likely interpretation.
Every one of these had a defensible alternative. The trade-off is stated because the trade-off is the point.
This is the whole design. It bounds the blast radius of a model error to a mis-routed question — visible, recoverable — rather than a wrong number presented as certified.
Cost: questions the semantic layer cannot express simply cannot be answered, rather than being approximated.
Showing the resolution — measure, dimension, filter — turns an opaque answer into a checkable one. In testing this was what moved finance users from interested to willing.
Cost: more interface to design, and it exposes the semantic model's naming to people who had never seen it.
Ambiguous questions get a clarifying question back. The alternative — choosing the most probable reading — is precisely how a plausible wrong answer enters a review meeting.
Cost: it feels less impressive in a demonstration, which is a real cost when you are still trying to win backing for the work.
Rather than building a parallel definition set for the agent, I pointed it at the governed measures already in production. The agent inherits governance instead of duplicating it.
Cost: the agent is only as expressive as the semantic layer, so its limits are the model's limits.
I optimised the first version for demonstration rather than for the boring questions people actually ask. It handled an impressive multi-part comparison and stumbled on what was margin last month, because I had not spent enough time on the resolution of relative dates. Frequency beats sophistication in what you harden first.
I would also instrument the refusals from the start. The questions the system could not resolve turned out to be the most valuable output of the whole first version — they are a direct, unfiltered list of what the semantic layer is missing. I was discarding that signal for the first several weeks.