A system that has to be able to say "I don't know"

An MRO engineer stands beside an aircraft with a laptop open, looking for a procedure right now. A confident wrong answer there is not a worse answer than no answer. It is a different kind of event.

TL;DR — the gates fail closed, so the failure mode is stop rather than best effort. Then I had to prove the thing still answers, and that is where the numbers disagreed with me. One person, toy scale, synthetic XML.

I inverted the constraint a consumer chatbot is built on. Inverting it creates a second problem, and it took me longer to see that.

A package that fails validation is rejected with the deviation named. This step matches hazard keywords with no warning before it. This reference points at a document that is not in the package. Every finding carries a fix line. None are applied automatically. The machine detects. The human decides.

When the system answers, the answer is welded to its source. Not a footnote — an XPath into the document, plus the verbatim quote that has to be found in the cited chunk for the citation to stand. The contract makes the model cite by chunk id and quote. The document code and the XPath are filled in afterwards by the system, so an accepted citation never rests on a source label the model made up.

A question the corpus cannot answer is refused, and the refusal names which gate fired. In the recorded demo that happens before the model is called at all. The trace says llm_called: false, because nothing cleared the retrieval threshold and no llm span was written.

There is no third outcome. There is one documented exception, and it applies to the fail-closed chain, not to the answer-or-refuse rule. If the graph database is unreachable, the expansion route returns nothing and logs a warning, so retrieval carries on as plain hybrid. The later fact-injection path still aborts rather than quietly running on degraded context. The evaluation path does not inherit the fallback, so a benchmark row is never measured without the route it claims to test.

That was the easy half. A system that refuses everything satisfies every rule I have just described and is useless.

So refusal gets scored like anything else. Twenty golden queries, fixed seed: fourteen of the sixteen answerable ones answered, a false-refusal rate of 0.125, all four traps refused. Every answered row carried citations that survived the verbatim-quote check.

And the scoring overturned something I believed. I had assumed the retrieval threshold was the main defence, the gate that stops a bad question before it costs anything. At this corpus size the answerable and trap score distributions overlap, so that gate is honestly a cost guard. What carries the load is the model's own call on whether the evidence answers the question, and the verbatim quote.

I would not have found that by reasoning about the design. I found it because refusal had a number attached.

Fail-closed is not caution. It is deciding in advance which failure you are willing to ship.