Back to the series

Act 2 · The system refuses · 06

The runtime decides what the model may read, and what it may write

2998 characters#AIEngineering#LLM#AppSec#AgentSafety
Evidence card: four guards, what each claimed, what the red team did to it, and the two items still open.
Evidence card

Below is the post exactly as published on LinkedIn, unchanged.

The repair agent here can rewrite an aircraft maintenance document. On the normal path, not without a human approving that patch.

The interesting part is where that rule lives. It is not in the prompt.

TL;DR — clearance is enforced inside retrieval, and the CLI writes approved patches to the source XML. Then the red team found the holes were in the fence. One person, toy scale, synthetic XML.

The first version had the read filter in the wrong place, and it looked right to me. Retrieve the candidates, drop the ones this caller is not cleared for, return the rest.

A red team pointed out what that misses. Another model reads this code, never the one that wrote it. The candidate set is what the reranker scores and what the model reasons over. By the time you filter the returned rows, a chunk the caller may not see has already shaped the answer.

So the filter moved inside the retrieval call. Offline, the corpus is built without inadmissible chunks, before any index exists. On the live path the constraint sits inside the query itself, alongside the similarity search, so those chunks are never candidates. It also fails closed on ambiguity. An unlabelled chunk is not admitted while a clearance is enforced, because treating unlabelled as unclassified would make the gate fail open on the exact input it exists to catch.

The write side came next, on the same principle. Dry-run is the default. Repair edits stay on temporary copies. Apply mode asks for approval one patch at a time, then writes the approved set. Some repair classes stay dry-run-only even there — XML syntax and structural repairs, and out-of-domain document fixes. Tools run in a jailed workspace on a temporary copy, with whitelisted commands and imports, and a timeout.

The review came back DO_NOT_MERGE, and almost nothing in it was about model quality. Most of it was about the fence. The jail blocked open and socket, and allowed pathlib — so any file the process could read was one allowed import away. The shell whitelist checked the command and never the arguments, so a whitelisted binary plus an absolute path walked out, and a whitelisted binary plus a URL walked out to the network. A symlink inside a source package was followed into the jail. The apply gate read the risk tier off the patch instead of recomputing it.

Every one of those is a guard failing at the exact thing it advertised.

Those four are fixed. Tests pin the import, argument, symlink and stray-backup cases. The forged-tier one still has no test that would catch it.

Two limits are open. This is scoping, not authentication — there is no identity model, so a caller states its own clearance and the runtime believes it. And the approval gate lives in the CLI around the write primitive rather than inside it, so calling that function directly skips the human.

The review that checked this post is the one that found the second.

Separate permitted to act from able to act. Then have someone else attack the thing you put it in.

What a reader takes away

Separate permitted to act from able to act, and enforce it where the model cannot argue with it — inside the query on the read path, behind an approval gate on the write path.

Check it yourself

Each pointer opens a real file on the LearnArken repository’s main branch.