A measurement is a statement about one corpus at one revision

The repo publishes a command for reproducing its headline table. Ten days later a re-verification run went through it. 12 of the 32 numbers came back different.

The code was not the cause.

TL;DR — the sample packages the demo runs on are also the packages the benchmark is scored against, and nothing said that editing one voided the other. I approved the edit, next to a note measuring the drop it caused, and did not connect the two. The project answers questions out of aircraft maintenance manuals — one person, toy scale, synthetic XML.

The retrieval code was the first suspect, since that was what had just changed. Ruling it out took a minute. One of the four search modes never goes near the vector database. Plain keyword matching, same process. It had drifted too, to the same numbers with the day's changes stashed back out.

So it was the material, and the diff was one I had signed off on. Day twelve added figure assets to two sample packages and touched the data modules that point at them, to get illustrations indexed.

And it hid nothing. It re-ran the day four questions with the new chunks in and reported a drop rather than a wash. That measurement is in the repo. What nobody did, me first, was walk two files over to the published table and ask what had happened to it.

Chunk ids here are a hash over the document code, its path, the chunking strategy and a digest of the file's bytes. Edit one module and every chunk id inside it changes. The chunk-id set in the corpus manifest, which is what a run is checked against, stops matching. Every number scored on it now describes a corpus that is no longer in the repo.

There was a guard, watching the wrong seam. It checks that the published tables match the frozen result file. Nothing checked that the frozen file still matches the corpus and the code that produce it.

My first instinct was to re-run everything and publish the new numbers. I am glad I did not. Overwriting a frozen result quietly changes what a published number means, and anyone who cited the old one is now citing something else.

So the table stays exactly as measured, labelled with the corpus it describes, and the rule went in beside it. When the source material changes, the earlier measurement is void. Not approximate. Not stale. Not in need of a refresh. Void. The automatic guard for it was offered, and I deferred it — take the lesson before the tooling.

Then I went looking for the next one. There is a rule here that I re-run any number a red team reports rather than take the implementer's run for it, so I picked another frozen result and ran it.

And this is the part I did not expect. On the fresh run every one of the eighteen decisions matched. Ten of the scores underneath them had moved, and not one decision had. A number can be void and still look right, and looking right was never the test.

A measurement is not a property of your system. It is a statement about the material you fed it, on the day you fed it.