My ablation disagreed with me

A reranker re-scores results the search has already found. This one cost 123 milliseconds a query, against 6 without it.

It also pushed relevant results out of the top ten.

TL;DR — before the run I approved the rule that would decide whether the reranker had earned its latency. The result came back against it twice. One of those was a bad rule, the other was a real verdict on the cost. The project answers questions out of aircraft maintenance manuals — one person, toy scale, 43 chunks of synthetic XML when these numbers were taken.

Day four was supposed to settle whether the reranker was worth its latency. It cannot invent a candidate, only reorder the chunks the earlier stages found. So the rule looked clean. Recall asks whether the right chunk is in the list at all. Ranking quality asks where in that list it sits. Only the second one is the reranker's business, and if recall moved, the experiment was wired wrong.

Recall at five moved. 0.93 to 0.99. Ranking quality hardly did. MRR and nDCG, the two standard scores for how high the right chunk lands, went 0.8496 to 0.8520 and 0.8829 to 0.8846.

So I went looking for the bug. There wasn't one. Reordering twenty candidates can lift a relevant chunk from rank seven into the top five, and push another from nine down to eleven. Recall only has to hold still at the full depth of the pool. Twenty is not in the table, so the rule was untestable at both reported cutoffs. The note in the repo offered recall at ten as pool-depth proof. Writing this post struck it out.

That cleared the wiring. The cost verdict stood. 123 milliseconds bought 0.0024 of MRR and 0.0017 of nDCG. And recall at ten went the wrong way, 1.00 down to 0.99. Reordering pushed relevant chunks out of the top ten that the pipeline without it kept in.

Then the plainest row in the table. The vector search on its own, no keyword arm and no reranker: best on both ranking scores, recall at ten of 1.00, at less than half the latency. Those p50s are cache-warm, which if anything flatters the reranked row. The reranker stayed on as the default for the question path anyway, and the reason is in numbers. It is the only mode that puts every part-number lookup in the top five, and it takes questions worded differently from the manual from 0.75 to 1.00. Seven queries and twelve. A reason, not a proof. "It is what everyone puts in the pipeline" is not even a reason.

And here is the part I did not see coming. On day eleven the project grew a third route, over the graph of references between manual sections. On questions whose answer spans two or three of them it lifted MRR 0.81 to 0.89. Turn the reranker on and both routes land on identical numbers. The 0.89 goes back to 0.71.

The stage I could not prove was worth its latency erased the one that had just cost me a day.

An experiment earns its cost when it argues with you. This one argued twice. Once about the pipeline, once about the rule I approved for judging it.