I had Claude write nearly all of the eval harness I use to decide whether changes to a production RAG app actually help. I made the design calls and reviewed what came back, which is how I build most things now. One of those design calls is the whole reason I caught the grader it wrote marking correct answers wrong.
The call was about how a grade gets recorded. Two things go into it: a judge model, which reads the answer and writes down a verdict plus a one-sentence explanation in its own words, and my code, which post-processes that before storing it. Both end up in the stored record. So when a rule in my post-processing misfired and forced a correct answer to wrong, it rewrote the verdict but had no way to rewrite the judge’s sentence, which still said the answer covered everything it needed to. The record contradicted itself, and a check comparing those two fields is what found it.
The harness exists because an afternoon of agent time gets you a working retrieval feature. Akaya is document Q&A over a policy corpus, in pilot at Portland State. It’s a client’s product, not mine. I contract on it. It had a dozen knobs and a plausible story for each: reranker, query rewrite, multi-hop, wider chunk overlap, metadata filters. I could build any of them in a day and I had no way to tell which of them did anything.
What I rejected
Every configuration I’ve tested is written up in a ledger with the runs behind it. Four of the losses:
Reranker, bge-reranker-base. Gained 3 answers on a 19-question pilot set. 34.8s per query against a 7.7s baseline. REJECTED on latency. MiniLM-L6 shipped instead at 12.5s for 2 of those 3 [measured, the flag rollout table]. I traded an answer for 22 seconds and I’d do it again.
Query rewrite. Recovered nothing the reranker hadn’t already, and cost one: a question that had been answering fine came back with no sources, because the wider merged candidate pool pushed its chunk out. REVERTED, net negative.
A multi-aspect specifics-retrieval pass. A deployment taking no live traffic, tested against the live one, two flags apart, three rounds each arm. Stable wins: none. Stable regressions: one. It failed what I started calling the two-sided bar: a feature has to show a gain AND create no regression of its own.
A prediction of mine. I expected an answer-concision change to fix the single stable failure I most wanted fixed. It got worse on both judges. FALSIFIED.
I’ve also gone back and marked up my own earlier entries. Several recorded wins are flagged as possibly-noise now that I know how wide the run-to-run spread is, and an old baseline is retired as stale.
Every one of those was a retrieval feature
Retrieval was not the problem. The bottleneck was answer generation and the test set, which is not where I was looking and not where the interesting work appeared to be.
The biggest single jump I have recorded came from a prompt rule telling the model that a method or a reference counts as a valid answer [measured, the rollout ledger]. I found it after hours on retrieval and reranking for a question whose document the system could already pull back. The document was in the candidate pool; the answer model was refusing to use it because there was no literal figure in it to quote. That number predates the three-rounds discipline below, so it’s a single-run figure I’d want to re-measure before leaning on it.
The other recurring finding is that most apparent failures were the test being wrong. A reference answer carrying a figure that didn’t appear in the document it cited. An expectation stricter than what the source page actually says. Both score as the app being wrong.
Why I trust a model to grade this at all
Retrieval is the easy half to score. Compare returned document ids against an oracle of expected ids, recall@k and MRR, no model in the loop, cheap enough to run on every change and wired to a weekly CI job against the deployed service. Answer quality is the contentious half: the judge model labels each answer correct, partial or wrong, and the headline score is (correct + 0.5 * partial) / N.
The standard objection is that the grader is itself a model. I think that objection is overblown. Across the ledger’s runs the grader being a model has cost me nothing I can point to. The grader being opaque cost me real time.
So each question carries named must_have, nice_to_have and wrong_if items, authored per question, and the judge returns its verdict, which items it covered, which it missed, which wrong-if triggers fired, and a one-sentence reason in its own words. A second judge scores the same answer against a short reference answer, and where the two disagree the case goes to a human queue instead of to whichever number I like better.
The bug was in the post-processing between the judge’s JSON and the stored verdict, not in the judging. When the app genuinely can’t answer, it says so, and a grade of “wrong” is the right call. My code decided whether that had happened by pattern-matching the giveaway phrasing anywhere in the answer text. So an answer that resolved the question and then honestly noted it couldn’t find support for one additional requirement got force-marked wrong, with every must-have moved to the missing list. In aggregate that reads as the app failing, and I spent time tuning retrieval against it. The comment that replaced that check names the specific questions it explains [measured, rubric_judge.js].
The fix stops guessing from prose and reads the flag the app itself sets when it bails, with regression tests. The general version: store the model’s own statement next to the verdict your code derives from it, over a vocabulary specific enough to check. “All must_have items are covered” is a falsifiable sentence only because must-have items are an enumerable list. My post-processing rewrote every field it knew about, and the one field it didn’t know about is what gave it away.
Agents write the questions, code decides which ones count
Question authoring splits the same way, for the reason I covered a while back: a generator grades its own output generously, and standing up a separate skeptical checker is more tractable than making the generator self-critical. Agents draft across four passes that write, harden, challenge and curate. Then deterministic code decides admission: schema validation, confirming every quoted span actually exists in the source document, dedup, difficulty quotas. Then a subject-matter expert passes over what survives. 60 cases, 20 each at easy, medium and hard.
One run is not a measurement
Early on, 4 of 19 cases flipped between answering and bailing across runs of near-identical configurations [measured, the rollout ledger]. The answer model makes borderline answer-versus-bail calls that vary run to run, which puts single-run swings of an answer or two inside the noise floor. Most of what I was chasing was an answer or two.
Everything runs three times now, written r1 / r2 / r3 → mean with the spread left visible. The corpus is pinned to a frozen snapshot, the judge is pinned to a dated model snapshot rather than a floating alias, and the seed is fixed, which on a hosted API is best-effort and is exactly why the answer is three rounds instead of one careful one. If a judge call fails past its retries the run exits and says the aggregate is polluted, because a partial score gets written down and then cited later as though it were a real one.
Two judges, three rounds, sixty questions: 360 judge calls per configuration, on top of 180 answer generations [derived, from the run protocol].
Swapping the judge model moved the headline 4 points
That was on identical saved answers. No system change, no new runs, pure re-scoring. The ledger entry says “measurement correction, NOT system improvement” in its own headline, because in a month I’d have read that line on a trend chart as progress. The secondary judge under the new model came out about 13 points more lenient on those same answers [measured, the rollout ledger], and it earns that by fixing one artifact while waving through a different failure class, so its levels aren’t comparable across judge models rather than offset by a constant. I stopped treating its absolute score as comparable and read its per-question verdicts instead.
Where it still can’t help me
The first one is the most serious.
Dual-judge agreement improved sharply after I tuned the judges, and stable disagreements dropped from 11 to 3 [measured, the rollout ledger]. Agreement is two models concurring, not either of them being right, and tuning two graders until they agree more is precisely the procedure that raises concordance without moving correctness. There is a partial anchor under it: I hand-adjudicated every disagreement case and spot-checked a random sample of the cases where both judges agreed. That’s me, and I’m not the domain expert. Expert sign-off is still outstanding. My most impressive number is the one with the least underneath it.
The retrieval oracle scores on a document axis. Right document retrieved, wrong passage inside it used, scores as retrieval success, so a whole class of answer failure is invisible to the retrieval score by construction. The CI gate has a matching hole: it fails only on NEW misses, with the known baseline misses allowlisted, so the cases most likely to be broken are the ones it can’t fail on. The ledger says to tighten that list once a pending parser fix lands. That hasn’t happened.
Latency killed the better reranker and appears nowhere in that score, so a change that quietly costs seconds passes clean. And for a stretch the harness’s own tests weren’t running anywhere but my machine: the eval workflow only existed on the feature branch, so a test that couldn’t even parse on Windows (a shebang plus a CRLF checkout leaving a stray \r) sat unnoticed until I ran the suite locally.
I’d build it the same way again
Including the parts that look like over-engineering. Every mechanism in here exists because something upstream of it lied to me: a regex in agent-written code, a reference answer with a figure its own source didn’t contain, a judge model swapped underneath a trend line, features that felt like gains and weren’t. None of them announced themselves.