The judge has a favourite seat

Somebody has to decide whether the new prompt is better than the old one, whether this model beats that one, whether the rewrite improved the answer. There is far too much output to read, so the job goes to another language model: show it two answers, ask which is better, count the wins. LLM-as-a-judge. It sits underneath public leaderboards, internal evals, reinforcement learning from AI feedback, and most "we tested it and ours won" claims you will read this year.

We use one ourselves. When this project compares models for its daily news briefing, a judge model reads both briefings and picks. That is the reason this page exists — not because grading with a model is a bad idea, but because we wanted to know what our own grader was actually responding to.

The way out is to stop trying to grade quality and start manufacturing pairs whose correct verdict follows from how they were made. Four kinds, from 24 questions — ordinary explanation questions with no single right answer, which is exactly the situation where people reach for a judge. Four of them are about Ghana, because the failure is global but the examples should be ours.

Then the part that does the work: every pair is judged twice, once as A-then-B and once as B-then-A. Same texts, same judge, same settings — only the order changes. Nothing about the answers has changed, so nothing about the verdict should.

Six judges, five from Anthropic and one from OpenAI.

Before anything else, the judges have to clear a bar. If they cannot detect a real, planted error, then every other number on this page is just noise and the page has no argument — "the grader is arbitrary" would be uninteresting if the grader were arbitrary about everything.

So each answer was paired with a copy of itself in which exactly one factual claim was replaced by a false one. Everything else — length, tone, structure, every other sentence — is identical.

They pass, and comfortably. So the judges are not broken. Keep that in mind for the rest of the page: everything that follows is a failure of a system that demonstrably works when there is something real to find.

Here are two answers to the same question. They came from the same model, with the same settings, in the same minute — two draws from one distribution. There is no better one. Read them and pick anyway.

Across all — equal pairs, this is how often each judge picked whichever answer it happened to read second:

The obvious reading of Section 3 is "these models prefer the answer they read last." I believed that for about an hour, and it is not safe, because of how the prompt is written. The answer shown second is also the one labelled B. A judge that likes the last thing it read and a judge that simply likes replying B produce identical data.

So the same equal pairs were run again with the labels pulled apart from the reading order: the answer presented first was labelled B, and the one presented second was labelled A. If the effect is positional, the judge keeps choosing what it read last. If it was the token all along, the judge keeps saying B.

"AI graders reward waffle" is the other thing everybody says. It is testable the same way: take an answer, and pair it against a rewrite of itself — median —× the length, under instruction to add no new facts, no new examples and no new numbers. Same content, more words. Averaged over both orders, so the seat cannot explain the result.

The standard advice, once someone shows you position bias, is to stop comparing. Score each answer on its own, one to ten, and rank by the scores. No seats, no letters, nothing to prefer. It genuinely removes the problem in Section 3 — you cannot have a position bias when there is only one position.

So the same texts were scored again, alone, one at a time. And the length result inverts.

The best-known result in this area is self-preference: a model asked to grade its own output against someone else's is said to prefer itself. It is easy to test and easy to get wrong, because a model whose answers really are better should win under every judge. The only honest measure is the gap — a model's win rate under its own judgement, minus its win rate under everyone else's, on exactly the same pairs.

None of this means stop using a model as a judge. It means stop reading a single verdict as a measurement.

Always judge both ways round, and count the disagreements. It doubles the cost and it is the only thing here that turns a verdict into evidence. On these equal pairs, swapping and averaging converted —

Offer the tie. Forcing a binary choice manufactures a preference where none exists. When "TIE" was allowed on the same pairs, —

Control length before you believe a win. If the winning answer is systematically longer, you have not learned which is better. Report the word counts alongside the win rate, or compare answers of matched length.

Do not switch to pointwise scoring and call it solved. Section 6 is the reason. If you do score pointwise, check the score against length before trusting the ranking.

Keep a planted-error control in your eval. A handful of pairs where one side is deliberately, checkably wrong tells you whether your judge is still working at all. It is the cheapest instrument on this page and the only one that catches a judge that has quietly stopped reading.

Ten questions on the method and the findings. The explanations are worth reading even when you get it right.