An LLM judge you cannot fully trust, and how we used it anyway

Our entry to the AI City Challenge 2026 produced two candidate answers for every question: one from a 9B reasoning specialist and one from a 27B answer specialist. A judge had to pick one. This post covers what we measured about that judge and how we designed around it. The numbers come from Sections 4.5 and 5.3 of our paper and from the repository README.

Key points

• Under a neutral rubric, the judge picked the last-shown answer on 65 to 79% of items, whoever wrote it.
• Only 51.6% of its verdicts survived an order swap, which is chance level.
• It preferred the answer from its own model family more often than an oracle would.
• A constrained, "winner-last" design turned the judge from a loss (0.4887) into a gain (0.5043).
• Per-question arbitration beats routing tables because the better model changes from question to question.

The set-up

The judge is a frozen, off-the-shelf Qwen 3.6 27B. For each question it receives the video frames, the question and both candidate answers, and it returns which one to submit. It has no fine-tuning and no precomputed routing table. Multiple choice with explanation bypasses the judge, because its answer is copied from the matching multiple-choice item.

We measured the judge on our validation and internal test splits before we trusted it with any decision.

Three measurements

Position bias

With a neutral "pick the better answer" prompt, the judge favoured the last-presented candidate on 65 to 79% of items, regardless of which model wrote it. When we swapped the order, only 51.6% of verdicts stayed the same. A judge that changes its mind when you change the order is reacting to the slot, not the content.

Self-preference

Averaged over both orders, the neutral judge selected the 27B candidate, which shares its model family, on 42.8% of items. The oracle, which always picks the higher-scoring answer, selects it on 37.8%. This matches earlier findings that LLM evaluators favour their own generations, and the position bias reported for LLM-as-a-judge.

Who can judge at all

A 9B sibling model returned the last-presented candidate on every item. The frozen 27B is the smallest family member that showed any sensitivity to content. Neither specialist can judge, because judging a pair that contains your own answer gives up independence.

How we constrained the judge

The paper describes the judge as a conservative verifier, not a free selector. The rules are:

• Winner last: the candidate from the specialist with the higher validation score on that task type is shown last. This aligns the position prior with the expected winner.
• Keep unless contradicted: the last candidate is kept unless the video clearly contradicts it. An override has to be earned by visual evidence.
• Per-task instruction: a short instruction states what separates a correct answer from an incorrect one for that task.
• Pre-filter: a deterministic step settles uncontested items, so the judge only sees real disagreements.

The repository README adds the operational detail. Each item is judged twice with the order swapped. Switching to the 9B answer requires both verdicts to agree, and the switch must then survive a separate refutation pass. Verdicts are cached on disk, so re-runs with changed settings cost little.

What the judge is worth

The neutral judge scores below the 27B alone. It overturns good defaults on position and verbosity more often than content earns a correct override. The difference between the two rubrics, 0.0156, is the value of constraining the judge. The constrained judge recovers roughly a fifth of the gap to the oracle: (0.5043 − 0.4958) / (0.5347 − 0.4958) = 0.22. The rest is the price of guarding against its biases.

Why arbitration, not routing

A routing table assigns each task type to the model that wins it on average. On validation, the 27B wins every free-text task on average, so a static router collapses to "always 27B" and its ceiling is 0.4958. Any gain above that must come from questions where the judge picked the 9B answer.

The headroom sits in flips: items where the better answer comes from the model that loses that task on average. A routing table discards these by construction.

Only video summarisation would be fully served by a routing table. The recoverable score sits in the reasoning-heavy tasks. That is why we arbitrate per question.

Lessons for anyone using a model as a judge

• Measure order sensitivity first. Swap the candidates and count how many verdicts survive.
• Measure family preference against an oracle on labelled data.
• Give the judge a default and make it earn an override, instead of asking for a free choice.
• Do not let a model judge a pair that contains its own answer.
• Report the neutral-judge baseline next to your designed judge, so readers can see what the design is worth.

The full system is in our paper explainer, and the levers that failed are in our negative results post. We apply the same caution to model outputs in products such as AI Search, where every match returns the frame, timestamp and plate as evidence. More work is on the Research page.

Frequently asked questions

Why not fine-tune the judge?

We kept the judge frozen and training-free so that it stays independent of both specialists. Its prompt and pre-filter were tuned on the validation split only.

Does the judge see the video?

Yes. It receives the video frames with the question and both answers, so an override can rest on visual evidence.

What decoding did the judge use?

All inference in the paper, including the judge, uses greedy decoding.