Your LLM-as-judge is lying to you

In the last post I argued that for open-ended tasks your success predicate is often another model call — an LLM grading whether an answer satisfies a rubric — and I flagged that this grader has failure modes of its own. This is that post. The short version: an LLM-as-judge is the only thing that scales for subjective quality, and it is a biased instrument you are reading as a ruler. It will hand you a confident 8/10 that moves with the length of the answer, the order you showed the options, and whether the text came from its own model family. If you don’t measure those biases, your eval number has an error bar you can’t see, and you’ll ship on it anyway.

Here’s what actually goes wrong with model graders, why the obvious sanity check (agreement with a human) hides most of it, and how to build a judge you can defend — with code.

Why we reach for a model judge at all

For a code agent you have a real predicate: the tests pass or they don’t. For “is this summary faithful,” “is this answer helpful,” “did the support reply resolve the ticket” — there’s no assert. Human grading works and it’s the gold standard, but it doesn’t scale past a few dozen cases and it’s the first thing to get skipped when you need to grade 500 outputs on every release.

So you reach for a model. Hand it the task, the answer, and a rubric; ask for a score or a pairwise winner. It’s fast, cheap relative to a human, and consistent in the narrow sense that it’ll give you the same style of answer every time. The problem is that “consistent” is not “correct,” and the ways it’s consistently wrong are specific and well-documented.

The biases that actually move the score

These aren’t hypotheticals. The MT-Bench / Chatbot Arena work (Zheng et al., 2023) that popularized LLM-as-judge measured several of these directly, and they show up in practice constantly.

Position bias. In a pairwise comparison — “which answer is better, A or B” — judges systematically favor one position, usually the first. Swap A and B and a nontrivial fraction of verdicts flip. If your eval always puts the new model’s output in slot A, you’ve baked a tailwind into every comparison.
Verbosity / length bias. Longer answers score higher, roughly independent of whether the extra words add anything. A judge reads thoroughness into length. This one is insidious because it creates a training-like gradient: optimize against a verbose-biased judge and your agent learns to pad.
Self-preference. A judge rates outputs from its own model family more generously. Grade Claude’s output with a Claude judge, or GPT with a GPT judge, and you get a quiet home-field advantage. If you’re comparing two vendors and grading with one of them, the grader is not neutral.
Leniency and the confident-wrong problem. Judges skew high and agreeable. Worse, a judge will hand you a fluent, well-structured 7/10 on an answer that is confidently and specifically wrong, because it’s grading fluency and surface plausibility, not truth — the exact failure that makes agents dangerous in the first place, now sitting inside your measurement instrument.
Format and style bias. Markdown structure, a confident tone, hedging language, the presence of a numbered list — all nudge scores independent of content. A judge partly grades register.

None of these mean “don’t use a judge.” They mean the judge is a sensor with a known bias profile, and you don’t trust a sensor you haven’t calibrated.

The sanity check that isn’t: raw agreement

The obvious way to validate a judge is to hand-grade a sample and see how often the judge agrees with you. People compute that, get “84% agreement,” and feel fine. That number is usually a lie of a specific kind: it’s inflated by class imbalance.

Say you’re grading pass/fail and 80% of outputs genuinely pass. A judge that just says “pass” every single time — a broken judge that isn’t reading anything — agrees with you 80% of the time. Your real judge scoring 84% is barely above the “always pass” floor, and raw agreement can’t see that, because it gives full credit for agreement that chance alone would produce.

The fix is Cohen’s kappa, which measures agreement above chance. It’s the number you should be reporting instead of accuracy.

def cohens_kappa(judge: list[str], human: list[str]) -> float:
    """Agreement above chance for two raters over the same items.
    Labels are categorical (e.g. 'pass'/'fail', or '1'..'5')."""
    n = len(judge)
    assert n == len(human) and n > 0
    labels = set(judge) | set(human)

    # observed agreement
    po = sum(j == h for j, h in zip(judge, human)) / n

    # expected agreement by chance, from each rater's marginal rates
    pe = 0.0
    for lab in labels:
        p_judge = sum(x == lab for x in judge) / n
        p_human = sum(x == lab for x in human) / n
        pe += p_judge * p_human

    return (po - pe) / (1 - pe) if pe < 1 else 1.0

Now the worked example bites. Start with the broken judge: it says “pass” on all 100 items and agrees with you on 80 — 80% accuracy — and its kappa is exactly 0.0, because once you subtract the agreement chance alone would produce, nothing is left. Now take a real judge that agrees with you on 84. With both of you passing about 80% of items, chance agreement is pe = 0.8² + 0.2² = 0.68, so kappa is (0.84 − 0.68) / (1 − 0.68) = 0.50 — only moderate on the usual bands (<0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial), and nowhere near the “substantial” you’d want anchoring a ship decision. The 84% that felt like a green light is a hair-and-a-half above a judge that isn’t reading anything. Report kappa and you see that; report accuracy and you don’t.

The practical rule: keep a small human-graded calibration set — a few dozen items you’ve scored yourself, refreshed occasionally — and every time you change the judge (model, prompt, rubric), recompute kappa against it. A judge you haven’t checked against human labels isn’t a measurement, it’s a vibe with a decimal point.

Hardening the judge: mitigations that actually move kappa

Validating tells you the judge is biased. These reduce the bias. In rough order of payoff:

Swap positions and require agreement. The single highest-leverage fix for pairwise grading: run every comparison twice, A-then-B and B-then-A, and only count a win if the verdict survives the swap. Disagreement between the two orders is the position bias, made visible and demoted to a tie.

def pairwise_verdict(judge_call, task, ans_a, ans_b) -> str:
    """Returns 'a', 'b', or 'tie'. Kills position bias by voting both orders."""
    v1 = judge_call(task, first=ans_a, second=ans_b)   # -> 'first' | 'second'
    v2 = judge_call(task, first=ans_b, second=ans_a)   # order swapped

    # Map each verdict back to the actual answer it picked.
    pick1 = "a" if v1 == "first" else "b"
    pick2 = "b" if v2 == "first" else "a"

    return pick1 if pick1 == pick2 else "tie"

The ties aren’t noise to be minimized — they’re honesty. A comparison too close for the judge to call the same way in both orders is a comparison you shouldn’t be reporting as a win.

Prefer pairwise over absolute scores. “Is A better than B” is a far more stable judgment for a model than “score A from 1 to 10.” Absolute scores drift between runs and cluster meaninglessly around 7–8; relative preferences are more reliable and are all you need for a regression test (“did the new version win against the old on the golden set”).

Force evidence, not just a verdict. Make the judge quote the specific span that justifies its call before it scores. “This answer is wrong because it states the deadline is Friday; the ticket says Monday” is checkable and disciplines the judge away from grading vibes. A bare score is unfalsifiable; a score with a required citation can be audited — and you can spot-check whether the quoted evidence actually supports the verdict.

Grade with a different family than the one you’re testing. Neutralize self-preference by not letting a model grade its own family in a vendor comparison. If you’re choosing between two providers, judge with a third, or at minimum run it both ways and look for the gap.

Control for length. If you suspect verbosity bias in absolute scoring, check whether score correlates with answer length across your set. If it does, either truncate to comparable lengths before grading or add an explicit rubric line (“do not reward length; a correct one-sentence answer beats a padded paragraph”) — and then re-measure, because a rubric instruction is a request, not a guarantee.

Pin the rubric and the model version. The judge is part of your test harness, so a silent model-version bump under it is a silent change to your ruler. Pin the model, version the rubric, and when either changes, treat it as a change that invalidates historical scores until you re-run the calibration set.

A judge you can actually defend

Putting it together, a defensible judge for regression testing looks like this: pairwise, position-swapped, evidence-required, run against a golden set, and periodically checked against human labels with kappa.

from collections import Counter

def judge_suite(cases, judge_call, k_repeats: int = 1) -> dict:
    """cases: list of (task, candidate_answer, baseline_answer).
    Returns win/loss/tie of candidate vs baseline, swap-verified."""
    tally = Counter()
    for task, cand, base in cases:
        # optional: repeat and take majority to damp sampling noise
        verdicts = [pairwise_verdict(judge_call, task, cand, base)
                    for _ in range(k_repeats)]
        v = Counter(verdicts).most_common(1)[0][0]
        tally[{"a": "candidate", "b": "baseline", "tie": "tie"}[v]] += 1

    decided = tally["candidate"] + tally["baseline"]
    return {
        "candidate_wins": tally["candidate"],
        "baseline_wins": tally["baseline"],
        "ties": tally["tie"],
        # win rate over *decided* comparisons; ties are abstentions, not wins
        "win_rate": tally["candidate"] / decided if decided else None,
        "tie_fraction": tally["tie"] / sum(tally.values()),
    }

Two numbers here earn their keep. win_rate over decided comparisons is your regression signal — it should move when the agent genuinely gets better or worse and stay put on a no-op change. tie_fraction is your bias smoke detector: if a large share of comparisons can’t survive a position swap, your judge isn’t discriminating and you should not be drawing conclusions from its verdicts, however confident each one sounds.

What I’d do

Report Cohen’s kappa, not accuracy. Raw agreement is inflated by class imbalance; kappa measures agreement above chance. A judge you haven’t scored against human labels is not a measurement.
Keep a small human-graded calibration set and recompute kappa every time you change the judge’s model, prompt, or rubric. Changing the judge changes the ruler.
Grade pairwise, and swap positions. Only count a win that survives A/B and B/A. Treat verdicts that flip as ties — they’re the position bias made visible.
Require the judge to quote its evidence before it scores. A citable verdict can be audited; a bare number can’t.
Don’t let a model grade its own family in a vendor comparison, and check whether score tracks answer length. Neutralize self-preference and verbosity before you trust the ranking.
Pin the judge model and version the rubric. A silent version bump under your judge is a silent change to every score you’ve ever reported.

An LLM-as-judge is genuinely useful — it’s the only way to measure subjective quality at the scale evals need. But it is a biased sensor, and the whole discipline is refusing to read a biased sensor as ground truth. Measure the bias, harden against it, and check the grader against a human before you let it anchor a ship decision. Otherwise the most confident number in your eval report is the one lying to you most fluently.

Why we reach for a model judge at all#

The biases that actually move the score#

The sanity check that isn’t: raw agreement#

Hardening the judge: mitigations that actually move kappa#

A judge you can actually defend#

What I’d do#