The last eval post argued that an LLM-as-judge is a biased instrument you’re reading as a ruler. The natural follow-up: how do you check the ruler? If a model grades your agent and you ship on its scores, then the judge is a load-bearing part of your release process — and an unvalidated judge is a measurement device you’ve never once compared against a known standard. This post is about meta-evaluation: evaluating the thing that evaluates. It builds on both what to actually measure when your agent works (which covers the multi-layer evaluation harness) and how LLM judges are biased (which covers the specific failure modes).

The stakes are specific. A noisy judge just widens your error bars; annoying, survivable. A biased judge is the real problem, because bias doesn’t cancel — it pushes your headline metric in one direction consistently, and every release looks better (or worse) than it is. You can average away noise. You cannot average away a systematic tilt you never measured.

Step 1: you need a gold set, and it has to be small enough to be real

Meta-evaluation requires ground truth the judge never sees during grading: a set of outputs with human labels. The instinct is to make it big. Resist. A gold set of 80 carefully-adjudicated examples beats 800 hastily-labeled ones, because the whole point is that these labels are more trustworthy than the judge — and label quality collapses when you rush volume. Include the hard cases on purpose: near-misses, outputs that are fluent but wrong, the adversarial shapes where you suspect the judge is weak. A gold set of only easy cases certifies a judge that will fail you exactly where it matters.

Adjudicate disagreements between human labelers explicitly and keep the ones that were genuinely ambiguous flagged — you’ll want to know later whether the judge is “wrong” on a case that two humans also split on.

Step 2: measure agreement with the metric that survives class imbalance

Here’s the trap that sinks most judge validations. You compare judge to human, get “92% agreement,” and ship. But if 90% of your outputs are “pass,” a judge that says “pass” to everything scores 90% agreement while being completely useless — it has zero ability to catch the failures you built the eval to catch. Raw agreement (accuracy) is inflated by the majority class and it hides the exact failure you care about.

Use Cohen’s kappa, which corrects for agreement expected by chance:

def cohens_kappa(judge, human):
    """judge, human: parallel lists of categorical labels."""
    from collections import Counter
    n = len(judge)
    po = sum(j == h for j, h in zip(judge, human)) / n   # observed agreement
    jc, hc = Counter(judge), Counter(human)
    pe = sum(jc[k] * hc[k] for k in set(jc) | set(hc)) / (n * n)  # chance agreement
    return (po - pe) / (1 - pe)

# raw agreement can be 0.92 while kappa is 0.31 — the gap IS the story

Rules of thumb: kappa below ~0.4 means your judge is barely better than a coin weighted to the majority class; 0.6–0.8 is usable; above 0.8 is strong. And always report kappa next to raw agreement — the gap between them tells you how much your accuracy number is just the class prior talking. This is the same discipline as measuring the agent itself: the headline number needs an error bar, and for a judge the error bar is how much its agreement is chance. The failures that a silent, wrong result creates in production are exactly why this validation on the judge matters — an invalid judge hands you incorrect confidence in a broken system.

Step 3: test for the biases directly, not just overall agreement

Kappa tells you the judge is accurate enough on average. It won’t tell you the judge is tilted, and a tilt is what actually corrupts your release signal. Probe the known bias directions with targeted perturbations on the gold set:

  • Position bias (for pairwise judges): grade (A, B), then grade (B, A). If the winner changes with order more than a few percent of the time, the judge is scoring position, not quality. Report the flip rate.
  • Verbosity bias: take correct answers and pad them with true-but-irrelevant filler. If scores rise, the judge is rewarding length. This one is insidious because it silently pressures your agent toward longer, more expensive outputs.
  • Self-preference: if the judge and the graded agent share a model family, grade a batch of outputs from a different family too and check whether the judge systematically scores its own family higher.

Each of these is a number you can track, and each is a direction your headline metric is being pushed. A judge with kappa 0.7 and a 15% position-flip rate is not a good judge — it’s an accurate-on-average judge that will hand you a fake win the moment your agent learns to exploit ordering.

Step 4: the judge drifts, so re-validate on a cadence

The failure that gets people who did all of the above correctly: they validate the judge once, at launch, and treat the kappa as permanent. It isn’t. The judge drifts out from under you for reasons that have nothing to do with your gold set — the provider silently updates the model behind the endpoint, you tweak the rubric prompt, or (most commonly) the distribution of what you’re grading shifts as your agent improves and starts producing outputs unlike anything in your original validation. A judge validated on last quarter’s agent outputs may be flying blind on this quarter’s.

Re-run the gold set through the judge on every judge-prompt change and on a calendar cadence regardless, and alert on kappa dropping, not just on the agent’s scores moving. If your agent’s eval scores jump, the first question isn’t “did the agent get better” — it’s “did the judge change,” and you can only answer that if you’re re-validating the judge as its own tracked metric.

What I’d actually do

  1. Build an 80–150 case human-labeled gold set weighted toward hard/adversarial cases. Small and trustworthy beats big and noisy; the labels must out-rank the judge.
  2. Report Cohen’s kappa beside raw agreement. The gap is how much of your “accuracy” is the class prior.
  3. Measure each bias as a direction, not just overall error. Position flip rate, verbosity delta, self-preference gap — a tilt corrupts releases in a way noise doesn’t.
  4. Re-validate on a cadence and alert on kappa, not just agent scores. The judge drifts; a score jump might be the ruler moving, not the thing measured.

An eval you never evaluated is a number with a hidden error bar you’re treating as exact. The judge is infrastructure — instrument it, validate it, and re-validate it — or every release decision downstream inherits a bias you chose not to look at.

Cohen’s kappa and the bias-probe methods are provider-independent measurement techniques; the position/verbosity/self-preference biases are documented across model families (see the MT-Bench / Chatbot Arena work). Kappa thresholds are conventions, not laws — calibrate them to how costly a judge error is in your pipeline.