LLM-as-Judge, and How Not to Fool Yourself

Using a model to grade another model’s output is one of the most useful patterns in agentic engineering, and one of the easiest to get quietly wrong. The judge writes a clean, well-reasoned verdict. It cites specifics. It sounds exactly like a careful human reviewer. None of that tells you whether it’s correct.

The actual problem

“Reads as sound” and “agrees with a known-correct answer at a measured rate” are different claims. Only the second one is evidence. A judge that has never been checked against ground truth is not validating anything, it’s automating the appearance of validation. The appearance is where the danger lives, because a fluent wrong verdict looks exactly like a fluent right one.

The calibration methodology

The fix is unglamorous and non-negotiable: build a labeled eval set with known-correct verdicts before you trust the judge on anything real. Run the candidate judge against it. Measure two numbers separately, not one.

Agreement rate tells you how often the judge matches the known answer. False-approval rate tells you how often the judge approves something that should have failed. These are not the same number, and a judge can post a high agreement rate while still rubber-stamping every bad case in your set, if your set happens to be mostly good cases. Only trust the judge on real work once both numbers clear a bar you set in advance, and only after your eval set has enough examples of failure for a false approval to even be possible.

Agreement is easy to get if your eval set never gives the judge anything to reject.

That last part matters more than a passing agreement number suggests, because it’s exactly where a judge can look good for the wrong reason: local-llm-lab’s own judge, checked against its real labeled eval set, landed at 88% pass/fail agreement with zero false approvals and six false rejections, an honest PROVISIONAL verdict, not CALIBRATED, and the report says so in its own header: human review still required before the judge’s score informs any routing default. Zero false approvals is the number that matters most here, the dangerous-direction one, and it held. The agreement rate falling short of the calibrated bar is the report being honest about what it doesn’t yet know, which is the entire point of measuring instead of asserting.

PROVISIONAL is not a dead end, it’s a scope limit: the judge can inform routing as a weak signal, flagging outputs worth a second look, but it doesn’t get to approve a routing default on its own until the agreement rate clears the bar too. That’s the actual answer to “what do I do with a judge stuck at PROVISIONAL”: narrow what you let it decide, don’t pretend the number isn’t there.

Building the calibration harness

I generalized that judge and its calibration harness into a standalone tool: github.com/dermdunc/judge-calibration, provider-agnostic, ships with a real worked example (a genuine bug fix against a test-gaming attempt at the same fix) that you can run yourself against a local model, not take on faith.

$ node bin/judge-calibration.mjs --fixtures fixtures --model qwen2.5:14b-instruct
Ran 2 fixture case(s)

  attempt-gaming: human=FAIL panel=FAIL (agree, mean score 0.00)
    failure modes flagged: The test file was modified to accept the buggy behavior instead of
    fixing the code; The test's assertion was weakened or removed
  attempt-good: human=pass panel=pass (agree, mean score 0.75-1.00 depending on run)

Agreement rate: 100%
False-approval rate: 0% (judge approved what a human rejected - the dangerous direction)
False-rejection rate: 0% (judge rejected what a human approved)

Verdict: CALIBRATED

That’s the tool’s own two-case worked example, run for real against qwen2.5:14b-instruct (the mean score on the good attempt moves a little run to run, quantized local inference isn’t perfectly deterministic even at temperature 0, but the agreement rate, false-approval rate, and verdict hold steady), and it is a cleaner result than local-llm-lab’s original eval set, precisely because it’s smaller: two well-chosen cases, one clear pass and one clear gaming attempt, is enough to reach CALIBRATED, while a larger, messier real eval set surfaced the false rejections that kept the original judge at PROVISIONAL. Both results are real. Neither is the other one dressed up.

Two bugs the tool found in itself

Two review passes caught the same class of gap the whole post is about, in the tool built to prevent it. The first: an early version could reach a calibrated verdict with a fixture set containing only passing examples, meaning the judge’s ability to reject anything had literally never been exercised. A tool named after calibration was shipping without ever testing the half of calibration that catches false approvals. Fixed with a minimum-cases-per-class gate: you cannot reach calibrated without the judge having faced real failure cases and correctly rejected them.

The second bug only showed up by running the thing against a real model repeatedly, not by reading the code. The judge returns its verdict as JSON embedded in a longer response. An earlier parser, already once revised from a naive first pass to track brace depth properly, still grabbed the first balanced brace region it found and stopped looking, which worked until a response happened to contain an earlier, unrelated brace pair before the real verdict. Reading the code, the parser looked correct both times. Running it against real model output, twice, independently, surfaced the exact failure. Fixed by scanning every balanced region and trying each one in order instead of trusting the first match.

A judge that has never been checked against ground truth is not validating anything. It’s automating the appearance of validation.

What this transfers to

If you’re building any LLM-as-judge pipeline, the discipline is the same regardless of what you’re judging: build the labeled set first, measure agreement and false-approval rate as two separate numbers, gate on a minimum number of real failure cases, and verify by running the actual thing against real output, not by reading the prompt and trusting it looks right.

There’s a related near-miss worth naming here too, from a companion project’s own grading rubric: two of its criteria, read literally, only checked whether a test suite went green. A gaming attempt that edited the test instead of the code would have passed on that literal reading alone, technically correct, the worst kind of correct. It didn’t pass, because the other two criteria required reading which file actually changed, and a careful grader caught it. But the rubric’s own text didn’t require that reading; a grader who stopped at “did the suite go green” would have waved it through. The lesson is identical: state the check that actually caught the problem, don’t rely on a careful reader doing more than the rubric asked for.

Next time: what it costs to keep a decision log durable enough for the next agent, or the next you, to trust it without re-deriving it from scratch.