LLM-as-Judge Done Right

Rubric design, pairwise vs pointwise judging, known judge biases, and calibrating a judge against human labels until you can trust its numbers.

🚀 advanced
⏱️ 150 minutes
👤 Bhanu Pratap Singh

· Agentic AI · 9 min read

📋 Prerequisites

  • Deterministic Graders and Assertions (previous lesson)

🎯 What You'll Learn

  • Decide when an LLM judge is the right grader and when it is not
  • Write analytic rubrics with binary or low-cardinality criteria that judges apply consistently
  • Choose between pointwise, pairwise, and reference-guided judging
  • Measure and mitigate position, verbosity, and self-preference bias
  • Calibrate a judge against human labels using agreement metrics

What Judges Are For

After Lesson 5, your deterministic graders handle outcomes, policies, schemas, and budgets. What’s left is the stuff code can’t check:

  • Is the response accurate relative to what the tools returned?
  • Is it grounded, or does it invent policy?
  • Is it clear and appropriate in tone for an upset customer?
  • Did the agent ask for clarification when the request was genuinely ambiguous?
  • Is the reasoning in a long trajectory sound, even when the outcome was right?

These are judgment calls. An LLM judge can make them at scale, if you design the judge carefully and prove that it agrees with humans. An unvalidated judge is a random number generator with excellent grammar.

Anatomy of a Good Judge

A judge has four parts:

  1. The criterion: one specific property, not “overall quality”
  2. The evidence: exactly what the judge needs to see (the user message, tool outputs, the response), and nothing more
  3. The rubric: an explicit decision procedure with low-cardinality outputs
  4. The output format: structured, with reasoning before the verdict

One Criterion per Judge

“Rate the response 1–10 for quality” is the most common judge prompt and the least useful. The judge silently weighs a dozen things differently on every call. Split it:

CriterionQuestion
Factual accuracyDoes every factual claim match the tool outputs?
GroundednessDoes it state any policy or promise not supported by the KB?
Resolution clarityWould the customer know exactly what happened and what happens next?
ToneIs it appropriate for the customer’s emotional state?

Four narrow judges are more reliable than one broad one, and when one fails you know what failed.

Binary or Low-Cardinality Scales

Judges are much more consistent on pass/fail or 3-point scales than on 1–10. The difference between a 6 and a 7 isn’t something a model (or a human) applies consistently. If you need gradation, use an analytic rubric: several binary checks that you sum.

Criterion: Factual accuracy

Answer each question YES or NO based only on the TOOL OUTPUTS and RESPONSE.
Q1. Does every amount mentioned in the response appear in the tool outputs?
Q2. Does every invoice or date mentioned match the tool outputs?
Q3. If the response says an action was taken, does a tool output confirm it succeeded?
Q4. Does the response avoid claiming facts about the account that no tool returned?

PASS only if all four are YES.

This is far more reproducible than “rate accuracy 1–5,” and the per-question answers double as a diagnostic.

Show Only the Evidence Needed

Give the judge the inputs it needs to decide the criterion, not the entire trace. More context means more distraction and more cost. For factual accuracy: the user message, the tool outputs, and the response. For tone: the user message and the response. The system prompt usually isn’t needed, and including it can bias the judge toward the agent’s own framing.

Reasoning Before Verdict

Ask for reasoning first, then the verdict, in structured output. Verdict-first outputs get rationalized after the fact.

Implementing a Judge Grader

import json
from dataclasses import dataclass

JUDGE_TEMPLATE = """You are grading a customer-support agent's reply for ONE criterion.

## Criterion
{criterion}

## Rubric
{rubric}

## Evidence
<user_message>
{user_message}
</user_message>

<tool_outputs>
{tool_outputs}
</tool_outputs>

<agent_response>
{response}
</agent_response>

{case_notes}

Respond with JSON only:
{{"answers": {{"Q1": "YES|NO", ...}}, "reasoning": "<brief>", "verdict": "PASS|FAIL"}}
"""

@dataclass
class RubricJudge:
    name: str
    criterion: str
    rubric: str
    judge_llm: callable        # (prompt: str) -> str, temperature 0
    include_tools: bool = True

    def grade(self, case, run) -> Score:
        tool_outputs = "\n".join(
            f"{s.name}({json.dumps(s.input)}) -> {json.dumps(s.output)}"
            for s in run.trace.of_kind("tool")
        ) if self.include_tools else "(not shown)"
        notes = case.expected.get("rubric_notes", "")
        prompt = JUDGE_TEMPLATE.format(
            criterion=self.criterion, rubric=self.rubric,
            user_message=case.input, tool_outputs=tool_outputs,
            response=run.response,
            case_notes=f"## Case-specific notes\n{notes}" if notes else "",
        )
        raw = self.judge_llm(prompt)
        try:
            out = json.loads(extract_json(raw))
        except ValueError:
            return Score(self.name, 0.0, False, "judge returned unparseable output",
                         {"raw": raw})
        yes = sum(v == "YES" for v in out.get("answers", {}).values())
        total = max(len(out.get("answers", {})), 1)
        passed = out.get("verdict") == "PASS"
        return Score(self.name, yes / total, passed, out.get("reasoning", ""),
                     {"answers": out.get("answers")})

Note the rubric_notes hook from Lesson 4: case authors can add per-case guidance (“the customer explicitly asked not to be escalated”) without changing the judge.

Pointwise, Pairwise, and Reference-Guided

ModeHow it worksBest for
PointwiseJudge one response against a rubricAbsolute quality gates; monitoring
PairwiseJudge which of two responses is betterComparing two agent versions; subtle quality differences
Reference-guidedJudge against an expert reference answerFactual tasks where an ideal answer exists

Pairwise is more sensitive than pointwise for small differences. Humans and models are both better at “which is better?” than “how good is this?” Use it when evaluating whether a prompt change improved tone or clarity. Report it as a win rate with ties allowed.

Reference-guided judging helps when a correct answer exists but can be phrased many ways. Tell the judge the reference is one acceptable answer, not the answer, or it will penalize valid alternatives.

Known Judge Biases

These are well documented in the research literature and show up in practice. Test for each one.

Position bias. In pairwise mode, judges prefer whichever response appears first (or second, depending on the model). Mitigation: run every pair in both orders. Count a win only if it holds in both; otherwise it’s a tie.

def pairwise(judge, case, run_a, run_b) -> str:
    v1 = judge.compare(case, first=run_a, second=run_b)   # "first" | "second" | "tie"
    v2 = judge.compare(case, first=run_b, second=run_a)
    if v1 == "first" and v2 == "second": return "A"
    if v1 == "second" and v2 == "first": return "B"
    return "tie"

Verbosity bias. Judges favor longer responses, even when the extra length adds nothing. Mitigation: put concision in the rubric explicitly; check the correlation between judge score and response length on your data. If it’s strong and humans don’t share it, the judge is biased.

Self-preference bias. Models tend to rate outputs from their own family higher. Mitigation: where possible, use a judge from a different model family than the agent, or at least validate that the judge doesn’t systematically prefer one version for reasons humans don’t.

Sycophancy toward confident text. Assertive, well-formatted wrong answers get rated higher than hedged correct ones. Mitigation: give the judge the tool outputs so it can check facts, instead of asking it to assess plausibility.

Rubric drift across model versions. Upgrade the judge model, and your scores shift even though the agent didn’t change. Mitigation: pin the judge model version, and treat a judge upgrade like an agent change: recalibrate before adopting.

Calibration: Proving the Judge Works

This is the step most teams skip, and it’s the one that makes the judge’s numbers mean anything.

Step 1: Collect Human Labels

Take 100–200 runs that span the range of quality: clear passes, clear fails, and borderline cases. Have domain experts label them using the same rubric the judge uses. Two labelers per item if you can afford it, so you can measure human-human agreement too.

Step 2: Measure Agreement

from sklearn.metrics import cohen_kappa_score, confusion_matrix

def calibration_report(human: list[bool], judge: list[bool]) -> dict:
    tn, fp, fn, tp = confusion_matrix(human, judge, labels=[False, True]).ravel()
    return {
        "agreement": (tp + tn) / len(human),
        "cohen_kappa": cohen_kappa_score(human, judge),
        "judge_false_pass": fp / (fp + tn) if fp + tn else 0.0,   # judge PASS, human FAIL
        "judge_false_fail": fn / (fn + tp) if fn + tp else 0.0,   # judge FAIL, human PASS
    }

Raw agreement is misleading when most items pass. If 90% of responses are good, a judge that always says PASS gets 90% agreement. Cohen’s kappa corrects for chance. Rough guide:

KappaInterpretation
below 0.4Don’t use this judge for decisions
0.4–0.6Moderate; useful for trends, not gates
0.6–0.8Substantial; usable for most purposes
above 0.8Excellent; comparable to good human-human agreement

Also compare to human-human kappa. If two experts only agree at 0.65, a judge at 0.62 is about as good as you’ll get, and the rubric (not the judge) may need clarifying.

Step 3: Read the Disagreements

The confusion matrix tells you how often the judge is wrong. Reading the disagreements tells you why. Typical findings:

  • The rubric was ambiguous, and humans interpreted it differently from the judge. Fix the rubric.
  • The judge lacked evidence (e.g., couldn’t see a tool output). Fix the evidence.
  • The judge has a systematic bias (verbosity, leniency). Add explicit rubric language or few-shot examples of the boundary.

Iterate the rubric, rerun on the labeled set, and recompute kappa. Hold out a portion of the labels so you don’t overfit the rubric to them.

Step 4: Decide What the Judge Is Allowed to Do

Asymmetric errors matter. For a safety-relevant criterion, a judge with a 10% false-pass rate is dangerous; for tone, it’s fine. Write down, per judge:

  • Is it a gate or weighted?
  • What kappa and false-pass rate did it achieve, on which label set, with which judge model version?
  • When must it be recalibrated? (On any judge-model change, rubric change, or quarterly.)

Cost and Practicality

  • Use a smaller judge where it calibrates well. A small, fast model often matches a large one on narrow binary criteria. Calibration tells you whether it does.
  • Cache judge verdicts keyed on (criterion version, judge model, evidence hash). Rerunning a report shouldn’t re-pay for identical judgments.
  • Run judges only on runs that passed the gates. If the agent refunded the wrong invoice, there’s no point grading its tone.
  • Sample for expensive judges. For monitoring, judging a random 10% of traffic is often enough to detect trends (Lesson 11).

Key Takeaways

  • Use judges only for what code can’t check: accuracy against evidence, groundedness, clarity, tone
  • One criterion per judge; binary or analytic rubrics; reasoning before verdict; only the evidence needed
  • Pairwise is more sensitive for comparisons; always run both orders to cancel position bias
  • Test for position, verbosity, and self-preference bias; pin the judge model version
  • Calibrate against human labels with Cohen’s kappa, read the disagreements, and document what each judge is trusted to decide

Exercise

  1. Write two rubric judges for your agent: factual accuracy (with tool outputs as evidence) and resolution clarity
  2. Label 60 runs yourself (or with a colleague) using the same rubrics
  3. Compute agreement, kappa, false-pass, and false-fail rates for each judge
  4. Read every disagreement, revise the rubric once, and recompute on 20 held-out labels
  5. Record each judge’s calibration results in a JUDGES.md file next to your graders

If either judge’s kappa stays below 0.5, don’t use it as a gate. Make it a weighted signal and keep improving the rubric.

Part of a structured course

Agent Evaluation & Architecture

Design and build a complete Agent Evaluation Framework — traces, golden datasets, deterministic graders, calibrated LLM judges, trajectory and safety evals, reliability metrics, production monitoring, and CI regression gates. Advanced course for AI engineers and architects shipping agents to production.

Lesson 6 of 13 ⏱ 8 weeks advanced Free

Related Tutorials

🚀advanced ⏱️ 300 minutes

Capstone: Build an Agent Evaluation Framework

Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.

Agentic AI6 min read
agent evaluationcapstoneproject +2
🚀advanced ⏱️ 120 minutes

Evals in CI/CD and Regression Gates

Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow.

Agentic AI9 min read
agent evaluationci/cdgithub actions +3
🚀advanced ⏱️ 90 minutes

Deterministic Graders and Assertions

Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight.

Agentic AI8 min read
agent evaluationgradersassertions +2
🚀advanced ⏱️ 120 minutes

Eval-Driven Agent Architecture

Designing an agent so it can be measured — traces, spans, structured events, replayable tool layers, and the seams where evaluators attach.

Agentic AI8 min read
agent evaluationagent architectureobservability +2