LLM judges need their own audit before they can certify agent quality

On 19 October 2026, Google Cloud published three methods for auditing LLM-as-a-judge evaluation systems: repeatability tests, agreement with human experts and ensembles of independent judges. The central warning is that a fluent automated grader can be systematically biased. An agent release gate that depends on such a judge may approve incorrect outputs while producing a reassuring numerical score.

Google names verbosity, position, agreeableness, halo and self-enhancement biases. It notes that swapping the order of candidate answers can reverse a model's preference in some tests, and that polished wording can distract a judge from factual errors. These are known evaluator failure modes rather than a claim that any single deployed judge fails at a fixed rate. The question is whether an organisation has measured its own evaluator.

Test stability before trusting precision

The first audit pillar is internal stability. Submit identical candidate outputs repeatedly and measure the spread of grades. For pairwise evaluations, swap the answer order and compare decisions. Vary irrelevant formatting and answer length while keeping the underlying facts unchanged. A large grade movement on an unchanged answer signals that the rubric or judging configuration needs work.

Record model version, temperature, prompt, examples and grading instructions with every result. Treat consistent but wrong scoring as a separate failure: repeatability is necessary for a useful metric, not evidence that the metric tracks truth.

An evaluator cannot serve as an independent release gate until its own errors are measured against known mistakes.

Human alignment needs difficult negative cases

Google's second pillar requires agreement with domain experts using chance-corrected measures, rather than relying on headline agreement accuracy. A test set dominated by acceptable answers allows a lenient judge to score well by passing almost everything. Teams should deliberately include invalid tool calls, unsupported claims, privacy violations and subtly wrong reasoning.

Build an expert-labelled calibration set, stratify by failure class and report sensitivity to unsafe or incorrect results as well as overall agreement. Resolve disagreements through adjudication, preserving the specific evidence that led an expert to reject an output. A release decision should explain which error classes remain underdetected.

An ensemble is a diagnostic, not a substitute for evidence

The third pillar is an ensemble: independent models judge the same sample, while decomposed prompts assign factual accuracy, safety and tone to different checks. Disagreement becomes an alert for review. Yet correlated training data and shared blind spots can cause two models to agree on the same wrong answer. Deterministic checks of tool execution and source citations still matter.

Production teams should periodically re-label fresh samples, monitor drift in judge disagreement and revalidate the system after any model or rubric change. Tie approvals to task outcomes and actual execution traces rather than the final text alone. This converts evaluation from a convenient scoring layer into a control that can withstand challenge.

Produced through Devence Lab's automated research and publishing workflow, without prepublication human review. Devence Lab retains editorial responsibility. Sources are linked; report factual errors to contact@devencelab.com.

Written by the Devence Lab research team.