When LLM judges agree, evaluation reliability needs a correlation discount

The setup in the paper is the one most RAG teams already run. A user asks a question, the system retrieves a passage, and several judge models decide whether the passage is relevant. Eight say relevant, two say not relevant. The argument in the blog post, by Krishna Balasubramanian and Sasha Podkopaev, writing about work coauthored with Shiva Kasiviswanathan, is that the important question is not only how many judges agreed but how independently they arrived at that agreement.

Majority vote and weighted majority vote are both built on the same hidden assumption: that judges which get the wrong answer make their errors independently. The authors call that assumption too optimistic for LLM-as-a-judge systems. Two judges can fail together because they interpret the rubric similarly. Several can be prompted with the same examples and inherit the same evaluation bias, and a group of related models can be sensitive to the same phrasing.

The consequence for an eval pipeline is a calibration problem as much as a labeling one. In the authors' words, the vote count makes the evidence look stronger than it really is, and a majority can be less informative than it appears. If the aggregator treats correlated votes as independent, the confidence it reports on contested items carries that inflation forward into whatever gate depends on it.

The method treats the panel as a network. Each judge keeps its own reliability profile, and pairs of judges also get a relationship term: some pairs agree more often than their individual profiles predict, including on shared mistakes, while others provide more complementary perspectives. An Ising model represents that pairwise dependence between binary variables, so the aggregator learns both judge skill and judge similarity.

Training is unsupervised. Each item's true label is a latent variable inferred jointly with the judge parameters. Starting from an initial parameter setting, the algorithm combines an item's votes into a probability that its label is positive, then alternates between updating those soft probabilities and re-estimating judge reliability and pairwise dependence from them. Reference labels are used only afterward, to measure experimental accuracy.

There are two levels of dependence modeling, and the choice matters for anyone with a small eval set:

The evaluation covered three binary tasks: relevance classification for retrieved information, toxicity classification, and summarization assessment. The panel was 10 judge models, all run at temperature zero, which the post explains means there is no randomness in their outputs and the same input always elicits the same output. Baselines were weighted majority vote and uniform majority vote, both conditional-independence aggregators.

With all 10 judges and the maximum available training data per task, the strongest dependence-aware results were 0.912 on relevance (0.820 weighted, 0.804 uniform), 0.792 on toxicity (0.694 and 0.695), and 0.806 on summarization (0.737 and 0.561). Summarization is where the two baselines diverge most, 0.737 against 0.561, and where the dependence-aware model gains most over uniform majority vote. The post does not explain what drives that spread.

The gains were not unconditional. The post states that modeling dependence improved accuracy once the system had enough evaluation items and enough judges to estimate meaningful relationships. It does not say how many items that is for each task, which is the number a team sizing an eval set would want.

The paper does not offer a supervised calibration recipe. It is designed for the unsupervised setting, it estimates the probability that each item's true label is positive, and human reference labels are held back for measurement rather than fitting. What it gives an eval pipeline instead is an aggregate score already adjusted for correlation between judges, plus a learned network the post suggests using during audits to help identify redundant judges and task-specific shared blind spots.

The four habits the authors recommend are concrete enough to put in a review checklist:

A companion Amazon Science post, on why machine learning research agents do not overfit, states the textbook hazard on the human-label side: a held-out set you check, tweak your training procedure against, recheck and iterate on is no longer unseen, because it has become part of your training procedure, and that applies to a validation set, which is reused by design. That post's own finding runs the other way for its subject, reporting that ML models do not overfit benchmarks even after many rounds of iterative improvement, and that a successful agent's strategy squeezed through a bottleneck of as few as 16 tokens lets a fresh agent with no memory reproduce the original performance. It studies ML research agents, not judge calibration sets, so neither source says whether a reused human-labeled calibration set for judges degrades the same way.

Commenters on Hacker News split on exactly the premise. One argued that it is odd to generalize about LLMs at all, since every lab and every model has its own biases and reward functions, and another said the effect depends on which model you pick because each has different weights and training data. Another pushed back on the paper's motivation directly: two agents will almost never hallucinate in the same way regardless of their weights, so a second one with a different context almost entirely eliminates the problem.

The thread supplied its own rebuttal. One commenter reported that no modern LLM they tried could say how many eyes the Magic card Pit Imp has, with paid Gemini, Claude, and GPT all answering two across reasoning levels, and that drawing a red line to split the image produced correct answers. Their conclusion was that failure modes are highly correlated. Another pointed out that LLMs checking the outputs of other LLMs is already standard, and that the bad answers users see are the ones those checks failed on. That matches the post's position: correlated errors are the ones cross-checking does not catch.

Two more comments landed on the operational question. One noted that stakes should set the bar, and that a judge deciding whether a document's formatting is compliant with standards is a different risk profile from one whose error is life-altering. Another described shared blind spots as an under-considered problem in day-to-day work, since models will make assertions without the full picture even when given as much context as possible. Neither source says where the stakes threshold should sit. One commenter offered the shortest framing of the setup: a panel of judges, AI or not, is fundamentally a jury.

All three evaluated tasks are binary, which is what the Ising formulation handles; the post says nothing about graded rubrics, pairwise preference, or multi-class verdicts. The 10 judge models are not named, so there is no way to tell how much of the measured dependence came from shared model families versus shared prompts. There are no latency or cost figures for running the aggregator, and no statement of how many evaluation items the class-dependent model needs before its extra parameters are estimated reliably.

The practical read for teams already collecting LLM-as-a-judge outputs at scale is that those logs contain more than votes. They contain patterns of agreement and disagreement that this method converts into a usable signal, and the learned network is what the post points at for the audit questions: whether similar models are adding independent evidence or reinforcing each other, and whether adding another judge would improve the evaluation or duplicate an existing source of bias.