LLM-as-Judge: The Reliability Number Everyone Quotes Is the Wrong One
The headline claim is ‘over 80% agreement with humans’. Correct for chance and it drops to moderate. A tour of which judge reliability numbers to trust, which biases still bite, and where a judge is actually legitimate.