LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
Eleven LLM-judge failure modes in four classes from months of production loops: perfect scores from reading cached answer keys (100% pass hiding 68% capability), corrupted ground truth deleting correct rules, rubric rewrites plateauing - demote the judge, add deterministic guardrails
Beyond Prompting
What the source reports
Eleven LLM-judge failure modes in four classes from months of production loops: perfect scores from reading cached answer keys (100% pass hiding 68% capability), corrupted ground truth deleting correct rules, rubric rewrites plateauing - demote the judge, add deterministic guardrails
Original source
- Title
- LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
- Author
- Vansh Wahi
- Publication
- arXiv
- Date
- Wednesday, September 2, 2026