AI essay graders still need calibration
Agreement is not validity. A grader can be steady, harsh, and wrong the same way each time. The audit gives educators a plain warning. When a model update moves the score scale, you replaced the rater. You did not simply upgrade it.
What the source reports
Veerendra Kumar Sunkavalli audits AI text model essay graders as raters. The study covers 2,377 essays, 12 judges, four providers, five version contrasts, and public essay sets in two languages. It checks severity, halo, reliability, version shifts, and differential functioning. Judge severity spans 219 points on ENEM’s 0-1000 scale. On ASAP, the panel spread is 15% to 33% of the score range. That compares with a between-trained-human gap near 1%. Judge-human correlations sit between .47 and .56. All five version contrasts shift severity, up to 133 points. Replication gives self-consistency, phi >= .80 at k <= 2, but not human-level accuracy.
Original source
- Title
- LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora
- Author
- arXiv, Veerendra Kumar Sunkavalli
- Publication
- arXiv
- Date
- Sunday, August 30, 2026
Also covered by the other newsletter.