A new AI grader may change the scores
An AI tool can keep the same scoring rules and still score differently after an update. Zhang and Yang checked three version changes on a fixed set of research proposals. One new version gave 93% of proposals the same score. That does not show how it would grade students.
What the source reports
Guijia Zhang and Harry Yang compare three version changes in an AI tool used to score research proposals. They kept the proposals fixed to check whether the scores still meant the same thing. The authors report changes in how the tool rated severity, criteria and solutions. In one transition, 93% of proposals received one score value. That makes it harder to tell different proposals apart. The result is about this proposal set and these tool versions. The researchers did not test student submissions. The AI Agency digest suggests checking a new version against the same examples before reading a trend as a real change. That is advice for a local test, not a validated district grading method.
Original source
- Title
- LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard
- Author
- Guijia Zhang, Harry Yang
- Publication
- arXiv cs.CY
- Date
- Tuesday, October 27, 2026