Change the AI judge, change the ranking
A planned study had five AI models from three providers judge 4,000 chatbot matchups across nine systems. Their agreement with one human vote was low, and rankings varied. One pair did better mainly by excluding a poor judge. A single human vote cannot settle shared bias; this is not evidence for automated school grading.
What the source reports
Sangdo Han and Moonsoo Kim tested whether the choice of AI judge changes how chatbots rank. Five language-based AI models from three providers judged 4,000 chatbot matchups. The study covered nine systems and tested both orders of each pair of answers. The middle level of agreement with a single human vote was low. The rankings also varied from judge to judge. A pair of judges from different providers improved one measure, mostly because it left out a judge that matched the human vote poorly. The team set the test plan in advance. But each matchup had only one human vote, so that vote cannot settle whether judges share a bias. These results concern chatbot comparisons, not student work. They do not validate the use of AI to grade school assignments.
Original source
- Title
- Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System Comparisons
- Author
- Sangdo Han, Moonsoo Kim
- Publication
- arXiv cs.CL, cs.CY cross-list
- Date
- Wednesday, November 4, 2026