An AI score can stay steady while its decisions move
Eight AI setups labeled 1,200 UK parliamentary questions. Their overall accuracy was similar, but 24.3% of questions changed label across repeated runs with the same setup. The paper also found wide differences in which cases went to human review. This was one UK task, not a school test.
What the source reports
Louise Yorke studied eight GPT-4o setups that labeled 1,200 UK parliamentary questions. Overall accuracy was close across setups. Yet 24.3% of questions changed label across repeats with the same setup. The share sent for human review also varied widely. The paper compares 37 UK and EU policy documents and argues that checks should cover stable results, shared errors, and what happens to each decision. The numbers come from one public-service sorting task. They do not measure a district assessment system. A steady average score alone cannot show whether the same person gets the same decision twice.
Original source
- Title
- Beyond accuracy: what AI assurance needs to establish in public services
- Author
- Louise Yorke
- Publication
- arXiv cs.CY
- Date
- Thursday, November 5, 2026