How well does AI grade real student answers?
We tested AI models against 30,000+ examiner-graded student answers, handwriting included, and we're open sourcing the dataset.
AI Exam Grading vs Human Examiners: What 30,000 Marked Student Answers Reveal
19 Jun 2026
Plenty of tools claim their AI can grade exam questions. The harder question is how well it grades real student answers. An AI tutor can only teach well if it marks accurately, and real responses look nothing like the clean model answers these systems are used to.
Real student work is messy. The working is scattered across the page, the first attempt is crossed out, and the final answer is circled almost as an afterthought.
We've just published our first research paper on this, and we're open sourcing a sample of the dataset behind it, so the whole field can improve how AI grades, and in turn how it teaches.
Why grading real answers is the hard part
You can't simulate student responses with AI. There's no reason to train a language model on wrong answers, so real student work is, by definition, missing from the training data. The models have studied the textbook and the mark scheme. They haven't seen what a Year 11 actually writes under time pressure.
That gap is why we built the largest bank of real student answers graded by expert human examiners across English, Maths and the Sciences.
Inside the dataset
We gathered more than 30,000 exam answers from over 900 students, and had every one graded twice, by two qualified examiners. Then we put the same answers through several AI models.
The dataset spans five subjects: English Language, Maths, Biology, Chemistry and Physics, with answers from across the ability range, top to bottom of the class. It's also multimodal. More than 5,200 of the answers are handwritten, which matters most in subjects like Maths, where almost half the answers were written by hand.
How we measured it
Because every answer was graded twice, we could measure how far the two examiners disagreed. That disagreement gives us a range: the band of grades that counts as expert human judgement on a given answer.
We then ran the AI models over the same answers and looked at where they fell against that human range. The pattern was different in every subject.
What we found
In English, AI grading was indistinguishable from human grading. Every reasoning model we tested landed inside the human range, even the smaller ones. That fits what you'd expect from a subjective subject, where examiners often disagree with each other anyway.
Maths and the Sciences were a harder test. Maths had the highest agreement between examiners, which left the narrowest range to hit, and the best models still hit it. They matched human examiners across the board.
The headline is that AI can grade as accurately as expert human examiners. Accuracy climbs sharply once a model is given the right harness, meaning the scaffolding and structure around how it marks.
What this means for teachers and schools
Should you trust AI to grade? Yes, but selectively. Models vary, and so do marking setups and subjects. Without a benchmark like this one, you'd have no way to tell which combination is genuinely reliable and which only looks it.
That's the case for measuring grading against real student work instead of tidy examples. It shows where AI marking can be trusted today, and where a human still needs to stay in the loop.
Open sourcing a sample
This is our first contribution to education research, and we want it to be useful beyond Medly. We're starting with a public subset, so anyone can test AI grading on real student answers with their own models.
You can read the paper here and explore the open dataset here.
Grading is step one
This dataset is only the start. We're building an AI tutor that doesn't just know the curriculum but understands how real students think, where they go wrong, and how to move them forward. Grading is the first step, and the data we're generating now feeds straight into it.
The reason behind it is simple. A grader that works on real student writing, handwriting included, means any student can get examiner-grade feedback instantly, whether or not their family can afford a tutor.
Try Medly AI for free
No credit card required. Start learning for free today.