We’ve been working on AI systems that evaluate handwritten exam papers, and one thing has surprised me throughout the process.
From the outside, it seems like grading should simply be a matter of giving an LLM the question, the student’s answer, and the marking scheme.
In practice, that has been one of the smaller parts of the problem.
Before a model can even begin grading, it often has to deal with things like:
- handwriting that’s difficult to read
- answers written out of order
- responses continuing across multiple pages
- diagrams mixed with text
- crossed-out work
- ambiguous or incomplete marking schemes
- deciding how much partial credit should be awarded
That made me realize AI exam grading is really a combination of document understanding, OCR, reasoning, and consistent evaluation—not just prompting an LLM.
For those who have worked on educational AI, document AI, or multimodal systems:
What has been the hardest technical challenge you’ve encountered?
I’m especially interested in hearing whether your biggest bottleneck was OCR, reasoning, evaluation consistency, or something completely different.
This challenge is not limited to grading systems or even AI systems in general; it applies to virtually any AI system that has to deal with real-world problems.
The fundamental issue is that an AI system is trained on a particular set of examples or patterns. When the system is deployed in the real world, however, the data it encounters may no longer fit the assumptions represented in its training data. This becomes particularly problematic when something genuinely new emerges—a new technique, a new way of solving a problem, or a new form of communication that was not represented in the training dataset.
For example, an AI system might be trained on thousands of different handwriting styles, or on many different structural approaches to writing and answering questions. It may become very good at recognizing the patterns it has seen during training. However, there will always be cases where someone uses a style, structure, or approach that falls outside those patterns. In such situations, the AI system can struggle or produce an incorrect interpretation.
This is essentially a broader challenge of generalization and distribution shift: the real world does not necessarily follow the same patterns as the data on which an AI system was trained.