Hi everyone,
I ran the prompt from the notebook exactly as provided and got an unexpected result.
The model correctly computes the actual solution as:
360x + 100,000
while the student’s solution is:
450x + 100,000
Since these are different, I expected:
Is the student's solution the same as actual solution?
No
Student grade:
incorrect
However, the model returned:
Is the student's solution the same as actual solution?
Yes
Student grade:
correct
Has anyone else experienced this? Is this due to model variability, or is there something I should modify in the prompt to make the evaluation more reliable?
Thanks!
Hi @Nayoneeka_Paul,
This is actually the expected behavior, not a bug. That’s exactly what the notebook is trying to demonstrate.
The first prompt in that section is the “naive” version. It asks the model to evaluate the student’s work in one shot, and the model tends to just skim and agree with the student without carefully checking the math. That’s the failure you’re seeing.
If you scroll down in the same notebook, there’s a second, improved prompt right after it. That one asks the model to work out its own solution first, then compare. With that prompt, the model correctly identifies the student’s answer as incorrect.
The whole point of this section (Principle 2, Tactic 2: “Instruct the model to work out its own solution before rushing to a conclusion”) is to show that prompt structure directly affects accuracy:
- First prompt → model rushes, agrees with wrong answer
- Second prompt → model solves independently first, catches the error
Run the next cell in the notebook and you’ll see the model correctly grade it as incorrect.