L6-Agents - Quite many errors when solving maths problems

Short Course Q&A LangChain for LLM Application Development
1

IT appears that the chances of getting an error when using “llm-maths” is quite high.

I tried serveral Grade 12 maths questions and got quite a few errors (different types of errors).

Examples:
" An examination consists of three parts. In part A, a student must answer 2 of 3 questions. In part B, a student must answer 6 of 8 questions and in part C, a student must answer all questions. How many choices of questions does the student have?"
" The triangle bounded by the lines y = 0, y = 2x and y = -0.5x + k, with k positive, is equal to 80 square units. Find k."

Wrong answers by chatgpt
2

Chat bots are not really designed as calculators or algebra truth machines.

3

That is totally true

4

LangChain attempts to find mathematical expressions that it can evaluate, so it probably is more reliable on computation than vanilla ChatGPT. But clearly still challenged with complex ‘word’ problems like your examples.

https://python.langchain.com/en/latest/_modules/langchain/chains/llm_math/base.html?highlight=math%20expressions

output = str(
numexpr.evaluate(
expression.strip(),
global_dict={}, # restrict access to globals
local_dict=local_dict, # add common mathematical functions
)
)

Here is a link to the doc page for numexpr, which is what LangChain is using to perform the evaluation (as shown at the page linked above)

https://numexpr.readthedocs.io/en/latest/user_guide.html#supported-functions

1 Like