L6-Agents - Quite many errors when solving maths problems

IT appears that the chances of getting an error when using “llm-maths” is quite high.

I tried serveral Grade 12 maths questions and got quite a few errors (different types of errors).

Examples:
" An examination consists of three parts. In part A, a student must answer 2 of 3 questions. In part B, a student must answer 6 of 8 questions and in part C, a student must answer all questions. How many choices of questions does the student have?"
" The triangle bounded by the lines y = 0, y = 2x and y = -0.5x + k, with k positive, is equal to 80 square units. Find k."

Chat bots are not really designed as calculators or algebra truth machines.

That is totally true

LangChain attempts to find mathematical expressions that it can evaluate, so it probably is more reliable on computation than vanilla ChatGPT. But clearly still challenged with complex ‘word’ problems like your examples.

https://python.langchain.com/en/latest/_modules/langchain/chains/llm_math/base.html?highlight=math%20expressions


output = str(
numexpr.evaluate(
expression.strip(),
global_dict={}, # restrict access to globals
local_dict=local_dict, # add common mathematical functions
)
)


Here is a link to the doc page for numexpr, which is what LangChain is using to perform the evaluation (as shown at the page linked above)

https://numexpr.readthedocs.io/en/latest/user_guide.html#supported-functions

1 Like