L5-Evaluation apply_and_parse error and gpt-3.5-turbo-0301 deprecation

On trying to get this cell to work I kept running into an error that I couldn’t understand, it seemed like the apply_and_parse function wasn’t able to parse the result the LLM output and nothing worked to fix it.
I realized that at the beginning there’s a cell that makes sure the gpt-3.5-turbo-0301 model isn’t used after the deprecation date, which was yesterday. Unfortunately, gpt-3.5-turbo’s output for this cell creates an error, the same for gpt-3.5-turbo-0613.

The deprecation of 0301 is set for today at the earliest, so it might continue to work, but it’s a shaky foundation so this should be reviewed to make sure the notebook doesn’t break completely.

Until then, try using 0301 and pray it still works :melting_face:

3 Likes

Bump

Hi. When running the following cell

new_examples = example_gen_chain.apply_and_parse(
    [{"doc": t} for t in data[:5]]
)

I got the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[18], line 1
----> 1 new_examples = example_gen_chain.apply_and_parse(
      2     [{"doc": t} for t in data[:5]]
      3 )

File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:257, in LLMChain.apply_and_parse(self, input_list, callbacks)
    255 """Call apply and then parse the results."""
    256 result = self.apply(input_list, callbacks=callbacks)
--> 257 return self._parse_result(result)

File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:263, in LLMChain._parse_result(self, result)
    259 def _parse_result(
    260     self, result: List[Dict[str, str]]
    261 ) -> Sequence[Union[str, List[str], Dict[str, str]]]:
    262     if self.prompt.output_parser is not None:
--> 263         return [
    264             self.prompt.output_parser.parse(res[self.output_key]) for res in result
    265         ]
    266     else:
    267         return result

File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:264, in <listcomp>(.0)
    259 def _parse_result(
    260     self, result: List[Dict[str, str]]
    261 ) -> Sequence[Union[str, List[str], Dict[str, str]]]:
    262     if self.prompt.output_parser is not None:
    263         return [
--> 264             self.prompt.output_parser.parse(res[self.output_key]) for res in result
    265         ]
    266     else:
    267         return result

File /usr/local/lib/python3.9/site-packages/langchain/output_parsers/regex.py:28, in RegexParser.parse(self, text)
     26 else:
     27     if self.default_output_key is None:
---> 28         raise ValueError(f"Could not parse output: {text}")
     29     else:
     30         return {
     31             key: text if key == self.default_output_key else ""
     32             for key in self.output_keys
     33         }

ValueError: Could not parse output: QUESTION: According to the document, what is the approximate weight of the Women's Campside Oxfords per pair?

ANSWER: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz.

Is this the same error you got?

I tried using the version gpt-3.5-turbo-0301 for the QAGenerateChain, and it fixed the problem for this cell. So you can do the following:

example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI(model="gpt-3.5-turbo-0301"))
3 Likes