# L5-Evaluation apply\_and\_parse error and gpt-3.5-turbo-0301 deprecation **URL:** \ **Category:** LangChain for LLM Application Development\ **Created:** [June 13, 2024, 8:38pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863 "2024-06-13T20:38:06Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![brunotc](https://avatars.discourse-cdn.com/v4/letter/b/b5a626/32.png) [@brunotc](https://community.deeplearning.ai/u/brunotc)\ **Post date:** [June 13, 2024, 8:38pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/1 "2024-06-13T20:38:07Z")
On trying to get this cell to work I kept running into an error that I couldn’t understand, it seemed like the apply\_and\_parse function wasn’t able to parse the result the LLM output and nothing worked to fix it. I realized that at the beginning there’s a cell that makes sure the gpt-3.5-turbo-0301 model isn’t used after the deprecation date, which was yesterday. Unfortunately, gpt-3.5-turbo’s output for this cell creates an error, the same for gpt-3.5-turbo-0613. The deprecation of 0301 is set for today at the earliest, so it might continue to work, but it’s a shaky foundation so this should be reviewed to make sure the notebook doesn’t break completely. Until then, try using 0301 and pray it still works 🫠 ---
**Author:** ![jforthai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jforthai/32/45967_2.png) [@jforthai](https://community.deeplearning.ai/u/jforthai)\ **Post date:** [June 21, 2024, 5:05am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/2 "2024-06-21T05:05:20Z")
Bump ---
**Author:** ![DanielOsorio01](https://avatars.discourse-cdn.com/v4/letter/d/f05b48/32.png) [@DanielOsorio01](https://community.deeplearning.ai/u/DanielOsorio01)\ **Post date:** [June 23, 2024, 10:31pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/3 "2024-06-23T22:31:32Z")
Hi. When running the following cell ```python new_examples = example_gen_chain.apply_and_parse( [{"doc": t} for t in data[:5]] ) ``` I got the following error: ```python --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[18], line 1 ----> 1 new_examples = example_gen_chain.apply_and_parse( 2 [{"doc": t} for t in data[:5]] 3 ) File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:257, in LLMChain.apply_and_parse(self, input_list, callbacks) 255 """Call apply and then parse the results.""" 256 result = self.apply(input_list, callbacks=callbacks) --> 257 return self._parse_result(result) File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:263, in LLMChain._parse_result(self, result) 259 def _parse_result( 260 self, result: List[Dict[str, str]] 261 ) -> Sequence[Union[str, List[str], Dict[str, str]]]: 262 if self.prompt.output_parser is not None: --> 263 return [ 264 self.prompt.output_parser.parse(res[self.output_key]) for res in result 265 ] 266 else: 267 return result File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:264, in (.0) 259 def _parse_result( 260 self, result: List[Dict[str, str]] 261 ) -> Sequence[Union[str, List[str], Dict[str, str]]]: 262 if self.prompt.output_parser is not None: 263 return [ --> 264 self.prompt.output_parser.parse(res[self.output_key]) for res in result 265 ] 266 else: 267 return result File /usr/local/lib/python3.9/site-packages/langchain/output_parsers/regex.py:28, in RegexParser.parse(self, text) 26 else: 27 if self.default_output_key is None: ---> 28 raise ValueError(f"Could not parse output: {text}") 29 else: 30 return { 31 key: text if key == self.default_output_key else "" 32 for key in self.output_keys 33 } ValueError: Could not parse output: QUESTION: According to the document, what is the approximate weight of the Women's Campside Oxfords per pair? ANSWER: The approximate weight of the Women's Campside Oxfords per pair is 1 lb. 1 oz. ``` Is this the same error you got? ---
**Author:** ![Jean\_Cupe](https://avatars.discourse-cdn.com/v4/letter/j/ee7513/32.png) [@Jean\_Cupe](https://community.deeplearning.ai/u/Jean_Cupe)\ **Post date:** [June 25, 2024, 9:02pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/4 "2024-06-25T21:02:52Z")
I tried using the version `gpt-3.5-turbo-0301` for the QAGenerateChain, and it fixed the problem for this cell. So you can do the following: ```auto example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI(model="gpt-3.5-turbo-0301")) ``` ---
**Author:** ![BasmaElshoky](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/basmaelshoky/32/92056_2.png) [@BasmaElshoky](https://community.deeplearning.ai/u/BasmaElshoky)\ **Post date:** [August 16, 2024, 1:30pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/5 "2024-08-16T13:30:57Z")
Thanks ---
**Author:** ![Priya\_M](https://avatars.discourse-cdn.com/v4/letter/p/7cd45c/32.png) [@Priya\_M](https://community.deeplearning.ai/u/Priya_M)\ **Post date:** [September 10, 2024, 11:50am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/6 "2024-09-10T11:50:33Z")
Thanks, I can confirm that this is still an issue when using the model “gpt-3.5-turbo”. As of 10th September, 2024 “gpt-3.5-turbo-0301” still works but unfortunately, this is due to be shutdown on 13th Sept as per [https://platform.openai.com/docs/deprecations](https://platform.openai.com/docs/deprecations) ---
**Author:** ![Kiaurash](https://avatars.discourse-cdn.com/v4/letter/k/b38774/32.png) [@Kiaurash](https://community.deeplearning.ai/u/Kiaurash)\ **Post date:** [September 14, 2024, 8:53pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/7 "2024-09-14T20:53:51Z")
Yes, I still got the same problem. ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [September 15, 2024, 1:47pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/8 "2024-09-15T13:47:25Z")
@Mubsi , deprecated tools? ---
**Author:** ![Mubsi](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/mubsi/32/94983_2.png) [@Mubsi](https://community.deeplearning.ai/u/Mubsi)\ **Post date:** [September 18, 2024, 5:50am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/9 "2024-09-18T05:50:03Z")
Thanks @TMosh. I just checked, the loading of the model didn’t run into an error for me, but later in the notebook I got an error. I have reported it to the team. ---
**Author:** ![lesly.zerna](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/lesly.zerna/32/47465_2.png) [@lesly.zerna](https://community.deeplearning.ai/u/lesly.zerna)\ **Post date:** [September 19, 2024, 2:45pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/10 "2024-09-19T14:45:16Z")
Thank you for bringing this up! Indeed it’s something about parsing and new compatible versions with OpenAI and Langchain, more specifically with QAGenerateChain. Just updated this code cell: > new\_examples = example\_gen\_chain.apply\_and\_parse( > #[{“doc”: t} for t in data[:5]] > [{“doc”: data[:5]}] > ) Try it again 🙂 ---
**Author:** ![omar.105](https://avatars.discourse-cdn.com/v4/letter/o/bbce88/32.png) [@omar.105](https://community.deeplearning.ai/u/omar.105)\ **Post date:** [September 23, 2024, 3:18am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/11 "2024-09-23T03:18:52Z")
cell 24 also have an issue: ```auto --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[25], line 1 ----> 1 graded_outputs = eval_chain.evaluate(examples, predictions) File /usr/local/lib/python3.9/site-packages/langchain/evaluation/qa/eval_chain.py:60, in QAEvalChain.evaluate(self, examples, predictions, question_key, answer_key, prediction_key) 50 """Evaluate question answering examples and predictions.""" 51 inputs = [ 52 { 53 "query": example[question_key], (...) 57 for i, example in enumerate(examples) 58 ] ---> 60 return self.apply(inputs) File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:157, in LLMChain.apply(self, input_list, callbacks) 155 except (KeyboardInterrupt, Exception) as e: 156 run_manager.on_chain_error(e) --> 157 raise e 158 outputs = self.create_outputs(response) 159 run_manager.on_chain_end({"outputs": outputs}) File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:154, in LLMChain.apply(self, input_list, callbacks) 149 run_manager = callback_manager.on_chain_start( 150 {"name": self. __class__. __name__ }, 151 {"input_list": input_list}, 152 ) 153 try: --> 154 response = self.generate(input_list, run_manager=run_manager) 155 except (KeyboardInterrupt, Exception) as e: 156 run_manager.on_chain_error(e) File /usr/local/lib/python3.9/site-packages/langchain/chains/llm.py:79, in LLMChain.generate(self, input_list, run_manager) 77 """Generate LLM result from inputs.""" 78 prompts, stop = self.prep_prompts(input_list, run_manager=run_manager) ---> 79 return self.llm.generate_prompt( 80 prompts, stop, callbacks=run_manager.get_child() if run_manager else None 81 ) File /usr/local/lib/python3.9/site-packages/langchain/chat_models/base.py:143, in BaseChatModel.generate_prompt(self, prompts, stop, callbacks) 136 def generate_prompt( 137 self, 138 prompts: List[PromptValue], 139 stop: Optional[List[str]] = None, 140 callbacks: Callbacks = None, 141 ) -> LLMResult: 142 prompt_messages = [p.to_messages() for p in prompts] --> 143 return self.generate(prompt_messages, stop=stop, callbacks=callbacks) File /usr/local/lib/python3.9/site-packages/langchain/chat_models/base.py:92, in BaseChatModel.generate(self, messages, stop, callbacks) 90 run_manager.on_llm_error(e) 91 raise e ---> 92 llm_output = self._combine_llm_outputs([res.llm_output for res in results]) 93 generations = [res.generations for res in results] 94 output = LLMResult(generations=generations, llm_output=llm_output) File /usr/local/lib/python3.9/site-packages/langchain/chat_models/openai.py:292, in ChatOpenAI._combine_llm_outputs(self, llm_outputs) 290 for k, v in token_usage.items(): 291 if k in overall_token_usage: --> 292 overall_token_usage[k] += v 293 else: 294 overall_token_usage[k] = v TypeError: unsupported operand type(s) for +=: 'OpenAIObject' and 'OpenAIObject' ``` ---
**Author:** ![Lulu\_Wang](https://avatars.discourse-cdn.com/v4/letter/l/a88e4f/32.png) [@Lulu\_Wang](https://community.deeplearning.ai/u/Lulu_Wang)\ **Post date:** [October 5, 2024, 11:33pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/12 "2024-10-05T23:33:16Z")
I am observing the same issue in the cell “eval\_chain.evaluate”, was this resolved? Tried a few things including the class definition to “langchain.evaluation.qa.eval\_chain”, but that didn’t work. ---
**Author:** ![Vlad\_Enache](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/vlad_enache/32/54406_2.png) [@Vlad\_Enache](https://community.deeplearning.ai/u/Vlad_Enache)\ **Post date:** [October 8, 2024, 10:41pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/13 "2024-10-08T22:41:33Z")
Thank you @lesly.zerna, your fix worked, but the notebook still breaks on line 24 `graded_outputs = eval_chain.evaluate(examples, predictions)`, near the end. Do you have any idea how it can be solved? Thx ---
**Author:** ![Mahisethuraman](https://avatars.discourse-cdn.com/v4/letter/m/c67d28/32.png) [@Mahisethuraman](https://community.deeplearning.ai/u/Mahisethuraman)\ **Post date:** [October 21, 2024, 11:18pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/14 "2024-10-21T23:18:03Z")
I am having the same issues. Has anyone gotten past this error? ---
**Author:** ![tttttt](https://avatars.discourse-cdn.com/v4/letter/t/8797f3/32.png) [@tttttt](https://community.deeplearning.ai/u/tttttt)\ **Post date:** [October 23, 2024, 10:49pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/15 "2024-10-23T22:49:40Z")
That worked for me: example\_gen\_chain = QAGenerateChain.from\_llm(ChatOpenAI(model=“gpt-3.5-turbo”)) new\_examples = example\_gen\_chain.apply\_and\_parse([{“doc”: data[:5]}]) ---
**Author:** ![sai\_trinath\_dubba](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sai\_trinath\_dubba](https://community.deeplearning.ai/u/sai_trinath_dubba)\ **Post date:** [October 28, 2024, 8:15am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/16 "2024-10-28T08:15:43Z")
Thank you , this worked ---
**Author:** ![HowardL](https://avatars.discourse-cdn.com/v4/letter/h/ba8739/32.png) [@HowardL](https://community.deeplearning.ai/u/HowardL)\ **Post date:** [November 4, 2024, 2:52am UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/17 "2024-11-04T02:52:41Z")
Did anyone resolve this issue? @lesly.zerna solution worked (thank you Lesly!) to generate predictions. But when it came to QAEvalChain, and applying that function in graded\_outputs, I got the error again: …/opt/anaconda3/lib/python3.8/site-packages/langchain\_core/language\_models/chat\_models.py in generate(self, messages, stop, callbacks, tags, metadata, run\_name, run\_id, \*\*kwargs) 424 for res in results 425 ] → 426 llm\_output = self.\_combine\_llm\_outputs([res.llm\_output for res in results]) 427 generations = [res.generations for res in results] 428 output = LLMResult(generations=generations, llm\_output=llm\_output) # type: ignore[arg-type] /opt/anaconda3/lib/python3.8/site-packages/langchain\_community/chat\_models/openai.py in \_combine\_llm\_outputs(self, llm\_outputs) 375 for k, v in token\_usage.items(): 376 if k in overall\_token\_usage: → 377 overall\_token\_usage[k] += v 378 else: 379 overall\_token\_usage[k] = v TypeError: unsupported operand type(s) for +=: ‘dict’ and ‘dict’ ---
**Author:** ![Dvf](https://avatars.discourse-cdn.com/v4/letter/d/cdc98d/32.png) [@Dvf](https://community.deeplearning.ai/u/Dvf)\ **Post date:** [November 4, 2024, 3:07pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/18 "2024-11-04T15:07:31Z")
Hi @lesly.zerna, correct me if I’m wrong, but your solution only generates one new example using 5 documents in data, while the goal is to generate 5 new examples, one for each document in data[:5]. ---
**Author:** ![lesly.zerna](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/lesly.zerna/32/47465_2.png) [@lesly.zerna](https://community.deeplearning.ai/u/lesly.zerna)\ **Post date:** [January 29, 2025, 7:30pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/20 "2025-01-29T19:30:47Z")
Hi! can you try again! We’ve updated the gpt version and it is currently all working! Give it one more check, please! and if something comes up, please let us know! > **[LangChain for LLM Application Development - DeepLearning.AI](https://learn.deeplearning.ai/courses/langchain/lesson/6/evaluation)** > > Use the powerful and extensible LangChain framework, using prompts, parsing, memory, chains, question answering, and agents. ---
**Author:** ![adn\_121](https://avatars.discourse-cdn.com/v4/letter/a/8dc957/32.png) [@adn\_121](https://community.deeplearning.ai/u/adn_121)\ **Post date:** [February 18, 2025, 8:50pm UTC](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863/21 "2025-02-18T20:50:59Z")
graded\_outputs = eval\_chain.evaluate(examples, predictions) * * * This line is giving an issue that is, TypeError: unsupported operand type(s) for +=: ‘OpenAIObject’ and ‘OpenAIObject’ [Next page](https://community.deeplearning.ai/t/l5-evaluation-apply-and-parse-error-and-gpt-3-5-turbo-0301-deprecation/646863.md?page=2)