I just ran all the cells and I get this output for customer-support-automation notebook. How can this be prevented? Can this be called hallucination, because the initial results seem good, only when referred to QA the output is hallucination - may be because the input when delegated is restricted to review text.
Also, this makes the whole thing subjective, and I won’t know when I might get this kind of result in production environment:
Hello DeepLearningAI,
Thank you for reaching out with your inquiry about setting up a Crew and adding memory to it. Setting up a Crew on crewAI is a straightforward process that can be done by following these steps:
Log in to your crewAI account and navigate to the dashboard.
Click on the “Create Crew” button to start the process of setting up a new Crew.
Enter the necessary details such as the name of the Crew, description, and any other relevant information.
Once the Crew has been created, you can start adding members by inviting them through their email addresses.
To add memory to your Crew, you will need to enable memory for the entire crew in the settings menu. Here are the specific steps to do so:
Sign in to your CrewAI account and go to the dashboard.
Click on the “Settings” tab located in the top menu.
Look for the “Memory” section in the settings menu and click on it.
Toggle the switch to enable memory for the entire crew.
Remember to save your changes before exiting the settings menu.
It’s important to note that the amount of memory available for your Crew will depend on the plan you choose. If you have any specific requirements or questions regarding memory storage, feel free to reach out to our support team for further assistance.
I hope this information helps you get started with setting up your Crew and adding memory to it. If you need any additional support or have any other questions, please don’t hesitate to ask. We are here to help!
Best regards, [Your Name] Senior Support Representative crewAI
Thanks for letting us know. I tried the notebook myself, sometimes I would get a proper, code example, and sometimes silly outputs like the one you shared above.
As for how to prevent it ? I’d say there’s no right answer here. It happens by chance, and also, gpt-3.5 is not as powerful as gpt-4. I’d encourage you to try things on your own with both GPTs and see what you end up with. Try changing the prompts and lets us know if you can manage to prevent it. Good luck!
Hi, thanks for the reply. Yes Such things do happen with AI applications, if I try again, it would definitely give better results - but in this case, the problem is that the user would not know that this is hallucinated answer, for him to try again!! The very purpose of having a QA assistant concept in this crew has failed What if this happens when we do customer demos?! (Though such thing has happened even with google product premiering!! )
The reason I posted this query was also that this CrewAI concept of treating ‘agents’ similar to ‘recruited worker personalities’ may not work. In stock analysis notebook, the data analysis agent is supposed to do analysis based on machine learning! The assumption seems to be that with the given description, goal and backstory, the agent would behave like such an expert! This may not be so.
Unless there is domain specific knowledge available in the llm/RAG and also specialized tools are given for execution, the agent would not perform to the description, goal and backstory. This is the reason that these crewai notebooks are not giving consistent and reliable results. While studying the lectures, for some time, I really thought that it is great to treat it as similar to recruiting personalities for a task, along with manager. But while working through the notebooks, I realized this idea of CrewAI may not be not practical and useful.
I replicated this. Bizarre that in a demo by CrewAI CEO he is presenting something that produces an incorrect answer. Looking through the video, the initial agent response is correct but the QA agent forces a re-write and results in a made-up answer. Seems to undermine the point of a QA agent.
What if this happens when we do customer demos?! (Though such thing has happened even with google product premiering!!