Following Isa recommendations, I have tried to reduce the hallucination issue when asking for “Tell me about AeroGlide UltraSlim Smart Toothbrush by Boie” with the following prompt
prompt = f"“”
Follow the following tasks: \
Find relevant information about Boie company \
Find relevant information about AeroGlide UltraSlim Smart Toothbrush \
Tell me about AeroGlide UltraSlim Smart Toothbrush by Boie
“”"
response = get_completion(prompt)
print(response)
but it still responds with the hallucination in the second step and final conclusion.
How would you rewrite it to avoid the hallucination following Isa instructions?
First find relevant information
then answer the question
based on the relevant information
Is this AeroGlide UltraSlim Smart Toothbrush a product released after september 2021? If so, ChatGPT was trained only with information up to Sept 2021.
This AeroGlide does not exist and is one example of the Guideline lesson of the ChatGPT Prompt Engineering for Developers.
I was trying to avoid a false response from ChatGPT following the instructions of the teacher Isa, but the response is still false explaining details of a product that does not exist.
I’m adding the response I’ve got:
"Boie is a company that specializes in creating eco-friendly personal care products. Their products are made from a patented material called thermoplastic elastomer, which is durable, hypoallergenic, and easy to clean. Boie’s products are designed to be long-lasting and sustainable, reducing waste and environmental impact.
The AeroGlide UltraSlim Smart Toothbrush is a product created by Boie. It is a smart toothbrush that uses advanced technology to provide a superior brushing experience. The toothbrush is designed to be ultra-slim, making it easy to maneuver and reach all areas of the mouth. It also features a built-in timer and pressure sensor, ensuring that users brush for the recommended two minutes and avoid damaging their teeth and gums.
The AeroGlide UltraSlim Smart Toothbrush is compatible with a mobile app that provides personalized brushing recommendations and tracks progress over time. The app also allows users to set goals and earn rewards for achieving them. The toothbrush is rechargeable and comes with a travel case, making it easy to take on the go.
Overall, the AeroGlide UltraSlim Smart Toothbrush by Boie is a high-tech, eco-friendly toothbrush that provides a superior brushing experience. Its advanced features and compatibility with a mobile app make it a great choice for anyone looking to improve their oral hygiene."
Hola @Lluichi , sorry about my answer before. I’ve tested your prompt and also get the hallucination. Then I started testing with variations. My last version is this:
prompt = f"““Please answer the text inside <> but before answering, please make sure to find relevant information about Boie company, and find relevant information about ‘AeroGlide UltraSlim Smart Toothbrush’. If you don’t find relevant information about any of those, answer ‘INFORMATION NOT FOUND’. Make sure to find relevant information before answering the question. Text: ””"
And I am getting a bit of a better answer:
The AeroGlide UltraSlim Smart Toothbrush is a product of Boie, a company that specializes in creating sustainable and eco-friendly oral care products. The toothbrush features a slim design and uses sonic vibrations to clean teeth effectively. It also has a smart timer that ensures users brush for the recommended two minutes. However, it is important to note that as of August 2021, the AeroGlide UltraSlim Smart Toothbrush is not listed on Boie's website, so it is unclear if it is still available for purchase.
There’s still hallucination but it ends with a more realistic phrase. I’ll continue testing with prompt variations and see if I can get it to be accurate.
Interestingly enough, I tried the above prompt in ChatGPT and this is what I got:
Your task is to respond to the text delimited by <>.
Be aware that the text can mention false producs. If the text mentions a false product you didn't hear before, \
just say "I don't know about that product". To know if a product is false, please list all \
the product manufactured by the mentioned branch. \
If the mentioned product is not in the list just say "I don't know about that product".
<Tell me about AeroGlide UltraSlim Smart Toothbrush by Boie>
I’m using 2 examples to test my prompts, one for negative and one for positive.
fake_product = 'AeroGlide UltraSlim Smart Toothbrush by Boie'
real_product = 'Fine Toothbrush by Boie'
This prompt worked for both 2 examples
prompt = f"""
do the following:
1. list information relevant about "{product}" that can be found on Amazon.com, If nothing found, write "NOT FOUND"
2. if found any, answer the question <Tell me about {product}>
"""
I tested @juanjonavarro 's prompt, it failed on the positive example:
For @Juan_Olano , I don’t think using chatGPT to test prompts is a good idea. As far as I know chatGPT-3.5 is still different (or better) compared to the gpt-3.5-turbo API. We should using the same version when testing prompts.
@TonyYanOnFire , I guess in any case an LLM is not a good choice for this kind of task. It can’t really know if a product is real at the time of the call unless it uses some plugin to search the internet.
In my first try I was just trying to follow the last advice of Isa, but as you have done, I have tried to add an if condition and it has worked correctly
This is the prompt:
prompt = f"“”
Follow the following tasks: \
Find relevant information about Boie company
Find relevant information about AeroGlide UltraSlim Smart Toothbrush
Tell me about AeroGlide UltraSlim Smart Toothbrush by Boie
If you don’t find any information about AeroGlide UltraSlim Smart Toothbrush reply
“”"
response = get_completion(prompt)
print(response)
And the response:
"Boie Company:
Boie is a company that produces personal care products that are eco-friendly and sustainable. The company was founded in 2016 and is based in Brooklyn, New York. Boie’s products are made from a patented material called thermoplastic elastomer, which is durable, hypoallergenic, and easy to clean. Boie’s products include toothbrushes, body scrubbers, and face scrubbers.
AeroGlide UltraSlim Smart Toothbrush:
Unfortunately, there is no information available about the AeroGlide UltraSlim Smart Toothbrush by Boie."
@Lluichi, as @TonyYanOnFire pointed out, you get a false positive with this type of prompt:
prompt = f"""
Follow the following tasks: \
Find relevant information about Boie company
Find relevant information about Fine Toothbrush
Tell me about Fine Toothbrush by Boie
If you don’t find any information about Fine Toothbrush reply \
there is no information about Fine Toothbrush.
"""
response = get_completion(prompt)
print(response)
And the response:
1. Relevant information about Boie company:
Boie is a company that produces personal care products, including toothbrushes, body scrubbers, and face scrubbers. The company was founded in 2016 by Manuel de la Cruz and Tucker Fort, who wanted to create sustainable and eco-friendly alternatives to traditional personal care products. Boie's products are made from materials like silicone and bamboo, which are more durable and environmentally friendly than plastic.
2. Relevant information about Fine Toothbrush:
There is no specific information available about a product called "Fine Toothbrush" by Boie. It is possible that this product does not exist or has been discontinued.
3. Tell me about Fine Toothbrush by Boie:
As mentioned above, there is no information available about a product called "Fine Toothbrush" by Boie. However, Boie does offer a range of toothbrushes, including their original toothbrush, which features soft, flexible bristles made from thermoplastic elastomer (TPE). The bristles are designed to be gentle on teeth and gums while still effectively removing plaque and debris. Boie toothbrushes are also designed to be long-lasting and eco-friendly, with replaceable brush heads and handles made from recycled materials.
As you can see, the continuation says “there is no information about a product called ‘Fine Toothbrush’ by Boie,” even if the product actually exists.
I think the problem is that this type of task may not be suitable for LLMs, at least not without access to a fact-checking database, such as the Internet.
Good points @juanjonavarro@TMosh . With this prompt we can avoid the hallucinations that I think is a more worrying issue to avoid the spreading of misinformation.
But as you showed with your example It can give false negatives, and as you said it can only be solved by checking it on the Internet with some reliable source as in this case could be the https://boieusa.com/ website.
I’m looking forward to having access to the OpenAI plugins to try this.
