Hi all,
Does anyone else have the problem that the response from the above query is actually wrong when compared to the actual CSV input file?
When querying this, not all shirts with this property are returned (e.g. shirt #28 is missing)! When querying only for shirts, again not all shirts are returned (especially not the ones that were previously found to have sun protection). I tried several ways (including the map_reduce method) because it seems the context generation is off, but I wonder how exactly… anyway, this shouldn’t happen in a tutorial example. As a result, I cannot trust LangChain to reliably infer from the provided documents when this rather simple example does not even work.
Thanks for your thoughts on this!
Sebastian
I think this goes down to the capabilities of CHATGPT or LLM used rather than the LangChain interface. These are statistical models after all but the most recent versions have been performing better than previous ones.
Hi, thanks for your reply and thoughts. However, I’m still more inclined to believe that LangChain is somewhat unable to provide the full relevant context to LLM used to answer this prompt, probably due to an intrinsic token limitation. Maybe trying to summarize the descriptions first would do the trick… I might try it when I find some time.
In my admittedly limited experience with these tools, this is indeed the correct takeaway. Businesses deploying systems today have to hope risk and cost of imprecision is less than benefit of time and salary saved by automating with an LLM.
1 Like