People assume the hard part is getting a smarter model.
It isn’t.
The hard part is making sure the model is reading the right information in the first place.
Ask an AI about a recent research paper, a niche topic, or something buried in academic literature. If it has the wrong source or no source at all.. it can sound incredibly confident while being completely wrong.
That’s not an intelligence problem.
It’s a data problem.
The more I read about how modern AI systems actually work, the more I realized that the biggest improvements often come from better retrieval, cleaner documents, and more reliable sources, not from swapping one model for another.
Kind of funny.
We spend weeks comparing models…
…and sometimes the real bottleneck is just getting the right document into the context window.
Curious.. what’s the biggest source of AI mistakes you’ve seen: bad reasoning, or bad information?
This is exactly why retrieval matters so much. A stronger model can’t compensate for missing or outdated sources. I’ve noticed that once you improve the quality of the documents being retrieved, the answers become far more reliable than simply switching between different LLMs.
I think it can be both. AI mistakes can come from the model itself—its architecture, complexity, capabilities, and the technology used to build and train it. The quality and diversity of the training data also play an important role.
At the same time, even a very capable model can produce a poor answer if the information it is given or retrieves is inaccurate, incomplete, outdated, or based on unreliable sources. If the relevant information is not available to the model, or if the retrieval process brings back poor-quality sources, the resulting output may be incorrect or may not make sense.
So, in my view, it is not necessarily a question of bad reasoning versus bad information. It can be a combination of both. The model’s inherent capabilities and the quality of the information it has access to can both determine the quality of the final response.
I’m not convinced that the source of AI errors is simply “bad data” or “bad reasoning”. I think thinking and data are too entangled to separate them that easily.
For me the better way to look at it is as a chain: question → grounding → interpretation → inference → generation. Every step in this chain can introduce errors. The system can use the wrong information, misunderstand the information it has, make the wrong conclusion, or express something incorrectly.
That is also why it is difficult to trust AI. We only see the final answer, but we usually don’t see what happened in the steps before that. We lack observability into the process: what data was used, what assumptions were made, how the conclusion was reached, and where uncertainty exists.
My approach basically is to not to blindly trust the answer. As a user I need to understand what I´m working with. If the data matters, provide it or point the system to the right source. Don’t let it guess. If the result matters, validate it.
The biggest error I see with AI is that it can’t see to tell the time. I use my AI agent as a news service every morning and at the end ask her to list some stories she thinks I would find interesting that haven’t been in the various subjects I cover. Yesterday (August 2nd) she told me that America is getting ready to celebrate its 250th birthday. You know, July 4th. Still somewhat undependable when you use it to dig into current events although having an interface giving you your news is normally up to date and let’s you explore things more deeply if they interest you. Sports, SpaceX, even spiritual thoughts each day. I even named it Cordelia News Service.
I don’t think it is a “data” problem if by that you mean, “if only we had more data” or “if only we could better clean up the data”. the problem is that we work with humans in ways that are hard to duplicate in our interactions with LLMs. for one thing, when you or i go look for an expert, we can do diligence on them: we can research their education, accomplishments, the projects they have completed, and what their peers say about them. we can ask them questions and understand how they approach projects and decide whether we want to work with them, whether we are a match. we can’t really do any of that with an LLM
a simple example: i just replaced the roof on my house. sure i wrote a spec that covered the work to be done, the materials to be used, and what a good result looked like, and that helped, but the real magic wasn’t my “prompt engineering”. it was talking to the contractor, who explained to me how he worked, looking at the projects he has done, talking to people he worked for, and reviewing the work specification with him, all before i hired him. and while he worked, i could see what he was doing and ask him questions about what was happening and he could ask me questions when he encountered some work that wasn’t in the contract. that’s how we get things done with humans, why we trust them, because we are interacting with them as we work together.
and all of that is really hard to do with an LLM. they just aren’t built for it and no increase in how much data, improved or otherwise, they ingest will change that. so how do i vet an LLM’s expertise on a topic? how does an LLM explain to me what it is doing and why? why should I trust anything an LLM says or does? i have to come up with a whole raft of metrics and structure to answer those questions and substitute for one of the most important parts of how we successfully work with other humans, the ways we evaluate and decide how and when to trust others in, essentially, real time.
I wonder if part of the problem is that we talk about “trusting an AI” as though trust were an intrinsic property of the model. I would argue that the context and source material it has to work from can greatly affect its performance, and that trust also needs to be scoped to a particular task and supported by evidence. I can reasonably trust the same model to reformat text while requiring extensive validation before trusting it to evaluate a consequential business decision.
The contractor analogy is useful because much of what establishes trust there is really evaluation: prior performance, observable results, agreed requirements, monitoring during the work, and intervention when something falls outside expectations. We may not be able to reproduce the human relationship with an LLM, but perhaps we do not need to. Testing, provenance, reproducibility, monitoring, and clearly defined failure or escalation criteria can provide another basis for justified trust.
I also think there is an odd expectation that AI should be correct 99.99% of the time simply because it is machine intelligence. We routinely tolerate uncertainty and error from human experts while judging an AI failure as evidence that the system cannot be trusted at all. That makes me think the AI trust problem is not only a data or reasoning problem. It is also an evaluation problem: what evidence do we require before deciding that reliance on a system is warranted for this particular use?
