When building a Retrieval-Augmented Generation (RAG) system, retrieving relevant documents is only part of the challenge. Sometimes the knowledge base contains outdated information or multiple documents that contradict each other.
I’m curious how developers handle these situations in practice. For example, should a retrieval system prioritize documents based on publication date, source reliability, or some combination of factors?
How do you prevent an LLM from confidently generating an answer based on outdated or conflicting sources? Are there effective strategies for document versioning, metadata filtering, or source prioritization?
I’d appreciate hearing about practical approaches and lessons learned from anyone working with RAG applications.
I think I’d be a little careful about prioritizing documents just because they’re newer.
For example, if someone asks about an older API version, the latest documentation might actually give them the wrong answer. So maybe the first step isn’t ranking by date, but figuring out which version or context the question belongs to.
I’m not entirely sure there’s a good universal rule for conflicting sources either. I’d probably keep the conflicting evidence visible and let the system say something like “these sources disagree” rather than forcing the LLM to pick one.
One thing I’d definitely want is to track which document version each answer came from. Otherwise even a correct answer becomes pretty hard to audit later when the knowledge base changes.