I’ve witnessed again and again companies trying to implement a RAG solution only to fail in the process. Most of these projects don’t fail because the technology is immature. They fail because the documents fed into them are contradictory, outdated, or simply shouldn’t be there.
The oversimplified flow of a RAG (Retrieval-Augmented Generation) solution looks like this: a user asks a question, the system retrieves relevant passages from a vector database, and those passages plus the question are sent to an LLM to formulate an answer. Why such initiatives most often fail is quite obvious to the external eye but not so obvious to the people leading them from within.
The goal is a noble one—to democratize internal knowledge or make siloed information available. But the implementation approach is naive: simply add the RAG solution to every document in the organization. The problem is that documents compiled by multiple people over time typically contain contradictions. A policy document from three years ago that states remote work is not allowed can sit next to a more recent HR update that says the opposite. Both may be retrieved for the same question. What is a RAG system supposed to do when the document set provides more than one “correct” answer?
Many organizations blame the technology, saying the system is hallucinating answers. When it is demonstrated that the real culprit is their own documents, the reaction is often one of genuine surprise. RAG systems retrieve; they do not adjudicate truth. When two passages conflict, the model has no reliable way to know which one reflects current policy. The result looks like hallucination even when the model is faithfully following the retrieved text.
This is not just a technical inconvenience. Conflicting answers erode trust in the system, waste budget, and create real risk if employees or customers act on outdated information. Once people lose confidence in the answers, the entire initiative loses credibility.
The recommendation for a good start with a RAG solution is quite simple: don’t put documents in the vector database that should not be there. Of course this is easier said than done, but the reward is worth the effort. In practice this means starting with a narrowly defined use case and a deliberately small, high-quality corpus rather than indexing everything. It means establishing clear ownership for what gets added and preferring living sources of truth—current policy portals and approved knowledge bases—over static document dumps.
A RAG system is only as trustworthy as the documents it is allowed to see. The first and most important design decision is not which vector database or embedding model to choose—it is which documents deserve to be in the system at all.