Every enterprise RAG demo looks the same: ask a clean question, get a clean answer, everyone nods. The failure mode nobody demos is a support engineer asking about a policy that was updated eighteen months ago, and the system confidently quoting the old one.
The Failure That Actually Costs You
We once audited a knowledge base for a financial services client where roughly one in eight retrieved answers were pulling from a superseded compliance document that had never been deleted, just replaced. The model wasn't hallucinating in the technical sense — it was retrieving real text and summarizing it accurately. The problem was upstream: the vector index had no concept of document lifecycle, so a stale document and its replacement carried equal retrieval weight. This is the failure mode that matters in production, and it's almost never the one people design against.
Most conversations about LLM accuracy start and end at "hallucination," as if the fix is a better model. In our experience, the large majority of enterprise RAG failures trace back to retrieval quality, not generation quality — and retrieval quality is a data engineering problem dressed up as an AI problem.
RAG vs. Fine-Tuning: The Decision We Actually Use
We get asked constantly whether to fine-tune a model on internal documentation instead of building a retrieval pipeline. The honest answer is that fine-tuning solves a narrower problem than most people think, and RAG solves a broader one than most people expect.
- Fine-tuning teaches a model a style, format, or narrow skill — it does not reliably teach it new facts, and facts injected via fine-tuning degrade unpredictably as the underlying knowledge changes.
- RAG keeps facts external and swappable — update the source document, and the next query retrieves the update, no retraining required.
- Fine-tuning earns its cost when the task is structural — extracting fields in a specific format, adopting a specific tone — rather than factual.
- Most production systems we build combine both: a well-prompted or lightly fine-tuned model for structure and tone, RAG for anything that needs to stay current.
The Question We Ask First
Before architecture, we ask one question: how often does this knowledge change? If the answer is 'constantly,' RAG is close to mandatory. If the answer is 'almost never,' fine-tuning — or even a well-crafted system prompt with a small curated context window — will outperform a full RAG pipeline on cost and latency.
What Breaks in Production That Never Breaks in the Demo
A RAG demo runs on ten hand-picked documents. Production runs on ten thousand documents written by dozens of people over several years, in inconsistent formats, with no shared structure. That gap is where most of the real engineering work lives.
- Chunking strategy matters more than embedding model choice — a poorly chunked 40-page policy document will work fine until someone asks a question that spans two chunks, and then it silently fails.
- Document lifecycle has to be modeled explicitly — superseded, draft, and approved versions need different retrieval weight, or the model treats a rejected draft as equally authoritative as the current policy.
- Re-ranking is not optional at scale — semantic similarity search alone returns plausible-sounding matches that aren't actually the best answer; a re-ranking step that scores retrieved chunks against the specific question catches this before it reaches the user.
- Retrieval confidence needs a floor — if nothing in the index is a strong match, the system should say so, not generate a fluent answer built on the closest-available-but-still-wrong chunk.
A Simple Readiness Check Before You Build
Before committing engineering time to a RAG pipeline, we run a manual test: take twenty real questions your team actually gets asked, and manually search your existing document repository for the answers. If a knowledgeable person struggles to find the right document in under a minute, an automated retrieval system will struggle too — the problem isn't the AI layer, it's the underlying information architecture, and no amount of model tuning fixes that.
The organizations that get real value from enterprise AI knowledge bases are the ones that treat retrieval quality as the primary engineering problem, and generation quality as secondary. Get the data layer right, and a reasonably good model will produce reasonably good answers. Get it wrong, and the best model available will confidently produce wrong ones.