Why RAG Fails in Production

The answer looked right. It cited a real document, the wording matched the user’s question, the model sounded confident, and nothing in the response looked obviously made up. The problem was the source.

Retrieval had pulled an old implementation checklist instead of the current policy. The checklist still existed, it used the same product words, and it described a real feature state from six months earlier. The model did exactly what the system asked it to do: answer from the provided context.

That’s what makes production RAG uncomfortable. The visible failure looks like generation, but the real failure usually happened earlier, when the system decided what evidence deserved to be in the prompt at all.

I have been working through versions of this problem while working at Tapestry, an AI-powered retail intelligence application. A retailer might ask why a category’s margin changed, whether a promotion worked, or what needs attention this week. Those questions sound conversational. The answer still has to be grounded in the right store, the right date range, the right product, and data the person is allowed to see.

The details are particular to retail. The failure mode is not.

Similarity is not relevance

Vector search finds text that’s semantically close to a query. That’s useful, but it isn’t the same as finding the right answer. A retail manager asks, “Which products hurt margin last week?” and the nearest chunks might mention margin, promotions, supplier terms, product ranges, sales reports, and category performance. All of them are related. Only one of them might actually answer the question. Similarity gives you a candidate set, it doesn’t decide which candidate should win.

This is where a lot of RAG prototypes quietly cheat. The demo corpus is small, the documents are clean, and the user asks in the same vocabulary the docs use, so the top result is obvious because there are only a few places it could be. Production strips all of that away. The documents overlap. An old promotion still looks relevant. A product’s packaging changes. Two stores use different names for the same range. And the user asks in their own words, not the language your data model uses.

In retail, the same phrase can also mean different things. “Margin” can mean a percentage or dollars. “This week” has to resolve to actual dates. “Top products” could mean sales, units, growth, or the biggest drag on a department. A model should not make those choices invisibly just because the question was phrased casually.

Retrieval is a product decision

The hard part of RAG isn’t storing embeddings. It’s deciding what counts as the answer. And that decision is full of product logic:

whether the current period beats an older report
whether a store view beats an aggregate group view
whether a supplier can see only its own performance
whether a product name is specific enough to use
whether the system should answer or ask for clarification

None of that lives inside cosine similarity. You have to encode it somewhere.

A naive retrieval function looks reasonable enough:

const chunks = await vectorSearch({
	query: userQuestion,
	limit: 8,
});

const answer = await model.generate({
	context: chunks.map((chunk) => chunk.text).join("\n\n"),
	question: userQuestion,
});

That can work for a demo. A production-shaped version is still small, but it makes the choices visible:

const candidates = await retrieveProductContext({
	query: userQuestion,
	shopId: "shop_a",
	includeInactive: false,
});

const sources = await rerank({
	query: userQuestion,
	candidates,
	limit: 3,
});

const context = sources
	.map((source) => `# ${source.label}\n${source.summary}`)
	.join("\n\n");

const answer = await model.generate({
	instructions: "Answer from the supplied sources. Cite them, or say when the evidence is insufficient.",
	context,
	question: userQuestion,
});

This is product-context retrieval, not a sales query. If the question needs exact sales, margin, or order figures, the application should calculate those with a structured query and pass the result in as evidence.

The important detail is not the helper names. It is the order: search the relevant product context within the shop, leave out inactive records, rank the results for the question, then give the model a small set of sources it can cite. That can tell the difference between a current product and an old promotion. The model alone cannot.

Ranking is where the system tells the truth

Most teams discover ranking later than they should. They start with embeddings, because embeddings are the obvious new part. Then users report wrong answers, so the team swaps embedding models, changes the chunk size, adds more chunks to the prompt, and rewrites the system message to say “use only the provided context.” That helps sometimes. But if the right source is ranked below the wrong one, the model has already lost.

At Tapestry, a useful answer needs more than text that happens to be related to the question. It needs the store or group in scope, the date range, the measure the user asked for, and the exact product or category where that matters. A response about sales is not useful if the user meant gross margin. A response about a network is not useful if they manage one store.

That is the broader RAG problem in miniature. Ranking is where the system says what matters most. Recency matters. Authority matters. Customer state matters. And exact matches still matter, especially for product names, supplier names, error codes, invoice IDs, feature flags, and policy names.

That’s why production retrieval usually ends up hybrid. Vector search finds the semantic neighbours, keyword search catches the exact terms, metadata filters drop documents that shouldn’t be considered at all, and a reranker orders the survivors with more context than the first pass had. It sounds less elegant than “just embed everything.” It also works better.

Context has to travel with the data

RAG systems often treat chunking as a storage detail. It isn’t. Chunking decides what context can travel together. Split a document too aggressively and you lose the conditions that made the answer true. Keep the chunks too large and retrieval returns noisy blocks that bury the one useful sentence.

That’s why context engineering matters here. RAG isn’t just retrieval bolted onto a model, it’s the part of the system that decides what world the model is allowed to see before it answers.

For retail intelligence, the relevant context can be a date range, a shop, a product, a supplier, a promotion, and the measure behind a result. Pull apart the relationship between them and an answer can stay technically correct while becoming operationally wrong. A margin change without the time period is not enough. A recommendation without a store is not enough. A product result without the full product identity may not be enough either.

Good chunking follows the shape of the domain. Policies need their conditions attached. API docs need examples next to the parameters. Retail data needs the scope and definitions that make a number meaningful. A chunk isn’t just text. It’s the smallest unit of meaning you’re willing to let the model reason over.

Freshness is not optional

Production RAG systems rot. Documents change, products change, pricing changes, policies change. Teams rename things. Old launch plans stay indexed long after the feature shipped in a different form. Retrieval doesn’t know about any of that unless you tell it.

The same is true of retail data. A recommendation only makes sense against a stated reporting window and known data freshness. A number from this morning and a number from last week should not be made to look equivalent. If the data is delayed or a product has changed category, the system needs a way to represent that rather than quietly carrying on.

So freshness needs design:

source timestamps
expiry dates
document status
ownership
reindexing guarantees
deletion paths

Without those controls, the index turns into a landfill with a search API in front of it. The system can still answer, and that’s the problem. Wrong answers from stale knowledge often look more credible than a refusal, because they cite real documents. The source exists, the text is genuine, and the conclusion just isn’t true anymore.

Evaluation has to measure retrieval first

Most RAG evals only look at the final answer. That matters, but it hides the failure. When an answer is wrong, you need to know whether the model ignored good context or never received it in the first place, because those are different fixes.

For a retail question, I want to know whether the system used the right store or group, applied the intended date range, interpreted the requested measure correctly, and kept the answer within the user’s data boundary. Those checks are closer to product risk than a general score for writing quality.

So measure the retrieval step directly. For each test question, ask:

did the candidate set include the right source?
did the right source rank near the top?
did the context include the conditions and exceptions?
did the system exclude sources the user should not see?
did the model cite evidence that actually supports the answer?

This is why evals are not tests: a passing answer today doesn’t prove the system will keep answering correctly after the docs change, the product changes, or the query distribution shifts. You need evals that watch the moving parts, not just the prose at the end.

Permissions are part of retrieval

A RAG system that ignores permissions isn’t unfinished. It’s dangerous. The model should only answer from information the user is allowed to see, which sounds obvious right up until the index holds public docs, internal support notes, account-specific contracts, sales call transcripts, and engineering runbooks all together.

In a retail product, those boundaries may be between a store, a group, a supplier, and the wider network. Filtering after generation is too late. The restricted text has already entered the context window, and the answer can paraphrase it even after you strip the citation. Access control belongs before retrieval, or inside it.

The candidate set has to respect the user’s permissions from the start. Otherwise, a helpful assistant becomes a leak with friendly prose on top.

The model is usually blamed last and fixed first

When a RAG answer fails, the model gets blamed, because the model produced the visible text. That makes sense emotionally. It makes less sense architecturally. The model can only work with the context it receives, and if that context is stale, conflicting, low-authority, or incomplete, it’s choosing between bad options.

Changing models is sometimes useful, and so is changing embedding models, but those are rarely the first production fixes. The first fixes are usually boring:

clean the corpus
add metadata
separate source types
track document status
use hybrid retrieval
rerank candidates
evaluate retrieval before generation
refuse when evidence is weak

Those are information-management problems. Building retail intelligence has made that concrete for me. The model makes data easier to ask about, but it does not remove the need to decide what each answer means or whether the evidence is strong enough to support it.

Make the right evidence easier to use

RAG works when the system makes the right evidence easy to retrieve and the wrong evidence hard to use. That takes engineering work outside the model: product decisions about authority, freshness, permissions, and refusal, plus content hygiene and evals that inspect the whole pipeline instead of just the prose at the end.

That is what I have found useful about working on Tapestry. The interesting part is not getting a model to produce a sentence about retail data. It is building enough structure around the question that a retailer can trust the sentence enough to use it.