RAG is probably the most over-hyped and under-understood pattern in AI right now.
The short version: RAG lets you give an LLM access to your own documents at query time, without fine-tuning. You retrieve relevant chunks, shove them into the prompt, and let the model answer.
Simple in theory. Surprisingly subtle in practice. Most tutorials show you the happy path — embed some PDFs, query a vector store, get a clean answer — and skip the part where you have to decide whether RAG was even the right call, and the part where your retrieval quietly degrades three weeks after launch and nobody notices until a user complains.
The three problems RAG actually solves
Problem 1: The context window limit
LLMs have a fixed context window. Even with the larger windows available today, you can’t dump an entire knowledge base — a wiki, a codebase, a support ticket archive — into every prompt. It’s slow, it’s expensive, and past a certain point the model starts losing track of what’s actually relevant buried in the middle of a huge context. RAG lets you be selective: retrieve only the handful of chunks that matter for this specific query, and leave the rest of the knowledge base untouched until it’s needed.
Concretely, this is the difference between paying for (and waiting on) 50,000 tokens of context on every single call versus paying for 1,500 tokens of the 5 chunks that are actually relevant. At scale, that’s not a minor optimization — it’s the difference between a system that’s usable and one that times out.
Problem 2: Stale training data
The model doesn’t know what happened after its training cutoff, and it doesn’t know anything that was never public in the first place — your internal runbooks, your product’s actual pricing tiers, last week’s incident postmortem. RAG pulls from a live data source you control. Your docs update, the answers update, with zero retraining involved. This is the single biggest reason teams reach for RAG over fine-tuning: the knowledge changes faster than any retraining cycle could keep up with.
Problem 3: Hallucinations on proprietary data
A model trained on the public web knows nothing about your internal systems, and when you ask it anyway, it doesn’t say “I don’t know” — it confabulates something plausible-sounding instead. RAG grounds the model’s answers in your actual documentation, and critically, it lets you show which document the answer came from. That citation is often more valuable to the end user than the answer itself, especially in any domain where someone might need to verify the source before acting on it.
When RAG is the wrong tool
This is the part most RAG tutorials skip entirely, and it’s the part that actually matters once you’re past the demo stage.
Your knowledge base is small and mostly static. If what you’re grounding against is a handful of FAQ documents or a product spec that changes twice a year, you don’t need a vector database, an embedding pipeline, and a retrieval step. You need a system prompt with the content pasted directly into it. RAG infrastructure has real operational cost — something to deploy, monitor, and keep in sync — and that cost only pays for itself once the knowledge base is too large or too dynamic to paste into a prompt by hand.
The task is single-turn and latency-sensitive. RAG adds a retrieval round-trip before generation even starts: embed the query, search the vector store, re-rank if you’re doing it properly, then build the prompt. That’s real latency stacked on top of whatever the LLM call itself takes. For a chat interface where a user is willing to wait a couple of seconds, that’s fine. For an autocomplete-style feature or anything in a tight request path, it often isn’t — and there’s no version of RAG that makes that retrieval step instant.
The task is structured extraction or classification, not open-ended Q&A. If you’re pulling specific fields out of an invoice, classifying support tickets into five known categories, or summarizing a fixed-format document, you don’t have a retrieval problem — you have the entire input already in front of you. Bolting RAG onto this just adds a search step that searches across documents you weren’t even going to use. A targeted prompt against a smaller, faster model — or in some cases no LLM at all, just a deterministic parser — beats RAG on cost, latency, and reliability for this category of task every time.
You can’t easily evaluate whether retrieval succeeded. This is the failure mode that’s genuinely hard to debug, and it’s worth calling out on its own. When a RAG answer is wrong, there are two completely different possible causes: the retrieval step pulled the wrong chunks, or the retrieval was fine and the generation step misused good chunks. Without instrumentation that logs exactly what was retrieved for a given query, you can’t tell which one happened — which means you can’t fix it either. If you’re not prepared to build that observability, you’re signing up for a system you can’t debug when it inevitably gets something wrong in front of a user.
RAG shines when:
- Your knowledge base is large and changes often enough that re-prompting by hand isn’t sustainable
- Queries are unpredictable — you genuinely can’t pre-write a prompt for every case in advance
- You need citations or source attribution as part of the answer, not just the answer itself
Common ways RAG pipelines fail in practice
Once you’ve decided RAG is the right tool, most of the actual difficulty is operational, not architectural. A few failure modes show up constantly:
Chunking strategy chosen arbitrarily. A fixed character-count chunk size (e.g. “every 500 characters”) will cut sentences, tables, and code blocks in half at random points, and the embedding for a half-sentence is not a useful embedding. Chunking should respect document structure — split on headings, paragraphs, or logical sections — and chunk size should be tuned against your actual content, not copied from a tutorial that was grounding against a different kind of document entirely.
Embedding model mismatch between indexing and querying. Whatever model embedded your documents at index time has to be the same model embedding the query at search time — same model, same version, same dimensionality. Swap the embedding model later (a common move when chasing better quality or lower cost) and every existing vector in your store is now incompatible with new query embeddings, silently. Nothing crashes. Retrieval quality just quietly gets worse, and there’s no error message pointing at the cause.
No retrieval evaluation, only end-to-end evaluation. Teams often only test “does the final answer look right,” which conflates retrieval quality with generation quality. Without a separate retrieval-only test set — a list of queries with known correct source chunks — you have no way to catch a regression in the retrieval step specifically, and you’ll waste time tuning prompts to compensate for a search problem.
Stale indexes. Source documents change. If nothing re-embeds and re-indexes the updated content, the vector store keeps serving the old version indefinitely, and Problem 2 from the section above — the entire reason most teams adopted RAG in the first place — quietly stops being true. A RAG pipeline without a defined re-indexing trigger (on a schedule, on document change, or both) is a pipeline that degrades by default, not one that stays correct by default.
RAG vs. fine-tuning: a question you’ll get asked directly
If you’re weighing RAG against fine-tuning for the same problem, that’s usually a sign you’re dealing with a knowledge gap rather than a behavior gap — and the deeper breakdown of that distinction is worth reading before you commit to either path. The short version: RAG is almost always the right first move when the model needs access to facts it doesn’t have. Fine-tuning earns its cost when the problem is how the model formats or structures its output, not what it knows.
What you’ll build in Path 1
Project 3 in Path 1 is a full RAG pipeline on AWS — OpenSearch Serverless for the vector store, Bedrock for embeddings and generation. You’ll set up the execution-role permissions your retrieval service needs to actually call OpenSearch and Bedrock, then hit every tradeoff above with real infrastructure: chunking decisions that affect real retrieval quality, an embedding model choice you have to live with, and a re-indexing step you have to design yourself.