What is retrieval-augmented generation (RAG)?
RAG is a technique where an AI model first retrieves relevant passages from your own documents, then uses them to write its answer. Responses are grounded in your data instead of the model’s memory, which reduces hallucinations, keeps answers current without retraining and lets the system cite its sources.
How RAG works, step by step
- Ingest and chunk: your documents, such as policies, manuals, tickets and web pages, are cleaned and split into passages.
- Embed and index: each passage is turned into a vector that captures its meaning and stored in a searchable index.
- Retrieve: when someone asks a question, the system finds the passages most relevant to it.
- Augment: the question and the retrieved passages are placed together in the model’s prompt.
- Generate and cite: the model writes an answer from those passages and points to the sources it used.
RAG vs fine-tuning vs prompting
| Approach | Best for | Strengths | Limits |
|---|---|---|---|
| Prompting only | General tasks and quick prototypes | Fastest and cheapest to start | No knowledge of your private or recent data |
| RAG | Answers from changing company knowledge | Current, citable, no retraining | Quality depends on retrieval and content |
| Fine-tuning | Style, format or specialized behavior | Consistent tone and task skill | Costly to update, not ideal for facts |
When RAG is the right choice
- Your answers live in documents that change, such as policies, product docs and support articles.
- You need sources shown so people can verify answers.
- Different people should see different information based on permissions.
- You want to update knowledge by changing documents, not by retraining a model.
Common pitfalls
- Poor chunking that splits answers across passages or buries them in noise.
- Weak retrieval that returns plausible but irrelevant text.
- No evaluation, so quality problems go unnoticed.
- Stale or contradictory source documents.
- Ignoring permissions, so users see content they should not.
- Prompt injection: instructions hidden in documents that try to hijack the model.
Security and privacy
Treat a RAG system like any system that exposes company data. Enforce access control at retrieval time, so a user can only retrieve what they are allowed to read. Redact sensitive fields where possible, keep data in the region and environment your policies require, and log queries and sources for review. Choose model providers whose terms keep your data out of public model training.
A practical checklist to start
- Pick one narrow, valuable use case and one set of trusted documents.
- Write 30 to 50 real questions with known good answers.
- Build a baseline and measure it against those questions.
- Improve chunking, retrieval and prompts, re-measuring each time.
- Add citations, guardrails and a hand-off to a person for low-confidence answers.
- Monitor quality, cost and latency in production.
Key takeaway: RAG reduces hallucinations, it does not remove them. Measure answer quality on real questions before and after launch.
Frequently asked questions
Does RAG eliminate hallucinations?
No. It reduces them by grounding answers in retrieved text, but models can still misread or over-generalize. Citations, guardrails and evaluation are what keep quality high.
What data can RAG use?
Documents, web pages, knowledge bases, tickets, wikis and database records can all be indexed, as long as they are cleaned, current and you have the right to use them.
Do I need a vector database?
A vector index is central to most RAG systems, but it does not always need a dedicated product. Many projects run well on PostgreSQL with pgvector, and larger systems may use specialized vector databases.
How do you evaluate a RAG system?
Build a test set of real questions with known good answers and measure retrieval quality, answer accuracy, citation correctness, latency and cost, then track them over time.