3:6
Retrieval-augmented generation (RAG) is a technique for improving large language model responses by first retrieving relevant information from an external knowledge source and inserting it into the model's prompt before generating an answer. Instead of relying purely on what the model memorized during training, RAG lets an application ground its responses in up-to-date, private, or domain-specific documents.
An LLM's knowledge is fixed at the point its training data was collected, and it has no inherent awareness of private company documents, recent events, or anything outside its training set. Asking a model directly about such information typically results in either a refusal or a confidently wrong guess (a hallucination). RAG addresses this by supplying the model with relevant source material as part of the prompt, so it can answer based on real, current data rather than relying solely on memorized parameters.
A typical RAG system has two phases. In the indexing phase (done ahead of time), documents are split into chunks, converted into vector embeddings, and stored in a searchable index. In the query phase (done per user request), the user's question is embedded the same way, the most relevant chunks are retrieved by similarity search, and those chunks are inserted into the prompt sent to the LLM alongside the original question.
1. User asks a question
2. Embed the question into a vector
3. Search the vector index for the most similar document chunks
4. Insert those chunks into the prompt as context
5. Send the augmented prompt to the LLM
6. Return the generated answer to the userSource documents are usually too large to fit entirely into a model's context window, so they are split into smaller chunks - often a few hundred tokens each, sometimes with overlap between consecutive chunks so that information near a chunk boundary isn't lost. Chunk size is a meaningful design decision: chunks that are too small lose surrounding context, while chunks that are too large dilute the relevance of the retrieved content and waste context window space.
Each chunk of text is converted into a numeric vector, called an embedding, using an embedding model. Texts with similar meaning end up close together in this vector space, even if they don't share the same exact words. At query time, the user's question is embedded the same way, and a vector database or index is searched for the chunks whose embeddings are closest to the question's embedding, typically using cosine similarity.
query_vector = embed(user_question)
results = vector_index.search(query_vector, top_k=5)
context_chunks = [r.text for r in results]Once relevant chunks are retrieved, they are combined with the original question into a single prompt, often with explicit instructions to answer only using the provided context and to say so if the answer isn't contained in it - reducing the chance of the model falling back on unrelated memorized knowledge.
Answer the question using only the context below.
If the answer is not in the context, say you don't know.
Context:
{retrieved chunk 1}
{retrieved chunk 2}
Question: {user question}RAG and fine-tuning solve related but different problems. Fine-tuning adjusts a model's weights to change its behavior, tone, or specialized skills, and is relatively expensive to update. RAG doesn't change the model at all - it changes what information the model has access to at request time - which makes it much cheaper to keep current: updating the knowledge base is as simple as re-indexing new or changed documents, with no retraining required.
By: Tomas Silny
Edited: 2026-08-13 06:49:58