What Is RAG (Retrieval-Augmented Generation) and How It Works

LLMs only "know" the data they were trained on — they have no idea about your company's internal documents, the latest data, or your private knowledge base. RAG (Retrieval-Augmented Generation) solves this: relevant information is "retrieved" and given to the model before it generates an answer.

The 2 Parts of a RAG Pipeline

1. Retrieval — finding documents/chunks relevant to the user's question from your knowledge base.
2. Generation — feeding the retrieved information into the prompt, and having the LLM generate an answer based on that context.

Embeddings — the Core of Retrieval

Text is converted into an "embedding" (an array of numbers) that represents its semantic meaning. Texts with similar meaning end up with similar embeddings — this allows "meaning-based" matching instead of just "keyword" matching.

from openai import OpenAI  # or any other embedding provider

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I request a refund?"
)

vector = response.data[0].embedding  # a list of numbers

Storing and Searching in a Vector Database

Each document is embedded and stored in a vector database (like Pinecone, Weaviate, or pgvector). When a user asks a question, its embedding is computed and the database is searched for the "closest matching" documents.

results = vector_db.query(
    vector=query_embedding,
    top_k=3  # the 3 most relevant chunks
)

Putting Retrieved Context into the Prompt

context = "\n\n".join([doc.text for doc in results])

prompt = f"""Answer the question based only on the information below.
If the answer isn't in the information, say "I don't know."

Information:
{context}

Question: How do I request a refund?"""

Now the model answers from your actual documents, not just its "general knowledge" — this significantly reduces hallucination (made-up, incorrect answers), and keeps the data always up to date (just update the documents — no need to retrain the model).

Key Takeaways

• RAG = Retrieval (finding relevant data) + Generation (using that data to produce an answer).
• Embeddings enable semantic (meaning-based) search, not just keyword matching.
• RAG lets a model give accurate answers using private/updated data — without retraining.