RAG Tutorial: Build a Chatbot on Your Own Data (Step by Step)

Every enterprise has a mountain of private knowledge: internal wikis, legal contracts, product manuals, research reports. A stock LLM — even GPT-4o — can't touch any of it. You could fine-tune a model on that data, but fine-tuning is expensive, slow to update, and notoriously bad at memorising facts. Retrieval Augmented Generation (RAG) is the pragmatic alternative: fetch the relevant passages at query time and hand them to the LLM as context. The model doesn't need to "remember" anything — it just needs to read and reason.

By the end of this tutorial you'll have a working RAG pipeline that can answer questions from any PDF you throw at it. We'll go phase by phase: indexing, retrieval, generation, then evaluation — the part most tutorials skip.

What is RAG and Why It Matters

Large language models are trained on a massive public snapshot of the internet, frozen at a cutoff date. They have no awareness of your company's Q3 earnings call, last week's policy update, or the 200-page specification doc that lives in Confluence. When you ask a vanilla LLM about private data, it does one of two things: it hallucinates a plausible-sounding answer, or it correctly admits it doesn't know. Neither is useful.

RAG patches this gap in three steps:

1. Retrieve — Given the user's question, find the most relevant text passages from your document corpus using semantic (vector) search.
2. Augment — Inject those passages into the LLM's prompt as context, alongside the question.
3. Generate — Let the LLM produce an answer grounded in the retrieved evidence.

This means the LLM acts as a reasoning engine, not a knowledge store. Updating your knowledge base is as simple as re-indexing a new document — no retraining required.

RAG vs. Fine-Tuning — Which Should You Use?

Criterion	RAG	Fine-Tuning
Knowledge update speed	Minutes — re-index the doc	Days–Weeks — retrain
Factual grounding / citations	✅ Source chunks are attached	❌ Hard to attribute
Cost	Low — only inference + vector DB	High — GPU compute for training
Best for	Dynamic private knowledge bases	Style, tone, domain behaviour
Hallucination risk	Lower (bounded by retrieved context)	Higher for obscure facts
Document volume limit	Essentially unlimited	Bound by training data size

RAG Architecture: Three Phases

Every RAG system — regardless of framework — consists of two offline phases and one online phase:

1. Indexing (offline) — Load documents → split into chunks → embed each chunk → store embeddings in a vector database. This runs once (or whenever your corpus changes).
2. Retrieval (online) — Embed the user's query → do approximate nearest-neighbour search in the vector DB → return the top-k most similar chunks.
3. Generation (online) — Stuff the retrieved chunks into a prompt → call the LLM → stream the answer back to the user.

Here are the six building blocks you need to implement those phases:

Phase 1: Document Indexing

Let's start coding. Install the dependencies first:

bash
pip install langchain langchain-openai langchain-community langchain-chroma \
            pypdf chromadb python-dotenv tiktoken

Create a .env file at your project root:

.env
OPENAI_API_KEY=sk-your-key-here

Now build the indexing pipeline. This script loads a PDF, chunks it, embeds the chunks, and saves everything to a local ChromaDB vector store on disk:

Python
# indexer.py — run this once to build the vector store
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

load_dotenv()

# ── 1. Load PDF ──────────────────────────────────────────────────────────
# PyPDFLoader splits the PDF into one Document per page automatically.
# Replace the path with your own file.
loader = PyPDFLoader("docs/company_handbook.pdf")
pages  = loader.load()
print(f"Loaded {len(pages)} pages")

# ── 2. Split into overlapping chunks ─────────────────────────────────────
# chunk_size=1000 characters ≈ 250 tokens — safely under most LLM context limits.
# chunk_overlap=200 ensures sentences that straddle a boundary aren't lost.
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],  # hierarchy: paragraphs → sentences → words
    length_function=len,
)
chunks = splitter.split_documents(pages)
print(f"Split into {len(chunks)} chunks")

# ── 3. Embed + persist to ChromaDB ───────────────────────────────────────
# OpenAIEmbeddings uses text-embedding-3-small by default (1536-dim, very cost-effective).
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",   # saved to disk — reuse without re-indexing
    collection_name="company_handbook",
)
print(f"Indexed {vectorstore._collection.count()} chunks into ChromaDB ✓")

After running indexer.py, you'll see a chroma_db/ folder appear. Inside is a SQLite file holding all your chunk embeddings. For a 100-page PDF you can expect roughly 300–500 chunks at these settings.

Phase 2: Smart Retrieval

Retrieval is where most RAG pipelines silently fail. The default similarity search is good — but not always good enough. Let's look at the two main strategies.

Basic Similarity Search (k=4)

Re-open the persisted vector store and run a similarity search:

Python
# retrieval_demo.py
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings  = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings,
    collection_name="company_handbook",
)

# Similarity search — returns top-4 most similar chunks
query   = "What is the company's remote work policy?"
results = vectorstore.similarity_search(query, k=4)

for i, doc in enumerate(results, 1):
    print(f"\n── Chunk {i} (page {doc.metadata.get('page', '?')}) ──")
    print(doc.page_content[:300])  # preview first 300 chars

# With scores (lower cosine distance = better match)
scored = vectorstore.similarity_search_with_score(query, k=4)
for doc, score in scored:
    print(f"Score: {score:.4f} | {doc.page_content[:80]}...")

Why k=4? Retrieving 4 chunks typically gives the LLM ~1000–1500 tokens of context — enough signal without bloating the prompt. Going higher increases cost and can actually hurt quality by introducing noisy chunks that confuse the model (the "lost in the middle" effect).

MMR: Maximum Marginal Relevance for Diversity

Standard similarity search can return near-duplicate chunks — all from the same paragraph, just slightly differently worded. MMR balances relevance to the query against diversity among the returned chunks:

Python
# MMR retriever — get diverse, relevant chunks
mmr_retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 4,           # chunks to return
        "fetch_k": 20,    # candidates to consider before MMR re-ranking
        "lambda_mult": 0.6,  # 0=max diversity, 1=max relevance
    },
)

docs = mmr_retriever.invoke("What is the company's remote work policy?")
print(f"Retrieved {len(docs)} diverse chunks")

# Standard similarity retriever (for comparison)
similarity_retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4},
)

Phase 3: Answer Generation

Now we wire retrieval to generation. LangChain's create_retrieval_chain handles the plumbing: it calls the retriever, injects the returned chunks into the prompt as {context}, and calls the LLM — all in one chain.

Building the RAG Chain

Python
# chatbot.py — the full RAG chatbot
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain

load_dotenv()

# ── LLM ─────────────────────────────────────────────────────────────────
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)
# temperature=0 → deterministic, factual answers (ideal for Q&A over docs)

# ── Vector store ─────────────────────────────────────────────────────────
embeddings  = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings,
    collection_name="company_handbook",
)
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 4, "fetch_k": 20, "lambda_mult": 0.6},
)

# ── Prompt template ──────────────────────────────────────────────────────
# {context} will be filled by the retriever's output.
# {input} is the user's question.
system_prompt = (
    "You are a helpful assistant for answering questions about company documents. "
    "Use ONLY the following retrieved context to answer the question. "
    "If the answer is not in the context, say 'I don't have that information in the provided documents.' "
    "Keep your answer concise and factual. Cite the relevant page number when possible.\n\n"
    "Context:\n{context}"
)

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
])

# ── Assemble the chain ───────────────────────────────────────────────────
# create_stuff_documents_chain: joins retrieved docs and passes to prompt
# create_retrieval_chain: orchestrates retriever → doc chain → answer
document_chain  = create_stuff_documents_chain(llm, prompt)
rag_chain       = create_retrieval_chain(retriever, document_chain)

# ── Ask a question ───────────────────────────────────────────────────────
question = "How many vacation days do full-time employees receive?"
response = rag_chain.invoke({"input": question})

print("Answer:", response["answer"])
print("\nSource chunks:")
for doc in response["context"]:
    print(f" • Page {doc.metadata.get('page', '?')}: {doc.page_content[:120]}...")

Streaming the Answer Token by Token

For a chatbot UI, streaming makes the experience feel alive. Replace the invoke call with astream:

Python
import asyncio

async def stream_answer(question: str):
    print(f"Q: {question}\nA: ", end="", flush=True)
    async for chunk in rag_chain.astream({"input": question}):
        # The chain yields intermediate events; filter for the final answer text
        if "answer" in chunk:
            print(chunk["answer"], end="", flush=True)
    print()  # newline after stream finishes

asyncio.run(stream_answer("What is the parental leave policy?"))

Evaluating Your RAG Pipeline

Most developers deploy RAG pipelines without systematic evaluation. They vibe-check 3 questions and ship it. Then users encounter hallucinations and the whole project loses credibility. Don't do this.

The RAGAS framework (Retrieval Augmented Generation Assessment) provides four metric categories that give you quantitative signal on exactly what's broken:

Metric	What it measures	Score range	Tool
Faithfulness	Is the answer factually grounded in the retrieved context? (Detects hallucinations)	0 – 1 (higher = better)	RAGAS
Answer Relevancy	Does the answer actually address the question asked? (Penalises off-topic verbosity)	0 – 1 (higher = better)	RAGAS
Context Recall	Did the retriever fetch all the information needed to answer correctly? (Measures retriever coverage)	0 – 1 (higher = better)	RAGAS
Context Precision	What fraction of retrieved chunks are actually relevant? (Measures retriever noise)	0 – 1 (higher = better)	RAGAS

Running RAGAS evaluation is straightforward once you have a test dataset:

Python
# evaluate.py
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)

# Build an evaluation dataset: questions + ground-truth answers
eval_data = {
    "question": [
        "How many vacation days do employees get?",
        "What is the remote work policy?",
        "How do I submit an expense report?",
    ],
    "ground_truth": [
        "Full-time employees receive 20 vacation days per year.",
        "Employees may work remotely up to 3 days per week with manager approval.",
        "Expense reports must be submitted within 30 days via the Concur portal.",
    ],
}

# Collect RAG pipeline outputs for each question
answers, contexts = [], []
for q in eval_data["question"]:
    result = rag_chain.invoke({"input": q})
    answers.append(result["answer"])
    contexts.append([d.page_content for d in result["context"]])

eval_data["answer"]   = answers
eval_data["contexts"] = contexts

# Run RAGAS evaluation
dataset = Dataset.from_dict(eval_data)
results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
)
print(results.to_pandas())

Taking It to Production

A local Chroma store and a single Python script are fine for prototyping. Here are the four key levers you pull when moving to production:

RAG is not a one-and-done setup. The gap between a working prototype and a production-grade RAG system is wide — it lives in the evaluation loop. Build your evals first, then optimise. Every architectural change (different chunker, different retriever, different model) should be validated against your golden question set before it goes to users.