RAG Explained: A Chatbot That Answers From Your Own Documents

A general-purpose chatbot has read much of the public internet and knows nothing whatsoever about your company's refund policy, your fee structure, or the clause on page 14 of your standard contract. Retrieval-augmented generation — RAG — is the standard technique for closing that gap, and it is behind most of the "AI that knows our business" systems being sold today.

The idea in one paragraph

Instead of hoping the model happens to know your facts, you search your own documents for the passages most relevant to the question, paste those passages into the prompt, and instruct the model to answer using only what it was given. The model contributes fluency and reasoning; your documents contribute truth. When it does not find an answer, it should say so.

How it is actually built

Step 1: Collect and clean the source material

PDFs, Word files, spreadsheets, website pages, past support conversations. This step is boring and decisive — a system built on three conflicting versions of the same policy will confidently give three different answers. Deduplicate first, and mark which document wins when two disagree.

Step 2: Split documents into passages

Documents are cut into chunks of a few hundred words, ideally along real boundaries: sections, headings, table rows. Splitting mid-sentence or mid-table is the most common cause of nonsense answers we are called in to fix.

Step 3: Create embeddings

Each passage is converted into a vector — a list of numbers representing its meaning — and stored in a vector database. This is what lets a search for "can I get my money back" find a paragraph titled "Refunds and cancellations" even with no shared keywords.

Step 4: Retrieve at question time

The user's question is embedded the same way, the closest passages are retrieved, and the best results are placed in the prompt. Serious systems combine vector search with ordinary keyword search, because keyword matching still wins for product codes, names and numbers.

Step 5: Generate with citations

The model answers from the supplied passages and returns which document each claim came from. Citations are not decoration: they are how staff verify an answer, and how you keep trust when the system is occasionally wrong.

Where RAG systems go wrong

• Garbage sources. Outdated policies in the index produce outdated answers, delivered with total confidence.
• Bad chunking. Tables and forms need different handling from prose. Treating them alike breaks numeric answers.
• Retrieving too much. Stuffing twenty passages into the prompt raises cost and, counter-intuitively, lowers accuracy. Retrieve fewer, better passages.
• No refusal path. If the passages do not contain the answer, the correct output is "I could not find this — here is who to ask", not a plausible invention.
• Ignoring permissions. If salary documents are in the same index as the staff handbook, your chatbot has just become a data leak. Filter retrieval by the user's access rights.
• Never re-indexing. Documents change. Without a scheduled refresh, the assistant slowly drifts out of date.

What it costs to run

For a typical small or mid-sized organisation in Pakistan: a one-time build in the PKR 200,000–700,000 range depending on integrations and access rules, then roughly PKR 5,000–30,000 per month covering model usage, the vector database and hosting. Costs scale with questions asked, not documents stored, so a wide knowledge base with modest traffic stays cheap.

Is it worth it for your organisation?

RAG pays for itself when three things are true: staff or customers repeatedly ask the same questions, the answers exist in writing somewhere, and finding them currently takes real time. If your knowledge lives only in one senior person's head, fix that first — write it down. The AI cannot retrieve what was never recorded.

We build these systems for clients from our office in Kohat, and we teach the underlying techniques in our Data Science & AI course. If you want to scope one for your organisation, send us the details and we will tell you honestly whether it is worth building.