How to build a RAG chatbot, step by step: an assistant that cites its sources

What RAG is, and what it is not

A retrieval-augmented chatbot answers from passages it looks up at question time, not from what the language model happened to memorise. It has three parts: an index you build ahead of time, a retriever that finds the passages most likely to answer the question, and a generator that is told to answer only from those passages and to say which one each claim came from. The idea comes from the original RAG paper, which paired a sequence-to-sequence model with a dense retriever over Wikipedia. A business chatbot keeps the idea and adds the plumbing that makes it safe to put in front of customers: documents with owners and versions, two kinds of search, a rule for when not to answer, and a test set that runs before every change.

RAG is not a way to make a model know everything about your company. It narrows what the model may say to what your approved content says. If the content is wrong, thin or contradictory, the bot will be too, only faster.

Terms used in this guide

• Chunk: a passage small enough to retrieve and cite on its own, such as one FAQ answer or one plan description, stored with the heading it sits under.
• Embedding: a vector that places a passage near other passages with a similar meaning, so a question can find text that uses different words.
• BM25: the classic keyword ranking function. It rewards rare words that appear in both the question and the passage, and it is precise on names, plan titles and error codes.
• Reciprocal-rank fusion (RRF): a way to merge two ranked lists by position rather than by score, so keyword and vector results can be combined without calibrating one against the other.
• Coverage: how much of the question’s distinctive vocabulary the top passages actually contain. Low coverage means retrieval probably missed.
• Grounded answer: a reply whose every claim can be traced to a retrieved passage, marked with a citation such as [1].

The architecture at a glance

There are two pipelines. The offline one turns approved content into versioned chunks on every build, and an ingestion step loads them into the search indexes. The online one answers a visitor and runs on every message. The diagram above shows both: content on the left, the index in the middle, the query-time stages on the right.

• Offline: approved sources → structural chunks with headings → content hashes and versions → a keyword index and a vector index.
• Online: route the message → retrieve with both searches → fuse by rank → check confidence → answer from numbered sources → validate → cite → log.

Stage 1: ingestion from content you approve

The assistant on this site never crawls its own pages. A build script reads the same typed data the pages render from (service pages, published prices, the process, legal pages, concept builds and owner-reviewed guides) and turns it into documents. Crawling looks easier, but it picks up navigation, footers, cookie banners and draft pages, and it cannot tell a current price from an old blog post. Starting from structured data means every chunk has a known source and a known owner.

For a business deployment the sources are usually a help-centre export, policy documents and product sheets. Each enters the index with an owner who can correct it, a visibility and a status. An admin can switch a document to internal or archived, and it drops out of answers within a minute, without a rebuild.

Chunk along structure, not character counts

Our chunker splits each document along its sections: one FAQ answer, one plan, one process step. Only a section that is too long is split again, and then only at sentence boundaries. Each chunk keeps its heading, so a citation can point a visitor at the exact part of the page the answer came from. Fixed-size windows are simpler, but they cut question-and-answer pairs and tables in half, and a half-answer retrieved with confidence is worse than no answer.

Version everything

Every chunk carries a content hash, and every document a version number. When a document’s hash changes, its version goes up and ingestion replaces the old chunks and their search rows in one atomic batch; documents that left the site are deleted. Vector entries are re-checked against the current version at query time, so a stale embedding can never resurface a price that changed last week.

{
  "id": "kd_pricing",
  "title": "Pricing",
  "source_url": "/pricing/",
  "visibility": "public",
  "status": "published",
  "owner": "website",
  "version": 4,
  "content_hash": "…",
  "chunks": [
    {
      "id": "kc_pricing_4_2",
      "heading": "What is billed at cost",
      "content": "Model, telephony and hosting usage are billed at the provider's rates…",
      "content_hash": "…"
    }
  ]
}

Stage 2: hybrid retrieval

Keyword search and vector search fail in different ways. Keyword search finds “WhatsApp Business” or a plan name exactly, but misses a question phrased in words the page never uses. Vector search finds paraphrases, but it can rank a page about a similar topic above the page that actually contains the answer, and it is weak on codes, names and numbers. Using both is the standard fix: Anthropic’s write-up on contextual retrieval reports that embeddings combined with BM25 beat embeddings on their own, and Supabase documents the same pattern inside Postgres.

On this site the keyword side runs BM25 over the bundled chunks in memory, and SQLite FTS5 with its built-in bm25() ranking in Cloudflare D1 once the database holds the current knowledge version. The vector side queries Cloudflare Vectorize with bge-base-en-v1.5 embeddings. Questions are embedded with the model’s search instruction in front of them; passages are embedded without it, which is how that model family is meant to be used.

Fuse by rank, then cap per document

BM25 scores and cosine similarities are on different scales, so adding them is meaningless. Reciprocal-rank fusion ignores the scores and uses positions: each list gives a passage one over the constant plus its rank, and the totals decide the order. A passage that ranks well in both lists rises to the top. After fusion, no single document may contribute more than two chunks, so one long page cannot crowd out the page that holds the precise answer.

export const EMBEDDING_MODEL = '@cf/baai/bge-base-en-v1.5' // 768 dimensions
const QUERY_PREFIX = 'Represent this sentence for searching relevant passages: '
const RRF_K = 60            // reciprocal-rank fusion constant
const MAX_PER_DOC = 2       // chunks one document may contribute
const VECTOR_TIMEOUT_MS = 4000 // fall back to keyword-only if the vector call is slow
const BM25_K1 = 1.2
const BM25_B = 0.75

// Confidence: 'high' when the top-3 evidence covers enough of the question,
// or the best vector match is close enough with some keyword coverage.
export const CONFIDENCE = Object.freeze({ coverage: 0.5, vectorScore: 0.78, vectorCoverage: 0.25 })

function rrf(lists, k = RRF_K) {
  const score = new Map()
  for (const list of lists)
    list.forEach((id, rank) => score.set(id, (score.get(id) ?? 0) + 1 / (k + rank + 1)))
  return [...score.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id)
}

The timeout matters as much as the ranking. If the vector call is slow or the binding is missing, retrieval falls back to keyword search alone rather than making the visitor wait, and citations still work in every mode.

Stage 3: the confidence gate

Most wrong answers start as weak retrieval, not as a model misbehaving. So the decision to answer is made before the model is called. The assistant measures how much of the question’s distinctive vocabulary the top passages contain, weighting rare words more than common ones and treating words the site never uses as a warning sign. When nothing usable comes back, or coverage is low, or most of the question is vocabulary the site does not contain, it replies with a clarifying question and an offer to reach a person. No model call happens, so there is nothing to make up.

A router runs even earlier. Messages that try to override instructions, that contain card numbers or passwords, that ask for a human, or that are plainly off topic get fixed replies from the policy layer. That is cheaper than a model call, it behaves the same way every time, and it keeps the model away from the inputs most likely to cause trouble.

Stage 4: grounded generation and citations

The model receives the retrieved passages as numbered sources and an instruction to answer only from them, marking each claim with its source number. Retrieved text is treated as untrusted data: a chunk that contains instructions is dropped before it reaches the prompt. That is not paranoia. Indirect prompt injection, where an attacker plants instructions in content the model will later read, has been demonstrated against real LLM-integrated applications, and the OWASP Top 10 for LLM Applications lists vector and embedding weaknesses among its risks.

After generation an output validator checks the reply before the visitor sees it: no banned claims, no client results, no prices that are not in the published price list, and no links, email addresses or phone numbers outside an allowlist built from the knowledge itself. Citations are taken from the source markers in the reply; if the model forgot them, the top evidence is cited instead, so every model-written answer arrives with at least one source.

Keep the context short and ordered. Research on long contexts found that models use information at the start and end of the input better than information in the middle, which is one reason this assistant sends no more than six sources, strongest first, rather than everything that matched.

Stage 5: evaluation before every change

The assistant has a fixed regression set of 150 questions in 20 categories: services, pricing, conflicting prices, security, integrations, off-topic chat, direct and indirect injection, personal data, empty retrieval, multilingual questions and more. Each item states what a good reply looks like: which pages should be cited, what the router should decide, whether confidence should be high, whether a handoff should be offered, and strings that must never appear.

• Citation accuracy: replies that cite at least one of the expected pages.
• Source recall: the expected page was among the passages the model saw, even if it cited another.
• Groundedness: model-written answers that carry a citation.
• Unsupported-claim rate: replies the output validator would still have to change. The target is zero.
• Handoff and policy correctness: a person is offered, and fixed replies are used, exactly when they should be.

Offline, the set runs with deterministic stand-ins for the model and the vector store, which makes it fast and repeatable enough to run before any change to retrieval, prompts or thresholds. Online, it runs against a preview deployment with the real model and embeddings before a release. This guide publishes the method only. Measured results for the live assistant will be published with their date and test-set size once the online run is complete.

Worked example: one question, step by step

An illustrative question, not a logged conversation: a visitor asks whether we build WhatsApp chatbots and what one costs.

1. Router: no injection, no personal data, no request for a person. It detects a service and a price signal.
2. Keyword search ranks the chatbot service page high on “WhatsApp” and the pricing page high on “cost”.
3. Vector search adds a passage about channels that never uses the word “WhatsApp” but answers the same need.
4. Fusion puts the chatbot service section first and the pricing section second; each document contributes at most two chunks.
5. Coverage is high, so the model is called with those numbered sources.
6. The reply says yes, names the published starting price and cites [1] and [2].
7. The validator confirms the price string exists in the published price list, then citations and an offer to book a call are attached.

Build checklist

• Every source has an owner, a visibility and a status, and an admin can retire it without a rebuild.
• Chunks follow the document’s structure and keep their headings.
• Documents are versioned by content hash; a changed page replaces its chunks atomically.
• Keyword and vector retrieval both run, fused by rank, with a cap per document and a keyword-only fallback.
• A router handles injection, sensitive data, requests for a person and off-topic messages without a model call.
• A confidence gate asks a clarifying question instead of answering when evidence is weak.
• Retrieved text is treated as data, and replies are validated for prices, links, contact details and claims before display.
• Every answer carries a citation naming the page and the heading it came from.
• A regression set with expected citations runs offline on every change and online before release.
• Conversation logs are stored with an expiry, and a visitor can reset or delete their session.