Knowledge and retrieval: answers that show their sources
The Know layer of the Agent Engine, the architecture every build of ours runs on rather than a product you sign up for. Your documents are split with their source and version attached, searched by exact words and by meaning at the same time, and each answer points back to the passages it used. When a policy changes, the old version leaves the index in the same run that adds the new one.
Yes, if it is unworn with its tags. Outerwear follows the standard return window in the current returns policy [1]. Gift orders have a longer window, but yours was not marked as a gift [2]. Shall I open a return request?
[1] Returns policy v3 · Eligibility
[2] Gift orders · Return windows
Visitor
Do you price-match other stores?
Assistant
I could not find a price-match policy in the store’s published pages, so I will not guess. I can pass this to someone on the team with our chat so far.
Keyword search matched “jacket” in the catalogue, vector search matched “send back” to the returns policy, rank fusion kept the strongest passages, and both chips point at retrieved text.
The problem
Why this layer exists
A language model on its own answers from what it absorbed in training. That knows nothing about your returns policy, this quarter’s price list or the clause your legal team rewrote last month. Retrieval-augmented generation (RAG) changes where the answer comes from: before the model writes anything, the system finds the passages in your own material that bear on the question, and the model is told to answer from those passages only.
Most RAG demos stop there. Most RAG failures in production happen after it, in four places:
Wrong passage: the question names a plan, part number or error code that meaning-based search blurs into something similar.
Stale passage: last year’s policy still sits in the index beside this year’s, and the model quotes the wrong one.
Leaked passage: an internal page or an unpublished draft reaches a customer-facing answer.
No passage: nothing relevant exists, and the model fills the gap from general knowledge in a confident tone.
The Know layer handles all four before the model is involved. The mechanisms below are what we ship by default; the defaults and limits after them say what that buys you and what it does not.
How it works
The mechanisms, and their alternatives
Mechanism 01
Ingest with provenance
Each source (help-centre pages, policy PDFs, product sheets, internal wikis for internal builds) is converted to text and split on its own headings, so a chunk is a section a person would recognise rather than an arbitrary run of characters. Every chunk carries its document id, source URL and anchor, heading path, version, content hash, publish date and a visibility flag. Those fields are what make citations, filtering and supersession possible later.
knowledge index · chunks with provenanceSample data
returns-policyv3 current · split by heading
eligibilitypublic · hash 9f2c · cited as [1]
refund-timingpublic
exchangespublic
returns-policy@v2superseded · chunks deleted when v3 was ingested
product-catalogsynced on a schedule
outerwearpublic · one chunk per product family
ops-handbookinternal · never used in customer answers
What's the alternative? Fixed-size chunks
Splitting every few hundred tokens is simpler and suits uniform text such as call transcripts. On structured documents it cuts tables and lists in half and produces citations that point into the middle of nothing, so we use it only where a document has no usable structure.
Mechanism 02
Keyword and vector search, fused
Two retrievers run on every question. Keyword search, ranked BM25-style over a full-text index, is exact: it finds “Plan B2”, “ERR-401” or a product code even when nothing else in the sentence matches. Vector search over embeddings finds passages that mean the same thing in other words, such as “send it back” for “return”. Reciprocal rank fusion merges the two ranked lists by position rather than raw score, because the scores are not on the same scale. A cap on passages per document stops one long page from crowding out the short one that holds the answer.
Two rankings, one fused list
Keyword ranking · full-text index
Vector ranking · embeddings
Reciprocal rank fusion, capped per document
Numbered passages for the model
Confidence from coverage and match strength
What's the alternative? Vector search alone
It is the common default and handles paraphrased questions well. It is weakest on the exact strings customers paste in, such as order numbers, model names and error codes, and those are often the questions where a wrong answer costs most.
Mechanism 03
Filter before ranking, not after answering
Visibility, language, product line and, for internal assistants, the asking person’s permissions are applied as filters inside the retrieval query. A passage the user may not see never reaches the model, so it cannot be paraphrased into a reply. For internal builds the permissions come from your identity system or the document store’s own sharing settings, read through its API.
What's the alternative? One index per audience
Separate indexes for customers, staff and managers are easy to reason about, and we use them when the groups are few and stable. With many overlapping groups they multiply storage and re-indexing, and a document shared with two groups has to be kept in step twice.
Mechanism 04
Cite, or decline
Retrieved passages are numbered and handed to the model with an instruction to answer only from them and to mark each claim with its source. The reply is checked on the way out: every citation must resolve to a passage that was actually retrieved, links must be on an allowlist, and a factual claim with no citation counts as unsupported. When retrieval comes back empty or weak, the system says it does not have that information and offers a person instead of letting the model improvise.
What's the alternative? Answer first, cite afterwards
Some pipelines write freely, then search for passages that seem to support the text. It reads well, but the citation becomes decoration: the answer was not built from the source, so a wrong answer can still arrive with a plausible reference.
Mechanism 05
Supersede, never append
Each document has a stable id and a content hash. When a source changes, ingestion writes the new chunks and deletes the old version’s chunks from both the full-text and the vector index in the same run, so the two never disagree. Removed documents are deleted, not hidden. A scheduled drift check compares what is published with what is indexed and tells the content owner when they differ.
What's the alternative? Rebuild everything nightly
A full rebuild on a schedule is simple and always consistent, and for small knowledge bases it is what we do. As the corpus grows it gets slow and costly, and answers lag every edit until the next run, which is too long for prices and policies.
Mechanism 06
Live data stays in its system
Order status, stock, bookings and balances are not copied into the index; a stale copy is worse than none. The system calls the owning system’s API at the moment of the question, through a read-only tool, and the answer cites that lookup the way it cites a document. The orchestration page covers how those calls are allowed and retried.
What's the alternative? Sync records into the index
Copying records on a schedule makes them searchable by meaning, which helps for slow-moving data such as a catalogue, and we do it there. For anything a customer acts on, such as a delivery date, it produces answers that were true yesterday.
Defaults
Defaults we ship
Chunks split on document headings, each carrying source, anchor, version, hash and visibility.
Hybrid retrieval on every question: full-text and vector search merged by reciprocal rank fusion.
A per-document cap on passages, so one long page cannot fill the context.
Citations required on factual claims; a reply citing nothing that was retrieved is blocked.
An honest “I don’t have that” and a human option when retrieval is weak.
New versions replace old chunks in both indexes within one ingestion run.
Retrieved text treated as data: passages carrying injected instructions are dropped and flagged.
Drafts, internal pages and archived documents kept out of customer-facing answers.
Limits
Limits, stated plainly
Retrieval only finds what is written down. If the answer lives in someone’s head or in a scan with no readable text, the first job is documentation, not AI.
Citations make errors checkable, not impossible: a model can still misread a correct passage, which is why the evaluation set tests answers as well as retrieval.
Contradictory sources produce cautious answers. We flag conflicts found at ingestion, but deciding which document is right is your content owner’s call.
Complex tables and charts lose structure in text extraction; they often need a cleaned copy or a structured lookup tool instead.
Permission-aware retrieval is only as accurate as the permissions it reads. Over-shared folders in the source system stay over-shared.
Configuration
What the configuration looks like
knowledge.config.yaml: an illustrative Know-layer configuration (Sample data, not a client file).
# knowledge.config.yaml (Sample data: illustrative values)
sources:
- id: help-centre
type: web
include: ["/help/**"]
split: by_heading
visibility: public
- id: returns-policy
type: pdf
supersede: true # a new version deletes the old chunks
- id: ops-handbook
type: wiki
visibility: internal # never used in customer answers
retrieval:
keyword: { index: full_text }
vector: { index: embeddings, timeout_ms: 4000 }
fuse: reciprocal_rank # k = 60
top_k: 6
max_per_document: 2
answer:
cite: required
when_no_evidence: say_so_and_offer_a_person
freshness:
drift_check: daily
notify: content_owner
Which kinds of documents can the retrieval layer work from?
Anything we can turn into reliable text: web pages, help-centre articles, PDFs with a text layer, Word and Markdown files, wiki exports and structured records such as a product catalogue. Scanned images need OCR first, and the output is checked before it is indexed. Each source is listed in the build scope with its owner and how often it changes.
Why run keyword search at all when vector search understands meaning?
Because customers type exact strings: plan names, part numbers, error codes, clause references. Embeddings place those near similar-looking strings, which is wrong when one character changes the answer. Keyword ranking catches the exact match, vector ranking catches the paraphrase, and fusing the two covers both without a rule that has to guess which kind of question it is.
How soon does an edited policy show up in the answers?
When the next ingestion run for that source finishes. Runs are scheduled to match how often each source changes, from a webhook on publish for a help centre to a daily sync for a catalogue, and a manual re-index is one command. The old version’s chunks go in the same run, so there is no window in which both versions can be cited.
Can the assistant cite a page our customers cannot open?
Not in a customer-facing build. Every chunk carries a visibility flag that the retrieval query filters on, so internal pages and drafts never reach the model for a public answer, and outgoing links are checked against an allowlist of your public URLs. Internal assistants can cite internal sources, limited to what the asking employee may see.
Where do the search index and the embeddings live?
In accounts in your name by default, for example Postgres with vector search on Supabase, or Cloudflare D1 with Vectorize. Embeddings are created by a model provider from our build stack, or by a self-hosted open-weight model when documents must not leave your environment. The deployment options page compares those shapes.
Does retrieval work when customers write in another language?
Yes, with care. Multilingual embedding models can match a question in one language to a passage in another, while keyword search needs the same language on both sides. We include the languages you serve in the evaluation set, and where many customers write in one language we usually index a reviewed translation of the key pages rather than relying on cross-language matching alone.
Is this the same design as the assistant on this website?
Yes. The assistant on miniailabs.in answers only from this site’s published pages, with a per-document cap, required citations, versioned chunks that replace old ones and injection scanning on retrieved text. It fuses a full-text index with vector search when its database index is available, and falls back to keyword ranking over a bundled copy of the pages when it is not. Its releases are checked against a 150-question set across 20 categories.
Next step
Have a workflow in mind? Let’s scope it together.
Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.
