Skip to content

Platform · Control

Guardrails and human review: the gate every action passes

The Control layer of the Agent Engine, built into every system we deliver rather than sold as a separate product. No single control makes a language model safe; together these reduce what can go wrong and limit what a mistake can do. Every input is treated as untrusted, every consequential action waits for a named person, and an unsure answer goes to your team with the transcript instead of a guess.

Book a Strategy CallGet an AI Plan

The question matches published sources well. The model drafts a reply from the retrieved passages, and the output check confirms that every citation resolves to a retrieved passage, links are on the allowlist, prices match published ones and no personal data is echoed back. Then it is sent.

  • Evidence found and cited
  • Output check passed
  • Sent and logged with its sources

The problem

Why this layer exists

Guardrails are easier to judge once the threats have names. These are the four we design against in every build, and the controls that answer each one.

Threats and the controls that answer them
ThreatWhat it looks likeMain controls
Direct prompt injectionA user types instructions meant to override the system: reveal your prompt, ignore your rules, pretend to be staffA fixed router path for injection patterns, instructions kept apart from user text, output checks
Indirect prompt injectionInstructions hidden in a web page, email or document the system reads, aimed at the model rather than the human readerRetrieved text treated as data, scanning at ingestion and at retrieval, a tool allowlist so a missed injection has little to call
Data leakageA reply that includes another customer’s details, an internal page or text from the system promptPermission filters inside retrieval, session-scoped tools, redaction, a canary check for prompt leaks
OverreachThe system acts beyond its brief: promises a refund, quotes an unpublished price, sends a message nobody approvedApproval gates, published-price checks, allowlisted links, refusal rules for out-of-scope requests

No single control stops all four. The design goal is depth: an injection that slips past the scanner meets a tool allowlist, a call that passes the allowlist meets an approval gate, and a reply that passes both meets the output check.

How it works

The mechanisms, and their alternatives

  1. Mechanism 01

    Untrusted by default

    User messages, uploaded files, retrieved passages and tool results are all data. They sit in clearly delimited parts of the prompt, never spliced into instructions, and are scanned for text that addresses the model directly. Passages flagged at ingestion stay out of the index; passages flagged at retrieval are dropped from that answer and recorded. The system prompt carries a canary string, and any reply containing it is blocked as a prompt leak.

    What's the alternative? Rely on a stronger system prompt

    Telling the model to ignore instructions found in documents helps, and we include it. A prompt is not a security boundary, though: it lowers the odds of an attack working rather than removing them, which is why allowlists and approvals sit behind it.

  2. Mechanism 02

    Least privilege for every tool

    Each step can propose only the tools on its allowlist, each with a schema, a read or write scope and a credential that can do nothing else. A support assistant’s payment credential can create refunds and nothing more; it cannot read card data or change prices. If an attacker does steer the model, the damage is bounded by what that step was allowed to touch.

    What's the alternative? One admin key for everything

    A single powerful credential is quicker to set up and turns any successful injection or bug into full access. We do not ship it, and we ask for named, revocable roles in your systems instead.

  3. Mechanism 03

    Approvals set by consequence

    Every tool is classified when the build is designed. Reversible, low-stakes actions such as tagging a ticket or looking up an order run freely. Customer-visible or costly actions such as refunds, cancellations, outbound messages and data deletion wait for a named approver, who sees the proposed call, the conversation and the evidence, and can approve, edit or reject. Rejections become evaluation cases.

    What's the alternative? Auto-approve below a threshold

    Approving small refunds automatically cuts waiting time and is a sound rule once a flow has a track record. We launch with approvals and agree thresholds with you from the review history, rather than guessing them on day one.

  4. Mechanism 04

    Checks on the way out

    Every reply passes an output validator before it is sent: citations must resolve to retrieved passages, prices must match published ones, links must be on the allowlist, contact details must be yours, claims your business cannot make (certifications, guarantees, promised outcomes) are blocked, and card and ID numbers are redacted. A reply that fails is replaced with a safe fallback and logged.

    What's the alternative? A second model as the judge

    Asking another model to grade each reply catches tone and subtle errors that rules miss, and we use it in evaluation runs. Inline on every reply it adds cost and delay and is itself non-deterministic, so live checks are rules first, with model review on a sample.

  5. Mechanism 05

    Handoff with the transcript

    Low confidence, a complaint, a sensitive topic, a payment dispute or a plain “can I talk to someone?” hands the conversation to your team. The handoff carries the transcript, the detected intent, the sources the system looked at and any record it concerns, and lands where your team already works, such as the helpdesk or a shared queue, with a reference the visitor sees too.

    What's the alternative? Forward the chat to an inbox

    Emailing the transcript is easy and works at low volume. Without a queue nobody owns the case, nothing tracks whether it was answered, and the context arrives as a wall of text rather than a record.

  6. Mechanism 06

    Personal data removed early

    Card numbers (checked with the Luhn algorithm), government ID numbers, passwords, one-time codes and API keys are redacted before a message is stored or sent to a model. Transcripts are kept for the period you set, and visitors can reset or delete a conversation. For health and other regulated data, the build is HIPAA-aware and its data path is reviewed before launch.

    What's the alternative? Clean the logs afterwards

    Scrubbing stored logs protects storage but not the model provider, which has already received the data. Redaction runs before the model call, so sensitive values never leave the boundary at all.

Boundaries

What we will not automate

Some decisions stay with people in every build, whatever the confidence score says.
  • Decisions that deny someone a service, a claim, a refund or a job. The system can gather and summarise; a person decides.
  • Medical, legal or financial advice to an individual. Assistants share published information and route the rest to qualified staff.
  • Sending to many recipients at once without a person approving both the message and the list.
  • Deleting records, closing accounts or changing who has access to what.
  • Anything a regulator, a contract or your own policy reserves for a named person.

These boundaries are written into the scope and the tool configuration, so they do not depend on the model remembering them.

Defaults

Defaults we ship

  • Every input, document and tool result treated as untrusted data.
  • Injection scanning at ingestion and at retrieval, with flagged passages dropped and recorded.
  • A canary string in the system prompt; any reply that contains it is blocked.
  • Tool allowlists per step, with credentials scoped to one job each.
  • Named approval for refunds, cancellations, deletions and outbound messages.
  • An output validator on every reply: citations, prices, links, contact details and claims.
  • Redaction of card numbers, ID numbers, passwords and keys before storage or a model call.
  • Handoff with transcript, intent and sources, plus a reference the visitor can quote.

Limits

Limits, stated plainly

  • No combination of controls makes a language model immune to prompt injection. These lower the chance and the damage, and adversarial cases in every evaluation run test them.
  • Rule-based output checks catch what they are written for. New kinds of risky claim need new rules, which come from reviewing transcripts.
  • Handoff works only if someone picks it up. Staffing the queue and its hours is part of your rollout plan, and the assistant states those hours honestly.
  • Redaction catches structured identifiers well and free-text personal details less well; for regulated data we narrow what is collected in the first place.
  • Approval gates move risk to the approver. Reviewers need the evidence in front of them and time to read it, or approval becomes a rubber stamp.

Configuration

What the configuration looks like

control.policy.yaml: an illustrative Control-layer policy (Sample data, not a client file).
# control.policy.yaml  (Sample data: illustrative)
inputs:
  treat_as_untrusted: [user_message, retrieved_passage, tool_result, upload]
  injection_scan: { at_ingest: true, at_retrieval: true, on_hit: drop_and_flag }
handoff:
  when: [low_confidence, asked_for_human, complaint, payment_dispute, sensitive_topic]
  attach: [transcript, intent, sources, related_record]
  destination: helpdesk_queue
  tell_visitor: "Someone from the team will reply during business hours."
approvals:
  required_for: [issue_refund, cancel_order, send_bulk_message, delete_record]
  approvers: [support_lead, ops_manager]
output_checks: [citations_resolve, published_prices_only, allowed_links_only,
                own_contact_details_only, no_certification_or_outcome_claims]
redact_before_model: [card_number, government_id, password, otp, api_key]
retention: { transcripts_days: 30, visitor_can_delete: true }

FAQ

Frequently asked questions

Can you promise the assistant will never say anything wrong?

No, and we would be wary of anyone who does. Language models can misread a correct source or phrase something badly. The controls on this page make errors less likely, keep consequential actions behind a person and make every answer traceable to its sources, so a mistake can be found, corrected and turned into a test case.

Who on our side reviews approvals and handoffs?

People you name during scoping: usually a support lead for refunds and replies, and an operations owner for anything that touches orders or accounts. Each queue has an owner and a backup, and the approval screen shows the proposed action with the conversation and evidence beside it. Training those reviewers is part of the rollout.

How does the system decide it is not confident enough to answer?

From the retrieval evidence, not from the model’s own sense of certainty. It looks at how much of the question the top passages cover and how strong the best match is, plus fixed triggers such as a complaint or a request for a person. The thresholds are tuned on your evaluation set and re-checked whenever they change.

Can we choose topics that always go to a person?

Yes. Topics, intents and keywords can be set to hand over every time, for example billing disputes, cancellations, medical questions or anything from a named account. Those rules live in configuration, so changing them needs no new prompt, and the evaluation set gains cases for each one.

Do the guardrails make replies slower?

Slightly. Rule-based scans and output checks add little next to a model call, and many messages skip the model entirely because the router answers them on a fixed path. Approvals add real waiting time, which is why they are kept for actions with consequences, and the visitor is told when a person needs to look.

What happens to a passage flagged as an injection attempt?

It is left out of that answer, recorded with the document it came from and shown to the content owner in the admin view. If it came from a source you control, the fix is usually to remove or rewrite it; if it came from an outside page, the source can be blocked. Repeated hits from one source are a signal to review how it is ingested.

Next step

Have a workflow in mind? Let’s scope it together.

Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.

Book a Strategy CallGet an AI Plan

Free 30-minute call · no commitment

Send an RFP or security questionnaire