Designing AI agents with guardrails: patterns, permissions and when not to use one

What counts as an agent

Anthropic’s engineering guide to building effective agents draws a useful line. In a workflow, models and tools are wired together along code paths you define in advance. In an agent, the model directs its own process and decides which tool to use next. Both are useful, and the same guide recommends finding the simplest solution possible and adding multi-step agents only when simpler approaches fall short, because autonomy brings higher cost and the risk of compounding errors.

That gives a ladder of autonomy. Climb it one rung at a time, and only when the rung below cannot do the job.

Terms used in this guide

• Tool: a function the model can ask to run, such as “look up an order”. The model proposes the call; your code decides whether it runs.
• Scope: what a tool and its credentials are allowed to read or change. Narrow scope is the strongest guardrail you have.
• Approval gate: a pause where a person sees the exact action and its context and approves, edits or rejects it.
• Step budget: the most steps, tool calls, time and spend a single run may use before it stops and hands over.
• Untrusted content: anything the agent reads that did not come from you, including emails, web pages, documents and tool results.
• Excessive agency: the OWASP name for an agent holding more functionality, permission or autonomy than its task needs.

The pattern catalogue

In Anthropic’s terms every row except the tool-using agent is a workflow, although orchestrator-workers sits close to the line because the lead model decides how to split the work. Start at the top of the table; many business tasks never need to go past the router.

Guardrails, layer by layer

No single guardrail is enough, and the model is never one of them. Prompts that say “never refund more than the policy allows” are instructions, not controls; they can be ignored, misread or overridden by injected text. The controls live in your code and your systems’ permissions. This guide uses six layers, in this order.

Layer 1: scope the tools, not the prompt

Give the agent the fewest tools that complete the task, split read tools from write tools, and give each tool its own narrowly scoped credential. Constrain parameters in code: the refund tool accepts an order reference and an amount, checks the amount against the order and the policy, and can only pay back to the original method. The OWASP guidance on excessive agency says the same in general terms: minimise extensions and their permissions, require user approval for high-impact actions, and enforce authorisation in the downstream system rather than trusting the model.

agent: support-inbox
model_can_call:
  get_order:        { access: read,  scope: orders:read,  risk: low }
  search_policy:    { access: read,  scope: kb:read,      risk: low }
  draft_reply:      { access: write, scope: helpdesk:draft, risk: low }   # drafts only, never sends
  create_refund:
    access: write
    scope: payments:refund_original_method
    risk: high
    approval: required             # a person approves every call
    limits: { max_amount: order_total, per_day: 20 }
    idempotency_key: "ticket_id + order_id"
not_available: [send_email_external, change_customer_details, issue_credit]
budget: { max_steps: 15, max_tool_calls: 10, max_minutes: 10, max_spend_usd: 0.50 }
on_budget_exhausted: handoff_with_summary

Layer 2: treat everything the agent reads as data

Indirect prompt injection is the agent-era version of a phishing email: instructions planted in content the model will read later. Researchers demonstrated it against real LLM-integrated applications, where planted text led to data theft and manipulated behaviour. Simon Willison describes the most dangerous combination as a “lethal trifecta”: access to private data, exposure to untrusted content, and the ability to communicate externally. An agent with all three can be talked into sending your data to an attacker. The practical rule is to break at least one leg. Our support agent reads untrusted email and private order data, so it has no tool that can send anything outside the helpdesk, and every outbound action passes a person.

Layer 3: approval gates on actions that are hard to undo

Rank every write tool by how costly a mistake would be and how easily it can be reversed. Drafting is low risk; updating a record is medium; moving money, deleting data or contacting a customer is high. High-risk calls stop at a gate that shows the approver the exact action, its arguments, the evidence the agent used and anything a filter flagged. The approver can approve, edit or reject, and the decision is logged with their name. If nobody responds, nothing happens: a gate that times out into “approved” is not a gate.

Layer 4: budgets and stop conditions

Every run gets a ceiling on steps, tool calls, wall-clock time and spend, and a check for repetition, such as the same tool called with the same arguments twice in a row. When a limit is reached, the agent stops and hands over with a summary of what it found and what it was about to do. Budgets turn the failure modes of autonomy, loops and runaway costs, into a routine handoff.

Layer 5: validate outputs before they touch a system

Check every tool argument and every structured output against a schema and against business rules before execution: the order exists, the amount is within policy, the customer on the ticket owns the order. Reject and log anything that fails, and let the agent try once more with the error message, or hand over.

Layer 6: log every step for review

Record the plan, each tool call with its arguments and result, each filter flag, each approval and who gave it, and the final outcome, in one trace per run. Review a sample of traces on a fixed schedule. Logs are how you find the pattern of near-misses before it becomes an incident, and they are what you show a customer or an auditor who asks what the system did.

Worked example: the support-inbox agent

The run log at the top of this page, read as a design. It is Sample data, not a client system.

1. Pattern choice: refunds depend on what the order lookup finds, so a fixed chain is not quite enough. A tool-using agent with four tools is the smallest pattern that works.
2. Reads first: the agent looks up the order and the written policy with read-only credentials.
3. Injection handled as data: the ticket contains an instruction to refund everything to a new account. The input filter flags it, the agent keeps it as customer text, and the refund tool could not pay a new account anyway.
4. One proposal: a partial refund to the original payment method, within the policy.
5. Gate: the refund is a high-risk write, so the run pauses until the support lead approves.
6. Safe execution: the refund call carries an idempotency key built from the ticket and the order, so a retry cannot pay twice.
7. Close: the reply is written from the approved outcome, and the run ends well inside its budget with a complete trace.

Design checklist

• You tried the patterns below an agent on the ladder and can say why each falls short.
• Each tool is the narrowest that works, with its own scoped credential; read and write tools are separate.
• Limits on amounts, recipients and records are enforced in code and in the downstream system, not in the prompt.
• Untrusted content is labelled as data; the agent cannot combine private data, untrusted input and external sending without a person.
• Every write tool has a risk rank; high-risk calls stop at an approval gate that times out to “do nothing”.
• Each run has caps on steps, tool calls, time and spend, plus a repetition check, and hands over with a summary.
• Tool arguments and outputs are validated against schemas and business rules before execution.
• Writes carry idempotency keys, so retries cannot repeat an action.
• One trace per run records plans, calls, flags, approvals and outcomes, and someone reviews a sample regularly.
• A test set of tasks, including injection attempts and edge cases, runs before every prompt, tool or model change.