Orchestration and tools: a state machine around the model
The Reason layer of the Agent Engine, the delivery architecture under each system we build for a client. The model does the parts that need judgement. Everything around it is ordinary code we can test: a router that decides what kind of request this is, specialist steps with their own instructions, tools the model may propose but never run by itself, and retries that cannot charge a card twice.
00:00PLANRouterRouter › Intent: refund_request. No injection markers, no request for a person. Route to the orders specialist.
00:01PLANOrders specialistOrders specialist › Need the order and payment status first. Propose get_order with the order id from the signed-in session.
00:01TOOLRuntimeRuntime › get_order is on this step’s allowlist, read scope, no approval needed. Calling.
00:05ERRORRuntimeRuntime › Order system timed out.
00:06RETRYRuntimeRuntime › Retry 1 of 3 after backoff. Reads are safe to repeat.
00:07OKRuntimeRuntime › Order found: delivered, paid by card, inside the refund window.
00:08PLANOrders specialistOrders specialist › Eligible. Propose issue_refund for the order total, idempotency key refund:A1042:req-7.
00:08APPROVALRuntimeRuntime › issue_refund moves money. Held for a named approver; the customer is told a person is checking.
03:40APPROVEDSupport leadSupport lead › Approved in the review queue with the order and conversation beside it.
03:41TOOLRuntimeRuntime › issue_refund sent with the same key on every attempt; a repeat returns the first result.
03:42DONERuntimeRuntime › Reply drafted from the result, checked, sent. Every step stored in the trace.
The problem
Why this layer exists
Give a model a list of tools and the instruction “handle refunds” and it will usually do something reasonable. Usually is the problem. The same request can take a different path on a different day, a tool can be called with an amount the model inferred rather than read, and when a payment API times out nobody knows whether the refund went through. None of it is visible until a customer complains.
The Reason layer turns that into a flow with fixed edges. The model decides within a step; code decides which steps exist, which tools each step may propose, what a valid argument looks like, what needs a person and what happens when a call fails. Every step writes to a trace, so “why did it do that?” has an answer.
Who decides what inside a request
Decision
Made by
Why there
What kind of request this is
Router: rules first, then a small model
Cheap, fast and testable; fixed paths need no model call
What to say, from which evidence
Specialist step (model)
Needs language and judgement
Which tool to propose, with which arguments
Specialist step (model)
Needs a reading of the conversation
Whether the call is allowed
Runtime policy (code)
Allowlists, scopes and schemas are code, not prompts
Whether a person approves first
Runtime policy (configuration)
Set per tool by consequence
What happens on failure
Runtime (code)
Retries, backoff and idempotency must be deterministic
How it works
The mechanisms, and their alternatives
Mechanism 01
Route first, model second
Every message is classified before any expensive step runs. Greetings, thanks, a request for a person, an off-topic question and anything that looks like an injection attempt follow fixed paths that need no model call. The rest go to a classifier or a small, fast model choosing from a closed list of intents. The router returns a label from that list, never free text, so a new intent is a deliberate change with its own test cases rather than something the model invents.
What's the alternative? One big prompt for everything
A single call with every instruction and every tool is quick to prototype. It gets harder to test with each addition, spends a full model call even on “thanks”, and a fix for one intent quietly changes behaviour on another.
Mechanism 02
Specialists with narrow briefs
Each intent has its own step: its instructions, the retrieval filters it uses, the tools it may propose and the output it must return, usually a structured object rather than prose. A returns specialist cannot see a billing specialist’s tools. Narrow briefs mean shorter prompts, which are cheaper, faster and easier to evaluate one at a time.
What's the alternative? Autonomous multi-agent planning
Letting agents plan and delegate among themselves suits open-ended research. For business processes with known steps it adds cost and makes failures hard to reproduce, so we use it only inside a bounded step, such as researching an account, with a budget on calls and time.
Mechanism 03
Tools from an allowlist, with typed arguments
A tool is a function with a JSON schema, a read or write scope, the one credential it uses and an approval rule. The model may propose a call. The runtime validates the arguments against the schema, checks that the tool is allowed for this step, fills identifiers from the session where it can (the customer’s own order, not any number typed into the chat) and only then runs it. Rejected proposals are logged and the model is told why.
tools visible to the returns stepSample data
read
get_orderorder id taken from the session
search_policiespublic sources only
write
open_return_requestno approval · policy-eligible orders only
issue_refundapproval: named person · idempotency key required
What's the alternative? Hand the model the whole API
Broad API access plus a description of the endpoints makes the fastest demo. It also makes the model’s judgement the only control on what it reads and writes, and that is not a control we will put in front of your customer data.
Mechanism 04
Retries that are safe to repeat
Every call has a timeout. Reads retry on timeouts and rate limits with exponential backoff and jitter. Writes retry only with an idempotency key derived from the request, such as the order and the refund request, sent on every attempt, so the provider or our own write path returns the first result instead of acting twice. After the last attempt the step stops, the customer gets an honest holding reply and the failure goes to the operations queue.
What's the alternative? Retry everything, or nothing
Blind retries on writes are how duplicates happen; no retries turn every network blip into a failed conversation. Deciding per tool, by scope and by whether the downstream API honours idempotency keys, is more work up front and the only version we trust.
Mechanism 05
Approval gates where consequences are real
Tools that move money, message many people, delete data or change an account are marked approval-required. The proposed call, its arguments and the evidence behind it wait in a review queue; a named approver accepts, edits or rejects it, and the decision joins the trace. The guardrails page covers how those gates are set and reviewed.
What's the alternative? Review after the fact
Checking actions after they run keeps things fast and suits low-stakes, reversible writes such as tagging a ticket. For anything irreversible, finding the mistake afterwards is too late, so approval comes first.
Mechanism 06
A model per step, behind one interface
Router, specialists and summarisers can each use a different model, chosen by running the build’s evaluation set for quality, speed and cost. Providers from our build stack (OpenAI, Anthropic, Google Gemini, Mistral AI, or open-weight models such as Meta’s Llama) sit behind one interface with a tested fallback, so a provider outage or a better model is a configuration change plus an evaluation run.
What's the alternative? The largest model for every step
Using the strongest available model everywhere is simple and often the right start. It is rarely the right steady state: smaller models often route and extract as well for less cost and latency, which only an evaluation run on your cases can confirm.
Defaults
Defaults we ship
A closed list of intents, with no-model paths for greetings, handoff requests and injection attempts.
One specialist step per intent, each with its own tools and output schema.
JSON-schema validation of every tool argument before a call runs.
Identifiers filled from the session where possible, never trusted from free text alone.
Timeouts on every call, backoff with jitter on retries and idempotency keys on every write.
Approval required for tools that move money, message many people or delete data.
A full trace per conversation: route, prompt versions, tool proposals, decisions and results.
A fallback model for each step, tested on the same evaluation set.
Limits
Limits, stated plainly
A state machine handles the paths it was designed for. A genuinely new kind of request goes to the fallback path and a person until someone adds it.
Idempotency protects a write only where the downstream system honours a key or lets us check before writing; writes to systems that allow neither go behind approval.
Structured outputs cut parsing errors but do not make the content right; the evaluation set still has to check what the model chose.
Every approval gate adds a wait. We set gates by consequence, because too many of them turn an assistant into a form with extra steps.
Model behaviour changes between versions. Pinning versions and re-running the evaluation set before an upgrade is ongoing operations work, not a one-off.
Configuration
What the configuration looks like
tools/issue_refund.yaml: an illustrative tool definition (Sample data, not a client file).
# tools/issue_refund.yaml (Sample data: illustrative)
name: issue_refund
description: Refund a delivered order, in full or in part.
scope: write
credential: payments_refunds_only # scoped key: refunds and nothing else
approval: required # named approver in the review queue
arguments:
order_id: { type: string, source: session } # never from free text
amount: { type: number, max: order.total }
reason: { type: string, enum: [damaged, not_as_described, late, other] }
idempotency_key: "refund:{order_id}:{request_id}"
timeout_ms: 8000
retry: { attempts: 3, backoff: exponential, jitter: true }
on_final_failure: [send_holding_reply, notify_ops_queue]
Do you build on an agent framework or write the orchestration yourselves?
Either, depending on the build. Often the flow is plain TypeScript or Python, because a state machine with typed tools is not much code and is easy to test. Where a framework saves real work, LangChain from our build stack is the usual choice, and n8n, Make or Zapier when your team wants to edit the flow visually. The mechanisms on this page apply either way.
What stops the model calling a tool with an order number it made up?
The runtime, not the prompt. Arguments are validated against the tool’s schema, and identifiers that belong to the customer, such as their order or account, are filled from the signed-in session or an earlier lookup. A proposal that references something the session does not own is rejected and logged, and the model is asked to try again or hand over.
How many tools can one assistant use?
There is no fixed number, but each step sees only the handful it needs. Long tool lists make models choose worse and make prompts longer and slower, so a build that touches many systems gets more specialist steps rather than one step that can see every tool.
Can it check stock and delivery dates at the same time?
Yes. Independent reads inside a step can run concurrently, and the trace records each result. Writes run one after another, after any approval, so their order is predictable and each can be retried or reversed on its own.
What happens if the model provider has an outage?
Each step has a fallback model behind the same interface, already tested on the evaluation set, and calls move to it after repeated errors or timeouts. If every option fails, the fixed paths still work: the system tells the customer it cannot answer right now and offers a person or a callback.
Can our team see why the assistant chose a particular action?
Yes. Each conversation stores a trace: the route chosen, the specialist and prompt version, the passages retrieved, every tool the model proposed, whether the runtime allowed it, approval decisions with the approver’s name, and each result. Your team reads it in the admin view, and it is the same trace we debug from.
Next step
Have a workflow in mind? Let’s scope it together.
Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.
