Skip to content

Resources · Reference architecture

Inbound voice agent: a reference architecture

The design we start from for an AI agent that answers your business line, books or looks things up through your systems, and puts the caller through to a person with a summary when it should. It is a reference design, not a client system. Cost per minute and consent rules live in the voice agent buyer’s guide; the Twilio behaviour this page relies on is cited with the date we checked it.

Book a Strategy CallGet an AI Plan

Layer: all four, centred on Reach in real time

Sample data
Caller on the lineYour number on TwilioDone
Media streamAudio both waysRunning
Speech to textStreaming, end of turnDone
Model turnTool: calendar lookupRunning
Text to speechCleared on barge-inDone
Warm transferSummary firstIdle
Front deskTakes the callIdle
Call recordTranscript, outcomeIdle
The caller’s audio arrives over a media stream and is transcribed as it streams. At the end of the turn the model replies, calling a calendar tool, and text to speech plays the answer back on the same stream, where barge-in can clear it. A warm transfer to the front desk waits as a branch, and the call record is written when the call ends.

The problem

What this architecture solves

A phone call is the least forgiving channel. The caller cannot see a typing indicator, talks over the agent, mumbles a date and hangs up after a few seconds of silence. An inbound voice agent is a loop that runs once per caller turn, under a tight time budget, with a person one transfer away. Most of the engineering goes into turn-taking, interruptions and the transfer, not the model.

  • The caller hears a reply quickly enough that the call never feels broken.
  • Talking over the agent stops its speech at once.
  • Anything outside the agent’s job reaches a person with a summary, never a cold transfer.
  • Every call leaves a record: transcript, outcome, consent and what was booked.

Components

Components and the options for each

  • Telephony and number

    Answers your number, applies business-hours and fallback rules, and opens a media stream to the agent.

    • Twilio Programmable Voice
    • Your existing number, ported or forwarded

  • Media stream bridge

    A WebSocket service holding the call’s live audio in both directions; it can clear queued audio the moment the caller speaks.

    • Twilio bidirectional Media Streams
    • Twilio ConversationRelay (Twilio handles speech)

  • Speech to text

    Streams partial transcripts while the caller talks, tuned for names, dates and your product terms.

    • A streaming speech-to-text model
    • Speech handled by the relay

  • Turn detector

    Decides that the caller has finished, rather than paused mid-sentence, from silence and the partial transcript.

    • Silence plus transcript rules
    • Model-based end of turn

  • Dialogue model and tools

    Holds the call state, answers from approved content and calls narrow tools: find a slot, book it, look up a booking.

    • OpenAI
    • Anthropic
    • Google Gemini

  • Text to speech

    Turns each reply into audio in short chunks, so the first words play while the rest is generated.

    • A streaming voice model
    • Speech handled by the relay

  • Transfer rule

    Code that decides when a person takes over: the caller asks, an emergency word, a repeated misunderstanding or a task outside scope.

    • Warm transfer with a summary
    • Callback task in your CRM

  • Call record

    Transcript, outcome, consent answer and booking reference, kept for your retention period.

    • Supabase
    • Cloudflare D1
    • Your CRM via its API

Data flow

How data moves, step by step

  1. 01The call reaches your number; business-hours rules decide whether the agent answers.
  2. 02The agent says it is an AI assistant, and the media stream starts.
  3. 03Speech to text streams partial transcripts while the turn detector waits for the end of the turn.
  4. 04The dialogue model replies, calling a tool such as a slot lookup when it needs one.
  5. 05Text to speech plays the reply in chunks; if the caller talks over it, queued audio is cleared and the loop listens again.
  6. 06When the transfer rule fires, the agent tells the caller and puts them through with a summary.
  7. 07When the call ends, the record is written and any confirmation goes out on the channel the caller agreed to.

How it works

Design decisions

  1. Mechanism 01

    Interruptions stop the agent

    Callers talk over agents all the time. The bridge keeps listening while the agent speaks; when the caller starts, it sends Twilio’s clear message, which empties the buffered audio, and the model receives the interruption as part of the next turn.

    What's the alternative? Ignore speech while the agent talks

    Simpler, and callers end up shouting over a voice that will not stop.

  2. Mechanism 02

    Short replies, streamed

    Delay adds up across every stage. Replies are kept to a sentence or two, the first audio chunk plays while the rest is generated, and a slow tool gets a spoken filler (“let me check that”) instead of silence.

    What's the alternative? Generate the whole reply, then speak

    Easier to log, and the pause before every answer makes the call feel broken.

  3. Mechanism 03

    Transfer is a rule, not a mood

    The model can suggest a transfer, but code decides: an explicit request, an emergency word, two failed attempts at the same detail, or anything outside the task list. The person hears a short summary before the caller is connected.

    What's the alternative? Let the model decide when to transfer

    Polite and inconsistent: some callers loop, others are transferred for no reason.

Reference diagram

The whole design on one diagram

Drawn to the same legend as our other reference architectures, so you can compare them box for box.
Inbound voice agent · reference diagramReference design

An incoming call opens a media stream; speech to text and a turn detector pass each complete turn to the dialogue model, which calls booking tools on your calendar and sends reply text to text to speech for the caller. A transfer rule hands the call to the front desk with a summary, and the call record keeps transcript and outcome.

Diagram with 12 parts: Incoming call (source), Media stream (processing), Speech to text (model), Turn detector (control), Dialogue model (model), Booking tools (processing), Text to speech (model), Transfer rule (control), Your calendar (system of record), Caller (human), Front desk (human), Call record (system of record).

Connections:

  • Incoming call → Media stream (call audio)
  • Media stream → Speech to text (caller audio)
  • Speech to text → Turn detector (partial text)
  • Turn detector → Dialogue model (complete turn)
  • Dialogue model → Booking tools (tool call)
  • Booking tools → Your calendar (booking)
  • Dialogue model → Text to speech (reply text)
  • Dialogue model → Transfer rule (transfer intent)
  • Text to speech → Caller (audio)
  • Transfer rule → Front desk (summary + call)
  • Your calendar → Call record (booking reference)

How to read these diagrams

Source· in this diagram
Where a request or data enters: a chat widget, a WhatsApp message, a phone call, a webhook, a schedule or an approved document set.
Processing· in this diagram
Ordinary code we write and test: ingress endpoints, routing, retrieval, queues, workers and tool calls. It behaves the same way every time.
Model· in this diagram
A language, speech or embedding model call. Its output is treated as a draft and checked before it can reach a customer or a system.
Control· in this diagram
A gate that can stop, hold or reroute work: input checks, citation checks, send rules, transfer rules and dead letters.
Human· in this diagram
A customer or someone on your team. An arrow into a person is the answer, the escalation or the approval path.
System of record· in this diagram
Where state lives and survives a restart: your CRM, helpdesk or calendar through its API, a search index, a log.
  • Arrows run in the order data moves: left to right, or top to bottom on a phone.
  • A label on an arrow names what travels along it, such as a question, an event id or a ticket.
  • Named products are options from our build stack; “your … via its API” means the system you already run.
  • Each diagram is a reference design, not a drawing of any client’s system. Component choices are made per project.
Read the diagram as text
  1. Incoming call → Media stream (call audio)
  2. Media stream → Speech to text (caller audio)
  3. Speech to text → Turn detector (partial text)
  4. Turn detector → Dialogue model (complete turn)
  5. Dialogue model → Booking tools (tool call)
  6. Booking tools → Your calendar (booking)
  7. Dialogue model → Text to speech (reply text)
  8. Dialogue model → Transfer rule (transfer intent)
  9. Text to speech → Caller (audio)
  10. Transfer rule → Front desk (summary + call)
  11. Your calendar → Call record (booking reference)

Platform

Platform capabilities this design uses

Build or buy

Build, buy or combine

Hosted voice platforms put a demo on the phone in an afternoon. The question is what you need to control after the demo.
Build, buy or combine: when each option fits and what it costs you
OptionFits whenThe trade-off
A hosted voice-agent platformOne use, standard voices and languages, and you accept its console, its transcript export and its terms for your number.Bundled pricing, little control over turn-taking and transfers, and data in the vendor’s system.
Build on this reference designYou need your own tools, transfer rules or languages, or call data kept in your accounts.More parts to run, and measuring delay becomes your job.
Telephony relay plus your logicTwilio ConversationRelay handles speech; you build only the dialogue, tools and transfer.Fewer parts, less choice of voice and speech models.

Rollout

How we roll it out

Typical sequence for a first build; the weeks are typical ranges, set per project at scoping.

  1. Week 1

    Call audit and script

    With consent, review real calls, list the top call reasons, and write the disclosure and transfer rules.

  2. Weeks 1–3

    The turn loop

    Stream bridge, speech, end of turn and barge-in, timed stage by stage on test calls.

  3. Weeks 3–4

    Tools and transfer

    Booking tools with read-back, the warm transfer and the call record.

  4. Weeks 4–5

    Shadow, then after hours

    Staff call in with scripted scenarios; then the agent answers outside office hours only.

  5. After launch

    Daytime overflow

    Answer when the front desk is busy, tuned from reviewed calls.

Vendor facts on this page, with the date we checked

  1. Media Streams overview (unidirectional and bidirectional streams) (opens in a new tab) · Twilio Docs · accessed
  2. Media Streams WebSocket messages (clear, mark, dtmf) (opens in a new tab) · Twilio Docs · accessed
  3. TwiML Voice: <ConversationRelay> (opens in a new tab) · Twilio Docs · accessed

Failure modes

What fails, and how it is handled

A long silence before the reply
Every stage is timed in the call record; slow tools trigger a spoken filler, and turns over budget show up in review.
The caller is cut off mid-sentence
End of turn waits longer after words that usually continue a sentence, tuned per line from reviewed calls.
A speech or model provider fails
The call falls back to a recorded message and a transfer or voicemail; it never rings out.
Names, dates or numbers misheard
The agent reads back anything it will act on (“Tuesday at three, is that right?”) and books only after a yes.
Nobody picks up the transfer
The agent comes back to the caller, takes a message and creates a callback task.
The caller presses keys instead of speaking
Keypad presses arrive as DTMF events on the stream and are handled as input.

Running cost

What drives the running cost

Drivers, not prices: vendor prices change, so we quote them per project with the date we checked.
Connected minutes
Telephony is billed per minute and so is the stream or relay; shorter calls cost less at every layer.
Speech in and out
Usually priced by audio time or characters; the relay option bundles both into one rate.
Model tokens per turn
Small per turn but multiplied by turns per call; a long prompt is paid for on every turn.
Transfers
A warm transfer usually opens a second call leg billed at the same time.
Recording and storage
Optional; recordings need consent and storage, and transcripts are often enough.

Defaults

Defaults we ship

  • The agent says it is an AI at the start of every call.
  • Barge-in clears the agent’s audio as soon as the caller speaks.
  • Read back and confirm before any booking or change.
  • Transfer rules live in code, with a summary for the person taking over.
  • Recording stays off unless consent is asked for and given; transcripts mask card numbers.
  • A fallback message and voicemail when any provider fails.

Limits

Limits and variants

  • Phone audio is narrowband, so accents, noise and spelling out names are harder than in chat; the read-back step stays.
  • Card payments are kept away from the agent: they go to a compliant payment flow or a person.
  • Long or emotional calls belong with people; the agent is built to transfer early.
  • Variant, speech relay: the telephony provider handles speech and the agent exchanges text, trading voice choice for fewer parts.
  • Variant, after hours only: the same design while the office is closed, with callbacks instead of live transfers.

Configuration

What the configuration looks like

Voice agent configuration (Sample data)
line: main-reception
answer:
  hours: outside-office-hours      # or: always
  disclosure: "You are speaking with our AI assistant."
stream:
  mode: bidirectional
  barge_in: clear-buffered-audio
turn:
  end_of_turn_silence_ms: 700
  max_reply_sentences: 2
  filler_after_ms: 1200
tools: [find_slot, book_slot, lookup_booking]
confirm_before_action: read-back
transfer:
  triggers: [asked-for-person, emergency-words, repeated-misunderstanding, out-of-scope]
  target: front-desk
  no_answer: take-message-and-callback
recording: off
record_fields: [transcript, outcome, consent, booking_ref]

FAQ

Frequently asked questions

What happens when a caller talks over the voice agent?

The agent stops. The bridge listens while the agent speaks, clears the queued audio as soon as the caller starts, and passes what they said into the next turn, so the conversation carries on from the interruption.

How does the agent know the caller has finished speaking?

A turn detector combines the length of the silence with the partial transcript: a pause after “I’d like to book for” waits longer than a pause after a full sentence. The setting is tuned for each line from reviewed calls.

What does the person taking a transferred call hear first?

A short summary before the caller is connected: who is calling, what they want and what the agent has already done, such as the slot it offered. The caller is told they are being put through, so nobody repeats themselves.

Where does the delay before each reply come from in this design?

From stages in series: end-of-turn detection, the model’s first words, any tool call and the first chunk of speech. We time each stage on every turn and stream the first sentence while the rest is generated; the buyer’s guide has the full latency budget.

Can we start with after-hours calls only?

Yes, and it is the rollout we suggest. The agent answers while the office is closed, books what it can confirm and leaves callbacks for the rest; daytime overflow comes once reviewed calls show it handles your top call reasons.

Does the agent need our booking system to have an API?

For live booking, yes: the tools find and hold slots through your calendar or practice system’s API. Without one, the agent takes a structured request and creates a callback task, and a person confirms the booking.

Next step

Have a workflow in mind? Let’s scope it together.

Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.

Book a Strategy CallGet an AI Plan

Free 30-minute call · no commitment

Send an RFP or security questionnaire