Platform · Control

Observability and operations: run logs your team can read

Part of the Control layer of the Agent Engine, which runs inside each client’s own deployment. A system that talks to customers and writes to your tools will sometimes fail: a provider times out, an API rejects a field, a webhook arrives twice. What matters is that the failure is visible, retried safely, parked where a person can see it and replayed without duplicates. We publish no uptime percentage; we describe what we watch, and a custom SLA is available for Enterprise scope.

Book a Strategy Call Get an AI Plan

Sample data

Webhook receivedSignature checkedDone

Outbox jobIdempotency key storedDone

CRM writeAttempt 3 · backoffRetrying

Dead-letter listPayload and error keptFailed

Owner alertedA named personDone

Manual replaySame key, no duplicateNeeds approval

The problem

Why this layer exists

Most AI incidents are ordinary integration incidents. The model call is one step among many: webhooks, lookups, writes to a CRM, emails, payments. When a lead goes missing, the useful question is never “is the AI working?” but “which step failed, for which record, and did anything run twice?”

What we monitor, and who hears about it
Signal	Why it matters	Who is alerted
Failed and retried runs	A rising retry rate is often the first sign of a provider or API problem	The named operations owner
Dead-letter list size and age	Parked jobs are customer requests nobody has finished yet	Operations owner, plus a daily digest
Model and provider errors	Timeouts, rate limits and refusals show up here before customers notice	Your owner, and us while we support the build
Handoffs created and handoffs waiting	A spike means the assistant is unsure about something new; a backlog means nobody is answering	The support lead
Output-check blocks and injection flags	Replies stopped by the validator and flagged passages need a human look	The content owner
Cost per conversation or run	A sudden change usually means a loop, a prompt change or a shift in traffic	The budget owner

Each signal has a named owner and a threshold that is tuned once real traffic arrives. Alerts go to a person, not a shared inbox, through a channel your team already watches.

How it works

The mechanisms, and their alternatives

Mechanism 01

Run-log anatomy

Every conversation and every automation run writes a structured log: a run id, the trigger, each step with an input summary, output summary, duration, attempt number and status, the model and prompt version used, and the idempotency key of each write. Personal data is redacted before anything is logged. One run reads top to bottom as a story, so an operator can answer “what happened to this lead?” without asking an engineer.

Sample data

Run log for one inbound lead
Criterion	Status	Note
09:14:03 webhook.received	ok	Signature verified
09:14:03 dedupe.check	ok: first delivery	Delivery id not seen before
09:14:04 crm.upsert · attempt 1	failed: rate limited	Backoff before the next attempt
09:14:09 crm.upsert · attempt 2	failed: rate limited	Longer backoff
09:14:29 crm.upsert · attempt 3	failed: timeout	Retries exhausted
09:14:30 deadletter.park	parked	Payload, attempts and last error kept
09:14:30 notify.owner	sent	Operations owner, by email
10:02:11 crm.upsert · replay	ok: one contact, same key	Replayed by the operator after the CRM recovered

What's the alternative? Plain-text application logs

Unstructured logs come free with every platform. Engineers who know what to search for can use them; the operations person who actually receives the customer’s complaint cannot.

Mechanism 02
Retries with backoff, only for transient failures
Timeouts, rate limits and a provider’s temporary errors are retried automatically with exponential backoff and random jitter, so a recovering service is not hit by every queued job in the same moment. Permanent failures (a rejected field, a missing record, a revoked credential) are not retried at all. They go straight to the dead-letter list with the error, because repeating them only burns attempts.
What's the alternative? Retry at a fixed interval
Retrying every few seconds is easy to write. When a dependency is struggling, fixed intervals line every job up into the same instant and keep it struggling longer.
Mechanism 03
Dead letters and manual replay
A job that exhausts its retries is parked with its full payload, every attempt and the last error. The owner is alerted and sees it in the admin view with a replay button. Replay re-runs the job with its original idempotency key, so if an earlier attempt did succeed downstream, the replay returns that result instead of creating a second contact, ticket or booking. Jobs can also be corrected and replayed, or discarded with a recorded reason.
What's the alternative? Drop the job after the last retry
Discarding failed jobs keeps queues tidy and loses customer requests silently. Parking them costs a little storage and turns every failure into something a person can finish.
Mechanism 04
An audit log of changes
Admin actions and automated writes are recorded apart from the run logs: who or what made the change, when, what changed and from which screen or step. Prompts, configuration and knowledge are versioned, so any answer can be traced to the exact configuration that produced it.
What's the alternative? Rely on each tool’s own history
Most business tools keep some history, and it helps. It is scattered across systems, expires on different schedules and rarely records which automated step made a change.
Mechanism 05
Analytics without the conversation text
Product analytics record what happened, not what was said: chat opened, source cited, handoff requested, lead submitted, keyed by a hashed session id and carrying no message text. That shows which flows work and where people drop out without copying conversations into an analytics tool.
What's the alternative? Send transcripts to analytics
Full transcripts in an analytics tool make funnels easy to explore and move personal data into a system with different access rules and retention. Transcripts stay in the system of record, under your retention setting.
Mechanism 06
Transcript review on a schedule
Logs show that a run finished, not whether the answer was good. Someone on your team, with our help early on, reviews a sample of conversations on a fixed cadence, weekly at first: handoffs, flagged replies, disputed answers and a random slice. Findings become fixes and new evaluation cases.
What's the alternative? Review only complaints
Complaints are a strong signal and a biased one: most people who get a poor answer leave without saying so. A random slice catches the quiet failures.

Incidents

When something breaks

The same five steps every time, written into the runbook we hand over.

Detect: an alert names the signal, the line it crossed and the runs affected.
Contain: a switch pauses the affected flow or tool, and the assistant falls back to “a person will get back to you” rather than guessing.
Communicate: the owner is told first, and where customers were affected the message to them is agreed with you.
Fix and replay: the cause is corrected, then parked jobs are replayed with their original keys.
Learn: the failing cases join the evaluation set, and a short note records what changed and why.

Response times for each step are agreed per engagement; Enterprise scopes can write them into a custom SLA.

Defaults

Defaults we ship

A structured run log for every conversation and automation, readable by non-engineers.
Redaction before logging, and transcripts kept only for the retention you set.
Automatic retries with exponential backoff and jitter, for transient failures only.
A dead-letter list with payload, attempts and last error, plus correct, replay and discard.
Original idempotency keys reused on replay, so a replay cannot duplicate.
An audit log of admin actions, with versioned prompts, configuration and knowledge.
Alerts to a named owner for each signal, never to a shared inbox.
Analytics events without message text, keyed by a hashed session id.

Limits

Limits, stated plainly

We do not publish an uptime figure. Availability depends on your hosting and on every provider in the path; a custom SLA is available for Enterprise scope.
Monitoring shows that something failed; it cannot fix the system downstream. If a CRM is down, jobs wait in the dead-letter list until it is back.
Alert thresholds need real traffic to tune, so expect some noise in the first weeks while we adjust them with your owner.
Run logs cover what runs through the engine. Steps your team does by hand in other tools are outside the trace unless they are recorded back.
After handover someone on your side owns the alerts. On a monthly plan we share that role; without one, it is yours.

Configuration

What the configuration looks like

ops/alerts.yaml: illustrative alert and retry rules (Sample data, not a client file).

# ops/alerts.yaml  (Sample data: illustrative)
alerts:
  - signal: deadletter.count
    when: "above 0 for 15m"
    notify: ops_owner                 # a named person, with a backup
  - signal: run.retry_rate
    when: "above twice the 7-day median"
    notify: ops_owner
  - signal: handoff.oldest_waiting
    when: "beyond the business-hours target"
    notify: support_lead
  - signal: provider.error_rate
    when: "above 5% for 10m"
    notify: [ops_owner, miniai_support]
retry:
  transient: [timeout, rate_limited, provider_5xx]
  permanent: [validation_error, not_found, auth_revoked]   # straight to dead letter
  backoff: { base_s: 2, factor: 2, max_attempts: 3, jitter: full }
retention: { run_logs_days: 90, transcripts_days: 30, audit_log_days: 365 }

FAQ

Frequently asked questions

Who watches the alerts once the system is live?

A person you name on your side, with a backup. During the post-launch period, and on a monthly plan, we watch them too and fix what is ours to fix. Alerts go to the channel that person already uses, such as email, a team chat or the helpdesk, so they are actually seen.

Can our operations team replay a failed job without a developer?

Yes. The dead-letter list in the admin view shows each parked job with its payload, attempts and last error, and offers replay, correct-and-replay or discard with a reason. Replays reuse the original idempotency key, so a job that partly succeeded before cannot create a duplicate.

Do you log the full text of conversations?

Transcripts are stored in the system of record with personal identifiers redacted, for the retention period you choose, and visitors can delete their own chat. Run logs keep summaries of each step rather than the full text, and analytics events carry no message text at all.

Which monitoring tools does a build use?

It depends on where the system runs. Builds on Cloudflare, Supabase or Vercel use the platform’s own logs plus the engine’s run log and admin views; where your team already runs a monitoring tool, we send structured logs and alerts to it. We avoid adding a new paid tool unless the volume justifies it.

How long are run logs and audit logs kept?

You decide during scoping, usually separately for transcripts, run logs and audit logs, because they serve different purposes. Audit logs are normally kept longest, since they answer who changed what. Expired records are deleted on a schedule, not just hidden.

What does an incident report from you contain?

What was detected and when, which runs and customers were affected, what was paused, the cause, the fix, which parked jobs were replayed and which evaluation cases were added so it cannot quietly return. It is written for your operations and leadership, not only for engineers.

Next step

Have a workflow in mind? Let’s scope it together.

Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.

Book a Strategy Call Get an AI Plan

Free 30-minute call · no commitment

Send an RFP or security questionnaire

Observability and operations: run logs your team can read

Why this layer exists

The mechanisms, and their alternatives

Run-log anatomy

Retries with backoff, only for transient failures

Dead letters and manual replay

An audit log of changes

Analytics without the conversation text

Transcript review on a schedule