Reliable workflow automation: retries, idempotency and dead letters

What reliable means for an automation

A reliable automation is not one that never fails. Every workflow that talks to other people’s APIs will see timeouts, rate limits, expired tokens and data it did not expect. Reliable means three things hold anyway: no item is lost, no item is processed twice in a way anyone can see, and when a person is needed, the right person learns which item, why, and what to do. Everything in this guide serves one of those three.

Terms used in this guide

• Idempotent: an operation that has the same effect whether it runs once or several times. “Set status to paid” is idempotent; “add a payment” is not.
• Idempotency key: an identifier sent with a write so the receiving system can recognise a repeat of the same request and return the first result instead of acting again.
• At-least-once delivery: the sender retries until it hears back, so the receiver may see the same event twice. Most webhooks work this way.
• Transient failure: one that may succeed if tried again later, such as a timeout or a rate limit. Permanent failure: one that will fail the same way every time, such as invalid data.
• Backoff with jitter: waiting longer after each failed attempt, with a random spread so many clients do not retry at the same moment.
• Dead-letter store: where an item goes when it cannot be processed automatically, kept with everything needed to fix and replay it.

Start with a failure inventory

Before building anything, write down every step of the workflow and, for each one, how it can fail, whether that failure is transient or permanent, and what the workflow should do about it. It is a short exercise, and it changes the design: you discover which writes need keys, which steps need a deadline and who owns each kind of failure. The table below is the starting list we use.

Make every write safe to repeat

Assume every event can arrive more than once and in any order. Stripe’s own webhook documentation says endpoints may receive the same event more than once and that delivery order is not guaranteed, and recommends logging processed event IDs and skipping ones already handled. That is the first defence: a table of processed event IDs, checked and written in the same transaction as the work.

The second defence is an idempotency key on every write that creates or changes something. Stripe’s API saves the result of the first request made with a key and returns that saved result to any retry with the same key, and it rejects a reuse of the key with different parameters. An IETF draft proposes the same pattern as a standard Idempotency-Key header for any HTTP API. The detail that matters is where the key comes from.

Derive the key from the business event, not the attempt

A random identifier generated on each attempt defeats the purpose: the retry carries a new key, so the target sees a new request. Build the key from what makes the operation unique in business terms (the source system, the record, the operation and the record version) and store it before the first attempt, so a retry, a replay next week and a duplicate webhook all produce the same key.

import { createHash } from 'node:crypto'

/** Same business operation → same key, on every retry and every replay. */
export function idempotencyKey(op: { source: string; recordId: string; action: string; version: number }) {
  const raw = [op.source, op.recordId, op.action, op.version].join(':')
  return createHash('sha256').update(raw).digest('hex').slice(0, 48)
}

/** Process an event at most once: claim its ID in the same transaction as the work. */
export async function handleOnce(db: Db, event: { id: string }, work: (tx: Tx) => Promise<void>) {
  await db.transaction(async (tx) => {
    const claimed = await tx.run(
      'INSERT INTO processed_events (event_id, received_at) VALUES (?, ?) ON CONFLICT DO NOTHING',
      [event.id, new Date().toISOString()],
    )
    if (claimed.changes === 0) return // duplicate delivery: already handled
    await work(tx)
  })
}

When the target API has no idempotency support

• Upsert on a natural key, such as email address or order number, instead of creating blindly.
• Write your own key into a field on the target record, and search for it before creating.
• Use an outbox: record the intended write in your own database in the same transaction as the business change, and let a worker send it and mark it done. A crash between the two steps leaves a pending row, not a lost or doubled write.
• Where none of these is possible, make the step a human approval rather than an automatic retry.

Retry only what can succeed later

Retries fix transient failures and make permanent ones worse. Retrying a validation error only fills logs and delays the alert. So classify first: retry timeouts, server errors and rate limits; never retry invalid data, missing records or permissions that were refused. Space retries with exponential backoff and random jitter, cap the number of attempts and give the whole step a deadline.

Decide where retries happen. Amazon’s Builders’ Library article on timeouts and retries recommends retrying at a single point in the stack, because retries at every layer multiply: three layers that each try a few times can turn one slow database into a flood. Google’s SRE book adds a retry budget per process and deadlines that are passed downstream, so work for a request that has already been abandoned stops.

step: crm_upsert
deadline: 120s              # give up and park the item after this, whatever the attempt count
retry:
  max_attempts: 5
  backoff: exponential      # 2s, 4s, 8s, 16s …
  base_delay: 2s
  max_delay: 60s
  jitter: full              # random delay between 0 and the computed backoff
  honour_retry_after: true  # a 429 with Retry-After waits that long instead
retry_on: [timeout, connection_reset, 429, 500, 502, 503, 504]
never_retry: [400, 401_after_refresh, 403, 404, 409, 422]
on_exhausted: dead_letter   # never drop, never loop forever

The same rules in n8n, Make or Zapier

Low-code platforms give you the parts; you still have to assemble them. In n8n, a node’s Retry On Fail setting reruns it when an execution fails, so limit it to nodes that call flaky services and keep it off nodes that write without a key. The node’s On Error setting can continue using an error output, which routes the failed item down a branch you design instead of stopping the run. A separate error workflow that starts with the Error Trigger node receives the execution ID, its URL, the error and the last node executed, which is exactly what an alert needs. Make and Zapier have their own error-handling features, so check what each step supports; whatever the tool, the classification table above decides which failures take which route.

Dead letters: park, explain, replay

When an item fails permanently or runs out of retries, it goes to a dead-letter store. It must never be dropped. This is a real risk with defaults: Cloudflare’s Queues documentation notes that without a dead-letter queue configured, messages that reach the retry limit are deleted permanently. A dead-letter store also needs an owner and a retention period, or it becomes a quiet archive of lost work.

• The original payload, untouched, and the step it failed at.
• The error class, the message from the target system and the attempt count.
• First and last failure times, and the run link that shows the full execution.
• The idempotency key, so a replay is recognised as the same operation.
• What already happened before the failure, such as “contact created, deal not created”.
• A status: new, being fixed, replayed or abandoned, with who changed it.

Replay is a first-class operation, not a copy-and-paste job. It re-runs the item from the failed step, with the same idempotency key, after someone has fixed the cause. Because the key is unchanged, a replay after a partial success finishes the work instead of repeating the half that already happened.

Alerts a person will act on

An alert that fires for every failed attempt trains people to ignore alerts. Alert on outcomes instead: an item reached the dead-letter store, a step has been failing for longer than its deadline, or a scheduled workflow did not run at all. That last one matters more than it looks; a workflow that silently stopped triggering produces no errors, only absence.

• Name the workflow, the item and the business impact in the first line (“lead not in CRM”, not “HTTP 422”).
• Include the run link and the dead-letter record, so nobody has to search logs.
• Say what already happened and what did not.
• Give one next action and who owns it.
• Page out of hours only for money and customer-facing failures; batch the rest into a daily digest.

Worked example: lead routing under failure

The diagram above, step by step. It is an illustration, not a log from a client system.

1. A website form posts a webhook. The form tool does not hear back in time and sends it again.
2. The workflow claims the event ID in its processed-events table; the second copy finds the ID already claimed and stops.
3. Enrichment times out. The step backs off, retries and succeeds on a later attempt, inside its deadline.
4. The CRM rejects the phone number format. That is a permanent error, so there is no retry.
5. The item is parked in the dead-letter store with the payload, the CRM message, the run link and the note “enriched, not in CRM”.
6. The sales-operations owner gets one alert with the link and the fix: the form sends local numbers without a country code.
7. The owner adds the country-code rule to the mapping and replays the item. The upsert carries the same idempotency key, so exactly one contact exists afterwards.

Reliability checklist

• A failure inventory exists for every step, with a class and a handling rule.
• Inbound events are deduplicated by event ID in the same transaction as the work.
• Every create or change carries an idempotency key derived from the business operation.
• Steps without idempotency support use upsert, a stored external key, an outbox or a human approval.
• Only transient failures are retried, at one layer, with exponential backoff, jitter, an attempt cap and a deadline.
• Exhausted and permanent failures go to a dead-letter store with an owner and a retention period, and replay uses the original key.
• Alerts fire on outcomes, include the run link and one next action, and respect severity.
• A heartbeat alert fires when a scheduled workflow has not run.
• Input is validated at the first step, so a schema change fails loudly instead of writing blanks.
• The failure paths were tested before launch: duplicate delivery, a forced timeout, a forced validation error and a replay.