In development ·Join early access
Engineering guide

Webhook retries and duplicate events

Design incoming event receipt around signature verification, durable acceptance, duplicate detection, ordering, and separately controlled side effects.

A webhook is a delivery attempt, not a unique business command

Webhook receivers need to assume that delivery can be repeated, delayed, or observed out of order according to the provider’s contract. The useful boundary is durable receipt followed by controlled processing. A successful acknowledgment should mean you can safely continue later, not that every downstream side effect already happened.

This guide concerns incoming event lifecycles. Retrying an outbound request has related ambiguity but a different decision point; the API retries and replay guide covers that broader topic. No webhook runtime is implemented on this marketing site.

Verify before accepting work

Use the provider’s documented signature mechanism and maintained verification library where available. Verification often depends on the original request bytes, a signing key, a timestamp, and the correct endpoint configuration. Reject invalid messages before treating their fields as trusted business input. Authentication does not make the event’s contents safe to render or log without validation.

Stripe’s webhook documentation describes signature verification using the raw request body, duplicate deliveries, and a lack of guaranteed delivery ordering. Those are concrete provider requirements, not proof that every service uses Stripe’s protocol. Check the provider you actually integrate.

Keep webhook signing secrets separate from outbound API credentials. Rotate them according to the documented overlap behavior. Enforce body-size limits before expensive parsing, bound verification work, and avoid dumping rejected payloads into logs. An incoming event should not be able to select an arbitrary tenant’s secret merely by naming that tenant in an unverified field.

Separate receipt from side effects

A conceptual receiver can atomically save an event and an outbox job. The uniqueness key includes the provider account and environment because an event ID may not be globally unique across your entire application.

// Provider-neutral pseudocode. Database primitives are illustrative.
async function receive(rawBody, headers, connection) {
  const event = verifyProviderSignature(rawBody, headers, connection);
  validateEventEnvelope(event);
  const key = [connection.id, connection.environment, event.id];
  await db.transaction(async (tx) => {
    if (await tx.receipts.exists(key)) return;
    await tx.receipts.insertUnique(key, {
      type: event.type,
      payloadRef: await protectedPayloadRef(event),
    });
    await tx.outbox.insertUnique(key, { kind: "process_provider_event" });
  });
  return { status: 200 }; // Durable receipt, not completed side effects.
}

In a real implementation, payload persistence must participate in the durability design: a reference to a file that was never committed is not a durable receipt. A database uniqueness constraint or transactional equivalent handles concurrent duplicates. A process-memory set does not survive restarts or multiple workers.

Respond within the provider’s delivery deadline after durable acceptance. If storage is unavailable, do not acknowledge work you will lose. Document the provider’s retry schedule and retention window rather than assuming delivery continues forever.

Processing needs its own idempotency boundary

A worker claims the outbox item, reads current application state, and applies an authorized transition. Recording “received” is not the same as recording “processed.” If a worker crashes between a remote side effect and its own completion record, simply rerunning the worker can repeat that effect.

For local business changes, use a transaction and an application-specific unique transition key. For a remote operation, preserve a logical action key and use destination idempotency where the destination supports it. If the remote outcome is unknown, reconcile before blindly resending. A unique incoming event does not create universal exactly-once behavior across another API.

Store processing state such as pending, completed, retryable failure, or manual review. Bound attempts and total elapsed time. An operator should see the application transition and downstream action separately from the original delivery attempt.

Handle ordering as a contract decision

A delayed “created” event can arrive after a later update. Do not automatically move application state backward because an older payload arrived last. Options include comparing a provider-supported version, retrieving authoritative current state, or applying a transition table that rejects stale changes.

Each option has costs. Fetching current state adds an outbound dependency and consumes quota. Timestamps can have insufficient resolution or reflect different clocks. Version numbers may exist only for some resources. Explicitly describe what your adapter knows and preserve an unknown state when it cannot safely infer order.

For a fictional account-status workflow, a later authoritative suspension must not be undone by an older activation event. That rule belongs to application policy; the delivery infrastructure cannot infer it from HTTP status codes.

Retain enough evidence without retaining everything

Deduplication records need to outlive the duplicate horizon your application is prepared to handle. Deleting them too early can turn an old event redelivery into a fresh action. Keeping complete payloads indefinitely creates a different problem: unnecessary sensitive-data retention.

Separate a small receipt key and processing result from protected payload storage. Choose retention based on provider behavior, operational recovery needs, and data obligations. Redact headers and sensitive fields before logs. Do not use an event’s arbitrary text as an analytics parameter or alert label.

Monitor oldest unprocessed receipt, repeated signature failures, durable-storage errors, and processing dead letters. A high count of successful acknowledgments can hide a stalled worker. Similarly, duplicate deliveries may be expected and harmless; alert on their operational consequence rather than their existence alone.

Test the awkward boundaries

Send the same verified fixture twice concurrently. Deliver an older fixture after a newer one. Crash a worker after its local transaction and separately after a simulated external effect. Make storage fail before acknowledgment. Rotate a signing-key reference in a sandbox. Ensure retries preserve the same logical action identity and do not log raw secrets.

The test should establish a business invariant, such as “one entitlement transition is committed for this event,” rather than merely verifying that a handler returned 200. For remote effects, state the exact destination guarantee and the remaining ambiguity.

A small checklist before launch

  • Verify authenticity using the correct raw body and endpoint key.
  • Persist receipt and pending work durably before acknowledging.
  • Scope deduplication by connection and environment.
  • Separate incoming delivery, processing, and remote side-effect outcomes.
  • Document ordering, retry horizons, retention, and reconciliation.
  • Give failures a bounded retry policy and an owner.

Invokation includes inbound event handling in its platform vision, alongside outbound connection profiles and controlled execution. See the payment and email architecture example for a workflow that needs both directions, and the developer preview for the proposed operation model.

In development · Shape the first product

Bring a real integration problem.

Share the constraints your team keeps meeting. No payment or commitment required.

Join early access