In this article
Follow a payment event from receipt to effect
Consider an illustrative subscription service. A payment provider sends a notification, and the application must activate access. Stripe’s documentation explicitly discusses signature verification, duplicate deliveries and event-ordering limitations. These are useful reminders to consult the contract of each provider rather than assume that one HTTP request means one business operation.
A proposed receiving path verifies the signature, records an event in a durable inbox and acknowledges receipt after that record is accepted. A worker performs the business change separately. Keep event identifier, account context, processing state and attempts together. A successful acknowledgement means the event was accepted; it should not be presented in operations tooling as proof that subscription activation already succeeded.
Assume delivery is imperfect
Webhooks can arrive late, more than once or out of order. Polling can miss a narrow change or repeatedly fetch the same data. Define an event identity, source version and accepted ordering rules before connecting business effects.
Acknowledge inbound delivery only after durable acceptance. Keep the raw provider reference and controlled metadata needed for replay, but avoid storing secrets or unnecessary personal payloads.
Make effects idempotent and observable
Use a stable idempotency key for each business command and bind it to the request payload. Reusing a key with different content is a conflict, not a retry. Database constraints should protect the invariant under concurrency.
Separate acceptance from provider delivery with an outbox when an external call follows a local transaction. Track pending, processing, delivered, retry and dead-letter states without making the caller wait for another vendor.
A duplicate event should not duplicate its effect
An illustrative delivery path with durable state.
Receive
Validate the incoming event and its identity.
Record
Persist acceptance and detect repeated deliveries.
Process
Apply the business effect with an idempotency boundary.
Reconcile
Compare expected and actual state after failures.
Reconcile, do not only retry
Retries solve transient transport failures, not unknown outcomes or semantic drift. Add bounded backoff, a dead-letter review, manual replay and a reconciliation job that compares canonical state with provider state.
Assign an owner to schemas, credentials, rate limits and breaking changes. Dashboards should answer which records are late, stuck or divergent and what safe action restores them.
The difficult case: the effect happened, the reply did not
Suppose the worker activates access and crashes before marking the event complete. A retry must recognise that the activation already happened. Enforce the operation key and the local state change in the same transaction where possible. A check followed by an unprotected write is insufficient: two workers can both pass the check.
If the effect belongs to a remote service, a local transaction cannot make the remote call atomic. Use that service’s idempotency mechanism when available, retain its operation reference and reconcile uncertain outcomes. A timeout means “unknown”, not necessarily “failed”. Create an explicit unresolved state instead of generating a new operation key and repeating an effect that may already have succeeded.
Acknowledge receipt, then control the effect
A simplified proposed processing path after signature verification. An uncertain remote result is reconciled before a new attempt.
Verify → durable inbox → acknowledge → worker
Business operation already completed?
Yes
- Return the stored outcome
- Mark this delivery handled; no repeated effect
No
- Claim operation key → apply protected change
- Store success; reconcile uncertain remote results
Build the incident view before the first incident
An operator should be able to find an event and answer: was it authenticated, stored, attempted, completed or parked for investigation? Show the last error class and the next planned attempt without exposing secrets or complete payment payloads. Give replay actions the same business protections as normal processing and record who triggered them.
Test duplicate delivery, two simultaneous workers, an older cancellation arriving late, a database failure and a remote timeout after success. After each test, inspect business records, not just HTTP codes. A reconciliation job can compare expected subscription state with the provider’s authoritative state and flag differences. This closes the gap that retry loops alone cannot resolve.
Sources & further reading
Documentation consulted on .
FAQ / DECISIONS
Frequently asked questions
Are webhooks always better than polling?+
No. Webhooks reduce latency and repeated reads; polling can simplify recovery and completeness. Critical integrations often use webhooks for speed plus periodic reconciliation.