Case study: a webhook retry storm that took down our own ERP

Field notes — a real production bug, project name withheld. Symptom → investigation → fix → lesson.

The symptom

On a Monday morning the ERP went down. Its job queue filled with tens of thousands of goods-receipt postings, all for the same few hundred orders, and the warehouse team could not post anything new. The ERP vendor blamed our Magento integration. Our Magento logs looked healthy: order volumes were normal, no deploy had gone out over the weekend, and the payment success rate was unchanged. The only anomaly was a payment webhook endpoint receiving far more traffic than the order count justified: roughly 40 times more callbacks than orders.

The timeline narrowed it quickly. A payment provider maintenance on Sunday evening had delayed webhook deliveries by about an hour. When deliveries resumed, our endpoint started returning HTTP 500 for a subset of callbacks. The provider retried every 30 seconds for up to 24 hours. By Monday morning, a few hundred initially failed callbacks had each been retried dozens of times, and every retry had triggered a full synchronous ERP posting. The ERP did not deduplicate, so it queued every duplicate as new work until its workers drowned.

Investigation

The webhook controller told the story in twenty lines. It verified the signature, loaded the order, posted the goods receipt to the ERP synchronously, then marked the order as notified:

public function execute()
{
    $payload = $this->verifySignature($this->getRequest());
    $order = $this->orderRepository->getByIncrementId($payload['order_id']);
    $this->erpClient->postGoodsReceipt($order); // synchronous, 4-8 seconds
    $order->setData('webhook_notified', true);
    $this->orderRepository->save($order);
    return $this->jsonResponse(['ok' => true]);
}

Three defects stacked. First, there was no idempotency key. The only duplicate guard was the webhook_notified flag, which was written after the ERP call. If the ERP call succeeded but the response was slow, or if two retries overlapped, both executions saw webhook_notified as false and both posted. The flag recorded intent after the fact instead of claiming the work upfront.

Second, the ERP call ran synchronously inside the webhook request. Each posting took 4 to 8 seconds because the ERP processes receipts serially per warehouse. The payment provider's webhook timeout was 5 seconds. So even successful postings frequently exceeded the provider timeout: the ERP recorded the receipt, but the provider saw a timeout, marked the delivery as failed, and retried. Every slow success manufactured its own retry. The access log proved it: the same order_id appeared every 30 seconds with identical payload hashes, and the ERP log showed one goods receipt per delivery.

Third, failures were fatal instead of isolated. When the ERP queue backed up on Monday, postings began throwing exceptions, the controller returned 500, and the provider retried harder. The retry traffic added load to an already saturated ERP, which made more postings time out, which triggered more retries. A feedback loop: the worse the ERP struggled, the more duplicate work we sent it. By 09:00 the Magento webhook workers were spending all their time re-posting the same 300 receipts while new orders queued behind them.

We confirmed the loop by replaying one payload manually. Posting the same order_id twice created two goods receipts in the ERP with different receipt numbers but identical line items. There was no unique constraint on the external reference, so the ERP accepted every duplicate as legitimate. The vendor later confirmed their API expects the caller to pass an idempotency key; we had never sent one.

Fix

The fix separated acknowledgement from processing and made processing idempotent. The controller now returns HTTP 200 within milliseconds and enqueues the real work:

public function execute()
{
    $payload = $this->verifySignature($this->getRequest());
    $key = $payload['event_id']; // provider-supplied unique id per event

    if (!$this->idempotency->claim($key, $payload['order_id'])) {
        return $this->jsonResponse(['ok' => true, 'deduped' => true]);
    }
    $this->queue->publish('erp.receipt', $payload);
    return $this->jsonResponse(['ok' => true]);
}

The claim method inserts the event id into a small webhook_event table with a unique key before any ERP work happens. The first delivery wins; every retry finds the existing row and returns 200 without touching the ERP. Because the claim happens first, overlapping retries cannot both proceed even if they arrive in the same second. A consumer then processes the queue asynchronously, posting to the ERP with the event id as the external reference so the ERP itself can reject duplicates as a second line of defense.

Two operational changes went with it. The queue consumer has a circuit breaker: if the ERP error rate exceeds a threshold, it pauses consumption with backoff instead of hammering a struggling system, and failed messages move to a dead-letter queue with an alert rather than blocking the webhook endpoint. And the deployment added a dashboard tracking webhook deliveries versus ERP postings; the two lines should match 1:1, and any divergence pages the integration owner. We also asked the ERP vendor to add a unique constraint on the external receipt reference, which they scheduled for their next release.

After the fix we replayed the storm: 5,000 duplicate deliveries for 100 orders produced exactly 100 ERP postings and 4,900 deduped 200s. End-to-end acknowledgement time dropped from 4 to 8 seconds to under 150 milliseconds, well inside the provider timeout, so slow successes no longer manufacture retries.

Lesson

Never do slow synchronous work inside a webhook handler, and never guard duplicates with a flag you write after the work. Webhook providers retry aggressively and concurrently, so the handler must acknowledge fast and deduplicate upfront on a provider-supplied event id with a unique constraint. Assume every delivery will arrive at least twice and design the downstream call to be idempotent too, because the second line of defense is the only one that survives a queue replay or a vendor-side resend. Monitor deliveries versus side effects as a ratio, not just error rates: a 200-per-retry endpoint with no idempotency looks healthy in Magento while it buries the ERP. The system that sends the retry is never the system that pays for it.

Illustrated in magento2-payments-idempotency.