Alerting on symptoms, not causes

Part 4 of the Platform SRE series — page on pain, not on CPU.

Most Magento alert rules fire on causes: CPU above 80 percent, disk above 90 percent, Redis memory high. These pages arrive at 3 AM for conditions customers never felt, and they stay silent when checkout conversion drops 20 percent while every server looks healthy. Symptom alerting inverts the hierarchy. The paging alerts measure customer-visible outcomes, checkout success rate, order placement latency, queue lag for order email, and the infrastructure signals become diagnostic data, not pages.

This post builds on the tracing practice in magento2-distributed-tracing and assumes the logging base from magento2-observability. Here is how to pick the symptoms, set burn-rate windows, and keep the pager quiet.

The four symptoms worth paging on

A commerce store has a small set of outcomes that directly cost money. Alert on these and nothing else at paging severity:

1. Checkout success rate. Ratio of completed orders to checkout starts over the window. A drop means customers cannot pay, whatever the cause.
2. Order placement latency. The 95th percentile of time from place-order click to success response. Slow checkout abandons carts silently.
3. Queue lag for order-critical consumers. Age of the oldest unprocessed message in the order, invoice, and shipment topics. Lag here means confirmation emails and ERP syncs stall even when the storefront looks fine.
4. Error rate on payment and inventory operations. Fraction of payment authorizations and inventory reservations that throw. These two operations fail before an order exists, so order counts alone miss them.

Everything else, CPU, memory, disk, Redis evictions, FPM queue depth, is a ticket or a dashboard, escalated to a page only when a symptom alert is already firing and the cause signal names the fix.

Burn-rate windows that work for commerce

Fixed thresholds like "error rate above 1 percent" flap during low-traffic nights and react too slowly during campaigns. Burn-rate alerting fixes both by measuring how fast the error budget is being consumed over two windows: a fast window that pages quickly on sharp drops, and a slow window that catches gradual degradation.

Concrete starting points for a mid-size store:

checkout success, fast burn: 5% budget consumed in 15 min  -> page
checkout success, slow burn: 10% budget consumed in 2 h    -> page
order latency p95, fast:     p95 > 5 s sustained 10 min    -> page
order latency p95, slow:     p95 > 3 s sustained 1 h       -> ticket
queue lag, order topics:     oldest message > 10 min       -> page
queue lag, order topics:     oldest message > 3 min        -> ticket
payment error rate:          > 2% over 15 min              -> page

Tune the absolute numbers to your baseline, but keep the structure: every symptom has a fast page and a slow ticket. The fast window catches the deploy that broke payment; the slow window catches the consumer that fell behind at noon and will miss the evening peak. Low-traffic nights need a minimum-volume guard, for example suppressing the success-rate page when fewer than 20 checkout starts occurred in the window, so a single failed test order cannot page anyone.

Alert routing and severity

Two severities are enough. Page means customer pain is happening now and someone must respond within minutes. Ticket means degradation is building and someone looks within business hours. Every page alert names the symptom, the current value, the window, and a link to the runbook. A page that says only "checkout alert firing" will be ignored the third time; a page that says "checkout success 82 percent over 15 min, runbook magento2-runbooks-oncall section 2" gets acted on.

Route pages to the on-call engineer and tickets to the team board. Never route cause signals to the pager. When FPM pool exhaustion accompanies a checkout latency page, the latency page fires first and the FPM dashboard is linked from the runbook as step three. The order teaches the team to confirm pain before chasing graphs.

Keeping the pager quiet

Alert fatigue is a reliability bug. Three practices keep pages meaningful:

• One page per symptom. If the fast and slow windows for the same symptom fire together, the slow one must auto-suppress. Dedupe by symptom, not by rule id.
• Maintenance windows for deploys. A deploy that restarts consumers briefly spikes queue lag. Silence the lag ticket during the deploy window, never the page, and keep the silence shorter than the window itself.
• Monthly alert review. Count pages per symptom, false positives, and time to acknowledge. Any rule that paged without customer impact gets its threshold tightened or its severity demoted. A rule nobody trusts is worse than no rule, because it trains the team to ignore the pager.

The resilience companion magento2-devops-resilience covers the self-healing that should resolve causes before symptoms fire. Alerting and resilience are two halves of one loop: resilience absorbs what it can, symptom alerts catch what it cannot.

Next in this series: magento2-runbooks-oncall