Runbooks and on-call for a small commerce team

Part 5 of the Platform SRE series — calm incidents are written in advance.

A three-person commerce team cannot staff a 24/7 operations center, and it does not need to. What it needs is a small set of runbooks that turn the symptom alerts from magento2-alerting-on-symptoms into deterministic first responses, plus a rotation that does not burn anyone out before the holiday peak. Incidents feel chaotic when the responder has to decide what to check first. A runbook removes the decision.

The one-page runbook template

Every runbook fits on one page and follows the same five sections. The fixed shape matters because at 2 AM nobody learns a new format.

1. Symptom:       what the alert said, in customer terms
2. Confirm (5m):  two queries that prove pain is real
3. Mitigate:      the fastest safe action that restores service
4. Diagnose:      where to look next, with links
5. Escalate:      when to stop and who to call

A filled example for checkout latency:

1. Symptom:  order placement p95 above 5 s for 10 min
2. Confirm:  checkout success-rate dashboard; one real place-order in staging
3. Mitigate: enable maintenance-safe CDN rule to shed bot traffic; restart
   stuck queue consumers; if payment gateway, switch to secondary acquirer
4. Diagnose: widest span in the slow traces (see distributed tracing guide);
   FPM pool status; DB process list for locked inventory tables
5. Escalate: no improvement in 30 min -> call gateway support with trace ids;
   checkout still failing at 45 min -> freeze deploys, post status page

Write one runbook per paging symptom, four to six total. More than that and nobody remembers they exist. Each runbook links the dashboards and queries directly, because searching for a dashboard name during an incident wastes the minutes that matter. Store runbooks next to the code, in the same repo as the infrastructure, so a deploy that changes consumer topology updates the runbook in the same pull request. Review runbooks quarterly and after every incident that deviated from them.

Mitigation before diagnosis

Small teams must internalize one ordering rule: mitigate first, diagnose second. The customer does not care why checkout is slow; they care that it works. Safe mitigations for Magento are few and should be pre-approved so the responder acts without waiting for permission: shedding non-human traffic at the CDN, restarting wedged consumers, clearing a poisoned full-page-cache tag, failing over to the secondary payment endpoint, and freezing further deploys. Each mitigation lists its blast radius and its rollback in the runbook. Anything with irreversible effects, manual database edits, inventory adjustments, lives under Escalate, never under Mitigate.

A rotation three people can survive

The sustainable pattern for a team of three is a weekly rotation with a hard handoff ritual. The on-call engineer carries the pager for seven days, then hands over with a written note: open incidents, known degradations, scheduled jobs like reindex or import, and anything that paged overnight. The handoff note is mandatory even when the week was quiet, because "nothing happened" is itself information the next person needs. Keep a shared on-call log, one dated entry per page received, so patterns like recurring Sunday-night import failures become visible within a month instead of living in one person's memory.

Protect the rotation with three rules. First, the on-call engineer does no planned feature work that week, only support and small fixes; splitting attention between a deadline and the pager guarantees both suffer. Second, nights and weekends page only on the symptom alerts defined in the previous post; everything else waits. Third, after any night with more than one page, the responder takes the next morning off, no discussion. These rules look generous until you price the alternative, which is your only Magento specialist quitting in November.

Compensation must be explicit. On-call time is work. Whether it is paid extra,交換ed as time off, or rotated with visible credit, the arrangement is written down and honored. A team that treats on-call as invisible labor will silently stop responding.

Practice with game days

A runbook that has never been executed is a hypothesis. Once a quarter, pick the quietest morning, announce a game day, and simulate one paging symptom in staging: inject latency into the payment mock, stall a consumer, expire a cache tag storm. The on-call engineer follows the runbook literally while the others observe. Every point where they hesitated, guessed, or opened the wrong dashboard is a runbook edit filed the same day. One hour per quarter buys more reliability than any dashboard, because it tests the responder, not just the system.

The resilience patterns in magento2-devops-resilience reduce how often these runbooks fire. The tracing practice from the earlier post tells the responder where to look at step four. Runbooks are the glue between the two.

Next in this series: magento2-postmortems-to-gates