Blameless postmortems and turning incidents into gates

Part 6 of the Platform SRE series — every incident pays for the next prevention.

An incident that ends with "we fixed it and will be more careful" will happen again. An incident that ends with a merged automated check cannot happen the same way twice. The blameless postmortem is the mechanism that converts the first outcome into the second. It asks what allowed the failure, never who caused it, and its deliverable is always one or more gates in CI or monitoring, not a promise of vigilance.

This finale assumes the earlier parts of the series: traces that show what broke, symptom alerts that caught it, and runbooks that guided the response. The postmortem closes the loop by making the recurrence structurally impossible.

The blameless postmortem template

Hold the review within three business days, while memory is fresh but adrenaline is gone. Everyone who responded attends, plus one engineer who did not, to ask the naive questions. The facilitator is never the person whose change triggered the incident. The document follows a fixed shape and stays under two pages:

1. Summary:        one paragraph, customer impact in numbers
2. Timeline:       UTC timestamps of detection, mitigation, resolution
3. Root cause:     the mechanism, stated without names
4. Contributing factors: what made the mechanism possible
5. What worked:    which alert, trace, or runbook step helped
6. What failed:    which alert was missing, which step misled
7. Action items:   each with an owner, a due date, and a gate type

The language rule is absolute: no names in sections 3 and 4. "The deploy removed the consumer restart step" is allowed; "Lan forgot the restart" is not and gets edited out on sight. Blame drives details underground; the next incident then arrives with no warning because the person who saw it coming stayed silent last time. Impact numbers are mandatory in section 1: failed orders, minutes of degraded checkout, delayed confirmation emails. Without numbers, severity is opinion and prioritization stalls.

From action items to gates

Every action item must name its gate type, or it will decay into a wiki page nobody reads. Three gate types cover nearly all Magento incidents:

CI quality gates. The incident caused by a code pattern becomes a static check. A cache tag storm from an observer that invalidated too broadly becomes a test asserting the invalidated tag set for that observer. A fatal from an unhandled payment exception becomes a test covering the gateway timeout path. The rule is that the regression test fails against the pre-fix code and passes after, proven by running both. This is the same quality-gate philosophy as the CI pipeline, extended with incident-derived cases.

Deploy gates. The incident caused by an unsafe rollout becomes a precondition. A deploy that restarted consumers mid-message and spiked queue lag becomes a deploy script that drains consumers before restart. A schema change that locked the inventory table becomes a migration checklist enforced in the pipeline: online DDL only, with lock-wait timeout set. Each gate is a script, not a document, because documents are skipped under pressure and scripts are not.

Monitoring gates. The incident that evaded detection becomes a symptom alert. A silent ERP sync stall becomes a queue-lag alert on that topic with the thresholds from the alerting post. A slow totals regression that never paged becomes a latency alert on the affected route. Every monitoring gate links back to the postmortem, so the future engineer who wonders why the threshold exists finds the story in one click.

Track action items like production bugs, with owners and due dates inside two sprints. An action item older than a month is either done or explicitly deprioritized with a recorded reason. "Accepted risk" is a legitimate outcome when written down; silent neglect is not.

What not to do

• Postmortems for every ticket. Reserve the full ritual for paging incidents and near-misses with customer impact. Overuse turns it into paperwork.
• Action items without owners. Shared ownership means no ownership. One name, one date.
• Gates that test the fix instead of the mechanism. Asserting "this exact order no longer fails" is brittle; assert the invariant, like "consumer restart preserves in-flight messages."
• Publishing blame-adjacent detail. Customer names, engineer names, and chat excerpts stay out of the stored document. The postmortem is read by future hires; write it for them.

Closing the series

The Platform SRE arc moved from visibility to prevention. Tracing made every request followable across PHP-FPM, queues, and the CDN. Symptom alerts paged on customer pain instead of server graphs. Runbooks gave a small team deterministic first responses and a rotation it could survive. Postmortems turned each incident into a gate that holds permanently. Together they are the operating system under the store: see everything, page on pain, respond calmly, and never pay for the same failure twice.

This was the final post in the Platform SRE series.