Resilience: backups, rollback, and incident response

Part 3 of the Magento 2 Performance/DevOps series — plan for the outage you hope never happens.

Resilience is a practiced habit, not a checkbox

The question is not "do we have backups" but "have we restored from them." A backup nobody has tested is a hope, not a control. Resilience has three legs: a backup you trust, a rollback that is fast, and a runbook people can follow at 3 a.m. without thinking.

Backups you can restore

• Database — regular mysqldump or physical backup, plus point-in-time if the host supports it. Customer and order data is the thing you cannot recreate.
• Media and code — media under pub/media, and code via version control (so "redeploy the commit" is always an option).
• Test the restore. Quarterly, restore to a throwaway environment (a Govard env is perfect) and confirm the store boots and orders load. An untested backup is the most expensive gamble on the books.

A fast rollback path

Code deployments should be reversible in minutes:

• Tag every release so you can redeploy the previous known-good commit.
• Database changes need a down-path. A schema migration that cannot be reversed turns a bad deploy into a data incident. Prefer additive, backward-compatible migrations; if you must break, stage it across two releases.
• Cache flush as a lever, not a crutch. A full cache flush fixes "stuck" pages but hides the root cause; use it to recover, then find why.

An incident runbook

A runbook is only useful if it is boring and specific:

## Symptom: checkout returns 500
1. Check deploy log — last release hash?
2. If post-deploy: redeploy previous tag, flush FPC.
3. Check DB connection + disk on host.
4. If data corruption suspected: restore last verified backup to staging, confirm, then prod.
5. Post-incident: write the timeline.

The runbook beats tribal knowledge because the people who wrote it are rarely the ones on call.

Pitfalls

• Backups but no restore test. Confidence without evidence.
• Rollback that touches the database destructively. Always have a pre-change snapshot.
• A runbook nobody has read. Walk through it in a game day, not during the incident.
• Treating cache flush as the fix. It recovers the symptom; log the root cause.

What's next

Resilience protects the running system; the next post optimizes the edge — CDN and caching strategy so the origin stays quiet under load.

Next in this series: Edge, CDN, and advanced caching strategy — /blog/magento2-edge-cdn-caching