Structured logging that survives a peak-season incident

Part 2 of the Platform SRE series — grepable logs beat clever dashboards at 2am.

Why this matters

Peak-season incidents have a shape: checkout errors spike, the error budget burns, three engineers open three dashboards, and nobody can answer the only question that matters, which orders failed, for which reason, on which request. The usual cause is logs written for developers reading one line at a time: timestamped prose with interpolated variables, inconsistent field order, and identifiers scattered across system.log, exception.log, payment gateway callbacks, and the web server access log with no shared key.

Structured logging fixes the join problem before the incident. Every significant event is emitted as one JSON object with a stable taxonomy and a correlation ID that follows the shopper from edge to order. During the incident the team runs two or three prepared queries instead of improvising regex; after the incident the same fields feed the postmortem counts without re-parsing prose.

How it works

Three pieces: a JSON taxonomy every emitter follows, correlation IDs that stitch the layers, and a small set of rehearsed peak queries.

One JSON taxonomy for every emitter

Standardize on a single event shape, enforced by a shared logger or Monolog processor rather than by convention documents nobody re-reads:

{"ts":"2026-11-28T01:14:02Z","lvl":"error","svc":"magento-checkout","env":"prod",
 "req_id":"3f9a41c2e7b84d2a","quote_id":"482913","order_id":null,
 "route":"checkout/payment","evt":"payment.authorize.failed",
 "gateway":"adyen","code":"E_TIMEOUT","ms":2140,"msg":"authorize timed out"}

Field rules that pay off: ts always UTC ISO-8601; lvl from a fixed set (debug, info, warn, error); svc naming the Magento area or integration (magento-checkout, magento-queue-consumers, erp-export); evt a dotted stable name, never free prose, so payment.authorize.failed is countable and Payment failed for quote is not; code a machine-stable failure class (E_TIMEOUT, E_DECLINED, E_INVENTORY_SHORT, E_TOTALS_MISMATCH) with human detail confined to msg. Numeric durations in ms as numbers, never embedded in strings. Include quote_id on every checkout-path event from add-to-cart onward and order_id once placed; the quote-to-order join is how support answers "did my payment go through" while the shopper is still on the line.

Apply the taxonomy to the three Magento log sources that matter: application events (payment, inventory reservation, totals calculation, async consumers), PHP exceptions (logged with evt, req_id, route, and the exception class in code, plus a truncated trace, not a five-hundred-line dump per occurrence), and the edge/access layer (emit or enrich with req_id, route, status, and ms so slow-page analysis joins to application events on req_id).

Propagate one request ID end to end

Generate req_id at the edge (Fastly/Varnish or the first Nginx hop) and forward it as a header (X-Request-ID) through every layer: Varnish to Nginx to php-fpm, Magento to payment gateway metadata fields where supported, Magento to queue messages as a message property, and consumers back into their log lines. Log the ID in the access record too, so any line in any log with the same req_id describes the same shopper action.

Two implementation notes specific to Magento: many checkout failures span two requests (place-order POST, then the gateway webhook minutes later), so propagate quote_id/order_id into the webhook handler's logs as the cross-request key, since req_id legitimately differs there. And queue consumers processing in bulk must log per-message fields (evt per outcome) rather than one summary line per batch, or the single failing order in a batch of five hundred becomes invisible exactly when it matters most.

Keep three peak queries rehearsed

Write these before November, test them on staging with real log volume, and pin them in the incident channel:

# 1. Failing checkouts by cause, last 15 min
lvl=error evt=payment.authorize.failed env=prod | stats count() by code, gateway

# 2. Full story of one shopper session
req_id=3f9a41c2e7b84d2a OR quote_id=482913 | sort ts

# 3. Slow routes correlated with errors, last hour
svc=magento-checkout lvl in (warn, error) ms>2000 | stats count(), p95(ms) by route, evt

Complement with a tail-mode watch during the event: stream lvl=error grouped by evt and code in 1-minute buckets so new failure classes surface within a minute of deploying a promotion rule change. The discipline is that nobody invents syntax mid-incident; the queries exist, the fields exist, and the only variable is the ID.

Volume control keeps the pipeline affordable: debug off in production except behind a per-route sampling flag, full exception traces sampled (one full trace per code per minute, counts for the rest), and health-check and static-asset requests excluded from application logging at the edge. Set retention explicitly (hot searchable for 14 days covering the peak window, cold archive for 90) so nobody discovers during the postmortem that the decisive hour already aged out.

Pitfalls

• Logging PII into the incident trail. Card numbers, full emails, and addresses do not belong in logs just because they are convenient join keys. Log truncated references (order ID, quote ID, gateway reference) and resolve PII through the database under access control.
• Clock skew across nodes. Timestamps from four web nodes disagreeing by seconds break sort ts narratives. Run NTP everywhere and treat the edge timestamp as canonical when ordering cross-node events.
• Unbounded message fields. Logging full gateway responses or entire quote objects per line multiplies volume tenfold and leaks PII. Log the reference plus the outcome code; fetch the payload from the gateway dashboard when needed.
• Sampling the errors away. Sampling is for success traffic. Every error and every checkout-path warn ships unsampled; sample only info/debug and high-volume success events like catalog views.
• Two taxonomies after an acquisition. A new payment module logging its own prose format re-fragments the join. Gate new integrations in code review on emitting the standard shape with req_id, quote_id, and a dotted evt before they merge.

Closing

Structured logs are the incident-time memory of the store: one JSON shape, one request ID across layers, quote and order IDs across requests, and three rehearsed queries that turn a budget fire into a countable list of causes. With failures countable, the next step is seeing where time goes inside the slow ones, which is what distributed tracing adds.

Next in this series:

magento2-distributed-tracing