Distributed tracing across PHP-FPM, queues, and the CDN

Part 3 of the Platform SRE series — one request, one trace, end to end.

A slow checkout in Magento 2 is never slow in one place. The CDN serves the page shell, PHP-FPM renders blocks and totals, a queue consumer sends the confirmation email and syncs the ERP, and each layer logs in its own format with its own clock. Without a shared trace id, an incident becomes three engineers reading three dashboards and arguing about which layer is guilty. Distributed tracing fixes that by attaching one identifier to the request at the edge and carrying it through every hop until the last consumer finishes.

This post assumes the foundation from magento2-observability is in place: structured logs, request ids in Nginx, and a place to query spans. What follows is the practice layer on top, how the id moves, what each span should record, and how to read the result.

The trace id starts at the edge

Every trace needs a single origin. The CDN is the right place because it sees the request first and can generate an id when the client did not send one. Configure the CDN to accept an incoming traceparent header (W3C format) and to generate one when absent, then forward it as both traceparent and a simpler X-Trace-Id to origin. The dual header matters: traceparent keeps compatibility with OpenTelemetry collectors, while X-Trace-Id is trivial to read in PHP, Nginx logs, and queue payloads without a parsing library.

Cache behavior must be explicit. A cache HIT at the CDN terminates the trace at the edge, which is correct, because no origin work happened. A cache MISS or PASS must forward the headers untouched. Audit this with a curl that sends a known X-Trace-Id through the CDN and confirms the same value appears in the origin access log. If the CDN strips or regenerates headers on MISS, every downstream span is orphaned and the trace is fiction.

Propagating through PHP-FPM

Nginx should log the trace id on every origin request alongside response time and cache status. In PHP, read the header once at bootstrap, early in index.php or in a front-controller plugin, and store it in a request-scoped holder. Every log line the request emits must include it. The practical pattern is a Monolog processor that injects trace_id into the log context, so no developer has to remember to pass it.

The span structure inside PHP-FPM deserves discipline. Create a root span for the request with the route name, then child spans for the expensive operations: layout rendering, each totals collector run, inventory reservation, payment authorization, and order placement. The rule is to span boundaries you can act on. A span around "checkout" tells you nothing; spans around quote.totals.collect, inventory.reserve, and payment.authorize tell you which one regressed after a deploy. Keep span attributes small and typed: route, store id, quote item count, customer group, and cache hit or miss for full-page cache. Never put emails, addresses, or payment tokens in span attributes.

Sampling keeps the cost sane. Trace 100 percent of errors and of requests slower than a threshold (for example, slower than 2 seconds on checkout routes), and sample a fixed fraction, 1 to 5 percent, of everything else. Head-based sampling, decided at the edge, keeps the trace coherent: either the whole trace is kept or none of it is.

Crossing into the queue

The queue is where most Magento traces die. The web request publishes a message to RabbitMQ or MySQL queue, the consumer picks it up minutes later in a different process, and without propagation the consumer starts a fresh, unrelated trace. The fix is to serialize the trace context into the message envelope at publish time: traceparent, trace_id, and the originating route or topic. Magento message queue publishers accept additional data, so wrap the publish call in a helper that always injects these three fields.

On the consumer side, extract the context before processing and start the consumer span as a child of the original trace, with a link rather than a parent when the delay is long. Record the enqueue timestamp as an attribute so the trace shows queue wait time separately from processing time. That distinction decides the fix: long wait means add consumers or raise prefetch, long processing means optimize the handler. Consumers for order email, ERP sync, and inventory bulk updates each get their own span name keyed by topic, plus the message retry count, so a poison message looping through retries is visible as one trace with N attempts rather than N mysterious errors.

Failures need care. When a consumer throws, record the exception on the span, keep the trace id in the error log, and let the retry preserve the same trace context. A new trace per retry destroys the one thing you need, the count of attempts against one order.

Reading a trace when checkout is slow

A healthy checkout trace reads top to bottom in time order: edge receive, origin PHP-FPM root span, child spans for totals, reservation, and payment, then a gap for queue wait, then consumer spans for email and ERP sync. The investigation order is fixed. First check the gap between edge receive and PHP start, which is CDN and network. Then the widest child span inside PHP, which is the application bottleneck. Then the queue wait versus processing split. In practice, the widest span names the owner: a wide payment.authorize span is the gateway, a wide inventory.reserve span is the database, a long queue wait is capacity.

Set a team convention that every incident starts by pasting the trace id. No trace id, no investigation. The habit matters more than the tooling, because it forces the propagation chain to stay working. If someone cannot find a trace for a complaint, that is itself a bug in propagation, filed and fixed like any other.

Pitfalls

• Generating a new id per layer. Each hop must forward, never regenerate, except at the edge when none arrived.
• Tracing everything at full fidelity. Unsampled tracing on a catalog category page during a campaign will cost more than the infrastructure it monitors.
• Spans with no actionable boundary. A span that cannot name an owner is overhead, not observability.
• Dropping context at the queue. The publish helper must be the only way to publish, enforced by code review, or someone will publish a bare message and orphan the trace.
• PII in span attributes. Emails and addresses leak into every trace backend retention window. Attributes stay typed and anonymous.

For the resilience patterns that keep these layers alive under failure, see magento2-devops-resilience. Tracing shows you where it broke; resilience decides whether it breaks at all.

Next in this series: magento2-alerting-on-symptoms