Bulk and async APIs for high-volume product import

Part 7 of the Catalog at Scale series — the import path that survives real volume.

Synchronous saves do not scale to feed volume

The previous posts in this arc built the case in stages: the catalog data model, attribute discipline, indexation cost, and a staged import pipeline with validation and rollback. This finale covers the transport those ideas point to. The synchronous product endpoints process one SKU per request in the web process, holding the connection open through the full save. At fifty or a hundred thousand SKUs, that design means hours of sequential requests, fragile to any timeout, and every failure leaves a partially applied batch. The bulk and async APIs exist to break that coupling.

The bulk endpoints, precisely

Magento exposes bulk variants under /async/bulk/V1/products and related routes. Instead of one product per request, the client posts an array of operations in a single call. Magento validates the request envelope, persists each item as a message in the message queue (RabbitMQ in production, database queue in development), and returns immediately with a bulk_uuid. The HTTP request finishes in milliseconds; the actual product saves happen later, in background consumers.

Status tracking runs through the bulk status endpoints: query by bulk_uuid to see total, processed, failed, and pending counts. Each failed operation carries its error message, so the feed operator gets per-SKU failure detail without re-running the batch. This is the foundation the staged pipeline from Part 5 needs: the apply step becomes one bulk submission plus status polling, not fifty thousand blocking calls.

RabbitMQ consumers and throughput tuning

Background processing runs through queue consumers (bin/magento queue:consumers:start), typically supervised per queue topic such as async.operations.all. Throughput comes from running multiple consumer processes per topic and sizing the message batch, not from bigger web servers. Practical levers:

• Consumer count per topic. Start with one consumer per CPU-heavy topic and scale until the database, not the queue, becomes the constraint. Product saves are write-heavy; beyond a few parallel consumers, lock contention on inventory and index tables caps the gain.
• Separation of topics. Keep product-save consumers apart from order and notification consumers so a large import cannot starve checkout-related queues. Dedicated supervisor groups per topic make this explicit.
• Indexer interplay. Bulk saves still trigger indexation events. Schedule the import behind Update on Schedule indexers and let the changelog drain after the batch, or throttle consumers during peak storefront hours. A bulk import that invalidates the price index at noon is a self-inflicted outage.
• Message TTL and retries. Transient failures (deadlocks, brief replica lag) resolve on retry; permanent ones (bad attribute values that slipped validation) never will. Configure a bounded retry count, then route exhausted messages to the failure log instead of looping forever.

Failure replay without re-running the world

The bulk status record is what makes failures operational instead of catastrophic. A completed batch with 2 percent failures is a normal outcome: extract the failed SKUs from the bulk operations log, fix the underlying data (usually a mapping gap or a missing attribute option), and resubmit only those items as a new bulk request. The 98 percent that succeeded are never touched.

Build the replay path before the first production import: a script that takes a bulk_uuid, dumps failed operations with their error payloads, and emits a corrected resubmission file. Pair it with the rate-of-change guards from Part 5 so a replay cannot accidentally widen its own scope. Every import run ends in one of three states: fully applied, partially applied with a tracked replay batch, or rolled back from the pre-import snapshot. There is no fourth state where nobody knows what applied.

Operating bulk import in production

• Idempotent submissions. Tag each bulk request with the source batch key from the staging pipeline. If a submission is retried after a network timeout, the key prevents double application.
• Size the batches. Ten thousand operations per bulk request is a workable default; beyond that, status tracking and failure triage get unwieldy. Split a 100,000-SKU feed into sequenced bulks with verification between them.
• Monitor the queue, not just the endpoint. Alert on consumer process death, queue depth growth, and bulk failure rate, not only on HTTP errors from the submission call. The submission succeeding while consumers are down is the quietest failure mode.
• Keep the synchronous path for humans. Admin edits and small API updates stay synchronous; bulk is for feeds. Routing everything through the queue adds latency where immediacy matters.

Closing the series

The Catalog at Scale arc started from the data model and ended here: attribute sets that stay disciplined, indexers scheduled around reality, a staged import pipeline with validation and rollback, search relevance tuned on measured queries, and bulk transport that moves feed volume through background consumers with replayable failures. No single layer carries a large catalog; the pipeline, the index, the relevance function, and the queue each hold part of the weight. Build them as one system, verify every batch, and the catalog stops being the fragile center of the store.

This was the final post in the Catalog at Scale series.