Import pipelines: CSV, API, and the case against both

Part 5 of the Catalog at Scale series — the import path decides catalog quality long before the storefront renders it.

Every catalog has an import pipeline, even the accidental ones

Most Magento catalogs grow through a mix of admin CSV uploads, a supplier feed dropped on a schedule, and a few API scripts nobody remembers writing. Each path works in isolation. Together they produce the classic symptoms of a large catalog in trouble: duplicate SKUs, prices that change and then change back, images attached to the wrong simple, and stock values that disagree with the warehouse. The problem is rarely the data source. It is the absence of a pipeline with validation, ordering, and rollback.

What CSV import is good at, and where it breaks

The native CSV import (System > Data Transfer > Import) is the fastest way to create or update thousands of products. It handles the full product row, including custom options and images, and its validation pass reports errors per row before anything is written.

Its limits appear at scale. The import runs synchronously in the admin request or a single cron consumer, so a 200,000-row file can time out or exhaust memory. Error reporting stops at the row level: a bad attribute value in row 40,000 aborts or skips in ways that leave half the file applied. There is no diff, no dry run against production data, and no rollback. Re-running the same file after fixing three rows reprocesses everything, including the rows that already succeeded. For a weekly supplier feed, that means every import is a full rewrite with full risk.

CSV still earns its place for one-off migrations and small, human-reviewed batches. As a recurring integration path, it is a liability.

What the synchronous REST API is good at, and where it breaks

The REST endpoints (POST /V1/products, PUT /V1/products/:sku) give per-product control, proper error codes, and a natural fit for middleware. For dozens or hundreds of updates per hour, they are the right tool.

At catalog scale they fall over for a structural reason: one HTTP request per product, each bootstrapping the full product save pipeline including indexation hooks. A feed of 50,000 SKUs becomes 50,000 sequential saves, each taking hundreds of milliseconds. Rate limits, PHP timeouts, and indexer backlog turn a nightly sync into a job that is still running when the next one starts. Parallelizing the calls helps throughput but multiplies the lock contention on shared tables, and a failure halfway through leaves a partially applied batch with no transaction boundary.

The pipeline that replaces both

Treat every import as a staged batch, regardless of source:

1. Land. The supplier file or API payload lands in a staging table, untouched. Never transform in flight.
2. Validate. A standalone validator checks SKU existence, attribute set membership, attribute option values, website and store assignments, price and quantity types, and image paths before Magento touches anything. Reject the batch or quarantine bad rows here, with a report the supplier can act on.
3. Normalize. Map supplier columns to Magento attributes in one versioned mapping file. When the supplier renames a column, the diff is in the mapping, not scattered across scripts.
4. Apply in order. Simple products first, then configurables and their associations, then prices, then stock, then images. Each stage commits separately so a failure in images does not roll back prices that were correct.
5. Verify. After the apply, a reconciliation query compares counts, price ranges, and stock totals between staging and catalog. Alert on drift above a threshold instead of discovering it from customer complaints.
6. Rollback. Snapshot the affected SKUs (attribute values, links, stock) before the apply. A failed batch restores the snapshot, and the incident is a reverted import, not a catalog outage.

This pipeline can still use CSV or REST as the transport for the final apply step. The difference is that neither is the system of record anymore; the staging area is.

Import validation that actually catches damage

Row-level syntax checks are not enough. The validations that save catalogs are semantic:

• Referential checks. Every configurable_variations SKU must already exist as a simple in the same batch or in the catalog. Every attribute option value must exist in the attribute definition. Orphan links are the most common source of invisible storefront products.
• Scope checks. Price changes scoped to the wrong website, or a status flip applied to all store views instead of one, typically come from a file that assumed the wrong default scope.
• Cardinality checks. A configurable with zero associated simples, or a simple associated to two configurables in the same attribute set, fails the batch before it reaches the database.
• Rate-of-change guards. If a feed updates more than a set percentage of prices or disables more than a set number of SKUs in one run, pause and require human approval. Runaway supplier files have wiped entire categories; a threshold turns that into a held batch.

Scheduling and idempotency

A recurring import must be safe to re-run. Key each batch by source plus date, and record per-SKU application state so a retry skips rows that already applied. Run imports against a replica-friendly window, sequence them behind indexer schedules, and never let two import jobs overlap on the same SKU set. A lock per batch, released only after verification, is cheaper than debugging interleaved writes.

Pitfalls

• Editing the supplier file by hand each week. The manual fix becomes tribal knowledge and the mapping drifts.
• Applying images in the same transaction as product data. A missing image file should not fail ten thousand price updates.
• No quarantine path. Without a way to hold bad rows and apply good ones, one malformed row blocks the whole catalog refresh.
• Trusting the green checkmark. Native validation passing means the rows parse, not that the catalog is correct. Reconciliation is the real gate.

Next in this series: magento2-catalog-search-relevance