Technical SEO architecture for Magento 2

Part 1 of the Storefront Conversion Engineering series — the URL foundation everything else stands on.

The catalog save that took down the sitemap

A merchant with 40,000 SKUs in three store views saved a top-level category on a Friday afternoon. By Monday, url_rewrite had grown past 2.5 million rows, category pages were responding in seconds instead of milliseconds, and Google Search Console was reporting thousands of new duplicate URLs. Nothing in the code had changed. The architecture had simply never been designed for scale, and one routine admin action exposed it.

That is the core of Magento technical SEO: it is not a marketing checklist. It is a data architecture problem. Every product, category, and store view combination materializes as rows in a table, URLs on the site, and entries in a sitemap. Get the foundation wrong and every subsequent SEO effort, from faceted navigation policy to hreflang, inherits the mess.

One canonical URL per entity

Magento can serve the same product under multiple paths: the short product URL (/running-shoe-nike-air.html), the category-scoped URL (/men/shoes/running-shoe-nike-air.html), and variants per store view. Without a canonical policy, Google indexes all of them and splits ranking signals across duplicates.

Decide the policy once, at project start:

• Use Categories Path for Product URLs: No. This single configuration value (catalog/seo/product_use_categories) is the highest-leverage SEO setting in Magento. Setting it to No means each product has exactly one URL regardless of which category the shopper navigated through. Category context still renders in breadcrumbs, but the URL stays stable.
• Canonical Link Meta Tag: Yes for both products and categories (catalog/seo/product_canonical_tag, catalog/seo/category_canonical_tag). Even with a single-URL policy, parameters, sorting, and tracking codes append query strings. The canonical tag consolidates them.
• URL suffix: decide and freeze. Whether you use .html or no suffix matters far less than never changing it. The suffix is stored in catalog/seo/product_url_suffix and catalog/seo/category_url_suffix. Changing it regenerates every URL in the catalog, which means a full redirect plan or months of 404s.

A quick production check shows whether the policy holds:

SELECT request_path, COUNT(*) AS c
FROM url_rewrite
WHERE entity_type = 'product' AND store_id = 1
GROUP BY request_path HAVING c > 1 LIMIT 20;

Any rows returned are duplicate request paths competing for the same slot. On a healthy catalog this returns nothing.

The url_rewrite table is a growth system

Every product save, category move, or bulk import writes rows to url_rewrite. With N products, M categories, and S store views, the worst case approaches N x S rows plus category-product associations. A 50,000-SKU catalog in 3 store views routinely crosses a million rows.

Three practices keep it under control:

1. Monitor row count as a metric, not a curiosity. Track it alongside order volume. Alert when a single import adds more than a few percent:

   SELECT store_id, entity_type, COUNT(*) FROM url_rewrite GROUP BY store_id, entity_type;
   

   Run this before and after every bulk import. If the count jumps disproportionately, the import regenerated rewrites instead of updating them.

2. Regenerate surgically, never blindly. Full URL regeneration extensions walk the entire catalog and rewrite history. Prefer targeted regeneration for affected entities only, and always snapshot the table first (mysqldump of url_rewrite is cheap insurance).

3. Clean dead rewrites on a schedule. Disabled products, deleted categories, and failed imports leave orphaned 301 chains. Audit quarterly: find rewrites whose target_path no longer resolves to a live entity, and remove or consolidate chains longer than two hops. Each extra hop costs crawl budget and link equity.

Index health matters too. After large catalog operations, check that URL_REWRITE request-path lookups stay in the millisecond range. Slow category saves are often the first symptom of a bloated rewrite table, long before SEO reports show it.

Sitemaps that scale with the catalog

Magento's default sitemap generation writes a single XML file per store view. Past roughly 50,000 URLs or 50 MB uncompressed, that file violates the sitemap protocol and Google silently ignores the overflow. Large catalogs need split sitemaps.

The practical setup:

• One sitemap per store view, generated on a nightly cron, not on admin save. Generation is I/O heavy; tying it to catalog saves couples editorial workflow to infrastructure load.
• Cap each file and use an index. Split by entity type (products, categories, CMS pages) so a surge in one type does not push the whole file over limits. Verify the index references every child file with absolute URLs.
• Exclude what should never rank. Disabled products, out-of-stock products with no backorder path, internal test categories, and staging-only CMS pages must be filtered before generation. Every junk URL in the sitemap spends crawl budget that should go to revenue pages.

Validate after each generation cycle: fetch the sitemap index in Search Console, confirm the discovered URL count matches expectations within a few percent, and investigate any sudden drop (usually a generation failure) or spike (usually junk inclusion).

Robots.txt as infrastructure

Magento serves a default robots.txt that is rarely sufficient for a large catalog. Treat it as versioned infrastructure:

User-agent: *
Disallow: /*?*p=*&*
Disallow: /*?*product_list_order=*
Disallow: /*?*product_list_dir=*
Disallow: /checkout/
Disallow: /customer/
Disallow: /wishlist/
Allow: /media/catalog/product/
Sitemap: https://www.example.com/sitemap.xml

The query-string disallows cover sorting and pagination parameters that generate infinite crawlable variants. The checkout, customer, and wishlist paths have no indexable value. The media allow keeps product images discoverable for image search.

For multi-store setups, serve a per-domain robots.txt with the correct sitemap absolute URL per store. A single shared file pointing at the wrong sitemap silently starves secondary store views of crawl attention.

Search Console validation steps

Architecture work is only done when measurement confirms it:

1. Submit each store view sitemap in Search Console and record the indexed-versus-submitted ratio. A healthy catalog indexes the majority of submitted product URLs; a ratio under half signals duplication or quality filters.
2. Check the Page indexing report for Duplicate without user-selected canonical and Alternate page with proper canonical tag. Both should trend down after the canonical policy lands.
3. Inspect a sample of product URLs with the URL Inspection tool. Confirm the user-declared canonical matches the Google-selected canonical. Mismatches mean Google is overriding your signal, usually because of conflicting internal links.
4. Monitor the Crawl stats report. Pages crawled per day should stabilize; a sustained climb without catalog growth means new duplicate paths are being discovered.

Pitfalls

• Changing URL suffixes without a redirect plan. Flipping .html on or off regenerates every product and category URL. Without 301s from every old path, rankings reset and inbound links break. Freeze the suffix decision in the project kickoff document.
• Same product reachable under five category paths with no canonical. This is the default when category-path URLs are enabled and canonical tags are off. Fix the configuration before writing a single line of custom code.
• Sitemap including disabled and out-of-stock pages. Every dead URL submitted trains Google to trust the sitemap less. Filter at generation time, not after submission.
• Treating robots.txt as a one-time edit. New modules add new query parameters. Review crawler behavior quarterly and extend disallows for parameter families that generate no indexable value.
• Regenerating all rewrites to fix a few broken URLs. Full regeneration on a million-row table locks writes and risks duplicates. Fix surgically, snapshot first.

Closing

Technical SEO architecture is the foundation the rest of this series builds on. Canonical URLs, a bounded rewrite table, scaled sitemaps, and a maintained robots.txt turn the catalog from a duplicate-generating machine into a crawlable asset. Next, we tackle the largest remaining source of crawl waste: faceted navigation.

Next in this series:

magento2-faceted-navigation-crawl-budget