Faceted navigation and crawl budget control

Part 2 of the Storefront Conversion Engineering series — filters are for shoppers, not for crawlers.

When filters outnumber products ten to one

A fashion store with 8,000 products and six filterable attributes (size, color, brand, material, price, style) discovered Search Console tracking 120,000 indexed URLs. The catalog had not grown. Layered navigation had manufactured fifteen filter URLs for every real product page, and Google was diligently crawling combinations no shopper would ever bookmark: three colors plus two sizes plus a price band, sorted by newest, page four.

This is the faceted navigation problem in one number. Each filter combination is a distinct URL with near-duplicate content. Unchecked, filters generate more crawlable pages than the catalog itself, and crawl budget that should refresh product pages is spent on combinatorial noise.

The policy: one base page, controlled exceptions

The default rule is simple: filter pages canonical back to the base category, and crawlers are kept out of the combinatorial space. Exceptions are granted deliberately, never by accident.

Canonical discipline. Every filtered view of a category carries a canonical tag pointing at the unfiltered category URL. A shopper filtering jackets by red and medium still gets a working page with working URLs, but the ranking signal consolidates on /men/jackets.html rather than fragmenting across hundreds of variants. Implement this in the layered navigation block, not per template, so custom category layouts inherit it.

Robots and parameter policy. Canonical alone does not stop crawling; it only consolidates ranking. To protect crawl budget, block the parameter space:

User-agent: *
Disallow: /*?*cat=*&*cat=*
Disallow: /*?*color=*&*size=*
Disallow: /*?*price=*
Disallow: /*?*product_list_order=*
Disallow: /*?*product_list_dir=*
Disallow: /*?*p=*&*

The first two lines target multi-select combinations, the most explosive part of the space. Single-filter URLs may remain crawlable; two-or-more-filter combinations are blocked. Price ranges are blocked entirely since they produce near-infinite bands with thin content. Sorting and pagination parameters are blocked beyond the first page.

Complement this with Google Search Console URL parameter settings for legacy coverage: mark color, size, price, and sort parameters as narrowing content without changing it, so Google deprioritizes their combinations even when discovered through old links.

The allowlist. A small number of filter combinations deserve to rank: searches with real volume like "men leather jackets black" or "running shoes size 12 Nike." Maintain an explicit allowlist of high-value filter landing pages, each with a curated title, description, and introductory copy. These pages get self-referencing canonicals, internal links from category content, and sitemap entries. Everything not on the allowlist follows the default block-and-canonicalize rule. The allowlist is reviewed quarterly against search analytics; pages with no impressions graduate back to blocked status.

AJAX filters and URL state

Modern layered navigation updates results via AJAX without a page reload. That is good for shoppers and dangerous for SEO if mishandled in either direction.

For blocked combinations, AJAX-without-URL is ideal: the filter state lives in JavaScript, no new crawlable URL is created, and the crawler never sees the combinatorial space. For allowlisted landing pages, the opposite is required: each page needs a stable, server-rendered URL with full HTML content. A landing page that only renders its products after JavaScript execution is invisible to most crawlers and flaky for the rest.

The implementation pattern: filter interactions update the URL fragment or push state for shareability, but the server renders the base category plus any allowlisted combination as complete HTML. Test each allowlisted URL with JavaScript disabled. If the product list is empty, the page is not indexable regardless of what the canonical says.

Protecting server performance from crawler traffic

Faceted pages are expensive. Each combination runs layered navigation aggregation queries, and facet counts (the per-option product numbers) often require additional aggregation per filter. When crawlers hit thousands of combinations, the database absorbs load that converts to zero revenue.

Three defenses:

1. Cache filtered pages aggressively. Blocked combinations can carry long full-page-cache lifetimes since only shoppers (not ranking) depend on them. Vary cache by the exact parameter set, and cap the cache key space so pathological combinations do not evict productive entries.
2. Compute facet counts once per base category where possible. Live counts per combination are a nice shopper touch but multiply query cost. Consider static or periodically refreshed counts for deep combinations.
3. Rate-limit crawler impact at the edge. Monitor bot traffic share in access logs. If crawlers exceed a healthy fraction of layered navigation hits, tighten robots disallows before scaling hardware. No server tier fixes an unbounded URL space.

Search Console validation steps

1. Compare indexed URL count against catalog size. Indexed URLs should track products plus categories plus the allowlist, not multiples of it. A ratio above 3:1 on a filtered catalog signals leakage.
2. Review the Page indexing report for Crawled - currently not indexed. A large and growing count here usually means Google is discovering filter combinations, crawling them, and discarding them. Each one still cost budget.
3. Inspect an allowlisted landing page and a blocked combination with URL Inspection. The allowlisted page should show self-referencing canonical and indexed status; the blocked one should show the base category as canonical.
4. Check Crawl stats for URL patterns. If parameterized URLs dominate crawled pages while product pages stagnate, the robots policy has gaps. Close them parameter family by parameter family.

Pitfalls

• noindex on pages that already earned links. If a filter combination attracted backlinks, noindex discards that equity. Redirect the URL to the closest allowlisted page or the base category instead, preserving the link value.
• AJAX filters with no URL state for the pages you do want indexed. Allowlisted landing pages need stable server-rendered URLs. A JavaScript-only filter state cannot rank.
• Facet counts computed on every crawler hit. Aggregation queries per crawl multiply database load. Cache counts and cap the cache key space.
• Blocking everything including the money pages. A blanket disallow on all parameters also blocks the allowlist. Write disallows for combinations and keep single-filter landing URLs reachable.
• Letting marketing create filter links ad hoc. Every emailed or banner-linked filter URL becomes a crawl seed. Route campaigns through allowlisted pages only.

Closing

Faceted navigation is a shopper feature with crawler side effects. Canonical discipline, parameter blocking, a curated allowlist, and cached aggregation turn it from a budget sink into a controlled asset. With crawl waste contained, the next lever is earning rich results through structured data.

Next in this series:

magento2-structured-data-schema-org