Case study: a reindex took the search page down

Field notes — a real production bug, project name withheld. Symptom → investigation → fix → lesson.

The symptom

Shortly after lunch on a normal weekday, the search page broke. Every query, from a single SKU to a generic term like "shoes", returned the same empty result set: "Your search returned no results." Category pages were fine, product pages were fine, checkout was fine. Only search was down, and it stayed down for almost an hour before recovering on its own. No deploy had gone out that morning, no config change, no traffic spike. The support queue filled with screenshots of an empty search page from customers who could see the products they wanted in the navigation but could not find them through search.

What made it confusing was the timing. Nothing in the release log explained it, and the application logs were clean. PHP-FPM was healthy, MySQL was healthy, OpenSearch was responding to pings. The search request itself returned HTTP 200 with a valid layout, just zero hits. It looked like the catalog had been emptied, but the admin still showed 40,000 searchable SKUs.

Investigation

The first clue came from the cron schedule. A teammate noticed that indexer_update_all_views had started at 12:40, right when the tickets began. Someone had clicked "Reindex" on catalogsearch_fulltext from the admin that morning to fix a missing product, and a full reindex was still the default behavior for that button. On a 40,000 SKU catalog with several store views, a full reindex takes 40 to 50 minutes.

The second clue came from OpenSearch itself. Listing the indices during the next reproduction showed the pattern clearly:

magento2_product_1_v12  docs.count: 0
magento2_product_1_v13  docs.count: 41208
magento2_product_2_v8   docs.count: 0

Magento creates a new versioned index on every full reindex and points the alias at it only when the build finishes. Checking the alias confirmed the failure mode:

curl -s localhost:9200/_cat/aliases/magento2_product_1?v

During a healthy reindex the alias keeps pointing at the old, fully populated index while the new one builds. In this case the alias had been moved to the fresh, still-empty index within the first minutes. Every search request from the storefront then queried an index with zero documents and correctly returned zero results. The storefront was working as coded; it was faithfully reporting what the empty index contained.

Why did the alias move early? Two settings combined. The indexer's batch configuration used a small bulk size with synchronous alias switching, and a custom search module had overridden the reindex flow to "prepare" the new index by moving the alias first so that partial results would appear faster. The intent was to make new products visible sooner. The effect was the opposite: shoppers saw nothing at all until the bulk import caught up. On a small staging catalog with 500 products the gap was a few seconds and nobody noticed. On production with 40,000 products and three store views, the gap was 45 minutes of downtime.

There was a compounding factor. The OpenSearch cluster ran with zero replicas and a single data node to save cost, and the bulk thread pool saturated during the import. Search queries that did arrive competed with the bulk indexing threads for the same node, so even the requests that hit the older replica path timed out. The opensearch.log showed EsRejectedExecutionException on the write pool and slow query warnings on the search pool at the same timestamps as the support tickets. A monitoring gap hid all of this: the uptime check hit the homepage, not a search URL, so the status page stayed green while search was empty.

We reproduced the whole sequence on staging by triggering a full catalogsearch_fulltext reindex at noon with production-like data volume and watching the alias. The search page went empty within two minutes and stayed empty until the bulk import finished, exactly matching the production timeline.

Fix

The fix had three layers: stop the early alias swap, stop full reindexes during business hours, and give the storefront a fallback when the index is unhealthy.

First, we removed the custom early-alias logic and restored the stock behavior: build the new versioned index fully, then flip the alias atomically. The relevant change was in the module that wrapped the indexer action:

// Before: alias moved before bulk import so "new products appear fast".
$this->aliasResolver->switchIndexAlias($newIndexName);

// After: build first, verify document count, then switch.
$this->bulkImporter->import($newIndexName, $documents);
$this->assertDocumentCount($newIndexName, $expectedMin);
$this->aliasResolver->switchIndexAlias($newIndexName);

The document-count assertion was new. Before flipping the alias, the job now compares the new index's docs.count against a floor derived from the catalog size. If the new index holds fewer than 90 percent of the expected documents, the switch aborts and pages the on-call engineer instead of emptying search.

Second, we changed how reindexes get triggered. Full catalogsearch_fulltext reindex on schedule was disabled; the indexer was set to update on schedule with the changelog-based partial path, and full rebuilds were restricted to a nightly window:

bin/magento indexer:set-mode schedule catalogsearch_fulltext

The admin "Reindex" button path was left in place but now enqueues a background job with a warning banner instead of running inline during the request. A one-off full rebuild runs at 02:00 local time, when a 45-minute build affects nobody, and the bulk size was raised from 100 to 500 with two worker processes to shorten the window.

Third, we added a storefront guard. The search block now checks the result metadata before rendering the empty state. If the index reports zero documents for a store view that should hold tens of thousands, the controller logs a critical, keeps the previous alias, and serves a degraded category-browse fallback with a notice instead of a hard "no results" page. A synthetic monitor now queries a known SKU through the search URL every two minutes, so an empty index pages within minutes instead of surfacing through customer tickets.

Lesson

Treat the search alias as a deploy: never point live traffic at an index that has not been verified. The versioned-index pattern only gives zero-downtime reindex if the alias flip is the last step and is gated on document count. Any shortcut that moves the alias early turns a 45-minute background job into 45 minutes of downtime, and the gap scales with catalog size, which is why it passes on staging and fails on production. Schedule full search rebuilds like migrations, off peak and unattended, and monitor what the customer sees, a real search query with expected hits, not just the homepage. An uptime check that never searches will happily report green while every search returns nothing.

Illustrated in magento2-search-opensearch.