Case study: customers lost their carts every Tuesday

Field notes — a real production bug, project name withheld. Symptom → investigation → fix → lesson.

The symptom

Every Tuesday morning the support inbox filled with the same complaint: "my cart was full last night and now it is empty." Guest carts, logged-in carts, wishlist-adjacent quote items, all gone. The pattern was precise enough to set a watch by: complaints clustered between 08:00 and 10:00 on Tuesdays, never on other weekdays, and the affected customers had all been active the previous evening. The store ran Magento 2 with Redis for sessions and page cache, sessions had a 24-hour lifetime, and nothing in the release calendar shipped on Mondays. The first theory was a botched deployment or a midnight cache flush, but the deploy log was clean and nobody admitted to flushing anything.

What made it slippery was that the sessions were not expiring, they were vanishing. A normal session expiry leaves the quote row orphaned in quote with an old updated_at, recoverable and visible. Here the quotes were still in MySQL, intact, but the frontend session that pointed at them was gone, so customers arrived as strangers and Magento created a fresh empty quote. Logged-in customers who signed back in sometimes got their cart back (because the persistent customer quote reattached), which made the team chase a frontend cookie bug for two weeks. Guest customers never got anything back, which kept the complaint volume high.

The Tuesday-only cadence was the thread worth pulling. Nothing in Magento cares what day it is. Something outside Magento did.

Investigation

The investigation started with Redis itself during a quiet hour:

redis-cli -n 2 INFO memory
redis-cli -n 2 INFO stats

The output told the story in two lines. used_memory_human sat at 7.2 GB against maxmemory_human of 8 GB, and evicted_keys was non-zero and climbing. The instance held sessions (database 2), default cache (database 0), and full-page cache spillover (database 1) all on one Redis node, a layout inherited from a single-server setup that had never been revisited after the move to dedicated infrastructure. The eviction policy, read from the config, was the default the hosting panel shipped:

redis-cli CONFIG GET maxmemory-policy
# 1) "maxmemory-policy"
# 2) "volatile-lru"

volatile-lru evicts keys with an expiry, least-recently-used first, when memory is full. Session keys have a TTL, so they are eligible for eviction. Cache keys with no TTL are not. That single setting inverted the intended priority: when memory ran short, Redis protected the never-expiring page-cache entries and ate live shopping sessions instead.

The Tuesday trigger was the backup and report job. Every Tuesday at 02:00 a cron chain ran a full catalog export, a sitemap regeneration, and an offsite backup that snapshotted Redis with BGSAVE. The export warmed hundreds of thousands of cache entries (every product page touched for the sitemap), memory usage spiked from ~6 GB to the 8 GB ceiling within minutes, and Redis started evicting. INFO stats after the fact showed evicted_keys jumping by 40,000-60,000 exactly in the 02:00-02:30 window each Tuesday. Cross-referencing with the Magento session lifetime (admin > Stores > Configuration > Web > Session plus env.php session: lifetime: 86400) confirmed the victims: evening shoppers whose session keys were the least recently touched at 02:00 were the first evicted.

Two corroborating details sealed it. First, the redis-cli --bigkeys scan showed the page-cache database holding multi-hundred-kilobyte entries with no expiry, accumulated over weeks, acting as the memory ballast that left no headroom. Second, the env.php session configuration had disable_locking: 0 with a short break_after_frontend: 5, so during the eviction window the session fallback path thrashed: each request for a missing session created a new empty one, wrote it back, and pushed memory pressure marginally higher. The access log showed a burst of fresh PHPSESSID values issued between 02:00 and 03:00 on Tuesdays, matching the eviction counts.

The cookie theory died cleanly. The frontend cookie domain, path, and SameSite attributes were identical across all days, and Monday-night sessions that survived past 03:00 behaved perfectly. Only sessions alive during the 02:00 spike disappeared, and only on the node where cache and sessions shared memory.

Fix

The fix separated what should never have shared memory, in this order.

First, stop the bleeding the same day: raise maxmemory from 8 GB to 12 GB and switch the Tuesday export to run with cache warming disabled (--no-cache-warm equivalent in the custom export script, plus sitemap generation pointed at a read replica instead of the live cache backend). Evictions dropped to zero the very next Tuesday, confirming the diagnosis before any architecture changed.

Second, split Redis by purpose. Sessions moved to their own Redis instance with maxmemory-policy noeviction and persistence enabled:

# session instance
maxmemory 4gb
maxmemory-policy noeviction
save 60 1000
appendonly yes

// app/etc/env.php
'session' => [
    'save' => 'redis',
    'redis' => [
        'host' => 'redis-session.internal',
        'port' => '6379',
        'database' => '0',
        'disable_locking' => '0',
        'max_concurrency' => '6',
        'break_after_frontend' => '15',
    ]
],

With noeviction, a full session store returns OOM errors on write instead of silently deleting shoppers, which is the correct failure mode: loud and pageable beats quiet cart loss. Cache and FPC stayed on the original node with volatile-lru (later allkeys-lru with TTLs enforced on every entry), where eviction is harmless and expected.

Third, cap the ballast. Page-cache entries got explicit TTLs, the sitemap job stopped writing through the live cache backend, and a weekly redis-cli --bigkeys review became part of ops. Memory on the cache node stabilized around 60 percent of max instead of riding the ceiling.

Fourth, add the alert that should have existed from day one: a monitor on evicted_keys rate plus used_memory_rss ratio for the session instance. Any eviction on the session node now pages, because the correct number of session evictions is zero, forever.

Verification ran over three Tuesdays: zero evicted_keys growth on the session instance, zero cart-loss complaints in the support queue, and the Tuesday export completing with cache-node memory never crossing 75 percent.

Lesson

Never share a Redis instance between sessions and cache, and never run session storage under an evicting maxmemory-policy. volatile-lru sounds safe until you realize session keys are exactly the keys with TTLs, which makes them the first victims when a backup or export spikes memory. Session storage must be noeviction on its own instance with persistence, so full means loud errors instead of silent cart loss. And when a bug respects the calendar, look outside the application: Magento does not know it is Tuesday, but the backup cron does. INFO memory, INFO stats, and the evicted_keys counter around the suspect window will name the culprit in minutes.

This case illustrates the cache architecture in /blog/magento2-cache-varnish-redis.