Why Your Cache Is Not
Improving Performance

A 65% cache hit rate sounds decent until you do the math. If your application handles 10,000 requests per second, 3,500 of those requests are full cache misses. Every single one of those misses goes to the database with the full latency penalty. You did not eliminate the bottleneck. You reduced it by two-thirds. Your database is still handling 3,500 requests per second instead of 10,000.

The hit rate problem compounds during traffic spikes. When load increases by 3x, your database suddenly faces 10,500 miss requests per second instead of 3,500. The cache did not protect the database from the spike. It just reduced the multiplier. And because cache eviction accelerates under memory pressure during spikes, hit rates often drop when you need them most. The system degrades exactly when performance matters most.

Static TTLs are the root cause. When you set a 5-minute TTL on a key, that key expires whether it is being accessed 100 times per second or zero times per second. Hot keys that should stay cached expire unnecessarily. Cold keys that should be evicted sit in memory consuming space. The result is a cache that wastes memory on data nobody needs while evicting data everyone needs.

Learn specific techniques to diagnose and fix low hit rates in our guide on how to increase cache hit rate.

Here is the math that most teams skip. Your database query takes 5ms. You add Redis. Redis returns the cached result in 2ms. You saved 3ms. That is a 60% improvement on that single query, which sounds good. But your page makes 8 API calls, each backed by a cache lookup. You just added 16ms of network overhead (8 calls at 2ms each) to serve a page that previously took 40ms in database time. Your net improvement is 40ms minus 16ms = 24ms of database time saved, but you added 16ms of cache network time. Total wall time went from 40ms to 32ms. A 20% improvement. Not 10x.

Network latency to Redis varies dramatically depending on deployment topology. Same-AZ Redis adds 0.3-0.5ms per round trip. Cross-AZ adds 1-2ms. Cross-region adds 5-15ms. Connection pooling overhead, serialization, and TCP handshake costs add another 0.2-0.5ms on top. Under load, these numbers increase as connection pools saturate and Redis single-threaded processing creates queuing delays. The cache that was supposed to eliminate latency has become a new source of it.

The fundamental issue is architectural: an out-of-process cache introduces a network hop that an in-process cache eliminates entirely. When your cache runs inside the application process, the round-trip time drops from milliseconds to microseconds. There is no serialization, no TCP overhead, no connection pooling. Just a memory lookup.

See how to eliminate network overhead from your cache layer in our deep dive on reducing Redis latency.

LRU (Least Recently Used) is the default eviction policy in Redis and most cache systems. It sounds logical: evict the data that was accessed least recently. The problem is that recency is a poor predictor of future access. A key that was accessed 3 seconds ago might never be accessed again. A key that was accessed 10 seconds ago might be needed 50 more times in the next minute. LRU cannot distinguish between these patterns. It evicts based on a single timestamp, not on actual access probability.

This problem gets worse with scan operations and batch jobs. A single background job that iterates over 50,000 keys will push every one of those cold keys to the top of the LRU list, causing mass eviction of genuinely hot keys. When the next burst of user traffic arrives, those evicted hot keys generate a wave of cache misses. The system recovers eventually, but the damage is done: a 30-second scan job caused 5 minutes of degraded performance for real users.

LFU (Least Frequently Used) partially addresses this by tracking access counts, but introduces its own problems. Keys that were popular last week but are no longer relevant accumulate high frequency counts and resist eviction. New keys that will become hot cannot displace them because they start with zero frequency. Every static eviction policy makes a trade-off that fails for some class of workload. The only way to evict correctly is to predict future access, not summarize past access.

Explore how machine learning replaces static eviction in our guide on reducing cache misses.

A cache stampede occurs when a popular key expires and hundreds or thousands of concurrent requests simultaneously discover the miss, all attempting to regenerate the cached value at the same time. Instead of one request hitting the database, 500 requests hit the database with the same query. The database spikes. Response times spike. Timeouts cascade. The cache that was supposed to protect the database from load just amplified it by 500x for a brief, devastating window.

Stampedes are invisible during testing because they only occur under production-scale concurrency. Your staging environment with 50 concurrent users will never trigger a stampede. Your production environment with 5,000 concurrent users will trigger stampedes on every popular key expiration. Traditional mitigations like lock-based recomputation (only one request regenerates the value) or jitter (randomizing TTLs) reduce the severity but do not eliminate the problem. Lock-based approaches introduce contention. Jitter just spreads stampedes over a wider window.

The correct solution is to never let popular keys expire in the first place. Predictive pre-warming detects when a key is approaching its TTL and proactively refreshes it before expiration. The key is always warm. No miss occurs. No stampede is possible. This requires knowing which keys are about to expire and which of those keys are hot enough to warrant pre-warming. Static rules cannot make this determination. Machine learning can.

Learn about stampede prevention techniques in our deep dive on cache stampede prevention.