Caching strategies

Why this matters

Caching is the fastest way to scale reads — and the easiest way to introduce silent correctness bugs. Every senior engineer has a story about caching gone wrong. The skill is designing caches with eyes open about staleness, invalidation, and failure modes.

From first principles: why does caching work?

Two reasons:

1. Locality of reference: a small fraction of data is accessed disproportionately often. The 80/20 rule: 20% of your data gets 80% of the requests. Caching that 20% in fast memory makes 80% of requests fast.
2. Predictability of access patterns: yesterday's hot keys are likely today's hot keys. The cache adapts to access patterns automatically.

Caching trades freshness for speed. Every cache is, by definition, possibly serving stale data. The system design question is: how much staleness can the user tolerate, and how do we manage it?

Where caches live

Browser cache (HTTP cache headers)
   ↓
CDN (Cloudflare, Akamai, CloudFront) — closest to user
   ↓
API gateway cache (Varnish, NGINX) — at the edge of your infra
   ↓
App-tier cache:
   - In-process (Caffeine, sync.Map, EhCache) — nanosecond access, per-process
   - Off-box (Redis, Memcached) — millisecond access, shared across processes
   ↓
DB-internal cache (Postgres buffer pool, MySQL InnoDB buffer pool)
   ↓
DB persistent storage

Each layer trades freshness for speed. The closer to the user, the faster; the closer to the DB, the more authoritative.

The four classic cache patterns

flowchart TB
    subgraph CA["Cache-aside (lazy load)"]
        CAa[App] -->|"1 read"| CAc[(Cache)]
        CAc -.miss.-> CAa
        CAa -->|"2 read on miss"| CAd[(DB)]
        CAa -->|"3 fill"| CAc
    end
    subgraph WT["Write-through"]
        WTa[App] -->|write| WTc[(Cache)]
        WTc -->|sync write| WTd[(DB)]
    end
    subgraph WB["Write-back (write-behind)"]
        WBa[App] -->|write| WBc[(Cache)]
        WBc -.async flush.-> WBd[(DB)]
    end
    subgraph RA["Refresh-ahead"]
        RAa[App] --> RAc[(Cache)]
        RAc -.background refresh
before TTL.-> RAd[(DB)]
    end

Pattern 1: Cache-aside (lazy loading)

The most common pattern. Application explicitly manages cache.

get(key):
  if cache.has(key): return cache[key]
  value = db.get(key)
  cache.set(key, value, ttl=300)
  return value

set(key, value):
  db.set(key, value)
  cache.delete(key)  // or update — see invalidation discussion

Pros: simple; cache only contains what's been requested.

Cons: cold cache → cache stampede on hot keys.

Pattern 2: Write-through

Writes go to cache and DB synchronously.

set(key, value):
  cache[key] = value
  db.set(key, value)  // synchronous

Pros: cache always consistent with DB.

Cons: slower writes (must hit both); pollutes cache with data that may never be read.

Pattern 3: Write-back (write-behind)

Writes go to cache; DB updates asynchronously.

set(key, value):
  cache[key] = value
  enqueue async db.set(key, value)

Pros: fast writes.

Cons: data loss risk if cache fails before flush. Rarely application-appropriate for durable data; used by OS page caches and some session stores.

Pattern 4: Refresh-ahead

Refresh hot keys before they expire.

get(key):
  value, age = cache[key]
  if age > threshold * TTL:
    async refresh(key)  // refresh in background
  return value

Pros: predictable latency for hot keys (no cold misses).

Cons: wasted refresh work for keys that aren't subsequently read.

Bonus: Write-around

Writes go directly to DB, skipping cache.

set(key, value):
  db.set(key, value)  // next read populates cache

Avoids polluting cache with write-once data. Good for bulk imports, audit logs, data rarely read soon.

Cache stampede — the textbook production failure

sequenceDiagram
    participant C1 as Client 1
    participant C2 as Client 2
    participant Cn as Client N (=1000)
    participant Cache
    participant DB

    Note over Cache: Hot key 'home_feed' just expired
    C1->>Cache: GET home_feed
    Cache-->>C1: MISS
    C2->>Cache: GET home_feed
    Cache-->>C2: MISS
    Cn->>Cache: GET home_feed
    Cache-->>Cn: MISS

    Note over C1,Cn: All 1000 clients now hit the DB
    C1->>DB: rebuild home_feed
    C2->>DB: rebuild home_feed
    Cn->>DB: rebuild home_feed
    Note right of DB: DB overloaded ❌

A popular cached key expires. 10,000 concurrent requests all miss simultaneously. They all try to recompute / hit the DB. DB CPU spikes. DB slows. All 10,000 requests queue. Latency goes from 5ms to 30s. Cascading failure.

Three fixes:

Probabilistic early expiration (XFetch)

def xfetch(key, recompute_fn):
    cached = cache.get_with_metadata(key)  // returns value, expires_at, recompute_cost_ms
    if cached:
        now = current_time()
        # As expiry approaches, increasing fraction of reads recompute proactively
        if now + cached.recompute_cost_ms * beta * (-math.log(random.random())) >= cached.expires_at:
            new_value = recompute_fn()
            cache.set(key, new_value, recompute_cost=time_taken)
            return new_value
        return cached.value
    # Cold miss
    return recompute_fn_and_cache()

With beta = 1, probability of early recompute scales gently as TTL approaches. Distributes recompute load smoothly in time.

Singleflight (request coalescing)

def get(key):
    if key in in_flight:
        return in_flight[key].await()  // wait for the other one
    promise = new Promise()
    in_flight[key] = promise
    try:
        value = expensive_db_call(key)
        promise.resolve(value)
    finally:
        delete in_flight[key]
    return value

Per-process map of in-flight cache fetches. Concurrent requests for the same key share one DB call. The Go singleflight package is the canonical implementation. Reduces N concurrent misses to 1.

Soft TTL + Hard TTL

Soft TTL = "data is stale but usable while we refresh"
Hard TTL = "data must be evicted"

def get(key):
    value, age = cache[key]
    if not value: return refresh_and_set(key)  // hard miss
    if age > soft_ttl: async refresh(key)  // serve stale, refresh in background
    return value

Used by HTTP's "stale-while-revalidate" (RFC 5861), Next.js Incremental Static Regeneration.

Eviction algorithms

Caches are finite. What do you evict when full?

For most production caches: Caffeine's W-TinyLFU (in-process Java) or Redis's sampled-LRU/LFU (off-box). Caffeine's blog and benchmarks are excellent: github.com/ben-manes/caffeine/wiki/Efficiency.

Invalidation — the hard problem

"There are only two hard things in computer science: cache invalidation and naming things." — Phil Karlton

Three approaches, each with tradeoffs:

TTL-based

Everything expires after N seconds. Simplest; tolerable for eventually-consistent reads.

The catch: every read between writes returns stale data up to TTL. For TTL=60s and a user who just updated their profile, they see their old profile for up to 60 seconds. Sometimes fine, sometimes not.

Event-based invalidation

On write, emit an invalidation event; cache subscribers evict.

app: writes data to DB
app: publishes invalidate(key) to Kafka / Redis Pub/Sub / events bus
caches everywhere: subscribe, evict on event

Pros: near-real-time freshness.

Cons: race conditions (event delivered before DB write committed visible). Solution: emit invalidation event after commit, possibly via CDC tailing the WAL.

Versioned cache keys

Embed a version in the key. Bumping the version effectively invalidates all old entries.

Cache key: user:123:v3
On update of user 123: bump version to v4
New reads use key user:123:v4; old key user:123:v3 expires via TTL

Pros: atomic; no event-distribution complexity.

Cons: need a counter / version source of truth.

CDC-driven invalidation

Tail the DB's WAL via Debezium → emit row-change events to Kafka → cache subscribers invalidate.

Pros: invalidation is causally tied to DB writes (no missed events). Used by Facebook TAO, many large-scale systems.

Cons: complexity; lag from WAL→Kafka→subscriber (typically <1s).

Cache topologies at scale

Real production cache architectures

Facebook TAO

The cache for Facebook's social graph. Two tiers: per-region L1 (web tier-local), shared L2 (regional). Backed by sharded MySQL. ~99% hit rate at L1. Invalidation via TAO's own pub/sub system. 1 billion+ requests per second at peak. Whitepaper: TAO: Facebook's Distributed Data Store for the Social Graph (USENIX ATC 2013).

Twitter's timeline cache

Timeline materialized on write. When you tweet, your tweet is pushed into the cached timelines of all followers. Reads are pure cache lookups. Tradeoff: massive write amplification for celebrities (Lady Gaga problem). For high-fan-out users, switch to fan-out-on-read.

Netflix EVCache

Memcached-based with global replication. Each region has a complete copy; writes propagate. Optimized for read-heavy workloads (recommendations, metadata). Open-sourced: github.com/Netflix/EVCache.

Discord's session cache

Hot user session data in Redis sharded by user_id. Backed by Cassandra for durability. Real-time presence events update the cache directly.

Cache hit ratio targets

Below 50% means your cache strategy is wrong, your data has poor locality, or your TTL is too short. Investigate before adding capacity.

Negative caching

Cache "not found" responses too, with short TTL.

def get_user(id):
    cached = cache.get(f"user:{id}")
    if cached == "NOT_FOUND_SENTINEL":
        return None
    if cached:
        return cached
    value = db.get_user(id)
    if value is None:
        cache.set(f"user:{id}", "NOT_FOUND_SENTINEL", ttl=30)
    else:
        cache.set(f"user:{id}", value, ttl=300)
    return value

Without negative caching, attackers (or buggy clients) can DDoS your DB by repeatedly requesting non-existent IDs. Used heavily by DNS resolvers. Underused in app caches.

Cache penetration, breakdown, avalanche

Interview drills

Red flags

• "Just add Redis" without analysis of hit ratio, invalidation, eviction.
• Doesn't mention TTL.
• No story for invalidation on writes.
• Ignores cache stampede.
• Caches everything indiscriminately.
• "Write-through always" — ignores that it slows writes.