Caching

Keep the popular stuff close, and everything feels fast

Caching is the single most common trick for making a slow system fast, and the idea is something you already do every day. You keep the things you use most within arm's reach: the coffee mug on your desk, not in the cupboard across the room. A cache does exactly that for data. It is also, famously, one of the easiest things to get subtly wrong, which is why it is worth understanding properly.

You will not write a caching layer. But "we added a cache and it got ten times faster" and "users are seeing old data" are both caching stories, and they come up constantly. Here is the whole picture: how caches work, where they live, what they throw away, how they handle writes, and the genuinely hard problem at the center of it all.

What a cache is, and hit versus miss

A cache is a small, very fast store that holds copies of data you ask for a lot, so you do not have to fetch it from the slow place every time. The slow place is usually the database or a faraway server. The cache sits in front of it, in fast memory (the common tool is Redis).

The flow is simple. A request comes in, and the system checks the cache first. If the data is there, that is a cache hit: it is returned instantly, and the database is never touched. If it is not there, that is a cache miss: the system goes to the database (slower), gets the data, and on the way back drops a copy in the cache so the next request is a hit.

The number everyone watches is the hit rate, the share of requests served from the cache. A 95 percent hit rate means only one request in twenty ever bothers the database, which is why a good cache does not just make things faster, it dramatically reduces load and cost on everything behind it. Think of a librarian who keeps the ten most-requested books on the front desk: most people are handed one instantly, and only the rare request sends them walking back into the stacks.

Why caching works at all: the 80/20 rule

Caching is not magic; it works because of a pattern that shows up almost everywhere: a small slice of the data gets the overwhelming majority of the requests. Roughly 80 percent of traffic asks for 20 percent of the content. A few videos go viral while millions sit unwatched; a handful of products are bought constantly; the same few pages get loaded over and over.

That lopsidedness is the whole opportunity. If you cache just that hot 20 percent, you serve the large majority of requests from fast memory while holding only a small amount of data. You get most of the benefit for a fraction of the cost. If requests were perfectly random across billions of items, caching would barely help, but they almost never are.

Caches live at every layer

It is tempting to think of "the cache" as one thing, but in reality a request passes through a whole series of them, and each one it satisfies early is a win. Your browser caches files it already downloaded, so a second visit does not re-fetch them. A CDN caches static content at the edge, close to users. The application keeps a cache of hot results in fast memory. Even the database has its own internal cache of recent queries.

A request tries each in turn, and the earlier it is answered, the faster and cheaper it is. The very best request is the one that is satisfied by the browser's own cache and never travels anywhere at all. When engineers optimize performance, a lot of the work is pushing answers to live as early in this chain as possible.

The cache is full: what gets thrown out?

A cache is deliberately small and fast, that is the entire point, so it fills up quickly, and then adding something new means throwing something out. The rule for what to throw out is called the eviction policy.

The most common is LRU, least recently used: evict whatever you have not touched in the longest time, on the bet that if you have not wanted it lately, you probably will not want it soon. It is exactly how you would clean a crowded desk, the papers you have not touched in weeks go first. A close cousin is LFU, least frequently used, which keeps the items asked for most often regardless of when. Either way the goal is the same: hold on to the hot data and let the cold data fall out. A well-tuned eviction policy is a big part of keeping the hit rate high.

Writing: keeping the cache and the database in step

Reading from a cache is easy. The interesting decisions come when data changes, because now the copy in the cache and the truth in the database can disagree. There are three common strategies, and the choice is a real tradeoff.

Write-through updates the cache and the database together on every write. The cache is never stale, which is safe, but every write pays the cost of two updates. You use it for data that must always be correct, like a balance.

Write-back updates the cache immediately and writes to the database a little later, in the background. This makes writes very fast, but it carries a risk: if the system crashes before the delayed write happens, that change is lost. You use it for high-speed data where a rare loss is tolerable.

Write-around skips the cache on writes entirely, sending them straight to the database; the cache only fills up later, on reads. This avoids filling the cache with data nobody reads back, at the cost of the first read after a write always being a miss. You use it for write-heavy data that is rarely read again.

None is universally right. The choice depends on how much you care about speed versus freshness versus the small chance of losing a recent write.

The hard part: staleness and invalidation

Here is the trap at the heart of caching, the one that causes the "why is it showing old data?" bugs. A cache holds a copy. The moment the real data changes, that copy is wrong, and until something fixes it, users see stale information: an old price, a deleted post that still appears, a profile that will not update.

There are two ways to keep the copy honest. The first is a TTL (time to live): each cached copy is stamped with an expiry, say sixty seconds, after which it is thrown away and re-fetched fresh. Simple and reliable, but the data can be stale for up to that whole window. The second is invalidation: actively clearing or updating the cached copy the instant the underlying data changes. Always fresh, but much harder to get right, because you have to make sure every code path that changes the data also remembers to clear every place it was cached.

This is why there is a famous engineering joke: there are only two hard things in computing, cache invalidation and naming things. When you hear a team debating TTLs or chasing a stale-data bug, this is the territory: the eternal tension between fast (keep the copy) and correct (refresh the copy).

When the cache itself becomes the problem

Caches usually save you, but they have their own failure modes, and the classic one is worth knowing: the cache stampede, also called the thundering herd. Picture a very popular item that everyone is reading from the cache. Its TTL expires, and it vanishes. Now, in the same instant, every one of those requests misses, and they all rush to the database at exactly the same moment to rebuild it. The database, which was happily idle behind a 99 percent hit rate, suddenly takes the full flood and can fall over, the very thing the cache existed to prevent.

The fixes are clever but simple in spirit: use a lock so that only the first request rebuilds the item while the others wait, or serve the slightly stale copy while one request refreshes it in the background. The lesson for a TPM is that "we have a cache" is not the end of the reliability story, how the cache behaves when it expires or empties matters just as much as when it is full.

What actually goes wrong

The recurring caching problems are all variations on the themes above:

Stale data. The cache was not invalidated when the data changed, so users see an old value. The most common caching bug there is.

A low hit rate. The cache is too small, or the eviction policy is wrong, or the data is not actually skewed, so most requests miss anyway and the cache adds overhead without much benefit.

The stampede. A hot item expires and the herd floods the database. Without a lock or stale-serving, the cure becomes a new outage.

Caching the wrong thing. Per-user or constantly-changing data gets cached when it should not be, causing either staleness or a hit rate so low the cache is pointless.

Thrashing. The working set is bigger than the cache, so items are evicted just before they are needed again, and almost everything is a miss.

Why a TPM should care, and what to ask

Caching is one of the cheapest, highest-leverage performance wins there is, and also a common source of subtle bugs, so it is worth a few sharp questions:

This is slow and read-heavy, do we cache it, and if not, why not?

What is our hit rate, and what is the plan if it is low?

When this data changes, how does the cache find out, TTL, or active invalidation? (This is the staleness question, and it is the one that prevents "old data" incidents.)

What happens when a hot cached item expires, are we protected against a stampede on the database?

Are we caching anything per-user or sensitive that we should not be?

Ask those, and you will catch both the easy ten-times-faster win that nobody set up, and the stale-data bug that is about to confuse every user, which are the two halves of getting caching right.