Scaling and Distributed Systems

What to do when one machine is no longer enough

Every system starts simple enough to run on a single computer. Then it grows, more users, more data, more requests, and that one machine begins to strain. Scaling is the craft of handling that growth, and once you commit to it past a certain point, you enter the world of distributed systems: many machines working together to act like one. That world has its own rules, its own vocabulary, and its own ways of breaking, and a TPM who understands them can tell the difference between a quick tuning task and a multi-quarter re-architecture.

You will not build a distributed database. But "we need to scale," "we have to shard," and "we lost the primary but failed over cleanly" are all conversations you will sit in, and this article gives you the map: the two ways to grow, how data gets split and copied, how machines agree, how they survive failure, and the order to reach for these moves.

Two ways to grow: up or out

There are exactly two ways to handle more load, and everything else builds on them.

Scaling up (vertical) means making your one machine bigger: more processors, more memory, faster disks. It is the simplest possible fix, because you change nothing about your software, you just buy a stronger computer. The catch is that there is a ceiling. The biggest machine money can buy is still one machine, and one day you hit its limit, with the price climbing steeply as you approach it. Scaling up buys you time, not a future.

Scaling out (horizontal) means adding more machines and spreading the work across them, with a load balancer in front sharing the requests. There is no ceiling, you just keep adding machines, which is how every large system on earth is built. But it introduces a new problem that the rest of this article is really about: now many machines have to coordinate, and coordination is genuinely hard.

The instinct should be: scale up while it is cheap and easy, scale out when you truly must, because scaling out is where the real engineering cost lives.

The catch with scaling out: state

Before the techniques, one idea makes scaling out possible or painful: state, meaning the data a server remembers between requests.

A stateless server remembers nothing about you between requests; everything it needs comes with each request or from a shared database. This is the dream for scaling out, because every server is interchangeable. The load balancer can send you to any of them, and if one dies, another picks up instantly, no one notices. A stateful server, by contrast, remembers something about you (your session, your data) in its own memory, which means you have to keep coming back to that exact machine, and if it dies, your state dies with it.

This is why modern systems work so hard to keep servers stateless, pushing all the real state into shared databases and caches. Stateless servers are what let you add and remove machines freely. When you hear "we need to make this service stateless," that is what they are buying: the freedom to scale out cleanly.

Splitting the data: sharding

Stateless servers are easy to multiply. The database is the hard part, because the data has to live somewhere, and eventually it is too big or too busy for one machine. The answer is sharding: splitting the data across several databases, where each one, a shard, holds only a slice.

The trick is choosing how to split. A simple example: put users with names A through H on the first shard, I through P on the second, Q through Z on the third. A router knows which shard owns which key, so a request for a given user goes straight to the right one. Now each machine holds a third of the data and handles a third of the load, and crucially, this is how you scale writes, since each shard accepts writes for its own slice independently.

The hard part of sharding is picking a split that stays even. If you shard a social app by celebrity and one celebrity gets a billion followers, that shard becomes a "hot shard," overloaded while the others sit idle. Choosing a good shard key, and dealing with hot spots, is one of the genuinely difficult parts of scaling, which is why "we need to re-shard" is never a small task.

Copying the data: replication

Sharding handles size. Replication handles two other things: read load and survival. With replication, you keep complete copies of the data on several machines. The usual arrangement is one leader that accepts all the writes, and several followers that each keep a copy and serve reads.

This buys you two big wins. First, read scaling: a read-heavy app can spread its reads across many followers, so you can serve enormous read traffic by just adding more copies. Second, survival: if the leader machine dies, one of the followers can be promoted to become the new leader, and the system keeps going.

That promotion, called failover, is the heartbeat of high availability. A leader can crash or be taken down for maintenance, and after a brief blip, a follower steps up and writes resume, with no data lost. The cost of replication is the consistency lag we met in the database article: a write reaches the leader instantly but takes a moment to stream to the followers, so a read from a follower might briefly see slightly old data. For most apps that is a fine trade for the scale and resilience it buys.

Growing without reshuffling: consistent hashing

Here is a subtle but important problem. When you spread data across machines by hashing keys to them, the naive method (take the key, divide by the number of machines, use the remainder) has a nasty flaw: the moment you add or remove a machine, the number changes, and almost every key suddenly maps to a different machine. Adding one server would force you to move nearly all your data, an operation so disruptive it could take the system down.

Consistent hashing is the clever fix, and it is worth knowing by name because it appears everywhere caches and databases scale. Picture all the machines arranged around a ring, and each key placed at a point on that ring, owned by the next machine clockwise. Now when you add a new machine, it slots into one spot on the ring and only steals the slice of keys between it and its neighbor. Everything else stays exactly where it was.

The payoff is that you can grow and shrink the fleet smoothly, moving only a small fraction of the data each time instead of reshuffling everything. You do not need the math. You need to recognize that "we use consistent hashing" means the system was built to scale up and down gracefully, and its absence is why some systems dread adding capacity.

Getting machines to agree: consensus

Once you have many machines, they sometimes need to agree on a single fact, like which one is currently the leader, or whether a particular write really happened. This is called consensus, and getting independent machines to agree despite failures and delays is one of the deepest problems in computer science.

The everyday version you will hear is a quorum: a majority of the machines must agree before anything counts. With five machines, at least three must say yes. This majority rule is what keeps a distributed system from splitting into two halves that each think they are in charge (a "split brain"), which would be catastrophic. The well-known algorithms here are called Raft and Paxos, and tools like ZooKeeper exist just to do this agreement reliably. You will not implement them, but when a system needs an odd number of nodes (three, five) and talks about "losing quorum," this is why.

Designing for failure: the distributed-systems reality

The single biggest mindset shift in distributed systems is this: in a fleet of hundreds of machines and a network connecting them, something is always broken. A machine is down, a disk is failing, the network between two data centers is slow or briefly cut. The naive assumption that the network is reliable and instant is so common, and so wrong, that engineers have a famous list called "the fallacies of distributed computing" warning against it.

So good distributed systems are designed to expect failure rather than be surprised by it. They replicate data so no single loss matters. They use timeouts and retries (carefully, to avoid the retry storms from the API article). They build in fault tolerance so the system as a whole keeps working even as individual parts fail. The goal is never "nothing fails," it is "things fail all the time and users never notice." That is the real meaning of reliability at scale, and it is why a one-machine app and a thousand-machine system are different kinds of thing, not just different sizes.

The scaling ladder

Put it together and there is a sensible order to reach for these moves, from cheapest to most expensive. The mistake teams make is jumping straight to the hardest one.

First, make it cheaper before making it bigger: add a cache, add a missing index, fix the slow query. A huge amount of "we need to scale" turns out to be one bad query. Second, scale up to a bigger machine, simple and quick, and it buys real time. Third, add read replicas to spread the reads. Fourth, shard to split the writes across machines, a serious step. Fifth, and only if you truly must, go fully distributed with the whole apparatus of consensus and coordination, the big re-architecture.

Each rung is harder and more expensive than the last. The discipline is to climb only as high as you actually need. When a team proposes jumping to step five, the useful question is often "have we done steps one through three?"

What actually goes wrong

The classic distributed-systems failures all come from coordination:

The premature re-architecture. A team rebuilds for massive scale they do not have yet, paying enormous complexity costs to solve a problem a cache would have fixed.

The hot shard. The data was split unevenly, so one shard takes most of the load while the rest idle, and the "scaled" system is bottlenecked on one machine anyway.

The split brain. Without proper consensus, a network partition leaves two machines both believing they are the leader, and they accept conflicting writes. Data corruption follows.

The retry storm. Something slows down, every client retries at once, and the retries pile onto the struggling system until it collapses. Scale makes this worse, not better.

The failover that was never tested. Replication exists, but nobody verified the promotion works, so when the leader actually dies, the failover fails too.

Why a TPM should care, and what to ask

Scaling is where cost, complexity, and reliability collide, and the decisions are expensive to reverse, so a few sharp questions pay off:

Before we re-architect, have we tried the cheap wins, caching, indexing, a bigger machine?

When we say "scale," do we mean more reads (replicas) or more writes (sharding)? They are very different projects.

If a machine or a whole zone dies, does the system fail over automatically, and have we actually tested that?

Are our servers stateless, so we can add and remove them freely?

How big a project is this really, a tuning task, or a true distributed-systems re-architecture with a multi-quarter timeline?

Ask those, and you will tell the difference between a team that needs an afternoon and a team that needs two quarters, which is exactly the judgment a scaling conversation demands.