Replication

Why this matters

Replication is the foundation of every high-availability system. Without it, a single server crashing means data loss and downtime. With it, you can survive failures, scale reads, place data near users, and roll out updates without downtime.

Replication is also the topic where interviewers probe hardest. Every "design system X at scale" question routes through replication choices: how do you keep multiple copies in sync? What's the failure mode when one crashes? How does a write get to all replicas?

From first principles: what problem are we solving?

You have data. You want to keep it accessible if any single machine fails. So you keep multiple copies on multiple machines. Easy in concept.

The hard part: keeping the copies in sync. Specifically:

• When a write happens, how does it propagate?
• What if two writes happen at the same time on different copies?
• What if the network between copies fails?
• What if a copy is offline for a while and then returns?
• What does a read see if copies disagree?

The history of distributed databases is essentially the history of answering these questions in different ways.

Three replication architectures

flowchart LR
    subgraph S1["Single-leader"]
        SC1[Client] -->|writes| SL((Leader))
        SC2[Client] -.reads.-> SR1((Replica))
        SL -->|WAL stream| SR1
        SL -->|WAL stream| SR2((Replica))
    end
    subgraph S2["Multi-leader"]
        MC1[Client] --> ML1((Leader))
        MC2[Client] --> ML2((Leader))
        ML1 ---|"replicate both ways"| ML2
    end
    subgraph S3["Leaderless (Dynamo-style)"]
        LC[Client] -->|"write W=2"| LN1((Node))
        LC -->|"write W=2"| LN2((Node))
        LC --> LN3((Node))
        LC2[Client] -->|"read R=2"| LN2
        LC2 --> LN3
    end

Single-leader (primary/replica)

One node is designated the primary (also called leader, master). All writes go to it. The primary propagates writes to one or more replicas (also called followers, slaves, secondaries). Reads can go to the primary (always fresh) or to replicas (possibly stale).

          Writes →  [Primary]
                       ↓ async or sync replication
                  [Replica 1] [Replica 2] [Replica 3]
          ← Reads can go here too

This is the simplest and most common pattern. Used by:

• Postgres (with hot standby replicas)
• MySQL (with binlog replication)
• MongoDB (in replica-set mode)
• Redis (master-replica)
• Most managed databases (Aurora, Cloud SQL, RDS)

Multi-leader (multi-master)

Multiple nodes accept writes. Each leader replicates to other leaders.

     Writes ↘    ↙ Writes
          [Leader A] ⇄ [Leader B]
              ↓             ↓
         [Replica]     [Replica]

Used for:

• Multi-region active-active (write locally in any region, replicate globally)
• Offline-first apps (local DB on device, sync when online)
• Datacenter redundancy with bidirectional flow

The hard problem: conflict resolution. Both leaders accept a write to row X "at the same time" (concurrent in network terms). Which wins?

Leaderless

All nodes are equal. Clients write to multiple nodes simultaneously and read from multiple nodes.

Client writes to W nodes:  [N1] [N2] [N3] [N4] [N5]
Client reads from R nodes: any R of {N1...N5}
If R + W > N: read and write sets overlap → at least one read sees the latest write

This is the Dynamo model. Used by:

• Amazon Dynamo (original, now DynamoDB internally)
• Cassandra (with tunable per-request consistency)
• Riak

Single-leader in depth — the workhorse

sequenceDiagram
    autonumber
    participant C as Client
    participant L as Leader
    participant R1 as Replica 1
    participant R2 as Replica 2
    C->>L: write(x = 1)
    L->>L: append to WAL
    par async to replicas
        L-->>R1: stream WAL
        L-->>R2: stream WAL
    end
    L-->>C: ack
    R1->>R1: apply
    R2->>R2: apply

Synchronous vs asynchronous replication

The most important configuration knob. Determines latency and durability.

How replication actually moves data

There are three implementations, in roughly decreasing order of "decoupling" between primary and replica:

Statement-based: replay the SQL statement on the replica. Fragile because of nondeterminism: NOW(), RANDOM(), sequences, triggers can all produce different results on different nodes. Largely abandoned for primary replication; sometimes used for analytics.

Physical / WAL shipping: ship the raw WAL records byte-for-byte. The replica is essentially a bit-by-bit copy of the primary's storage. Postgres physical replication, MySQL binary log in row format (sort of).

• Pros: fastest, identical replicas, simple.
• Cons: tightly coupled — replica must run same major version and identical schema. Cannot replicate across versions during upgrades. Cannot replicate to a different database.

Logical (row-based): decode the WAL into row-level change events: "row X in table Y changed from {a:1, b:2} to {a:1, b:3}." Replicate these change events. The replica applies them as regular inserts/updates.

• Pros: decouples primary and replica storage formats. Can replicate across major versions (zero-downtime upgrades). Can feed change events to non-database consumers (Kafka, Elasticsearch, data warehouses).
• Cons: slower than physical (more processing per row). Doesn't replicate DDL by default. More complex to operate.

Use physical replication for HA replicas. Use logical replication for everything else — online migrations, version upgrades, feeding analytics pipelines, building search indexes.

Change Data Capture (CDC) — the killer use of logical replication

If you can ship row-level changes from the WAL, you can build downstream pipelines:

Postgres → Debezium (CDC tool) → Kafka → Elasticsearch (search)
                                         → Snowflake (analytics)
                                         → Cache invalidation
                                         → Audit logs

This is the foundation of modern data architectures. The transactional database is the source of truth; CDC feeds every downstream system, eliminating dual-writes and inconsistency.

Replication lag and what it means for your application

In async setups, replicas lag behind the primary. The lag matters because it determines what your reads see.

Real numbers:

• Healthy Postgres lag: <100 ms intra-DC, <500 ms cross-region.
• Under load: 1–5 seconds is common.
• During VACUUM, autovacuum on large tables, or after a big DML: 10s–minutes.
• If replication breaks and resumes: hours.

The four "consistency levels" for users

If you read from replicas, what guarantees does the user see?

Most production: pin a user to the primary briefly after a write. Simple and gets read-your-writes for free.

Failover — where things really break

When the primary fails, you need to promote a replica. This sounds simple but has many failure modes.

The failover sequence

1. Detection: a sentinel (Patroni, Stolon, AWS RDS failover) heartbeats every primary. After 3-5 missed heartbeats, declare primary dead.
2. Election: pick the replica with the highest LSN (most recent data). If multiple sentinels disagree, they need consensus (typically Raft).
3. Promotion: tell the chosen replica "you're now the primary." It opens itself to writes.
4. Reconfiguration: other replicas need to know to follow the new primary. Service discovery (Consul, etcd, DNS) is updated.
5. Client retargeting: clients need to find the new primary. Connection poolers (pgbouncer) reconnect.
6. Old primary handling: if the old primary comes back, it MUST NOT accept writes. It needs to be reset to follow the new primary. This is called fencing.

The split-brain problem

What if the old primary doesn't know it's been deposed? It might still be accepting writes from clients that haven't switched. Now you have two primaries — split-brain. Data divergence ensues.

Prevention:

• STONITH ("shoot the other node in the head") — sentinel forcefully kills the old primary (e.g., via IPMI or cloud API).
• Fencing tokens — every write carries an epoch number. Old primary has an old epoch. Storage layer rejects writes with stale epochs.
• Quorum-based agreement — sentinels must agree (Raft) before promoting. Prevents two sentinels from electing two primaries simultaneously.

Multi-leader: when and how

When you actually need it

• Multi-region active-active where local-write latency matters (users in Sao Paulo can't tolerate 200ms cross-region writes).
• Offline-first applications (mobile, IoT) where the client is a "leader" while disconnected.
• Cross-datacenter redundancy with bidirectional traffic.

Conflict resolution strategies

Last-writer-wins (LWW): each write carries a timestamp; conflicts resolved by picking the later one. Simple but lossy — clock skew or near-simultaneous writes silently drop data. Used by Cassandra.

Multi-value (siblings): keep both versions; the application reads both and resolves. Used by Riak. Pushes complexity to the app.

CRDTs (Conflict-free Replicated Data Types): data structures with the mathematical property that concurrent updates always merge correctly. Examples: counters (PN-Counter), sets (OR-Set), maps. Used by Redis Enterprise CRDB, Automerge, Riak.

Application-defined resolution: app provides a merge function. Used in git-like systems where the user can review conflicts.

Conflict avoidance via partitioning: assign ownership of each key to one leader. Effectively single-leader-per-key. Most "multi-master" deployments in practice.

Leaderless / quorum: the Dynamo model

How quorums work

You have N replicas. You write to W of them and consider the write successful. You read from R of them. If W + R > N, your read and write sets overlap by at least one replica — that replica saw the latest write.

N = 3 replicas
W = 2, R = 2  →  W+R = 4 > 3  →  reads see latest writes
W = 1, R = 1  →  W+R = 2 ≤ 3  →  reads may see stale writes
W = 3, R = 1  →  writes are slow but reads are fast
W = 1, R = 3  →  writes are fast but reads are slow

Tuning gives you a knob between availability (smaller W and R make the system more available under partition) and consistency (larger W and R bring stronger guarantees).

The crucial caveat

Quorum overlap alone doesn't give you linearizability — it just gives you "at least one replica with the latest write." If you read R replicas and they have different values, how do you know which is latest? You need:

• Versioning (timestamps or vector clocks) on each write so reads can pick the latest.
• Read repair: when a read sees disagreement, write the latest value back to lagging replicas.
• Hinted handoff: when a replica is down, another node holds writes for it and replays when it returns.
• Anti-entropy: periodic full-replica comparison and reconciliation.

So Cassandra's "quorum consistency" gives you eventual consistency with high probability of seeing recent writes — not true strong consistency.

Interview drills

Red flags

• "Replication is for backups." Wrong; backups are for point-in-time recovery, replication is for availability and scale.
• "Synchronous replication = no data loss." Wrong, as we discussed.
• Can't distinguish physical and logical replication.
• Designs multi-master without explaining conflict resolution.
• "Just use quorum" without explaining W+R>N, read repair, or hinted handoff.
• No mention of split-brain handling.