Worked Example 15.5 · 5 of 15
Design Dropbox / file synchronisation
The setup
"Design Dropbox. Users have files on their devices that sync to the cloud and across multiple devices. Focus on: sync efficiency (don't re-upload entire files when small parts change), conflict resolution when multiple devices edit offline, and offline support. 500M users; average user has 10 GB of data; ~10% of users active daily."
Clarifying questions
| Question | Story answer |
|---|---|
| Max file size? | 50 GB per file. |
| Sync latency target? | Within a minute when both devices online. |
| Cross-user dedup? | Yes — important for cost. |
| Version history? | 30 days. |
| Sharing / permissions? | Out of scope — personal sync only. |
The central insight: content-addressed chunks
The naive design uploads each file as a monolithic blob. This is catastrophic when a 1 GB file's first MB changes — the whole 1 GB re-uploads.
The right approach: files are split into chunks; each chunk is content-addressed (identified by its hash); files are stored as ordered lists of chunk hashes.
This unlocks four wins simultaneously:
- Dedup: identical chunks (same file shared by 1000 users, unchanged portions of a modified file) are stored once.
- Delta sync: only modified chunks re-upload.
- Resumable upload: each chunk is independently uploaded; resume from last successful.
- Cheap copy: copying a file is just copying the metadata.
Design Decision 1: Fixed-size vs content-defined chunking
| Approach | How | Tradeoff |
|---|---|---|
| Fixed-size chunks (e.g., 4MB every 4MB) | Split at offset 0, 4MB, 8MB, 12MB ... | Simple. Fatal problem: inserting 1 byte at start of file shifts every chunk's boundary by 1 byte. Every chunk hash changes. Entire file re-uploads. |
| Content-defined chunking (CDC) | Use a rolling hash (Rabin fingerprint, FastCDC) over file content. Chunk boundaries placed where rolling hash hits patterns. | More complex; but insert/delete locally affects only neighboring chunks. Most chunks unaffected. |
Pick CDC. Without it, the entire dedup architecture falls apart.
The data model
users: user_id (uuid), email, ... files (metadata only): file_id, user_id, path, size, modified_at, chunk_hashes: [hash_1, hash_2, ..., hash_N] revision: monotonically increasing version number file_revisions: Older revisions of files (for 30-day history) Same as files but with revision number chunks (in object store, S3-class): hash → binary blob chunk_refs (for GC): hash → reference_count When count drops to 0, eligible for deletion devices: device_id, user_id, last_seen_at, sync_token sync_state: device_id, file_id, last_synced_revision
The sync flow (upload from client)
1. Client app watches local filesystem (using inotify / FSEvents / etc.) 2. User modifies file X. 3. Client computes CDC chunks of new X 4. Client compares new chunk list with previously-uploaded chunk list for X → Identifies new vs unchanged chunks 5. For each new chunk: a. Compute hash b. Query backend: "do you have this hash?" c. If yes: skip upload (dedup!) d. If no: upload chunk binary to object store 6. Update file metadata via API: new chunk_hashes list, new modified_at 7. Backend: stores file revision; bumps revision number 8. Backend: pushes "file X changed to revision N" notification to user's other connected devices
The sync flow (download from another device)
1. Device receives notification or polls for changes since last sync 2. Backend returns: file X has new revision N; chunk hashes are [a, b, c, d] 3. Device compares with current local chunks of X 4. For chunks it doesn't have locally: a. Request chunk from object store (signed URL or via API) b. Download 5. Reassemble file from chunks 6. Atomically replace local file
Design Decision 2: Conflict resolution
Interviewer: Two devices modify the same file offline. They both come online. What happens?
Server detects the conflict: both updates reference the same parent revision but produce different new revisions (different chunk lists). Naive merge is impossible for arbitrary file types — you can't safely 3-way merge a binary file like .docx.
Dropbox's strategy: keep both. Original file gets one device's update. The other device's update becomes "filename (conflicted copy from device B).txt". User decides which to keep. Simple, lossy-free, predictable, type-agnostic.
Follow-up: Can we do better for text files — e.g., 3-way merge?
A: For specific file types (text, JSON, source code), yes. But determining if a file is "safely mergeable" requires content inspection and an opt-in policy. For binary files (.docx, .pdf, images), automatic merge can corrupt data silently — far worse than a conflicted copy. The conservative "save both" approach is universal and correct.
A: For specific file types (text, JSON, source code), yes. But determining if a file is "safely mergeable" requires content inspection and an opt-in policy. For binary files (.docx, .pdf, images), automatic merge can corrupt data silently — far worse than a conflicted copy. The conservative "save both" approach is universal and correct.
Follow-up: What if 10 devices all modify offline?
A: First sync wins (becomes the "canonical" version). Each subsequent sync becomes its own conflicted copy. User ends up with 10 files. Messy but predictable. Better than data loss.
A: First sync wins (becomes the "canonical" version). Each subsequent sync becomes its own conflicted copy. User ends up with 10 files. Messy but predictable. Better than data loss.
Cross-examination round 1: storage cost
Interviewer: 500M users × 10GB = 5 exabytes nominal. With your dedup, what's actual storage?
Dedup ratios: cross-user dedup catches duplicates of public content (same Linux ISO downloaded by 1000 users), shared documents, common files. Typical dedup ratio: 2-5x. So 5 EB → 1-2.5 EB actual storage. Within a single user's files, intra-user dedup is also significant (same chunks across versions of a file as it's edited). Combined: 5-10x. At cloud storage costs ($0.02/GB-month for hybrid hot/cold tiering), storage budget is in the hundreds of millions per year — but dedup-enabled.
Cross-examination round 2: notification scale
Interviewer: 500M users, each with multiple devices, each connected to your sync notification service. How?
Same WebSocket gateway pattern as messaging. Devices connect via WSS to a gateway tier. When a user's file changes on device A, the change event is published to a pub/sub topic for that user; gateways with subscribed devices forward to those devices. ~100M concurrent connections globally; ~3K-5K per gateway node; ~30K gateway nodes. This is identical infrastructure to the WhatsApp-style messaging design.
Cross-examination round 3: garbage collection
Interviewer: A user deletes a file. Its chunks: do you delete them from object storage?
Maintain reference counts per chunk. When a file revision is deleted (after 30-day history elapses), each of its chunk hashes has its ref-count decremented. When a chunk's ref-count hits zero, it's eligible for deletion. Garbage collection job runs periodically (daily): finds chunks with ref-count=0 and older than 7 days; deletes them. The delay is for safety — if there's a bug in the ref-counting logic, we don't immediately lose data.
Follow-up: What if ref-counting is inconsistent across replicas?
A: The ref count is maintained transactionally in the metadata DB (Postgres or Spanner-class). All decrement/increment operations are atomic. Cross-region replication of metadata uses logical replication. If anything looks inconsistent, GC is conservative — only deletes chunks that have been "ref-count=0 for >7 days" — by which point any inconsistency would have been reconciled.
A: The ref count is maintained transactionally in the metadata DB (Postgres or Spanner-class). All decrement/increment operations are atomic. Cross-region replication of metadata uses logical replication. If anything looks inconsistent, GC is conservative — only deletes chunks that have been "ref-count=0 for >7 days" — by which point any inconsistency would have been reconciled.
Failure mode analysis
| Failure | Mitigation |
|---|---|
| Upload interrupted mid-file | CDC + chunked upload makes resumption natural. Client knows which chunks succeeded; resumes from next unsent chunk. |
| Object store regional outage | Chunks stored in multiple regions with cross-region replication. Reads fail over to backup region. |
| Metadata DB outage | Sync stops. Files on local devices still accessible. Once metadata is back, sync resumes from last_synced_revision. |
| Client clock skew on conflict detection | Don't rely on client timestamps for conflict resolution. Use server-assigned revision numbers; conflicts are detected by "both updates reference same parent revision." |
Takeaway
Dropbox's central insight is content-addressed storage. Every modern sync product (Google Drive, OneDrive, iCloud) uses some version of this pattern. The deeper lesson: when you can identify the unit of work (a chunk) and make it immutable (content-addressed by hash), the system becomes dramatically simpler. Chunks are immutable; files are mutable references to chunks; sync is reconciliation of references. This pattern avoids the entire class of "two writes to the same byte" coordination problems.
The other lesson: conflict resolution is a UX decision, not a technical one. The technical problem (detecting the conflict) is straightforward. The hard part is deciding what to do — auto-merge, ask user, save both. Most production systems pick "save both" because it's universal and never destroys data. Smart auto-merge for specific types is a feature, not the default.