Dropbox / Cloud File Sync

What is a Cloud File Sync System?

A cloud file synchronization system allows users to store files locally in a designated directory on their devices, automatically backing up and synchronizing the changes to a cloud storage center and all other connected client devices. The main technical challenge is Delta Synchronization (uploading and downloading only the modified chunks of a file) to reduce network transit and disk overhead.

Real-World Applications

• Dropbox, Sync.com, and Nextcloud.

Core API Contracts

1. Inquire Server Workspace Version (Sync check)

GET /api/v1/sync/workspace/version?device_id=dev_desktop_1
Headers:
Authorization: Bearer <auth_token>

Response (200 OK):
{
  "latestNamespaceVersion": 2981,
  "changesPending": true,
  "updates": [
    {
      "filePath": "/Documents/work.docx",
      "action": "UPDATE",
      "version": 2981,
      "deleted": false
    }
  ]
}

2. Commit Version Updates

POST /api/v1/sync/commit
Content-Type: application/json

{
  "filePath": "/Documents/work.docx",
  "parentVersion": 2980,
  "fileSize": 10485760, // 10MB
  "chunks": [
    { "index": 0, "hash": "sha256_e3b0c442..." },
    { "index": 1, "hash": "sha256_8f43a921..." }
  ]
}

Response (201 Created):
{
  "status": "committed",
  "version": 2981,
  "committedAt": "2026-06-21T14:20:00Z"
}

Functional Requirements

• Real-Time Client Sync: Any file created, modified, or deleted within the local sync directory must sync to the cloud and other client devices automatically.
• Offline Sync Queue: Track modifications made offline and sync them immediately upon reconnection.
• Conflict Handling: Detect and resolve concurrent modifications by creating conflicted copy files.
• Bandwidth Optimization: Only upload modified blocks of edited files (Delta Sync).
• Revision History: Support recovering deleted files or rolling back to historical versions.

Non-Functional Requirements

• Sub-Second Latency Notification: Propagate file change notifications to other online devices in under 1 second.
• Data Integrity: Guarantee zero file corruption during parallel sync actions.
• Bandwidth Minimization: Limit background network traffic using compression and chunk checks.
• Infinite Scalability: Support hundreds of millions of users and petabytes of files.

Capacity Estimates (Back-of-the-Envelope Math)

1. Registered Storage capacity:
   • 500 Million users, with 50 Million Daily Active Users (DAU).
   • Assume average storage per user = 20 Gigabytes.
   • Total storage capacity needed: $500 ext{M} imes 20 ext{ GB} = 10 ext{ Exabytes}$.
2. Delta Sync Savings Math:
   • 50M active users edit 4 files daily. Average size of these files = 20 Megabytes.
   • Without Delta Sync: Ingress rate = $50 ext{M} imes 4 imes 20 ext{ MB} = 4 ext{ Petabytes per day}$.
   • With Delta Sync (averaging 5% modification per file edit $approx 1 ext{ MB}$):
   • Optimized Ingress rate: $50 ext{M} imes 4 imes 1 ext{ MB} = 200 ext{ Terabytes per day}$.
   • Key Insight: Delta Sync reduces bandwidth and cloud ingress storage costs by $95%$.

How to Approach This Problem

When designing a cloud sync service like Dropbox, focus on client-agent architecture and incremental synchronization:

Step 1: Divide the Architecture (Client vs Cloud)

Unlike simple web apps, the Sync Client Agent running on the user's computer is the brain:

• Keep the local folder metadata indexed in a local database (SQLite).
• Use OS file system event listeners to capture file updates.
• Split files, compute SHA-256 hashes, and manage delta-upload states.

Step 2: Implement a Two-Phase Commit sync protocol

• Phase 1 (Chunk Check): Client queries the server for missing chunk hashes.
• Phase 2 (Upload & Commit): Client uploads the missing chunks directly to object storage, then calls the API to commit the new metadata version.

Step 3: Architect the Real-Time Notification Channel

• Use WebSockets or HTTP Long Polling for the client-to-cloud notification channel.
• When the server commits a metadata change, broadcast the update event containing only the file metadata to other online client devices.

Step 4: Resolve Offline Conflicts

• Use version counters. If a client attempts to commit a change with an outdated parent version (because another device updated it first), reject the commit and create a conflicted copy file locally.

Core Design Intuition

Dropbox scales by treating files as mutable collections of immutable chunks, keeping sync logic client-side to minimize server CPU usage.

Dynamic Sync Protocol & Pipeline

Key Insights

1. Local SQLite Index: Running dynamic scans recursively on a directory containing 100k files burns CPU. The local client maintains a local SQLite database that stores paths, sizes, modifications, and chunk hashes. It compares this database with OS event notifications to detect local changes instantly.
2. Global Deduplication: If multiple users upload the same file (e.g., a popular software installer), the chunk hashes will match existing server blocks. The client does not upload any data, saving network bandwidth.
3. P2P LAN Synchronization: When multiple computers are on the same local network, they discover each other via UDP broadcast and sync chunks directly over the LAN, bypassing S3 entirely.

System Architecture & Data Flow

Core Components

1. Client Sync Agent: Runs locally, monitors directories, and manages delta syncs.
2. Metadata DB (PostgreSQL): Stores namespaces, folders, files, and historical version maps.
3. Block Storage (S3): Houses the immutable 4MB data chunks.
4. WebSocket Notification Gateway: Broadcasters pushing update alerts to clients.
5. Sync Microservice: Coordinates file metadata version commits and permission checks.

Relational Database Schema

``sql -- Namespaces (Shared Folders) CREATE TABLE namespaces ( namespace_id VARCHAR(64) PRIMARY KEY, owner_id VARCHAR(64) NOT NULL, version INT DEFAULT 1 );

-- File Registry CREATE TABLE files ( file_id VARCHAR(64) PRIMARY KEY, namespace_id VARCHAR(64) REFERENCES namespaces(namespace_id), file_path TEXT NOT NULL, latest_version INT DEFAULT 1, is_deleted BOOLEAN DEFAULT FALSE );

-- Chunk Index per File Version CREATE TABLE file_chunks ( file_id VARCHAR(64) REFERENCES files(file_id), version INT NOT NULL, chunk_index INT NOT NULL, chunk_hash VARCHAR(64) NOT NULL, -- SHA-256 PRIMARY KEY (file_id, version, chunk_index) );

### Execution Flow: Delta Sync
1. Client edits a document. The client agent splits it into 4MB chunks and hashes them.
2. The agent queries `/sync/commit` with the parent version.
3. The server checks the metadata database. If the version matches, it returns a list of chunk hashes that do not exist in the block store.
4. The client uploads the missing chunks directly to S3 and calls the server to commit.
5. The server updates the database and publishes a sync event to Kafka.
6. The Notification Service pushes a WebSocket frame to other client devices.
7. Other client devices download only the modified chunks from S3 and reconstruct the file locally.

Advanced Scaling & Engineering Edge Cases

1. Dynamic Chunking with Rabin Fingerprints

• Static Chunking (Fixed-Size): Splitting a file every 4MB is simple but vulnerable to shifts. If a user inserts 1 byte at the beginning of a file, every single chunk boundary shifts down by 1 byte. All SHA-256 hashes change, rendering delta-sync useless.
• Dynamic Chunking (Variable-Size): Use Rabin Fingerprints. A sliding window calculates a checksum over the bytes. If the checksum matches a specific mathematical pattern (e.g., hash % 4MB == 0), a boundary is created. Adding a byte at the start only changes the first chunk; the remaining chunks retain their boundaries and hashes.

2. Handling Offline Modifications and Conflicts

• When a client is offline, it queues modifications in its local SQLite database.
• Upon reconnection, it attempts to commit the offline changes.
• Conflict Detection:
  • The server checks if the client's parent version matches the latest database version.
  • If the server has a newer version, it rejects the commit.
  • The client agent downloads the newer version, keeps it, and saves the offline local version as a separate conflicted file: document (conflicted copy).docx.

3. Chunk Garbage Collection

• As files are updated, older version chunks are orphaned.
• A background worker scans the metadata DB periodically to find chunks that are no longer referenced by any active file version.
• These chunks are deleted from S3 to reclaim storage space.

Trade-offs & Architectural Decisions

Succeeding in the Interview

What Interviewers Look For

1. Detailed Client Architecture: Explain the local agent's responsibilities: disk watching, SQLite tracking, and dynamic chunking.
2. Delta Sync & Rabin Fingerprints: Pointing out the byte-shift issue of fixed chunking and suggesting dynamic chunking (Rabin Fingerprints) is a major positive signal.
3. Idempotency and Dedup: Highlight how content-addressing using SHA-256 hashes enables instant deduplication.
4. Pre-signed URLs: Show that you understand performance bottlenecks by having the client upload chunks directly to S3/block storage rather than proxying raw bytes through the web server.

Common Mistakes

• Assuming E2EE by Default: Dropbox is not End-to-End Encrypted (Dropbox needs to scan files for viruses and search indexing). Clarify this distinction with the interviewer.
• Proxying Uploads: Proxying multi-gigabyte file uploads through your API Gateway will consume all server threads and cause timeouts. Always use pre-signed upload URLs.
• Using NoSQL for Hierarchy: Relational databases are better suited for folder hierarchies due to recursive queries and permission inheritances. NoSQL tables struggle here.