Forty Billion Messages a Day With a Tiny Team: Process-per-Connection, Islands and RAM-First Storage
A process per connection, supervisors that let things crash, replicated islands and memory-resident data: the choices behind a very lean messaging backend.
On this page
Messaging has no forgiving failure mode. A delayed message feels broken and a lost one breaks trust. The surprising part of this design is not exotic technology. It is how consistently it prefers the option that is easiest to reason about at 3 a.m.
Design principles
| Principle | What it looks like |
|---|---|
| Clarity over cleverness | Small focused components, minimal dependencies, a small blast radius |
| Async by default | Processes hand off work and move on, absorbing spikes instead of amplifying them |
| Isolation | Backends partitioned into islands that fail independently |
| Seamless upgrades | Code changes without restarting services or disconnecting users, through discipline about state and interfaces |
| Quality through focus | Early on, every backend line reviewed by the founding engineers, which kept the system lean and understood |
A connection is a process
A phone opens a persistent TCP connection to a frontend server, and that connection is managed as one lightweight process: it owns the socket and the session state, and it exits when the user goes away. No pooling and no multiplexing, just a one-to-one mapping.
- If a packet is dropped or a client crashes, the process dies and its memory goes with it. There is nothing to clean up.
- A single chat server can hold upwards of a million concurrent connections, because lightweight processes cost kilobytes, not threads.
- The session process is not a dumb pipe: it authenticates, checks blocking and permissions, and fetches pending messages and notifications, in parallel, at the edge.
- Frontends do less work per connection: presence and typing updates are batched and rate-limited, acknowledgements are tiny protocol messages, and idle sessions are culled.
That keeps frontend load proportional to real activity, not raw connection count. It also reflects a quiet rule: not every message deserves the same delivery guarantee. A typing indicator can be dropped. A message must not be.
Why the runtime matters
Backends: narrow, redundant, isolated
Divide by function. Over forty clusters, each doing one thing: message queues, authentication, contact sync, presence, media, push notifications, spam filtering. If spam filtering crashes, message delivery does not. Teams ship changes to one backend without risking the others, and memory-bound and CPU-bound services run on suitable hardware.
Islands. Groups of nodes each own a subset of the data:
- Two or more nodes per island.
- Each partition is deterministically assigned a primary and a secondary.
- Replication is one-way, primary to secondary. If the primary fails, the secondary takes over immediately.
- Islands do not replicate to each other, so most failures stay inside one island.
Bidirectional replication brings locking, contention and reconciliation. One primary per partition and one-way replication removes the whole problem class. You give up multi-writer flexibility, and you gain failure recovery that is nearly trivial to reason about.
Data: memory first, disk later
Almost all access is key-value, with predictable keys and compact values, so the storage layer can be simple and fast.
| Choice | Effect |
|---|---|
| In-memory tables in the runtime (ETS-style) | No network hop or seek for the hot path, throughput stable under load |
| Database logic embedded in the VM | Fewer moving parts and no external round trip |
| DB fragments | The keyspace is split into fragments, each accessed through a single process, so access is serialised per key with no locks and more fragments means more throughput |
| Asynchronous writes in “dirty” mode | Updates acknowledged without waiting on transactions or disk, so slow disks do not slow messages |
| Parallel transaction managers and spread disk writes | One lagging writer does not stall the others |
| Write-back offline cache | Messages for offline phones sit in memory and are flushed only if they linger |
During holiday peaks, the offline cache is a pressure valve: reportedly over 98 percent of messages were served from memory before ever reaching disk, because most are delivered and deleted within moments.
What goes wrong at the edge of scale
The reported incidents are instructive because they are mundane:
- Hash collisions in a hot table. A hashing seed produced uneven bucket distribution under load. Fixing it gave a several-fold throughput gain.
- Selective receive bottlenecks. A process scanning a long mailbox for specific messages slowed as the queue grew, a classic hidden quadratic. Solved by restructuring how messages are consumed.
- A network-level cascade. A failure in shared network infrastructure rippled through nodes that all depended on it, a reminder that “independent” components sharing a router are not independent.
None of these was a failure of the grand design. They were hot spots found by measuring, which is the real ongoing job.
Taking it with you
Do
Give each unit of failure its own boundary, decide per message type how much delivery guarantee it needs, keep one writer per partition, serve short-lived data from memory with write-back, and measure for hidden quadratic behaviour in queues and tables.
Don't
Defend against every possible error inside one big process, or add multi-writer replication “for flexibility”. The first hides failures, the second multiplies them.
Takeaway
Scale at this level is mostly the disciplined refusal of complexity: isolate failures, restart instead of repair, keep one writer per piece of data, treat memory as the primary store for transient data and disk as a lagging durability layer. A small team can run a very large system when almost every part of it is easy to understand.
Frequently asked questions
Why is one process per connection a good model?
In a runtime with cheap, isolated, preemptively scheduled processes, each connection gets its own state and mailbox. A slow or broken client affects only itself, lifecycle is simple because dropping the connection frees the process and its memory, and no shared pool or multiplexer needs tuning.
What does 'let it crash' mean?
Instead of defending every code path against every error, you write the happy path, isolate components in separate processes, and have supervisors restart failed ones in a known clean state. Failures stay local, and recovery is uniform and tested rather than scattered through error-handling code.
How can a system avoid conflicting writes without consensus?
By giving each piece of data exactly one writer. Assign every partition deterministically to a primary node, replicate one way to a secondary, and fail over by promoting the secondary. With a single writer per partition and one-way replication there are no concurrent conflicting updates to reconcile.
Why keep data in memory and write to disk asynchronously?
For short-lived data such as undelivered messages and presence, most items are read soon and then deleted. A write-back memory layer serves them without touching the disk, and flushes only items that linger. Disk speed then affects durability windows, not latency.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.