Real-Time Chat That Never Lies: Persist Before You Broadcast, and Cache the Cold Start
Why send flows must store a message before showing it, how an approximation of atomic broadcast works, and why session bootstrap needs an edge cache.
On this page
Users judge a messaging product by one test: when I see my message on screen, is it really there, and does everyone else see the same thing in the same order? Everything below is a consequence of taking that test seriously.
Origins leave fingerprints
The product grew out of an internal tool built by a team making a browser game. Two habits carried over. Separate real-time event handling from business logic, the way game servers do, and push updates as state changes rather than waiting to be asked. WebSockets were the foundation, not an optimisation. At peak weekday hours there are millions of simultaneous long-lived sessions, each carrying messages, typing indicators and presence.
The first design
| Component | Responsibility |
|---|---|
| Web app (a typed PHP dialect on a monolith) | Authentication, permissions, storage, API endpoints, job queues, session bootstrap |
| Channel server (Java) | WebSocket connections, real-time fan-out, ordering of messages, typing and presence |
The channel server settled disputes: when two people hit send together, it decided which message came first. This was quick to build, and it creaked later. The monolith got brittle to test and deploy, the channel server held state and was harder to recover, and failures were messy because each depended on the other in odd ways.
What a chat promise really means
Chat is not radio. In older protocols, a message is gone once it scrolls past. Here, messages are more like email: stored, searchable, replayable. So three guarantees must hold:
- A message that appears in a channel is visible to everyone in it.
- A message that appears in the UI is in stable storage.
- Everyone scrolling back sees the same history in the same order.
That is atomic broadcast: validity (a sent message is eventually delivered), integrity (nothing appears that was not sent) and total order (everyone sees one sequence). It is as hard as consensus, and consensus cannot be perfectly guaranteed under every failure. Production systems approximate it: one ordering authority, deferred work, and recovery from inconsistency instead of perfection.
Two send flows
Benefits of the new flow:
- Crash safety. No acknowledged message can be missing.
- Stateless channel servers. No local buffers or retry logic, so they scale and recover simply.
- Latency preserved. The broadcast is still immediate while background persistence continues.
- No WebSocket needed to send. A mobile client replying to a notification can use a plain HTTP request, avoiding a full session setup.
A user who sees “failed to send, retry” loses seconds. A user who sees “sent” for a message that evaporates loses trust. Order your system so the optimistic UI can only be wrong in the loud direction.
The cold-start problem
When a client connects, it needs the team’s current state: users, channels, memberships and unread counts. The first method sent a single, giant snapshot (an RTM-start style payload) as the starting keyframe, after which real-time deltas kept the client current.
| Team size | Effect |
|---|---|
| Under 100 users | Small and fast |
| 10,000+ users | Payload of tens of megabytes, tens of seconds to parse and build local caches |
| After a network blip | Thousands of clients reconnect at once and each triggers a fresh build in one data centre |
The payload grew roughly quadratically with team size (users times channels), computed in a single region, so distant users paid latency on top. A bad deploy or dropped connection could hammer the control plane and cascade.
Flannel: a cache for startup
A dedicated service holds a pre-warmed, in-memory cache of team metadata, subscribes to the same real-time events a client would so it stays current, and serves bootstrap data from regional replicas close to users. Clients get the slice they need, quickly, without the web app rebuilding the world each time.
The reusable idea: if many clients ask the same expensive question when they start, answer it from a cache that is kept warm by the event stream.
Retries and duplicates
Retries are unavoidable: networks fail, apps resume, users tap twice. Without protection, a retry after a lost acknowledgement creates two messages. The fix is simple: the client generates a unique id per message before sending, and the server deduplicates on that id. The same idea makes any write endpoint safe to retry.
Durable log plus working memory
A common arrangement for the data path is a durable log (such as Kafka) as the ledger of record, and an in-memory store (such as Redis) as working memory for in-flight state: presence, recent messages, fan-out bookkeeping. The log gives replay and recovery, the memory store gives speed, and the two have different failure semantics, so keep their roles clear.
Checklist for your own chat or feed
Do
Persist before acknowledging, give messages client-generated ids, keep real-time servers stateless, cache the startup snapshot regionally, and rate-limit or stagger reconnects after an outage.
Don't
Show a message as sent on the strength of an in-memory broadcast alone, or rebuild a whole team’s state per connection. Both work in a demo and collapse the first time a data centre blinks.
Takeaway
Real-time products earn trust by making the visible state a subset of the durable state. Store first, broadcast second, dedupe with client ids, keep the fan-out tier stateless and answer the cold-start storm from a warm cache. You give up a few milliseconds of apparent speed, and you stop lying to your users.
Frequently asked questions
Why must a chat system store a message before confirming it?
Otherwise a crash between confirming and storing leaves the sender looking at a message that no longer exists. Persist-first means any message the user sees as sent is in stable storage, and a failure produces a clear error, not a silent loss that users discover later.
What is atomic broadcast?
A guarantee that all participants receive the same messages in the same order, and that only sent messages are delivered. It is equivalent in difficulty to consensus, which cannot be perfectly guaranteed under all failures, so production systems approximate it with a single ordering authority and fast recovery.
How do you stop retries from creating duplicate messages?
Have the client generate a unique id for each message before sending and have the server deduplicate on it. A retry after a timeout carries the same id and is recognised as the same message, so the user sees one message regardless of how many attempts were made.
Why did full-state session bootstrap break for large organisations?
The initial payload grew roughly with users times channels, reaching tens of megabytes, built in one data centre on every connect. Reconnection storms after a network blip repeated that work for thousands of clients at once. A regional cache of team data removes the repeated computation.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.