Building an Event-Driven Platform: Schema Contracts, a Standby Store and Dead Letters
How a global retailer-scale event platform keeps events trustworthy: schema registry contracts, SDKs, a fallback store when Kafka is down, and domain sharding.
On this page
Event-driven architecture sounds like “publish and forget”. At company scale, forgetting is the danger. If nobody owns what an event looks like, every consumer guesses, and the first schema change breaks something unrelated three teams away.
What the platform has to do
A large digital estate sends events for very different jobs, and one platform must serve all three:
| Processing type | Example | Needs |
|---|---|---|
| Asynchronous | Tracking mobile order progress | Low latency, ordering per order |
| Transactional | Placing and confirming an order | No loss, reliable delivery |
| Analytical | Feeding marketing and reporting | Completeness, stable formats |
Add global deployment, real-time delivery and external partners, and the design goals follow: scale, high availability, performance, security, reliability, consistency of patterns, and simplicity for the teams building on it.
The components, and why each exists
The life of an event
- Define and register the schema
The owning team writes the event’s schema (fields, types, required versus optional) and registers it. This is the contract.
- Producer validates, then publishes
At startup the SDK caches the schemas it needs. Before sending, it checks the event against its schema. A valid event goes to the domain’s topic.
- Bad or fixable events go to a dead-letter topic
If validation fails or an error can be corrected later, the event is parked in a dead-letter topic for that application instead of being dropped or blocking the stream.
- Broker down? Save it
If publishing to Kafka fails because the cluster is unavailable, the event is written to the standby store and a retry job replays it later.
- Consumer validates and reads
The consumer SDK checks the schema version on the message, validates, and marks the event processed. Parked events can be repaired and replayed to the main topic.
The standby store changes the failure mode of a broker outage. Without it, producers either block (hurting customers) or drop events (hurting data). With it, events are late, but still arrive in order of retry.
Data contracts that survive change
A schema is a blueprint: which fields, what types, which are required. Messages carry the version number of the schema used to write them, so a consumer can interpret old and new messages correctly side by side. Schemas then evolve under rules.
| Change | Safe? | Why |
|---|---|---|
| Add an optional field with a default | Yes | Old consumers ignore it, new consumers read it |
| Add a required field | No, for backward compatibility | Old messages lack it |
| Remove an optional field | Usually, with defaults | Consumers must not depend on it |
| Rename a field | No | It is a remove plus an add |
| Change a field’s type | No | Existing readers break, use a new field |
Set a compatibility mode in the registry (backward, forward or full) so an incompatible schema is rejected when registered. That moves the discovery of a breaking change from production at 2 a.m. to a failed CI check.
Scaling the platform itself
Autoscale brokers on load. Managed Kafka grows storage automatically but does not necessarily add brokers when they are busy. A watchdog function watches each broker’s CPU, and when it crosses a threshold it adds a broker, then triggers a rebalance so partitions spread across the new capacity.
Shard by domain. Events are grouped by business domain, such as customer profiles, orders or loyalty, and each domain gets its own dedicated cluster. The event’s domain decides which cluster and topic it uses. Like separate mailrooms per department, failures and load stay local. Consumers subscribe to whichever domains they need, and the platform runs across several regions with high-availability configuration in each.
Practices worth copying
Do
Own a schema per event type, enforce compatibility in CI, ship one SDK per language with validation and retries built in, cache schemas locally for speed, and monitor dead-letter volume and standby store depth as health signals.
Don't
Let each team serialise events however it likes, or treat the dead-letter topic as a place where messages go to be forgotten. Unmonitored dead letters are silent data loss.
Where it tends to go next
Formal event specifications (so contracts cover semantics, not only shape), serverless or partition-autoscaling brokers to cut operational work, and better developer tooling for browsing, testing and replaying events. All of them continue the same principle: make the safe path the easy path.
Takeaway
An event platform is trusted when its contracts are enforced, its failure modes are boring and its clients behave the same everywhere. Put schemas in a registry with compatibility rules, validate in SDKs on both sides, park bad events in dead letters, catch broker outages in a standby store, and shard by domain so one team’s problem stays one team’s problem.
Frequently asked questions
What is a data contract in an event-driven system?
A versioned schema that defines which fields an event carries, their types and which are required, stored in a central registry. Producers validate against it before publishing and consumers validate on receipt, so teams can change their systems independently without breaking each other.
How should event schemas evolve safely?
Make additive, backward-compatible changes: new optional fields with defaults. Attach a schema version to every message so a consumer can use the right definition. Enforce a compatibility rule in the registry so incompatible changes are rejected at registration, not discovered in production.
What is a standby event store?
A fallback database where producers write events that could not be published because the broker was unavailable. A background job retries publishing from it once the broker recovers, so a broker outage delays events instead of losing them.
Why shard an event platform by domain?
Dedicated clusters per domain, such as orders or customer profiles, keep a noisy or failing domain from affecting others, let each cluster scale and be tuned on its own, and keep topic ownership clear. The trade-off is more clusters to operate and cross-domain consumers that read several.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.