How do you design a system so a bug like this stays contained to one region?
Put every new code path behind a feature flag rolled out region-by-region, not globally at once. Add a validation checkpoint before replicating configuration data, so malformed data can't propagate faster than a human can catch it. Handle null and malformed input defensively instead of trusting upstream data is always well-formed. And run your monitoring and status page on infrastructure independent of the system it's monitoring.
Answered in
The Google Cloud Outage That Started With One Missing Null CheckOne unprotected code path in Google Cloud's Service Control took down Spotify, Discord, and Cloudflare auth in minutes. Here's the anatomy of the failure.
Read the full analysisOther questions this article answers
More system design questions
- Why does a database need an index at all — why can't it just scan the table?
- When is a hash index better than a B-tree index?
- What is a composite index and why does column order matter?
- What's a covering index and why is it faster than a normal index?
- What's the real cost of adding an index, beyond disk space?
- What does ACID actually stand for, and why do all four properties matter together?
- What's the practical difference between pessimistic and optimistic concurrency control?
- What is a race condition in a database transaction, and how does isolation prevent it?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design beat.