How do you design a system so a bug like this stays contained to one region?
Put every new code path behind a feature flag rolled out region-by-region, not globally at once. Add a validation checkpoint before replicating configuration data, so malformed data can't propagate faster than a human can catch it. Handle null and malformed input defensively instead of trusting upstream data is always well-formed. And run your monitoring and status page on infrastructure independent of the system it's monitoring.
Answered in
The Google Cloud Outage That Started With One Missing Null CheckOne unprotected code path in Google Cloud's Service Control took down Spotify, Discord, and Cloudflare auth in minutes. Here's the anatomy of the failure.
Read the full analysisOther questions this article answers
More system design & architecture questions
- What does CodeDeploy RESTART actually deploy?
- Does CodeDeploy RESTART work with ECS or Lambda?
- Does RESTART remove an application from load-balancer traffic while it restarts?
- Does Elastic Beanstalk's Active Directory join require a domain password?
- Can an Elastic Beanstalk environment be healthy if the domain join fails?
- Will enabling Active Directory replace running instances?
- What is different about AWS Lambda Web Functions?
- Does Lambda Web Functions automatically cost less than ordinary Lambda?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design & Architecture beat.