The Google Cloud Outage That Started With One Missing Null Check

Most outage postmortems describe a chain of individually reasonable decisions that combine into an unreasonable outcome. This one is unusually clean: five specific, nameable engineering gaps, each one a known best practice that got skipped, stacked on top of each other. Strip away the incident-specific details and what’s left is a checklist every team running a shared, high-blast-radius system should be able to answer “yes” to — and Google, on that day, could answer “yes” to none of them.

What is Service Control, and why does one component matter this much?

Service Control is the component that sits between every client and every backend in Google Cloud’s API infrastructure. On every request, it verifies authorization, enforces quota limits, checks policy rules, and logs the request for billing and auditing. It is, functionally, the front door for nearly all Google Cloud traffic — which means its failure mode isn’t “one feature breaks,” it’s “everything behind the door stops answering.”

That’s the shape of any shared gatekeeper component: authentication services, API gateways, service meshes, centralized rate limiters. They concentrate risk by design, because concentrating the logic (one place to check auth, one place to enforce quota) is exactly what makes them useful. The tradeoff is that a bug in the gatekeeper isn’t scoped to the gatekeeper — it’s scoped to everything that walks through the door.

How one bug reached every region in seconds

The sequence, reconstructed from what Google shared publicly:

1. A new code path ships without a feature flag

   On May 29, 2025, Google shipped support for more advanced quota policy checks inside Service Control. The new logic activated only when a specific type of policy input was present — an input that hadn’t shown up during rollout testing, so the bug inside it went undetected. Critically, the new code path had no feature flag protecting it. It was live, in the binary, in every region, waiting for the right input.

2. A malformed policy update triggers the bug

   On June 12 at roughly 10:45 AM PDT, a policy update with blank or missing fields was written into Google Cloud’s regional Spanner databases. The new quota-checking logic read this malformed policy, hit a null pointer, and crashed the Service Control binary in that region.

3. Global replication does exactly what it's designed to do

   Google Cloud’s policy and quota metadata replicates near-instantly across every region — a feature, not a bug, for keeping global systems consistent. It worked as intended: the corrupted policy propagated everywhere within seconds. Every regional Service Control instance read the same bad data and crashed the same way, at nearly the same time.

4. Downstream systems fail as a wall, not a trickle

   With Service Control down, nearly all Google Cloud API traffic started returning HTTP 503. Spotify, Snapchat, Discord, Twitch, and Fitbit went down for end users. GitLab, Replit, Shopify, and Elastic degraded or failed outright. Cloudflare’s core CDN stayed up, but its authentication systems — which depend on GCP — failed, breaking login and session validation for many of Cloudflare’s own customers. Vertex AI, BigQuery, and Cloud Functions all halted.

Why recovery wasn’t instant, even after the fix was known

The team that built the new quota feature had, fortunately, also built an internal kill switch — a “red button” that could disable the specific broken code path without a full redeploy. Google’s SRE team identified the root cause and activated it within about 10 minutes, and it was fully rolled out across regions within 40 minutes of the incident starting.

Most regions stabilized quickly after that. One didn’t: us-central1, in Iowa, took roughly 2 hours and 40 minutes to fully recover. The reason is a second failure mode layered on top of the first — a herd effect. Once the kill switch let Service Control instances restart, all of them tried to reconnect to the same regional Spanner database for policy metadata at once. With no randomized exponential backoff on the restarts, the recovery attempt became its own overload event, and engineers had to manually throttle restarts and reroute load to multi-regional Spanner instances to stabilize the region.

The failure inside the failure: nobody could see what was happening

While engineers worked the incident, customers were mostly flying blind. Google’s own Cloud Service Health dashboard is hosted on the same infrastructure the outage affected, so it couldn’t be updated promptly — the first public acknowledgment came nearly an hour after the outage began. Worse, some customers relied on Cloud Monitoring and Cloud Logging to diagnose their own systems during the incident, and those tools were down for the same root-cause reason. Entire operations teams lost visibility into their own infrastructure at exactly the moment they needed it most.

This is a distinct lesson from the technical root cause, and arguably the more universal one: a monitoring and status system that depends on the infrastructure it monitors will go dark exactly when you need it. Status pages and alerting pipelines need an independent failure domain — different infra, different region strategy, different dependency chain — precisely so that “everything is broken” doesn’t also mean “and we can’t tell you anything about it.”

Five specific gaps, and what each one would have prevented

Gap	What it allowed	What would have contained it
No feature flag on the new code path	The bug shipped active globally from day one	Staged rollout to one region/percentage of traffic, instant rollback without redeploy
No null check on policy input	A single malformed record crashed the whole binary	Defensive handling of missing/malformed fields, treating upstream data as untrusted
No validation gate before global replication	Corrupted data reached every region in seconds	A checkpoint that rejects malformed config before it propagates, not after
No backoff on mass restarts	The fix triggered a second overload in one region	Randomized exponential backoff on any restart path many instances share
Status page shares infra with the monitored system	Customers had no visibility for nearly an hour	Independent infrastructure and failure domain for observability and status tooling

Takeaway

Nothing in this incident required a novel failure mode or an exotic edge case. Every gap on that list is a named, well-understood practice — feature flags, defensive null handling, replication validation, jittered backoff, independent observability — that most teams already know they should have. The outage happened because five of them were missing at once, on a component with the widest possible blast radius in the entire platform. The lesson isn’t “Google made a mistake” — every large infrastructure team eventually will. It’s that the value of these five practices scales with how many things sit behind the component you’re protecting, and Service Control sits behind nearly everything.