topic: software-engineering
author: Crashtech Editorial
published: Nov 17, 2026 · read: 3 min
---
The Weekly Release Machine: How Spotify Ships to 675 Million Users Without Breaking Production
How Spotify ships mobile apps to 675 million users every week without outages: trunk-based development, concentric rollout rings, and feature flagging.
On this page
In modern consumer technology, few engineering challenges compare to deploying client-side mobile applications at global scale.
Unlike backend cloud services where an engineer can push a fix, observe a metric drop, and roll back a container in 60 seconds, mobile app releases are irreversible physical distributions.
Once an iOS .ipa or Android .apk binary passes App Store review and downloads onto hundreds of millions of smartphones worldwide, you cannot roll it back. If a catastrophic crash-on-launch bug slips into production, the only cure is submitting an emergency hotfix binary and waiting days for Apple and Google to review and propagate the update.
At Spotify, over 675 million monthly active users listen to music and podcasts across iOS, Android, and desktop.
Every single week, hundreds of software engineers across dozens of autonomous squads contribute thousands of commits to the codebase.
Yet, despite this staggering velocity, Spotify ships a brand-new production release every single week with over 95% of releases shipping cleanly without emergency hotfixes.
Spotify achieved this not by being slow and conservative, but by engineering The Weekly Release Machine: a disciplined delivery system built on Trunk-Based Development, Concentric Rollout Rings, and Decoupled Feature Flagging.
+-------------------------------------------------------------------------------+
| SPOTIFY'S TWO-WEEK RELEASE TIMELINE |
+-------------------------------------------------------------------------------+
Week 1 (Development & Stabilization):
[Trunk-Based Commits on `main`] (Dozens of squads merge continuous changes)
|
v (Every Friday at 12:00 UTC: Automated Branch Cut)
[Release Candidate Branch Created: `release/2026.12`]
|
+---> Automated Regression Test Suites Execute in Parallel
+---> Nightly Builds dispatched to Spotify Internal Employees (Dogfooding)
|
Week 2 (Staged Rollout & Store Submission):
|
v (Tuesday: External Beta Ring - 50,000 public beta testers)
v (Wednesday: App Store & Google Play Review Submission)
v (Thursday: Phased 1% Production Rollout Begins)
v (Friday: Incremental Rollout expands: 1% -> 5% -> 20% -> 100%)
1. Pillar 1: Trunk-Based Development Over Long-Lived Branches
In legacy release models, feature teams worked in isolated “feature branches” for months. At release time, merging 20 divergent feature branches into a single candidate branch triggered catastrophic merge hell and untraceable integration bugs.
Spotify mandates strict Trunk-Based Development:
+-------------------------------------------------------------------------------+
| TRUNK-BASED DEVELOPMENT FLOW |
+-------------------------------------------------------------------------------+
[main branch (Continuous Green Build)]
^ ^ ^ ^ ^
| | | | |
(Small PRs merged daily, protected by automated tests and feature flags!)
- Engineers merge small, modular pull requests directly into
mainmultiple times per day. - A pull request cannot merge unless automated linters, static analyzers, and unit test suites pass completely.
- The Golden Rule: The
mainbranch must always be in a deployable, shippable state at any minute of any day.
2. Pillar 2: Decoupling Code Deployment from Feature Releases
How do hundreds of engineers merge unfinished, work-in-progress features into main every day without exposing buggy interfaces to 675 million listeners?
The answer is Feature Flagging (Remote Configuration):
+-------------------------------------------------------------------------------+
| FEATURE FLAG GATING PATTERN |
+-------------------------------------------------------------------------------+
// Code is shipped to production binaries, but hidden behind dynamic flags:
if feature_flags.is_enabled("new_podcast_player_v2", user_id) {
render_new_podcast_experience();
} else {
render_legacy_podcast_experience(); // Default path for 99% of users!
}
By decoupling code deployment (shipping the binary to the App Store) from feature release (flipping the flag on in production), shipping a mobile binary becomes a non-event.
If a new playback UI has a critical bug, engineers do not submit an emergency App Store binary update. They simply flip the remote feature flag to false in their internal configuration console. The feature deactivates globally in seconds without touching client code.
3. Pillar 3: Concentric Exposure Rings
When a new binary is cut, Spotify does not release it to 675 million users simultaneously. The release traverses Concentric Rollout Rings, each designed to catch specific classes of failures:
+-------------------------------------------------------------------------------+
| CONCENTRIC EXPOSURE RINGS |
+-------------------------------------------------------------------------------+
Ring 1: Internal Spotify Employees (10,000+ Dogfooders)
- Objective: Catch obvious crash-on-launch bugs and major UI regressions.
Ring 2: Public External Beta Testers (~50,000 Enthusiasts)
- Objective: Catch device-specific edge cases across diverse hardware/OS versions.
Ring 3: Staged 1% Production Rollout
- Objective: Telemetry auditing! Monitor crash-free user rates and battery drain.
Ring 4: Phased Expansion (5% -> 20% -> 50% -> 100%)
- Objective: Global deployment across 675 million users.
The Automated Rollout Circuit Breaker
During the staged 1% rollout phase, automated telemetry monitors watch Crash-Free User Rates.
If the crash-free rate drops below 99.9%, the rollout halts automatically, preventing the faulty binary from reaching the remaining 99% of users while developers isolate the regression.
Architectural Comparison Matrix
| Release Vector | Traditional Ad-Hoc Mobile Release | Spotify Weekly Release Machine |
|---|---|---|
| Release Cadence | Irregular (Every 1 to 3 months) | Fixed train: Every single week |
| Branching Strategy | Long-lived feature branches | Pure Trunk-Based Development |
| Merge Friction | Weeks of merge conflict stabilization | Continuous daily micro-merges |
| Rollback Capability | Impossible (Requires new store binary) | Instantaneous via remote feature flags |
| Risk Profile | High-stakes “bet-the-company” events | Routine, low-stress operational cadence |
Summary and Key Takeaways
- Decouple shipping code from releasing features: Use remote feature flags to hide incomplete work inside production binaries, making mobile deployments routine non-events.
- Enforce trunk-based development: Merge small pull requests directly into
maindaily to eliminate painful merge conflicts and integration drift. - Commit to a fixed release train: Cut release branches on a rigid, predictable calendar cadence regardless of whether specific features are finished.
- Deploy through concentric rings: Progress binaries through employees, beta testers, and phased percentage production rollouts to catch bugs before global impact.
- Automate rollout circuit breakers: Automatically halt staged rollouts if real-time crash-free user rates drop below strict SLA thresholds (99.9%).
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.