Why did Atlassian's Aurora clusters fail to start even though replication looked healthy?
Aurora's startup process enumerates every file in the cluster volume as a health check, and PostgreSQL stores every table, index, and sequence as at least one separate file. With roughly 5,000 files per Jira tenant database and up to 4,000 tenants per cluster, that's around 20 million files to enumerate — a process slow enough to exceed Aurora's internal startup timeout, causing the instance to fail even though the underlying data had replicated correctly.
Answered in
The Migration Trick That Moved 4 Million Databases Without a Bigger FleetAtlassian couldn't provision enough spare capacity to migrate 4M Jira databases at once. So they made the migration feed itself — a daisy chain, not a fleet.
Read the full analysisOther questions this article answers
- What does 'draining' mean in this migration, and why was it necessary?
- How does the 'daisy chain' approach avoid needing a large pool of spare infrastructure?
- Why did draining prioritize the smallest, least-used tenants first?
- Why keep the same instance count strategy (nearly tripling Aurora instances) if the goal was cost efficiency?
More system design & architecture questions
- How much spare capacity should a gateway keep?
- Why are bounded queues important for availability?
- Should a load balancer always move requests away from an unhealthy server?
- Can a circuit breaker prevent every cascading failure?
- Why does pagination improve API performance beyond just 'smaller responses'?
- Why is asynchronous logging faster than writing logs synchronously?
- When does caching actually help API performance, and when doesn't it?
- Why does payload compression trade CPU time for network time, and when is that worth it?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design & Architecture beat.