How OpenAI Scaled ChatGPT to 800 Million Users on Postgres: Single Primary and Read Scaling
How OpenAI runs millions of queries per second on a single Postgres primary writer: read replica topologies, PgBouncer pooling, and write offloading.
On this page
When an engineering team learns their application is about to onboard hundreds of millions of global users, the default architectural reflex is immediate horizontal database sharding.
The standard engineering playbook dictates deploying distributed databases: Vitess, Spanner, CockroachDB, or manual multi-tenant sharding across dozens of relational clusters.
OpenAI chose a radically different path. As ChatGPT scaled past 800 million active users, the primary metadata database powering user authentication, conversation trees, workspace accounts, and organizational permissions remained a single primary PostgreSQL database.
Running millions of queries per second against a single primary writer sounds like an invitation to catastrophic outages. Yet OpenAI achieved five-nines reliability by enforcing strict architectural discipline: offloading reads through a multi-tier replica tree, pushing high-throughput append writes to specialized stores, eliminating N+1 ORM queries, and insulating the database with aggressive PgBouncer connection pooling.
+-------------------------------------------------------------------------------+
| OPENAI POSTGRES SCALING TOPOLOGY |
+-------------------------------------------------------------------------------+
[800M ChatGPT Users & API Invocations]
|
v
[Stateless Application Tier (Kubernetes Pods)]
|
+---> Writes: Append Logs, Chat Messages ---> [Azure Cosmos DB / Blob Store]
|
+---> Writes: User Accounts, Billing, Orgs ---> [PgBouncer Write Pool]
| |
| v
| [Single Primary PostgreSQL Writer]
| |
| | Streaming WAL (Physical)
| v
+---> Reads: Auth Checks, Workspaces -------> [PgBouncer Read Pool Fleet]
|
+-------------------------+-------------------------+
| |
v v
[First-Tier Read Replicas] [Second-Tier Cascading Replicas]
1. Why OpenAI Avoided Premature Sharding
Database sharding is not a free performance upgrade; it is an organizational and operational tax of immense complexity.
When you horizontally shard a relational database by user_id:
- Cross-Shard Joins are Destroyed: A query joining
users,teams, andenterprise_billingrequires distributed two-phase commits or expensive application-side aggregation. - Resharding Migrations are Hazardous: Rebalancing shards as users grow requires running live dual-writing pipelines and data backfills that risk data corruption.
- Codebase Complexity Multiplies: Every database query across hundreds of microservices must be rewritten to include routing keys.
OpenAI evaluated the workload characteristics of ChatGPT and discovered a crucial insight: ChatGPT’s metadata workload is over 95% reads.
If the primary writer only processes genuine state mutations (creating a new user, updating a payment tier, changing an organization role), a modern bare-metal server with 128 CPU cores, 1TB of RAM, and NVMe storage can comfortably absorb tens of thousands of write transactions per second. The bottleneck was read traffic, not write throughput.
2. Pillar 1: Ruthless Write Offloading
To keep the single primary writer alive, OpenAI stripped away every write operation that did not strictly require relational ACID guarantees:
+-----------------------------------------------------------------------+
| DATA PLACEMENT ARCHITECTURE |
+-----------------------------------------------------------------------+
Data Type Storage Engine Reasoning
-------------------------------------------------------------------------
User Accounts & Orgs PostgreSQL Primary Requires ACID transactions
Conversation Content & Trees Azure Cosmos DB High-volume, partitionable by thread
Token Billing Metrics Apache Kafka -> S3 Append-only streaming logs
Prompt KV Cache Metadata Redis Cluster Ephemeral, volatile cache
Decoupling Conversation Logs
The raw text of chat conversations, user prompts, and streaming model completions represents massive data volume.
Storing every conversational turn in PostgreSQL would exhaust write-ahead logs (WAL) and trigger severe disk I/O bottlenecks. OpenAI pushed conversation threads into horizontally partitionable NoSQL systems (Azure Cosmos DB and blob storage).
PostgreSQL only tracks lightweight account references and organizational metadata.
3. Pillar 2: Cascading Read Replicas and Read Offloading
With writes minimized, the engineering challenge shifted to scaling read throughput to millions of queries per second.
Connecting thousands of application pods directly to a single PostgreSQL primary triggers replication overhead: the primary must spend CPU cycles and network bandwidth streaming WAL buffers to every listener.
OpenAI structured read scaling using Cascading Physical Replication:
[Single Primary Writer]
|
| Physical Streaming WAL (Sends once to tier-1 replicas)
v
[Tier-1 Intermediate Replicas (Dedicated WAL Forwarders)]
|
+-----------------------+-----------------------+
| |
v v
[Regional Read Replica 1] [Regional Read Replica 2]
(Serves Application Pods) (Serves Application Pods)
- Dedicated WAL Forwarders: The primary streams replication logs to a small set of intermediate tier-1 replicas. These replicas do not serve application queries; their sole job is to forward WAL streams to dozens of leaf read replicas.
- Read-Only Connection Routing: Application frameworks route all
SELECTqueries to regional read replicas by default. Only transactional mutations (INSERT,UPDATE,DELETE) are permitted on the primary connection string. - Replication Lag Defense: To prevent users from seeing stale data immediately after an update (read-your-own-writes consistency), the application tracks replication lag. If a user just modified their organization settings, their subsequent reads are pinned to the primary for 5 seconds until the WAL catch-up window elapses.
4. Pillar 3: Connection Pooling via PgBouncer
PostgreSQL implements a process-per-connection architecture: each incoming client connection spawns a dedicated operating system process.
Spawning 10,000 direct database connections consumes dozens of gigabytes of RAM in process memory, triggers extreme context-switching overhead on the Linux kernel scheduler, and degrades transaction throughput.
OpenAI deployed PgBouncer in Transaction Pooling Mode:
[10,000 Application Kubernetes Pods]
|
v (Holds persistent, lightweight client connections)
[PgBouncer Proxy Fleet]
|
v (Maintains fixed pool of only 200 physical database connections!)
[PostgreSQL Database Engine]
Transaction Pooling Mechanics
In transaction pooling mode, a physical database connection is only allocated to an application thread for the exact duration of an active transaction.
The moment a transaction issues COMMIT or ROLLBACK, the physical connection is instantly returned to the pool to serve another waiting request. A pool of only 200 physical database connections easily multiplexes 10,000 concurrent application clients with zero connection exhaustion.
5. Pillar 4: Query Hygiene and Killing the 12-Table Join
ORMs (such as Django ORM or SQLAlchemy) provide convenient abstractions that silently generate horrific SQL under the hood:
-- The Silent Killer: 12-Table Relational Join generated by naive ORM:
SELECT * FROM users
JOIN user_roles ON ...
JOIN permissions ON ...
JOIN organizations ON ...
JOIN subscriptions ON ...
JOIN invoices ON ...
WHERE users.id = 'usr_1024';
Under high concurrency, a 12-table join consumes massive CPU time in query plan optimization, scans multiple B-trees, and locks table buffers.
OpenAI instituted strict SQL query hygiene:
- Zero Complex Multi-Table Joins: Large joins were dismantled into independent, parallel key-value lookups (
SELECT * FROM users WHERE id = ?) utilizing primary-key indexes. - Application-Side Aggregation: Assembling a user’s full workspace profile happens in application memory via concurrent gRPC lookups, reducing database load to simple index scans.
- Automated Slow Query Killers: Statement timeouts (
statement_timeout = 250ms) automatically terminate any runaway analytical query before it starves the primary connection pool.
Architecture Comparison: Premature Sharding vs Single Primary
| Architectural Dimension | Premature Sharded Architecture | OpenAI Single-Primary Scaling |
|---|---|---|
| Operational Complexity | Distributed transactions, cross-shard 2PC | Standard single-instance relational ACID |
| Write Strategy | Sharded across dozens of database instances | Single primary writer; non-ACID writes offloaded |
| Read Strategy | Co-located shard lookups or distributed scatter-gather | Tiered cascading read replicas + PgBouncer |
| Development Velocity | Slow (Every query requires sharding key analysis) | Fast (Standard SQL queries, transparent reads) |
| Hardware Efficiency | Low (Resource fragmentation across empty shards) | Maximum (99% saturation of high-memory bare metal) |
Summary and Key Takeaways
- Exhaust single-primary scaling before sharding: Modern bare-metal servers paired with NVMe drives can handle massive write volumes when freed from read contention.
- Offload non-relational writes: Protect the primary database by pushing append-only conversation streams, logs, and metrics to specialized stores like Cosmos DB and S3.
- Use cascading replication trees: Shield the primary writer from WAL transmission overhead by utilizing intermediate replicas to fan out updates to leaf read nodes.
- Mandate transaction pooling with PgBouncer: Multiplex tens of thousands of application pods across a compact pool of physical connections to eliminate kernel context-switching stalls.
- Dismantle heavy relational joins: Replace complex multi-table joins with parallel indexed lookups and application-side assembly to preserve database CPU cycles.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.