---
topic: system-design
author: Crashtech Editorial
published: Nov 15, 2026 · read: 3 min
---

Pinterest Pacer: Scaling Asynchronous Jobs with Stateful Dequeue Brokers

Pinterest’s Pacer redesign assigns queue partitions to brokers to reduce contention in its Pinlater job system.

– –
A screen-printed queue diagram routes partitioned jobs through dedicated dequeue brokers to worker groups.
Conceptual illustration of the architecture discussed in the cited engineering account.

Background work is easy to describe [1] as “put a message on a queue, then process it.” At scale, dequeueing becomes its own distributed systems problem: many workers must find eligible jobs without repeatedly scanning the same storage or locking each other out.

Pinterest’s Pacer engineering post explains a redesign of its Pinlater asynchronous job platform. In the earlier arrangement, dequeue workers queried shards for work. Empty scans and lock contention made the system less efficient as the number of queues and workers grew. Pacer introduced stateful dequeue brokers that own queue partitions and serve work to worker processes.

Move partition ownership to brokers

The central design shift is to make queue ownership explicit. [1] Pacer assigns partitions to brokers, which maintain local buffers and hand jobs to workers. Pinterest describes using Apache Helix for cluster coordination, with assignment changes coordinated through ZooKeeper.

This creates a path from a broad job store to an owner responsible for a subset of the queue. When ownership is exclusive, competing brokers do not need to race to claim every item in the same partition. The model can reduce expensive scans and lock contention, though the system now has to manage broker lifecycle and ownership transitions.

The approach resembles partitioned stream processing, but the work unit is a scheduled or asynchronous job. Partition count and key selection shape both parallelism and ordering. A single partition can preserve a strict sequence more easily, while multiple partitions permit more concurrent work at the cost of global ordering guarantees.

The stateful trade-off

Buffers improve the hot dequeue path only if the system handles process loss and reassignment carefully. A broker restart must not silently lose jobs it has fetched but not completed. An assignment change must ensure another owner can resume safely, and acknowledgement semantics must distinguish delivery from successful work.

These are not implementation footnotes. They determine whether a fast queue remains correct under deployment, overload and worker failure. The publisher’s post reports Pacer’s architecture and performance observations; those measurements should be treated as Pinterest’s experience with its own traffic and workload.

When this pattern is useful

A broker layer may help when queue storage scans, hot partitions or lock contention dominate the system. Before adding it, measure empty polls, claim conflicts, queue age and processing latency. Then document ownership, acknowledgement, retry and rebalance semantics.

The general lesson from Pacer is that asynchronous platforms need a deliberate dequeue architecture. Partition affinity can turn repeated global competition into local work assignment, but it also makes ownership and recovery first-class parts of the queue contract.

Define delivery semantics before optimizing the poll

The queue’s contract determines what a broker may safely buffer. If a job is leased to a broker and the broker fails before a worker acknowledges it, the system needs a recovery path that makes it visible again without losing a successful completion. If acknowledgement is repeated, the handler may run more than once. Idempotency keys or deduplication can make that retry safe, but they do not create exactly-once side effects across arbitrary external systems.

Partitioning also chooses which jobs can progress independently. A key based on account or tenant can preserve local ordering while allowing unrelated keys to run concurrently. A skewed key can create a hot partition, so partition counts alone do not guarantee balanced throughput. Observe queue age and work distribution by partition, not just aggregate jobs per second.

Pacer’s stateful brokers move coordination into a dedicated tier. That can simplify the storage interaction seen by workers, but operating the broker tier requires dashboards for assignment churn, buffer depth, lease expiry and rebalance duration. A production rollout should test broker restart and worker crash cases while checking that the queue’s at-least-once or ordering promises remain true.

Advertisement

Frequently asked questions

What problem did Pacer address?

Pinterest’s engineering post describes contention and inefficient empty polling in the earlier Pinlater dequeue design. Pacer introduces stateful dequeue brokers that own queue partitions and coordinate work delivery.

How are broker assignments coordinated?

Pinterest describes using Apache Helix for partition assignment and ZooKeeper-backed coordination. The exact operational details are part of Pinterest’s implementation, not a requirement of every job queue.

Does Pacer guarantee FIFO ordering across all jobs?

No. Ordering depends on queue and partition design. Pinterest’s post notes that strict FIFO can limit parallelism when a queue is constrained to a single partition.

What is the main trade-off in a broker-based design?

Stateful brokers can reduce repeated database scans and contention, but add ownership, rebalancing, buffering and recovery concerns. Those states need monitoring and well-defined behavior during assignment changes.

Sources & further reading

/* Comments */