Workflow Orchestration at Scale: DAGs, Signals, Foreach Fan-Out and Why Single Leaders Stop Scaling
Replacing a single-leader scheduler with stateless services, queues and a distributed SQL store, plus signals, parameterised backfills and rollup views.
On this page
A scheduler looks like a cron table with a UI until workflows become the way a company moves data. Then the orchestrator is a distributed system whose failure stops everything downstream, and every design choice in its architecture decides how scary Monday midnight is.
Why the first design ran out
The earlier orchestrator coordinated everything through one leader. It was highly available, but it scaled only vertically.
| Symptom | Cause |
|---|---|
| Slowdowns at midnight UTC | Thousands of schedules fire at once, one leader processes them |
| On-call load | Engineers watch the system around the clock at peaks |
| Hardware ceiling | The team reached the largest available instance types |
| Doubling workload | Workflow count doubled year over year, so scaling up was a dead end |
The rule it illustrates: if any component sees all traffic, its machine size is your capacity ceiling.
The architecture that replaced it
Three services, each independently scalable:
Why signals beat clocks
A midnight schedule runs whether or not yesterday’s data has landed. The usual patch is padding: start late, retry, poll. A signal carries a small message (metadata and parameters) about an event, and a workflow subscribes to it.
upstream table updated --> signal: table=events, partition=2026-11-14
|
workflow subscribes: run when signal matches
or: step "load" waits until the signal arrives
Signals can start a whole workflow or hold back a single step. Lineage tracking records which signal triggered what, so you can read the dependency map between upstream and downstream pipelines instead of reconstructing it from tribal memory.
Scaling techniques
- Stateless services. Any instance can handle any request because state lives outside. Add instances to scale.
- Distributed queues. Components work at their own pace and absorb bursts.
- Strongly consistent storage. A distributed SQL database holds workflow definitions, step instances and state, scaling horizontally without losing consistency or taking on much operational complexity.
- Foreach with nesting. A single workflow may need to run the same job tens or hundreds of thousands of times: backfill a year of dates, sweep model parameters. Writing out steps is impractical, so a foreach step expands over a list, and loops can nest, supporting millions or billions of steps in one workflow instance while staying manageable.
Meeting users where they are
Users range from analysts to platform engineers, so execution is offered at several levels.
| Abstraction | For whom | What it gives |
|---|---|---|
| Predefined step types (Spark, SQL, data movement) | Most users | Supply parameters, the platform runs it. Improve the implementation once and every workflow benefits |
| Notebook execution | Data scientists | Run a notebook as a step, with parameters passed in |
| Container jobs | Advanced users | Any language and dependencies, packaged in an image |
| YAML, Python, Java DSLs, JSON API, UI, Python DAG libraries | Everyone | Define the same workflow the way each person prefers |
A parameterised backfill, step by step
- Compute the date range
A trivial first step produces the list of dates, say every day from 2021-01-01 to 2022-01-01, as an output parameter.
- Foreach over the dates
The workflow loops over that list, creating one iteration per date with the date as input.
- Run the job per iteration
Inside the loop, a notebook job runs for that date. One definition expands to hundreds of runs, each processing one day.
Huge fan-outs are unreadable as raw graphs. An aggregated view shows the state of a workflow as one summary across all its iterations and sub-workflows, and a rollup reduces counts of succeeded, failed and running steps up the tree. Design observability for the biggest run, not the typical one.
Events out, as well as in
The engine emits lifecycle events (workflow started, step failed, run completed) internally for tooling, and externally to message systems so other teams can react or build dashboards, which makes the orchestrator a platform component rather than a black box.
If you are choosing or building one
| Question | Why it matters |
|---|---|
| Can the scheduler scale horizontally, or is there a leader? | Capacity ceiling |
| At-least-once triggers plus deduplication? | Duplicate runs are the usual silent bug |
| Event triggers, not just cron? | Removes padding and polling |
| Parameterised fan-out? | Backfills are routine, not special projects |
| Lineage and rollup views? | Debugging at scale |
| Idempotent steps? | Retries and replays must be safe |
Do
Trigger on data readiness where you can, keep steps idempotent, version workflow definitions and keep run state in a store that scales out.
Don't
Pile everything onto midnight cron and add retries until it works. The failure just moves to the first night the upstream is late.
Takeaway
An orchestrator earns trust by having no single machine that matters. Stateless services, queues and a scalable consistent store remove the ceiling, signals replace clock guessing, foreach turns backfills into configuration, and rollup views keep enormous runs understandable. Get those right and the number of workflows can double without anyone staying up for it.
Frequently asked questions
Why does a single-leader orchestrator hit a ceiling?
One node coordinates all scheduling and state, so capacity grows only by buying a bigger machine. Eventually you reach the largest instance available, while workload doubles. Peak moments such as midnight UTC, when many schedules fire together, show it first as slow triggers and heavy on-call load.
What is the difference between time-based and signal-based triggering?
Time-based triggering starts a workflow on a clock, such as every midnight, regardless of whether its inputs are ready. Signal-based triggering starts it, or releases a waiting step, when an event arrives, such as a table landing or a file appearing. Signals remove the guesswork and the idle waiting.
How can a scheduler give exactly-once runs if triggers are at-least-once?
Make the engine deduplicate. The scheduler may fire the same trigger more than once, but each trigger carries an identity, such as workflow plus scheduled time, and the engine records which ones it has started. Duplicates are recognised and ignored, so effects happen once.
What is a foreach step used for in a pipeline?
To run the same sub-workflow for each item in a list computed at runtime: each date in a backfill window, each parameter set in a model sweep. One definition expands into thousands of runs, with the list produced by an earlier step, so nobody writes the steps by hand.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.