---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Workflow Orchestration at Scale: DAGs, Signals, Foreach Fan-Out and Why Single Leaders Stop Scaling

Replacing a single-leader scheduler with stateless services, queues and a distributed SQL store, plus signals, parameterised backfills and rollup views.

– –

A scheduler looks like a cron table with a UI until workflows become the way a company moves data. Then the orchestrator is a distributed system whose failure stops everything downstream, and every design choice in its architecture decides how scary Monday midnight is.

Why the first design ran out

The earlier orchestrator coordinated everything through one leader. It was highly available, but it scaled only vertically.

SymptomCause
Slowdowns at midnight UTCThousands of schedules fire at once, one leader processes them
On-call loadEngineers watch the system around the clock at peaks
Hardware ceilingThe team reached the largest available instance types
Doubling workloadWorkflow count doubled year over year, so scaling up was a dead end

The rule it illustrates: if any component sees all traffic, its machine size is your capacity ceiling.

The architecture that replaced it

Three services, each independently scalable:

Workflow engine the core
Manages definitions (versioned), runs workflows as DAGs of steps, and supports foreach loops, conditionals, subworkflows and reusable step templates. Dynamic parameters use a safe expression language. State machines are driven by the open-source Conductor library, and a timeline records every state change for debugging.
Time-based scheduler cron and intervals
Lightweight, triggers by cron expression or intervals, with an at-least-once guarantee. The engine deduplicates triggers, so runs happen once in effect.
Signal service event-driven
Triggers workflows or gates individual steps on events: an upstream workflow finishing, an object arriving in storage, a warehouse table updating. Tracks signal lineage.
Shared foundations how it scales
Stateless services, distributed queues between them and a strongly consistent distributed SQL database for definitions and run state.

Why signals beat clocks

A midnight schedule runs whether or not yesterday’s data has landed. The usual patch is padding: start late, retry, poll. A signal carries a small message (metadata and parameters) about an event, and a workflow subscribes to it.

upstream table updated  --> signal: table=events, partition=2026-11-14
                              |
          workflow subscribes: run when signal matches
          or: step "load" waits until the signal arrives

Signals can start a whole workflow or hold back a single step. Lineage tracking records which signal triggered what, so you can read the dependency map between upstream and downstream pipelines instead of reconstructing it from tribal memory.

Scaling techniques

  • Stateless services. Any instance can handle any request because state lives outside. Add instances to scale.
  • Distributed queues. Components work at their own pace and absorb bursts.
  • Strongly consistent storage. A distributed SQL database holds workflow definitions, step instances and state, scaling horizontally without losing consistency or taking on much operational complexity.
  • Foreach with nesting. A single workflow may need to run the same job tens or hundreds of thousands of times: backfill a year of dates, sweep model parameters. Writing out steps is impractical, so a foreach step expands over a list, and loops can nest, supporting millions or billions of steps in one workflow instance while staying manageable.
Advertisement

Meeting users where they are

Users range from analysts to platform engineers, so execution is offered at several levels.

AbstractionFor whomWhat it gives
Predefined step types (Spark, SQL, data movement)Most usersSupply parameters, the platform runs it. Improve the implementation once and every workflow benefits
Notebook executionData scientistsRun a notebook as a step, with parameters passed in
Container jobsAdvanced usersAny language and dependencies, packaged in an image
YAML, Python, Java DSLs, JSON API, UI, Python DAG librariesEveryoneDefine the same workflow the way each person prefers

A parameterised backfill, step by step

  1. Compute the date range

    A trivial first step produces the list of dates, say every day from 2021-01-01 to 2022-01-01, as an output parameter.

  2. Foreach over the dates

    The workflow loops over that list, creating one iteration per date with the date as input.

  3. Run the job per iteration

    Inside the loop, a notebook job runs for that date. One definition expands to hundreds of runs, each processing one day.

Make humans able to read a million steps

Huge fan-outs are unreadable as raw graphs. An aggregated view shows the state of a workflow as one summary across all its iterations and sub-workflows, and a rollup reduces counts of succeeded, failed and running steps up the tree. Design observability for the biggest run, not the typical one.

Events out, as well as in

The engine emits lifecycle events (workflow started, step failed, run completed) internally for tooling, and externally to message systems so other teams can react or build dashboards, which makes the orchestrator a platform component rather than a black box.

If you are choosing or building one

QuestionWhy it matters
Can the scheduler scale horizontally, or is there a leader?Capacity ceiling
At-least-once triggers plus deduplication?Duplicate runs are the usual silent bug
Event triggers, not just cron?Removes padding and polling
Parameterised fan-out?Backfills are routine, not special projects
Lineage and rollup views?Debugging at scale
Idempotent steps?Retries and replays must be safe

Do

Trigger on data readiness where you can, keep steps idempotent, version workflow definitions and keep run state in a store that scales out.

Don't

Pile everything onto midnight cron and add retries until it works. The failure just moves to the first night the upstream is late.

Takeaway

An orchestrator earns trust by having no single machine that matters. Stateless services, queues and a scalable consistent store remove the ceiling, signals replace clock guessing, foreach turns backfills into configuration, and rollup views keep enormous runs understandable. Get those right and the number of workflows can double without anyone staying up for it.

Advertisement

Frequently asked questions

Why does a single-leader orchestrator hit a ceiling?

One node coordinates all scheduling and state, so capacity grows only by buying a bigger machine. Eventually you reach the largest instance available, while workload doubles. Peak moments such as midnight UTC, when many schedules fire together, show it first as slow triggers and heavy on-call load.

What is the difference between time-based and signal-based triggering?

Time-based triggering starts a workflow on a clock, such as every midnight, regardless of whether its inputs are ready. Signal-based triggering starts it, or releases a waiting step, when an event arrives, such as a table landing or a file appearing. Signals remove the guesswork and the idle waiting.

How can a scheduler give exactly-once runs if triggers are at-least-once?

Make the engine deduplicate. The scheduler may fire the same trigger more than once, but each trigger carries an identity, such as workflow plus scheduled time, and the engine records which ones it has started. Duplicates are recognised and ignored, so effects happen once.

What is a foreach step used for in a pipeline?

To run the same sub-workflow for each item in a list computed at runtime: each date in a backfill window, each parameter set in a model sweep. One definition expands into thousands of runs, with the list produced by an earlier step, so nobody writes the steps by hand.

Sources & further reading

/* Comments */