Carrying Patches on an Open-Source Fork: Upstream-First vs Hotfix, Without Drifting Away

Using open-source software at scale ends in one of two places: you wait for upstream to fix what hurts you, or you start carrying patches. Carrying patches is fine. Carrying them without a system is how a team ends up on a five-year-old fork nobody dares to upgrade.

The scale that forces the question

Kafka at that size means:

Stock Kafka is excellent and was not tuned for clusters of 140+ brokers with millions of replicas each. The surrounding ecosystem tells you what a platform team ends up building: a REST proxy for non-Java clients, a schema registry for Avro, a mirroring tool for cross-cluster replication (Brooklin), a rebalancing and self-healing service (Cruise Control, now open source), a completeness audit to prove no data was lost, and a usage-metering tool for chargeback.

Release branches: fresh base, carried patches

Each internal release is a branch cut from a specific upstream version, such as an internal 2.3.0.x built on Apache 2.3.0. Patches on it fall into four groups:

1. Cut the new branch from the latest official release

   Never extend an old fork forward. Start from the newest upstream version so you inherit every upstream fix and feature.

2. Re-apply only what upstream lacks

   Walk the previous internal branch’s patches and carry over just those not yet merged upstream. Merged ones are dropped automatically, because they are already in the new base.

3. Mark patch status in commit messages

   A consistent note in each commit says whether it is merged upstream, pending, or internal-only, so a script can generate the carry-over list.

4. Track upstream continuously

   Pull in upstream changes regularly instead of once a year, so each integration is small.

5. Test with real data and real load

   Run the candidate branch on production-shaped traffic and compare performance and behaviour before it serves anything important.

Choosing the path for a change

The decision turns on three questions:

1. How urgent is it? A live production problem argues for the hotfix.
2. How long would upstream review take? Quick review favours upstream-first.
3. Is it a feature or a bug fix? Features go upstream first. Bugs may be either.

What kind of patches arise at scale

Scalability. Controllers that ran short of memory and were slow in huge clusters were fixed by reusing objects and reducing logging. Slow broker startup and shutdown improved by reducing lock contention between threads.

Operability. Draining a broker before removal is hard when new data keeps arriving on it. A maintenance mode for brokers, where no new replicas or traffic are assigned, makes safe removal straightforward.

Features. Per-user usage tracking for chargeback, enforcement of a minimum replication factor when topics are created, and resetting a consumer to the closest valid offset.

Contributions back. Better quota handling, detecting stale control messages and long-offline brokers, separating control and data channels, and bounding how far log compaction may lag.

Rules for any team that forks

If you are not at this scale, the same principles apply to a single patched dependency: pin the version, record why each patch exists, and open the upstream issue the same day you write the workaround.

Takeaway

Customising open source is sustainable when it is a process: fresh branches from official releases, patches labelled by status, upstream-first as the default, hotfix as the exception, and a standing habit of contributing back. Then your fork stays close to the project, upgrades stay routine, and the community carries part of your maintenance load.